Allow transcripts of umlauts

[sebix]

For all users of openthesaurus not using a german keyboard layout and not knowing how to use compose keys, it would be very nice to have an automatic conversion for umlauts:

• ue -> ü
• ae -> ä
• oe -> ö
• ss or sz -> ß

dict.cc is doing the same

[janschreiber]

This feature is now implemented and seems to work. However, I found one strange result today:

• Search for 'Evaskostum' finds 'Evaskostüm' via fuzzy search
• Search for 'Evaskostuem' finds nothing
• Search for 'Kostuem' finds 'Kostüm'

[sebix]

Great!

[danielnaber]

@janschreiber Thanks for the report. Unfortunately it's not easy to fix, as we apply some tricks to make it fast: the substring search needed here (as the item is im Evaskostüm) works on a memory table, but this table doesn't contain the normalized terms needed for this feature.

[janschreiber]

@danielnaber Thanks for your explanation. I'm not sure if the following suggestion makes any sense whatsoever, but wouldn't it be possible to apply the normalization to the search terms rather than to the searched data? I mean, isn't it possible to transform a search for words that contain "umlaut-ish" character combinations such as 'ae' to a search for (Cäsar|Caesar) before it is even sent to the search algorithm?

[danielnaber]

The normalization needs to be applied to both, but our in-memory database currently isn't a mapping, but just a list of words. We'd need to extend that to contain a mapping from normalized to original term. (Plus, we actually have two different ways of normalization.)