HTML strip normalizer on the tokenizer

[qdequele]

Sometimes I help users who send data that contains HTML without knowing that this dramatically affects the relevancy when searching, especially when words touch HTML tags.

Would it be possible to implement a normalizer that removes all HTML tags in document collections? Would it be possible to activate/deactivate it via settings or to activate it by default?

I found a crate to do it 🙂.

[gmourier]

Hey @qdequele 👋

This could be a first addition in the direction of having a transformation pipeline.

Note: There is a possible workaround. Doing this on the user-side before sending the documents (the workaround is limited to people that can manipulate the documents).

It's not obvious that this is a priority right now to be solved but the normalizer thing could be explored quickly.

@ManyTheFish What do you think?

[ManyTheFish]

Handling this in the tokenizer is possible but is harder than just adding a Normalizer because it passes after the Segmenter.
That means that the text <span><a href=\"#\">Summer</a> is nice</span> would be segmented as ["<", "span", ">", "<", "a", " ", "href", "=", "\"", "#", "\"", ">", "Summer", "<", "/", "a", ">", " ", "is", " ", "nice", "<", "/", "span", ">"] before the normalization phase.
However, it would be possible to pre-segment the text in order to isolate HTML tags and consider them as separators 🤔

[qdequele]

With this crate, it removes totally all HTML tags to keep only text.

[ManyTheFish]

Yes @qdequele! But it will completely break the highlight if we use this library in a preprocessing part.