Fragmenter doesn't properly filter HTML tags

[chang-zhao]

I search HTML documents and often get html tags or their parts in the highlighter results, like:

Абсолютная истина.<br />Не-противостояние и растворение напряжений</h1...Абсолютная истина</h4...В этом мире нет

Most often, these tags or tag pieces are:

    </p
    <br />
    </h1, </h2 and so on
    </em></strong>

I switched to using SentenceFragmenter (which is also more suitable for my needs):

results.fragmenter = highlight.SentenceFragmenter(
                            maxchars=240,
                            sentencechars='</>.!?',
                            charlimit = None
                            )

so it should filter all that out, but it doesn't work. I even tried to escape those characters like this:

sentencechars='\<\/\>.!?'

Nope. It seems I will have to resort to additional search and replace.

[chang-zhao]

Here's how I clean it: https://gist.github.com/chang-zhao/2a18dcab0b40e3011decefb65c91b4ca