Start, end of tokens in sanitized text

[zzwx]

trafficstars

This update is to keep track of original locations in the provided text, however I couldn't figure out how better to deal with sanitized (clean) text step inside Tokenize without breaking API here:

	clean, white := t.sanitizer.Replace(text), false
	length := len(clean)

Obviously clean becomes the actual source for Start and End, and not the original text.

Possible solution: Leave sanitizing up to caller so that they can have both the original string & the locations in it.

[nicolasassi]

I did not understand how this change is supposed to break the API. Could you please explain better? Your changes seems promising!

[zzwx]

Thank you for reviewing.

See, what is expected is that Start and End would refer to the original source string. However it is being sanitized right inside Tokenize and all calculations are now based on a possibly modified source, and the expectation here is not true anymore. (That's why I commented them with a reference to sanitized text, to remind of that). The break would be to remove sanitizing and leave it up to the caller to choose. Then he will have a reference to the correct source string.

The fear though is that the library depends on this sanitizing step.