`html5ever::tokenizer::Tokenizer::process_token_and_continue` needs to go away

[kevincox]

trafficstars

It will panic if you don't return Continue and the callee has no way of knowing if they are being called from a "normal" code path where you are allowed to return whatever you want and or a special panic-if-not Continue codepath. Not cool.

This is dangerous and basically means that returning anything but Continue is forbidden if you don't want panics.

At they very least this should be documented.

[jdm]

What are you trying to do where this is an issue?

[jdm]

Oh, I see that it compares against the result of calling process_token on the sink.

[kevincox]

I was implementing a TokenSink for rewriting some HTML in a streaming-ish fashion.

I'm not sure this is the best API for what I am trying to do but I didn't expect it to panic because I didn't return Continue.