syntax icon indicating copy to clipboard operation
syntax copied to clipboard

Provide a way to pass a character stream to the tokenizer or a way to preprocess the supplied text

Open fdutton opened this issue 8 years ago • 1 comments

PHP does not handle Unicode very well so I found it necessary to preprocess the supplied text by folding the Unicode characters into the ASCII character set using iconv('UTF-8', 'ASCII//TRANSLIT', $string).

Instead of passing a string to the parser, I would prefer passing a character stream or specifying a translation function so that I do not have to modify the generated code.

fdutton avatar Feb 14 '17 22:02 fdutton

Yeah, instead of a single generic parse function on a parser in the template, feel free to add parseFromString (and default parse will just call it), and the parseFromCharStream to PHP plugin.

DmitrySoshnikov avatar Feb 14 '17 23:02 DmitrySoshnikov