goparsec icon indicating copy to clipboard operation
goparsec copied to clipboard

Unicode support.

Open prataprc opened this issue 7 years ago • 2 comments

Default Scanner (SimpleScanner) and Terminal parser functions should have unicode support, along with utf8 encoding.

prataprc avatar May 16 '17 11:05 prataprc

While at it, add support for white spaces in unicode, for instance SimpleScanner.SkipWS().

prataprc avatar Jun 07 '17 04:06 prataprc

Skipping Unicode whitespace can be achieved by using SetWSPattern() API. By setting the necessary whitespace unicode-points.

There is also unicode category called, Separator, Spaces http://www.fileformat.info/info/unicode/category/Zs/list.htm

Can be used with golang's regexp \p{}: https://golang.org/pkg/regexp/syntax/

prataprc avatar Jul 03 '17 07:07 prataprc