Slimit/PLY explodes on unicode input

[autumnjolitz]

Slimit/PLY throws:

Traceback (most recent call last):
  File "vardetect.py", line 35, in collect_var_dec_globals
    tree = parser.parse(source)
  File "/Library/Python/2.7/site-packages/slimit/parser.py", line 93, in parse
    return self.parser.parse(text, lexer=self.lexer, debug=debug)
  File "/Library/Python/2.7/site-packages/ply/yacc.py", line 265, in parse
    return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
  File "vardetect.py", line 242, in     monkey_patch_parseopt_notrack
    tok = self.errorfunc(errtoken)
  File "/Library/Python/2.7/site-packages/slimit/parser.py", line 116, in p_error
    self._raise_syntax_error(token)
  File "/Library/Python/2.7/site-packages/slimit/parser.py", line 89, in _raise_syntax_error
    self.lexer.prev_token, self.lexer.token())
SyntaxError: Unexpected token (DIV, u'/') at 1:12757 between     LexToken(RPAREN,u')',1,12755) and LexToken(BXOR,u'^',1,12758)

On Javascript from URL: http://www.510.com/style1_3/510.js

Environment: Python 2.7, pip install git+https://github.com/rspivak/slimit.git#egg=slimit

monkey_patch_parseopt_notrack is a line by line copy of parseopt_notrack from yacc.py with try/except around the del errok statements. Assume it is same as yacc.py.

[rspivak]

Thanks. I'll look into it.