relation-network icon indicating copy to clipboard operation
relation-network copied to clipboard

Published 20 hours ago • verified publisher icon siddk

What is the "token"?

Open zlj63501 opened this issue 4 years ago • 0 comments

Hello,this is my question. In "reader.py" ,the 14line ,tokenize = lambda x: [token.strip().lower() for token in re.split(SPLIT_RE, x) if token.strip()] , I am not search the "token". So what is the "token"?

zlj63501 avatar Apr 25 '20 02:04 zlj63501