chevrotain
chevrotain copied to clipboard
TokenType structure and API
The TokenType structure has many optional properties.
export interface TokenType {
name: string
GROUP?: string
PATTERN?: TokenPattern
LABEL?: string
LONGER_ALT?: TokenType
POP_MODE?: boolean
PUSH_MODE?: string
LINE_BREAKS?: boolean
CATEGORIES?: TokenType[]
tokenTypeIdx?: number
categoryMatches?: number[]
categoryMatchesMap?: {
[tokType: number]: boolean
}
isParent?: boolean
START_CHARS_HINT?: (string | number)[]
}
I believe this is due to legacy reasons as in the past TokenTypes could be classes with static properties. The fact this structure only has a single mandatory property (name) means that any API that accepts TokenType (e.g CONSUME) could also accept (by mistake) other similar structures. For example a JS Function has name property and is therefore acceptable anywhere a TokenType is accepted.
I believe the TokenType structure should more closely match what is returned by "createToken" which means a-lot less if any optional properties.
Also getting rid of the upper case property names would also be advisable.
By definition this is a breaking change, however if the creation of TokenTypes is always done by calling createToken, does it matter if the API gets broken? does it really mandate an new major version?
I think this structure may actually be missing one of the more important properties (tokenIdx/Idx)
It may be a good idea to "hide" some of the internal properties using symbols instead of string properties.