Skipping whitespace tokens

[jarble]

Is it possible to skip tokens when defining a lexer? I want to split a string into a list of tokens without whitespace, but I don't know if Moo can do this:

Input string:

• "while ( a < 3 ) { a += 1; }"

List of tokens:

• ["while","(","a","<","3",")","{","a","+=","1",",";","}"]

[nathan]

const moo = require('moo')
const lex = moo.compile({
  ws: {match: /\p{White_Space}+/u, lineBreaks: true},
  word: /\p{XID_Start}\p{XID_Continue}*/u,
  op: moo.fallback,
})
;[...lex.reset('while ( a < 3 ) { a += 1; }')]
.filter(t => t.type !== 'ws')
.map(t => t.value)

[jarble]

@nathan The documentation doesn't describe this feature: does it need to be updated?

[tjvr]

The documentation needs to be updated to document moo.fallback (see #112).

As for the rest, I think Nathan's just demonstrating that since a moo lexer object is an Iterator, you can use filter() and map() which are built-in to JavaScript.