Allow more values for `#include <...>`

[Lutymane]

Value should allow path slashes \, /, different symbols $@- whitespaces. Actually just any characters.

#include <$lib>
#include <module/sub/file>
#include <@namespace>

I tried to do this similar to TOKEN.STRING:

TOKEN.MODULE = chevrotain.createToken({
  name: "MODULE",
  pattern: /<[^>]+>/,
});

//...

ALT: () => {
  // three.js style import
  //https://github.com/mrdoob/three.js/blob/98616257db739e50513437c6913156c17a6d40e4/src/renderers/webgl/WebGLProgram.js#L239
  // this.CONSUME(TOKEN.LEFT_ANGLE);
  // const t = this.CONSUME(TOKEN.PATH);
  // this.CONSUME(TOKEN.RIGHT_ANGLE);
  // str = "<" + t.image + ">";

  str = this.CONSUME(TOKEN.MODULE).image;
};

But it just randomly breaks at seemingly random places after this change. Could you look into implementing this pls?

[NaridaL]

So the difficulty with the pointy brackets is that the tokens as defined above overlap. So if we have

if (a < b && b > c) then if would parse < b && b > as a MODULE token....

[NaridaL]

Does it have to be <> rather than ""? "" doesn't get used for anything else so we don't run into any issues.

[NaridaL]

Actually, this works with only a slight hack. Have a look at #24 . I've kept it as an allow-list rather than [^>] to avoid issues with new lines or invalid characters.

[Lutymane]

Thank you! I realized the main issue in my edits was that my custom TOKEN(s) were defined after TOKEN.HASH/TOKEN.*_ANGLE, thus it broke. It must be defined before them.

Regarding new lines, good catch! I decided to stick to this regex for module includes. Mainly include new line chars into negative matching group. And also use space and \t instead of \s because it'd match new lines also: /[ \t]*#[ \t]*include[ \t]+<[^>\r\n]+>/

Also realized TOKEN.STRING could benefit from this too! Because if you have unclosed #include "something the parser would hang, with this regex it'd error (as intended): /"([^"\\\r\n]+|\\")+"/,

[NaridaL]

published in 0.2.2