Experiment: make GAP scanner/reader accessible from GAP

[fingolfin]

This experimental PR is something I've been telling people for a long time now; since I never seem to find time to clean it up and finish it, I thought I should post it here as-is; perhaps it motivates someone to join forces or to do their own thing.

This patch adds a way to take any input and run it through the parser, but without executing it, or printing any syntax errors etc. Instead the tokens as found by the scanner are recorded in a plain list, as is information about syntax errors.

To try it out, compile it, then ReadLib("tokenizer.g");

This has numerous potential application

• implement "perfect" GAP syntax highlighting
• implement a new REPL which allows for true multi line editing, context aware tab completion, syntax highlighting of the input while you type etc.
• the REPL could also detect syntax errors as you type and show them (e.g. by marking the offending bits in red or underlining them etc.)
• all the same could of course be used in alternative GAP frontends like Jupyter
• a better demo mode could be implemented
• yet another alternative implementation for Test could be written, in pure GAP (I don't see any reason to do that right now, I just mention it)
• could be used in alternative code testing frameworks
• AutoDoc could perhaps use it for more accurate parsing of GAP files
• ...

Of course to get there, more work should be done. For one, this proof of concept should be polished and turned into something nicer:

• instead of returning a dumb list of lists, one could return a list of records, or even objects (whether this transformation is done in the kernel or in a higher level GAP wrapper is up to the imagination of the implementor(s))
• actually try to implement something from the list above (or something not yet in there) to see if the code can do all that's necessary for it
• add tests
• documentation... :-)
• ...

[coveralls]

Coverage decreased (-0.003%) to 93.726% when pulling 0a0137e9527e56d49a4b593a68eef5def7fef677 on fingolfin:mh/scanner-tokenizer-accessible-from-gap into e62d39555838725badc50c716a29cdbf7cc576c4 on gap-system:master.

[codecov[bot]]

Codecov Report

Merging #4130 (8bca6c6) into master (b080f45) will increase coverage by 11.04%. The diff coverage is 73.07%.

:exclamation: Current head 8bca6c6 differs from pull request most recent head b8d83a7. Consider uploading reports for the commit b8d83a7 to get more accurate results

@@             Coverage Diff             @@
##           master    #4130       +/-   ##
===========================================
+ Coverage   82.32%   93.36%   +11.04%     
===========================================
  Files         679      716       +37     
  Lines      288109   812570   +524461     
===========================================
+ Hits       237184   758659   +521475     
- Misses      50925    53911     +2986     

Impacted Files	Coverage Δ
src/scanner.h	50.00% <ø> (+25.00%)	:arrow_up:
src/read.c	95.54% <36.36%> (-2.12%)	:arrow_down:
src/scanner.c	97.34% <59.25%> (-2.66%)	:arrow_down:
src/streams.c	81.62% <81.52%> (+0.85%)	:arrow_up:
src/baltree.h	0.00% <0.00%> (-97.53%)	:arrow_down:
src/dynarray.h	0.00% <0.00%> (-94.12%)	:arrow_down:
src/libgap-api.c	4.50% <0.00%> (-51.29%)	:arrow_down:
lib/memory.gi	43.12% <0.00%> (-39.92%)	:arrow_down:
src/sysjmp.c	52.17% <0.00%> (-30.44%)	:arrow_down:
src/io.c	59.29% <0.00%> (-28.64%)	:arrow_down:
... and 289 more