Hex escape sequence in string literal doesn't work

[daym]

Example:

Print["Hello q\.b4"]

(for example after starting the command mathics)

I found this when trying to get xAct to work.

Specifically, I tried xTensor/xTensor.m there.

[rocky]

This is a problem in the mathics-scanner project.

$ mathics3-tokens
In[]:= Print["Hello q\.b4"]


Traceback (most recent call last):
  File "/tmp/bin/mathics3-tokens", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/tmp/lib/python3.12/site-packages/mathics_scanner/mathics3_tokens.py", line 256, in main
    interactive_eval_loop(shell, args.CodeTokenize)
  File "/tmp/lib/python3.12/site-packages/mathics_scanner/mathics3_tokens.py", line 152, in interactive_eval_loop
    tokens(shell.feed(), code_tokenize_format)
  File "/tmp/lib/python3.12/site-packages/mathics_scanner/mathics3_tokens.py", line 169, in tokens
    token = tokeniser.next()
            ^^^^^^^^^^^^^^^^
  File "/tmp/lib/python3.12/site-packages/mathics_scanner/tokeniser.py", line 526, in next
    raise ScanError()
mathics_scanner.errors.ScanError

I noticed this a while ago during the last revision of the scanner. (I was hoping no one would notice too soon.)

[rocky]

For what it's worth, the revise-escape-sequence-scanning branches of mathics-core and mathics-scanner address the problem of handling this kind of escape sequence inside a string.

However, when I try this I am seeing other errors. Another scanner error seems to be handling boxed expressions inside strings. Recently, we addressed at least partially boxed expressions that are not in strings.

Something that may help track down the further problems. Mathics3 Get[] has a Trace->True option which will show the source text read. So instead of <<, using Get with that option can show you where things are going wrong.