Build error: FAILED: py/pxdgen_timefile

[thierry-FreeBSD]

Trying to build v0.4.1 on FreeBSD, I get the following error:

[  0% 2/364] cd /usr/ports/games/openage/work/openage-0.4.1 && /usr/local/bin/python3.8 -m buildsystem.pxdgen --file-list /usr/ports/games/openage/work/.build/py/pxdgen_sources --output-dir /usr/ports/games/openage/work/.build && /usr/local/bin/cmake -E touch /usr/ports/games/openage/work/.build/py/pxdgen_timefile
FAILED: py/pxdgen_timefile /usr/ports/games/openage/work/.build/py/pxdgen_timefile 
cd /usr/ports/games/openage/work/openage-0.4.1 && /usr/local/bin/python3.8 -m buildsystem.pxdgen --file-list /usr/ports/games/openage/work/.build/py/pxdgen_sources --output-dir /usr/ports/games/openage/work/.build && /usr/local/bin/cmake -E touch /usr/ports/games/openage/work/.build/py/pxdgen_timefile
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 479, in <module>
    main()
  File "/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 456, in main
    result = generator.generate(
  File "/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 373, in generate
    result = "\n".join(self.get_pxd_lines())
  File "/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 294, in get_pxd_lines
    self.parse()
  File "/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 269, in parse
    state = transitions[state](self, token, val, namespace_parts)
  File "/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 223, in handle_state_1
    raise self.parser_error(
__main__.ParserError: /usr/ports/games/openage/work/openage-0.4.1/libopenage/main.h:15 expected identifier after 'namespace'
ninja: build stopped: subcommand failed.

[TheJJ]

This is due to an update in pygments, which we fixed in 3aa91ea4d4a72a5f3b6f90ad0390b9e07de7aa77, which is not included in any release yet.

[thierry-FreeBSD]

Thanks, it fixed the error!

[thierry-FreeBSD]

Hmm... Either my tests were bad, or something has changed, but it still fails with pygments 2.9.0 and your patch:

[  0% 3/364] cd /wrkdirs/usr/ports/games/openage/work/openage-0.4.1 && /usr/local/bin/python3.8 -m buildsystem.pxdgen --file-list /wrkdirs/usr/ports/games/openage/work/.build/py/pxdgen_sources --output-dir /wrkdirs/usr/ports/games/openage/work/.build && /usr/local/bin/cmake -E touch /wrkdirs/usr/ports/games/openage/work/.build/py/pxdgen_timefile
FAILED: py/pxdgen_timefile /wrkdirs/usr/ports/games/openage/work/.build/py/pxdgen_timefile 
cd /wrkdirs/usr/ports/games/openage/work/openage-0.4.1 && /usr/local/bin/python3.8 -m buildsystem.pxdgen --file-list /wrkdirs/usr/ports/games/openage/work/.build/py/pxdgen_sources --output-dir /wrkdirs/usr/ports/games/openage/work/.build && /usr/local/bin/cmake -E touch /wrkdirs/usr/ports/games/openage/work/.build/py/pxdgen_timefile
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/wrkdirs/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 483, in <module>
    main()
  File "/wrkdirs/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 460, in main
    result = generator.generate(
  File "/wrkdirs/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 377, in generate
    result = "\n".join(self.get_pxd_lines())
  File "/wrkdirs/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 298, in get_pxd_lines
    self.parse()
  File "/wrkdirs/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 273, in parse
    state = transitions[state](self, token, val, namespace_parts)
  File "/wrkdirs/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 227, in handle_state_1
    raise self.parser_error(
__main__.ParserError: /wrkdirs/usr/ports/games/openage/work/openage-0.4.1/libopenage/main.h:15 expected identifier after 'namespace'

[TheJJ]

Hmmm. Can you run:

cd /wrkdirs/usr/ports/games/openage/work/openage-0.4.1
/usr/local/bin/python3.8 -m pdb -c continue -m buildsystem.pxdgen --file-list /wrkdirs/usr/ports/games/openage/work/.build/py/pxdgen_sources --output-dir /wrkdirs/usr/ports/games/openage/work/.build && /usr/local/bin/cmake -E touch /wrkdirs/usr/ports/games/openage/work/.build/py/pxdgen_timefile

this will run until the exception is raised. in the shell you can then evaluate variables and play around.

this we can use to see which condition would be correct, and what token really is. so then we can validate whether if token not in Token.Name: from my patch is right.

You can test e.g. by breaking in that line (b filename:linenumber), and evaluate in the pdb-shell:

• print val
• print token
• print token in Token.Name
• print token in Token.Name.Namespace The available tokens are listed here: https://pygments.org/docs/tokens/#name-tokens

For me (with pygments 2.11, the token is a Token.Name.Namespace, and token in Token.Name == True.

The place where it fails is when the cython-pxdfile-generator parses the c++ code and extracts the namespace nesting. Concretely: https://github.com/SFTtech/openage/blob/bed49bae2c92b80f1943a31f81d2b878fb428db0/libopenage/main.h#L14-L16 So the token order should be keyword=namespace, name=openage, {, but apparently name isn't matched.

[thierry-FreeBSD]

Sorry, my mistake: your patch works fine against Pygments-2.9.0, and the reported failure was on a machine running Pygments-2.11.2 !

[TheJJ]

Ok, but why does it work with my pygments 2.12, your 2.9, but not with 2.11?

[thierry-FreeBSD]

I don't know, maybe because we use a different Python version? (ATM FreeBSD ports still use Python 3.8). With Pygments-2.11.2 I have been able to reproduce the error on my workstation:

$ /usr/local/bin/python3.8 -m pdb -c continue -m buildsystem.pxdgen --file-list /usr/ports/games/openage/work/.build/py/pxdgen_sources --output-dir /usr/ports/games/openage/work/.build && /usr/local/bin/cmake -E touch /usr/ports/games/openage/work/.build/py/pxdgen_timefile
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/pdb.py", line 1703, in main
    pdb._runmodule(mainpyfile)
  File "/usr/local/lib/python3.8/pdb.py", line 1547, in _runmodule
    self.run(code)
  File "/usr/local/lib/python3.8/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 3, in <module>
    """
  File "/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 460, in main
    result = generator.generate(
  File "/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 377, in generate
    result = "\n".join(self.get_pxd_lines())
  File "/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 298, in get_pxd_lines
    self.parse()
  File "/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 273, in parse
    state = transitions[state](self, token, val, namespace_parts)
  File "/usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py", line 227, in handle_state_1
    raise self.parser_error(
ParserError: /usr/ports/games/openage/work/openage-0.4.1/libopenage/main.h:15 expected identifier after 'namespace'
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py(227)handle_state_1()
-> raise self.parser_error(
(Pdb) b /usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py:273
Breakpoint 1 at /usr/ports/games/openage/work/openage-0.4.1/buildsystem/pxdgen.py:273
(Pdb) print(val)
 
(Pdb) print(token)
Token.Text.Whitespace
(Pdb) print(token) in Token.Name
Token.Text.Whitespace
False
(Pdb) print(token) in Token.Name.Namespace
Token.Text.Whitespace
False

[TheJJ]

Interesting, so it really does parse a whitespace there, and not a namespace identifier. I suggest you update the code in pxdgen to see which tokens are processed one after the other, so we can see where it miss-parses things. Maybe it was a bug in pygments that's fixed already in newer versions?

[thierry-FreeBSD]

BTW, could you please tell me where you got pygments 2.12?

[TheJJ]

Uhm good point, no idea where I got it from... In my gentoo i have 2.11.2, i guess i've misread this as 2.12? But strange, I find no trace of 2.12 on my system, and I hardly believe I've seen ghosts :D

[danfe]

I believe commit d17159f fixes this once and and for all, and could probably explain why commit 3aa91ea was sufficient for Pygments v2.9, but not for v2.10 and above.