Pregenerated files in nyacc

[cosinusoidally]

I'm not too familar with mescc/nyacc but looking at https://git.savannah.nongnu.org/cgit/nyacc.git/tree/module/nyacc/lang/c99/mach.d?h=V1.00.2 there appear to be some pregenerated files that are used by the compiler. I know you are pretty strict about pregenerated files. Should these files be rebuilt as part of the bootstrap process?

According to https://git.savannah.nongnu.org/cgit/nyacc.git/tree/module/nyacc/lang/c99/README?h=V1.00.2

mach.d/
  cpp-act.scm	cpp expression parser actions, generated from cppmach.scm
  cpp-tab.scm	cpp expression parser tables, generated from cppmach.scm

  c99-act.scm	parser actions, generated from mach.scm
  c99-tab.scm	parser tables, generated from mach.scm

  c99x-act.scm	expression parser actions, generated from mach.scm
  c99x-tab.scm	expression parser tables, generated from mach.scm

  c99cx-act.scm	constant expression parser actions, generated from cxmach.scm
  c99cx-tab.scm	constant expression parser tables, generated from cxmach.scm

I tried removing the files, which then caused live-bootstrap to fail:

 +> /usr/bin/mes-m2 -e main /usr/bin/mescc.scm -D HAVE_CONFIG_H=1 -I include -I
include/linux/x86 -c lib/linux/x86-mes-mescc/crt1.c -o lib/x86-mes/crt1.o
error:include-from-path: not found: :(nyacc/lang/c99/mach.d/cpp-tab.scm)
exiting...
unhandled exception: error: (("include-from-path: not found: " "nyacc/lang/c99/m
ach.d/cpp-tab.scm"))
Backtrace:
  [b] (error _ _)

In https://git.savannah.nongnu.org/cgit/nyacc.git/tree/HACKING?h=V1.00.2#n27 it gives instructions on how to rebuild the files:

IF you modify module/nyacc/lang/c99/mach.scm you will need to rebuild the
files in the subdirectory mach.d.  To do this run, in guile, the following:
  (use-modules (nyacc lang c99 mach))
  (use-modules (nyacc lang c99 cppmach))

  (gen-c99-files ".")
  (compile-file "parser.scm")
  (gen-cpp-files ".")
  (compile-file "cpp.scm")

but I can't figure out how to run the above code snippet with mes:

# mes-m2
mes: boot failed: no such file: boot-0.scm

I'm sure there's some kind of module path logic for mes, but I've no idea how to set it.

[fosslinux]

From a quick glance, you are absolutely correct! Luckily, this one should be pretty easily fixable..

[stikonas]

From a quick glance, you are absolutely correct! Luckily, this one should be pretty easily fixable..

I'm not so sure about that. It does not run in mes-m2 right now.

[stikonas]

mes in some experimental branch has support for modules, not sure about mes-m2 though.

[fosslinux]

Update;

It is pretty easy to get the mach.d rebuild working from the nyacc and mes tarballs, just by adding the modules in using the same pattern that has been used for the rest of nyacc. A few other guile functions had to be added to guile.mes, but nothing super significant.

However, this rebuild uses psyntax.pp (https://gitlab.com/janneke/mes/-/blob/master/mes/module/nyacc/lalr.mes#L25) in https://git.savannah.nongnu.org/cgit/nyacc.git/tree/module/nyacc/lalr.scm#n127.

Based on IRC discussions the simplest stop-gap solution is likely to be rewriting the 1 syntax-case and 5 syntax-rules from lalr.scm into define-macro, but that is beyond my personal Scheme knowledge.

[schierlm]

@fosslinux: Do you happen to have a branch of mes uploaded, where the regeneration of the grammars already works (using psyntax.pp)? And maybe even a Makefile rule or script to invoke it? Would make it easier for contributors to take a stab at rewriting the syntax-case stuff and quickly testing whether it is still "good enough".

[Googulator]

From IRC discussion:

[Googulator] given the nebulous release date of 0.26.1 (which may or may not include the patch for regenerating Nyacc's files), we might need a 0.25-lb1 or 0.26-lb1 [stikonas] Googulator: no, there isn't and I'm not evne aware of some private patch well, 0.26 or 0.26.1 makes it easier to run guile modules there but I don't think it solves the problem anyway because the current guile method was depending on ppsyntax file (which would be pregenerated file in mes) otherwise unused in mescc and we delete it [Googulator] seems like it was Timothy Sample who achieved... something... without psyntax-pp: https://github.com/schierlm/guile-psyntax-bootstrapping/issues/1#issuecomment-1899297115 and I guess the issues he ran into are fixed in 0.26, hence the reference to 0.26 being a requirement for properly bootstrapping Nyacc [stikonas] oh, yeah, now I seem to remember this comment anyway, I don't know any more than in this ticket [Googulator] he did release https://gist.github.com/samplet/78a2e92d47dd37b17886b26dffa14ffd - but I'm not sure if this + double-execution fix on top of 0.26 is sufficient if it is, I'm inclined to roll a 0.26-lb1 release [stikonas] well, ask him I guess... and you can ask jannekefor 0.26.1 [Googulator] syntax-rules doesn't depend on psyntax-pp, right? only syntax-case [stikonas] no idea :( it's scheme but most likely you are right [mihi] Googulator & stikonas: samplet's patch is for nyacc, not for mes, and I doubt anybody wants to fork the old nyacc version mes uses to make a new release. So i'd keep the patch out-of-tree (either in mes or in live-bootstrap). And whether syntax-rules (and defmacro) requires psyntax or not depends on how macro support is implemented. In the way how mes do it, they don't need psyntax, in the way Guile does it, they do (which made bootstrapping psyntax on Guile slightly harder)

The referenced patch (for nyacc-1.00.2): https://gist.github.com/samplet/78a2e92d47dd37b17886b26dffa14ffd With this, plus mes-0.26, the following code can regenerate the files:

mes -L module gen-cpp-files.scm
# More RAM for this one!
MES_ARENA=32000000 mes -L module gen-c99-files.scm
mes -L module gen-c99cx-files.scm

(note the slightly misleading naming scheme; gen-c99-files.scm regenerates both c99 and c99x files)

Open questions:

• Does this truly work with psyntax.pp deleted? (In particular, lalr.scm uses syntax-rules, which requires psyntax.pp in Guile, but apparently not in Mes?)
• Is mes-m2 sufficient to execute this code? (Building full mes using mescc relies on these very files being present.)
  • If not, what needs to be added?
• How do we get this patch applied that early?
  • Can we use replace and/or simple-patch?
  • Can we get this patch upstreamed into nyacc, and then upgrade to the next nyacc release?
  • Do we need to roll our own nyacc-1.00.2-lb1 release, like we did for fiwix & lwext4?
  • Since mes-0.26 is required, which has the mescc double-execution bug, unless we can expect a 0.26.1 release soon, all of this also applies to the double-execution patch.