bddicken
Restructuring compilation, benchmarking
As I mentioned in #319 and comments, I built a series of patches on top of ca443ca (update, 2024-12-17) over the holidays. The rapid progression since that commit made a simple rewrite or rebase a bit trickier, so I've put off doing it, but I'd like to discuss the restructuring a bit first and suggest some ways it could be handled with latest main.
Compilation, Cleaning
The main idea is to move the compilation and clean commands to a POSIX-portable make syntax using suffix rules. Here's an example rule for a non-C language just to show it can be done:
.rkt:
raco make $<
raco demod -o $*.zo $<
raco exe $(RACO_FLAGS) -o $@ $*.zo
After adding .rkt to .SUFFIXES, then the idea is that cd bench && make racket/code would just do the right thing (in this case, compiling the file, demodularizing it, and packaging up a standalone executable). Further, artifacts could be listed for a PHONY clean target to remove, and all compiled programs would be dependencies of the (default by virtue of being first) all target, so that instead of cd bench && ../compile.sh you run cd bench && make.
The magic that makes this happen in the current series is putting a universal base.make in the root and then a simple shim in each benchmark:
.POSIX:
SHELL = /bin/sh
include ../base.make
Benchmarking
There are less big changes here. Primarily:
- removing unneeded
.shprefixes (that is typical of libraries but not programs) - not ignoring benchmark program failures (which should be a bug for the relevant program)
- throwing all commands into a single
hyperfinerun to get relative speed reports
I also tweaked hyperfine settings (the number of runs and warmups seemed low for statistics to me) and, against my better judgement, kept the sed command, which makes hyperfines output less useful by disabling any indicators of progress.
Long term, it would be nice to be able to run a named subset of programs (my script limits programs by checking whether the program can be invoked) to allow iterating on a specific language with the repo's machinery rather than extra-judicially (i.e., running programs directly in the shell). This means that contributors wouldn't need to install all the languages and pay for all the compilation time if they didn't want to (with a Makefile, you can always opt out of compiling some targets by specifying only those you want to build).
Implementation
My head start on this consists of the following commits: https://github.com/bddicken/languages/commit/5930c2ef4d95738522c9c1284d7b3a6e19df8dec (convert clean.sh to Makefile, 2024-12-18), https://github.com/bddicken/languages/commit/65e6d2297935f58299fa60ffa904d05b3049b92a (convert compile.sh to makefile, 2024-12-18), https://github.com/bddicken/languages/commit/95e7646b5ed191be8846fc8608fb794e47667e31 (convert run.sh to run, 2024-12-20), https://github.com/bddicken/languages/commit/04be35121a4a746cef22ef2b10ab4a64028b5864 (docs: mention new structures, 2024-12-20), https://github.com/bddicken/languages/commit/9b44b70fc65d532f6f433952e19144171dd59e92 (run: do not ignore failures when running benchmarks, 2024-12-20), https://github.com/bddicken/languages/commit/02f93564ba5bbfec55cb3a544e62f3181281a5d8 (loops: include an input.txt for ../run, 2024-12-20), https://github.com/bddicken/languages/commit/4cec75a01315abe24e6612068673e7d500b16bb8 (fib: include an input.txt for ../run, 2024-12-22).
The input.txt pieces may be less relevant; I haven't followed the changes to run.sh closely.
In comments, I proposed a way to attempt to migrate that series by adding one new mainline commit underneath it at a time. That's slow going, but with a pause in new merges would eventually catch up.
These are cool ideas! This is a lot of work, and it might help to try making several smaller PRs rather than one large one.
For example, the idea of allowing hyperfine to produce relative timing results is cool. Could we start by by making a PR with just that change? It is a small amount of work?
Understandable, the build / makefile restructuring is probably a big undertaking. Maybe we can hold off on that and pick out the other (smaller) good ideas first!
By the way, I'm also open to chatting about improvements like this over zoom. If you wanna set something up, ping me!
These are cool ideas! This is a lot of work, and it might help to try making several smaller PRs rather than one large one.
Agreed.
For example, the idea of allowing hyperfine to produce relative timing results is cool. Could we start by by making a PR with just that change? It is a small amount of work?
I'll take a quick look at what that would require; I might be able to cherry-pick those commits to the latest tip and then adjust for missing runners.
Understandabl[y], the build / makefile restructuring is probably a big undertaking. Maybe we can hold off on that and pick out the other (smaller) good ideas first!
Agreed—with the main problem being how many new languages have been added since I started that series 😅