Pull lisp files directly from Emacs

[Qkessler]

Opening this issue to discuss whether we can optimize the directory structure that we have on the project. The reasoning is that the lisp files are not something that will ever be modified as part of this project, but rather just a "fleeting" directory, where we add the files that are ready to be bootstrapped or similar.

I know that we have the bootstrap.el file with the tag RUNE_BOOTSTRAP and certainly that can stay, but what about having a git submodule of the Emacs core? That way, we can pull the changes that we need and even have a fork/branch with only the files working with Rune.

What do you think? Open to suggestions.

[CeleritasCelery]

I think that is a pretty solid idea. It would let us keep the git history for the lisp files and easily update them without having to blow them away and recopy them. My only concern is increasing the time to clone and run git operations. That is reason I didn't use a submodule in the first place. For example on my machine it takes 4 minutes to clone git://git.sv.gnu.org/emacs.git.

[Qkessler]

For the cloning issues, we could have a shallow repo, when cloning the submodule just to have the latest commit and that should theoretically improve that (need to check on details).

I'll try doing something like that and test what a first contributor would do.

[Qkessler]

Have been testing different commands, to see what gives us the best chance of getting contributors up to speed quickly:

❯ time git clone --depth=1 git://git.sv.gnu.org/emacs.git savannah-emacs
Cloning into 'savannah-emacs'...
warning: templates not found in /Users/enrikes/.git-templates
remote: Counting objects: 5514, done.
remote: Compressing objects: 100% (5032/5032), done.
remote: Total 5514 (delta 741), reused 2143 (delta 283)
Receiving objects: 100% (5514/5514), 48.44 MiB | 712.00 KiB/s, done.
Resolving deltas: 100% (741/741), done.
git clone --depth=1 git://git.sv.gnu.org/emacs.git savannah-emacs  3.84s user 2.15s system 4% cpu 2:09.41 total
❯ time git clone --depth=1 [email protected]:emacs-mirror/emacs.git github-emacs
Cloning into 'github-emacs'...
warning: templates not found in /Users/enrikes/.git-templates
remote: Enumerating objects: 5514, done.
remote: Counting objects: 100% (5514/5514), done.
remote: Compressing objects: 100% (4862/4862), done.
remote: Total 5514 (delta 646), reused 2399 (delta 453), pack-reused 0
Receiving objects: 100% (5514/5514), 48.60 MiB | 1.21 MiB/s, done.
Resolving deltas: 100% (646/646), done.
Updating files: 100% (5205/5205), done.
git clone --depth=1 [email protected]:emacs-mirror/emacs.git github-emacs  4.20s user 2.02s system 11% cpu 55.348 total
❯ time git clone --depth=1 --branch=master [email protected]:emacs-mirror/emacs.git github-emacs-29
Cloning into 'github-emacs-29'...
warning: templates not found in /Users/enrikes/.git-templates
remote: Enumerating objects: 5514, done.
remote: Counting objects: 100% (5514/5514), done.
remote: Compressing objects: 100% (4862/4862), done.
remote: Total 5514 (delta 646), reused 2399 (delta 453), pack-reused 0
Receiving objects: 100% (5514/5514), 48.60 MiB | 809.00 KiB/s, done.
Resolving deltas: 100% (646/646), done.
Updating files: 100% (5205/5205), done.
git clone --depth=1 --branch=master [email protected]:emacs-mirror/emacs.git   4.05s user 2.12s system 8% cpu 1:11.29 total

TL;DR We should use shallow submodules, where we can do something like this:

git clone --depth=1 --recurse-submodules --shallow-submodules [email protected]:CeleritasCelery/rune.git

That way, we'll have both the repo and the submodules shallow. We could potentially remove the --depth=1 option, to only have the shallow Emacs clone, which would still be pretty fast. At most, we would be looking at ~1m if pulling from the Github Mirror. I would prefer that to the Savannah one.

[Qkessler]

I'll make a commit on my fork and open a PR.

[CeleritasCelery]

At most, we would be looking at ~1m if pulling from the Github Mirror. I would prefer that to the Savannah one.

In my experience, cloning from Savannah is much faster than cloning from the Github mirror.

[Qkessler]

Interesting, haven't seen that myself. Probably I'm on a crappy mobile hotspot right now and that's playing tricks on me. If you feel Savannah is the place to be, I don't have a strong preferrence.

[CeleritasCelery]

It could just be the server load, but when I just tested it, cloning from Savannah took 4 minutes and cloning from Github took 30.

[Qkessler]

I need to think about overrides that we might want to do to the files. Couple of options come to mind:

1. We define a set of local files (can still be part of another submodule, to avoid having elisp and Rust mixed, though it's just personal preferrence) that have the same paths as in the Emacs submodule, and we would prefer those over Emacs'.
2. We would just start bootstrapping Emacs files and order them with "the ones that work" first. We can then group the ones that don't and work on those, to cover all the lisp subdirectory.

What do you think?

[CeleritasCelery]

Ideally what I would like to see from any solution is:

1. easily diff the the "bootstraped modified" version against the pure versions. We can either do this by doing a submodule of the original files and adding new commits to them, or by keeping two copies of the files.
2. Easily upgrade to new version of the elisp. This is where keeping two copies of the files would be tricky because we could update the "pure" files but then would have to manually merge over the changes from the bootstrapped versions.

Given all that I think the idea of having them in a submodule forked from GNU Emacs makes the most sense. It would be easy to diff the files with git and we could just do a git pull to uprev versions and resolve the conflicts.

If we wanted to make submodule cheaper, we could do a sparse checkout of just the lisp directory

[Qkessler]

Thanks! Question @CeleritasCelery, when running, where is the load-path initially pulled from? I'm playing with submodules and running cargo run and have issues:

unknown arg: target/debug/rune
Loading emacs/lisp/bootstrap.el...
Loading loadup...
MESSAGE: Dump mode: nil
Error: Signal
BEGIN_BACKTRACE
0: signal [error ("Could not find elisp load-path: searched ("/Users/enrikes/Documents/rune/lisp")") ]
1: load ["loadup" ]
END_BACKTRACE

[CeleritasCelery]

https://github.com/CeleritasCelery/rune/blob/72fbfca597011803b622aebd6b9aea66d633b343/src/lread.rs#L199

it is defined relative to CARGO_MANIFEST_DIR which is the directory that contains the top level cargo.toml file.