Add build info

[qinsoon]

This PR adds build info to mmtk-core, including mmtk-core version, git hash, and features. built is used to collect those information. This closes https://github.com/mmtk/mmtk-core/issues/634.

• use Rust build script to invoke built.
• add a public mmtk::build_info module.
• CI invokes setup scripts based on the architecture triple.

Related PRs:

• https://github.com/mmtk/mmtk-openjdk/pull/174
• https://github.com/mmtk/openjdk/pull/14

[qinsoon]

binding-refs OPENJDK_BINDING_REPO=qinsoon/mmtk-openjdk OPENJDK_BINDING_REF=tph-version

[wks]

I experienced some strange behaviours when building mmtk-core.

$ git clean -dfx
$ killall code # ensure no VSCode, rls or rust-analyzer are running in the background
$ cargo build
$ cat target/debug/build/mmtk-a8628ec218cf1b0a/out/built.rs | grep DIRTY # Your build hash may be different
pub const GIT_DIRTY: Option<bool> = Some(false);

It shows that the repo is not dirty. That's right.

$ echo 42 > foo.txt
$ echo "" > src/empty_file.rs
$ echo "" > tests/empty_test.rs
$ cargo build
   Compiling mmtk v0.14.0 (/home/wks/projects/mmtk-github/parallels/mmtk-version/mmtk-core)
    Finished dev [unoptimized + debuginfo] target(s) in 1.87s
$ cat target/debug/build/mmtk-a8628ec218cf1b0a/out/built.rs | grep DIRTY
pub const GIT_DIRTY: Option<bool> = Some(false);

At this moment, git status will show the repo is dirty, but built.rs is not updated.

$ echo "" >> src/lib.rs
$ cargo build
   Compiling mmtk v0.14.0 (/home/wks/projects/mmtk-github/parallels/mmtk-version/mmtk-core)
    Finished dev [unoptimized + debuginfo] target(s) in 1.74s
$ cat target/debug/build/mmtk-a8628ec218cf1b0a/out/built.rs | grep DIRTY
pub const GIT_DIRTY: Option<bool> = Some(true);

If, for some reason, it triggered rebuild, then built.rs will be updated.

Then I commit the changes.

$ git add .
$ git commit -m "Add a new line after lib.rs"
[mmtk-version-test ec788ff15] Add a new line after lib.rs
 4 files changed, 4 insertions(+)
 create mode 100644 foo.txt
 create mode 100644 src/empty_file.rs
 create mode 100644 tests/empty_test.rs
$ cargo build
    Finished dev [unoptimized + debuginfo] target(s) in 0.03s
$ cat target/debug/build/mmtk-a8628ec218cf1b0a/out/built.rs | grep DIRTY
pub const GIT_DIRTY: Option<bool> = Some(true);

And the built.rs file is not regenerated after I commit it.

I can force a rebuild by cargo clean

$ cargo clean
$ cargo build
....
$ cat target/debug/build/mmtk-a8628ec218cf1b0a/out/built.rs | grep DIRTY
pub const GIT_DIRTY: Option<bool> = Some(false);

The built tool is supposed to identify the case when mmtk-core is accidentally built when the repository is dirty by accident. From what I observed, it is not always detecting dirty repos, and may give false alarm when the repo has just been committed.

Is it possible to force regenerating built.rs at every cargo build? As for the overhead, I guess Rust's incremental build system can avoid rebuilding the entire mmtk-core if it detects the generated built.rs is not changed, or only built.rs changed.

[wks]

I also added some tests just to have a look at the version strings. I put my code here: https://github.com/mmtk/mmtk-core/compare/master...wks:mmtk-core:mmtk-version-test

Run with cargo test -- --nocapture test_build_info to see the output.

We can run the tests as smoke tests, i.e. they pass as long as they don't panic.

[qinsoon]

Is it possible to force regenerating built.rs at every cargo build? As for the overhead, I guess Rust's incremental build system can avoid rebuilding the entire mmtk-core if it detects the generated built.rs is not changed, or only built.rs changed.

According to https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorerun-if-changedpath, the default rebuild behavior is to scan the entire package for changes. Maybe when you have changes in the git repo but have no actual change to the cargo project, a rebuild won't be triggered.

[wks]

Maybe when you have changes in the git repo but have no actual change to the cargo project, a rebuild won't be triggered.

I added some printf! in build.rs to see if it is executed (requires cargo build -vv to see the printed messages), and I also tried to delete the generated built.rs to see if it re-generates it. So far, what I observed is, after touch foo.txt,cargo build -vv shows that

• build.rs is executed, and
• built.rs is regenerated, and
• libmmtk.rlib is regenerated, too.

But the re-generated built.rs still contains pub const GIT_DIRTY: Option<bool> = Some(false);

So it should be the built tool that decides the repo is still not really dirty.

[wks]

From the source of built, it ignores untracked and ignored files. See: https://github.com/lukaslueg/built/blob/d4e39e6587986c267de1cc3156c98143f9b06b77/src/util.rs#L80

In my case, after I run touch bar.txt, the file bar.txt is untracked. So if I run cargo build -vv, it will re-run build.rs, but built will skip bar.txt because it is untracked. Therefore, the generated built.rs has GIT_DIRTY being Some(false).

The tricky part is, I then run git add bar.txt. This time, bar.txt is officially tracked by Git. However, if I run cargo build -vv, cargo will refuse to rebuild mmtk-core because all files have the same timestamp. build.rs is not even run!

If I then run touch bar.txt again, and run cargo build -vv, it will regenerate built.rs with GIT_DIRTY being Some(true).

So in conclusion, built is having a hard time tracking untracked files. However, if I change an existing source file, it will always identify the change (at least I haven't found a way to cheat it). We probably won't care if it falsely reports a clean repo to be dirty, because in any production or benchmarking settings, we always build from clean repos.

~~One corner case is with bindings. Sometimes we edit Cargo.toml to depend on a local mmtk-core repo, and in this case built will consider the repo (of the binding, such as mmtk-openjdk, mmtk-jikesrvm...) dirty. I don't know if that would bother us.~~ This shouldn't be a problem, because we are not tracking git version of the bindings.

[qinsoon]

This shouldn't be a problem, because we are not tracking git version of the bindings.

Actually I just added that change to the OpenJDK binding. In the OpenJDK binding, we print the binding version as long with the mmtk-core version. As you said, if we modify Cargo.toml in the binding to point to a local MMTk path, it will show the binding's git as 'dirty'. However, I don't think we want to work around this in any way -- after all it is actually dirty.