Make rust build deterministic

[bmwiedemann]

Summary

We disable LTO and use codegen-units = 1 for deterministic build results of the rust code. See https://reproducible-builds.org/ for why this is good.

https://github.com/rust-lang/rust/issues/128675 states that the default codegen-units = 16 should be deterministic, but here it clearly is not.

Test Plan

[zsol]

Hey @bmwiedemann thanks for the PR!

here it clearly is not

Could you help me understand why not? How can I reproduce this nondeterminism?

[bmwiedemann]

I just did two clean builds as similar as possible. Same CPU, same username, same path, same timezone, locale, umask, date... and it still differed in the /usr/lib64/python3.12/site-packages/libcst/native.cpython-312-x86_64-linux-gnu.so

https://rb.zq1.de/compare.factory-20240904/diffs/python-libcst-compare.out shows a (partially filtered) diff.

[bmwiedemann]

hmm. For some reason the variations are gone now with rust-1.81.0

[bmwiedemann]

There are still variations - they just don't appear every time.

[zsol]

It looks like LTO gives us a significant ~20% performance win (as measured by cargo bench --bench parser_benchmark -- parse/all). I don't think disabling LTO in favor of reproducible builds is the right call for this project, I'd much rather have this solved in the rust toolchain.