racket-rash
racket-rash copied to clipboard
willghatch
rash build result varies from parallelism
While working on reproducible builds for openSUSE, I found that
our rash package varies when comparing builds from 1-core-VM and a 4-core-VM.
Two builds from 1-core-VMs are identical though, so there is some race going on during the build.
Here is an extract from the lengthy total diff:
/usr/share/racket/pkgs/shell-pipeline/shell/demo/compiled/more-pipeline-operators_rkt.zo differs at offset '41' (data)
--- a 2019-04-08 20:45:02.853104872 +0000
+++ b 2019-04-08 20:45:02.853104872 +0000
@@ -1,33 +1,33 @@
00000000 23 7e 03 37 2e 32 06 72 61 63 6b 65 74 44 02 00 |#~.7.2.racketD..|
00000010 00 00 12 00 00 00 11 63 6f 6e 66 69 67 75 72 65 |.......configure|
-00000020 2d 72 75 6e 74 69 6d 65 0f 89 00 00 9a 02 00 00 |-runtime........|
+00000020 2d 72 75 6e 74 69 6d 65 27 89 00 00 9a 02 00 00 |-runtime'.......|
00000030 38 00 00 00 00 00 00 00 00 00 00 00 4c 00 00 00 |8...........L...|
-00000040 c3 88 00 00 00 00 00 00 00 00 00 00 23 7e 03 37 |............#~.7|
-00000050 2e 32 06 72 61 63 6b 65 74 42 ae 89 99 2b 9f 9c |.2.racketB...+..|
-00000060 e5 3a 9d 55 7d ed 50 8a 32 5e 75 4f da 18 f2 00 |.:.U}.P.2^uO....|
+00000040 db 88 00 00 00 00 00 00 00 00 00 00 23 7e 03 37 |............#~.7|
+00000050 2e 32 06 72 61 63 6b 65 74 42 6a 31 80 57 72 da |.2.racketBj1.Wr.|
+00000060 ae 16 e0 34 bc 8e f6 ef f9 08 48 78 7f 33 f2 00 |...4......Hx.3..|
00000070 00 00 01 00 00 07 00 17 00 37 00 3c 00 42 00 4f |.........7.<.B.O|
...
/usr/share/racket/pkgs/rash/rash/demo/compiled/rc17-demo-modbeg_rkt.zo differs at offset '41' (data)
@@ -1,33 +1,33 @@
00000000 23 7e 03 37 2e 32 06 72 61 63 6b 65 74 44 02 00 |#~.7.2.racketD..|
00000010 00 00 12 00 00 00 11 63 6f 6e 66 69 67 75 72 65 |.......configure|
-00000020 2d 72 75 6e 74 69 6d 65 08 20 03 00 8b 02 00 00 |-runtime. ......|
+00000020 2d 72 75 6e 74 69 6d 65 0a 20 03 00 8b 02 00 00 |-runtime. ......|
00000030 38 00 00 00 00 00 00 00 00 00 00 00 4c 00 00 00 |8...........L...|
-00000040 bc 1f 03 00 00 00 00 00 00 00 00 00 23 7e 03 37 |............#~.7|
-00000050 2e 32 06 72 61 63 6b 65 74 42 22 58 4a f2 09 a4 |.2.racketB"XJ...|
-00000060 40 6f 08 3e ee 1d aa ae dd d0 52 72 b7 24 a5 07 |@o.>......Rr.$..|
-00000070 00 00 01 00 00 07 00 27 00 40 00 4e 00 51 00 55 |.......'[email protected]|
-00000080 00 6e 00 72 00 76 00 8c 00 a8 00 c0 00 c9 00 d1 |.n.r.v..........|
-00000090 00 df 00 e3 00 ed 00 f2 00 f8 00 04 01 11 01 26 |...............&|
-000000a0 01 34 01 43 01 48 01 4f 01 5e 01 67 01 77 01 82 |.4.C.H.O.^.g.w..|
-000000b0 01 8e 01 99 01 ba 01 da 01 f4 01 ff 01 0b 02 10 |................|
-000000c0 02 16 02 1b 02 24 02 2c 02 32 02 3c 02 4c 02 54 |.....$.,.2.<.L.T|
-000000d0 02 5c 02 73 02 7d 02 8a 02 98 02 b3 02 b9 02 c6 |.\.s.}..........|
-000000e0 02 d4 02 e6 02 f1 02 00 03 0c 03 1b 03 22 03 38 |.............".8|
-000000f0 03 51 03 5e 03 68 03 72 03 84 03 8a 03 9c 03 ae |.Q.^.h.r........|
-00000100 03 c4 03 c7 03 d3 03 d6 03 d9 03 dd 03 e1 03 e5 |................|
-00000110 03 ea 03 ef 03 f3 03 f9 03 07 04 0c 04 18 04 26 |...............&|
-00000120 04 2e 04 30 04 32 04 34 04 38 04 3a 04 41 04 43 |...0.2.4.8.:.A.C|
-00000130 04 46 04 48 04 4b 04 6b 04 85 04 9a 04 ac 04 bd |.F.H.K.k........|
-00000140 04 c6 04 db 04 f2 04 fd 04 0a 05 11 05 2d 05 31 |.............-.1|
-00000150 05 3f 05 64 05 80 05 9a 05 ae 05 b3 05 b8 05 cc |.?.d............|
-00000160 05 e7 05 eb 05 f7 05 13 06 15 06 18 06 1a 06 35 |...............5|
-00000170 06 37 06 53 06 57 06 66 06 6b 06 70 06 7a 06 8a |.7.S.W.f.k.p.z..|
-00000180 06 97 06 a2 06 a6 06 ad 06 b5 06 bb 06 c2 06 c8 |................|
-00000190 06 d2 06 e3 06 f2 06 08 07 18 07 28 07 30 07 35 |...........(.0.5|
-000001a0 07 3a 07 40 07 45 07 4a 07 4f 07 56 07 67 07 6e |.:[email protected]|
-000001b0 07 7a 07 8c 07 9d 07 a9 07 b5 07 c1 07 ca 07 de |.z..............|
-000001c0 07 e2 07 eb 07 f9 07 fe 07 0c 08 25 08 33 08 40 |...........%.3.@|
-000001d0 08 4b 08 56 08 72 08 7f 08 8c 08 99 08 a7 08 b3 |.K.V.r..........|
-000001e0 08 bf 08 cc 08 d2 08 d8 08 f0 08 fc 08 08 09 1d |................|
-000001f0 09 34 09 4a 09 5f 09 6c 09 83 09 91 09 a3 09 b4 |.4.J._.l........|
+00000040 be 1f 03 00 00 00 00 00 00 00 00 00 23 7e 03 37 |............#~.7|
+00000050 2e 32 06 72 61 63 6b 65 74 42 52 c2 82 78 e7 99 |.2.racketBR..x..|
+00000060 f6 f6 7f e2 4e ea 55 9d 34 88 49 0f ce a6 a5 07 |....N.U.4.I.....|
+00000070 00 00 01 00 00 07 00 28 00 41 00 4f 00 52 00 56 |.......(.A.O.R.V|
+00000080 00 6f 00 73 00 77 00 8d 00 a9 00 c1 00 ca 00 d2 |.o.s.w..........|
+00000090 00 e0 00 e4 00 ee 00 f3 00 f9 00 05 01 12 01 27 |...............'|
+000000a0 01 35 01 44 01 49 01 50 01 5f 01 68 01 78 01 83 |.5.D.I.P._.h.x..|
+000000b0 01 8f 01 9a 01 bb 01 db 01 f5 01 00 02 0c 02 11 |................|
+000000c0 02 17 02 1c 02 25 02 2d 02 33 02 3d 02 4d 02 55 |.....%.-.3.=.M.U|
+000000d0 02 5d 02 74 02 7e 02 8b 02 99 02 b4 02 ba 02 c7 |.].t.~..........|
+000000e0 02 d5 02 e7 02 f2 02 01 03 0d 03 1c 03 23 03 39 |.............#.9|
+000000f0 03 52 03 5f 03 69 03 73 03 85 03 8b 03 9d 03 af |.R._.i.s........|
+00000100 03 c5 03 c8 03 d4 03 d7 03 da 03 de 03 e2 03 e6 |................|
+00000110 03 eb 03 f0 03 f4 03 fa 03 08 04 0d 04 19 04 27 |...............'|
+00000120 04 2f 04 31 04 33 04 35 04 39 04 3b 04 42 04 44 |./.1.3.5.9.;.B.D|
+00000130 04 47 04 49 04 4c 04 6c 04 86 04 9b 04 ad 04 be |.G.I.L.l........|
+00000140 04 c7 04 dc 04 f3 04 fe 04 0b 05 12 05 2e 05 32 |...............2|
+00000150 05 40 05 65 05 81 05 9b 05 af 05 b4 05 b9 05 cd |[email protected]............|
+00000160 05 e8 05 ec 05 f8 05 14 06 16 06 19 06 1b 06 36 |...............6|
+00000170 06 38 06 54 06 58 06 67 06 6c 06 71 06 7b 06 8b |.8.T.X.g.l.q.{..|
+00000180 06 98 06 a3 06 a7 06 ae 06 b6 06 bc 06 c3 06 c9 |................|
+00000190 06 d3 06 e4 06 f3 06 09 07 19 07 29 07 31 07 36 |...........).1.6|
+000001a0 07 3b 07 41 07 46 07 4b 07 50 07 57 07 68 07 6f |.;.A.F.K.P.W.h.o|
+000001b0 07 7b 07 8d 07 9e 07 aa 07 b6 07 c2 07 cb 07 df |.{..............|
+000001c0 07 e3 07 ec 07 fa 07 ff 07 0d 08 26 08 34 08 41 |...........&.4.A|
+000001d0 08 4c 08 57 08 73 08 80 08 8d 08 9a 08 a8 08 b4 |.L.W.s..........|
+000001e0 08 c0 08 cd 08 d3 08 d9 08 f1 08 fd 08 09 09 1e |................|
+000001f0 09 35 09 4b 09 60 09 6d 09 84 09 92 09 a4 09 b5 |.5.K.`.m........|
00000200
+++ new//usr/share/racket/pkgs/rash/rash/demo/compiled/rc17_rkt.dep 2019-04-05 12:00:00.000000000 +0000
@@ -1 +1 @@
-("7.2" racket ("efe0dc5499c61488b11498a84f78962ffa64b116" . "408d55c1c3a6e91ef58aa3821ba4cc2704944c0e") (collects #"linea" #"read.rkt") (collects #"racket" #"base.rkt") (collects #"racket" #"runtime-config.rkt") (collects #"rash" #"private" #"lang-funcs.rkt") (collects #"syntax" #"module-reader.rkt"))
+("7.2" racket ("efe0dc5499c61488b11498a84f78962ffa64b116" . "b86f4cd74942f160d7cdb7d9b2fc87f18423b045") (collects #"linea" #"read.rkt") (collects #"racket" #"base.rkt") (collects #"racket" #"runtime-config.rkt") (collects #"rash" #"private" #"lang-funcs.rkt") (collects #"syntax" #"module-reader.rkt"))
/usr/share/racket/pkgs/rash/rash/demo/compiled/setup_rkt.dep differs (ASCII text, with very long lines)
Unfortunately I know too little about racket or rash to debug this without some help.
longer diff: http://rb.zq1.de/compare.factory-20190405/rash-compare.out (showing first 200 lines of each file's diff)
Thanks! This looks like a problem more generally with Racket's build system, so I've added an issue to the Racket repository as well: https://github.com/racket/racket/issues/2601
Or perhaps I'm wrong! I'll have to look through the implementation and see if/where I'm using some non-deterministic hash-table traversal or gensym...
I've removed some code that was likely the culprit (gensyms in macros that I was using for some reason). So if this were re-done using the master branch it would perhaps be reproducible now. I can't think of any other macros that could be doing anything non-reproducible.
@willghatch usually you can test this by compiling to zo on a few different machines (or with -j options to raco make) and seeing if you get the same contents.
@samth Thanks. After trying to think of a more convenient way and failing, I've been doing exactly that.
Which leads me to the bad news that it's still not reproducible. I think I've isolated it to a single file (git-info.rkt). But I haven't figured out what causes it yet. The only macro used there should be deterministic. It uses gensym at runtime to get a singleton value, but that shouldn't affect reproducible compilation (and removing it doesn't fix it). Anyway, I'll look at this some more after I eat some lunch.
So my last statement of isolation was way off, that was just an artifact of the way I first started re-kicking builds. At this point it seems that: Rash's main.rkt and its dependencies are reproducible. But any file using main.rkt via #lang rash that also uses begin-for-syntax, define-syntax, or otherwise directly has code at phase 1 is not reproducible (though using pre-defined Rash macros causes no problem). I'm confused as to how this is happening. My best guess is that it has something to do with how my #%module-begin macro is set up. But generally I'm mystified.
@samth Thanks for your help on this, by the way. Aside from gensym and hash-table traversals, are there any other features that come to mind? As far as I know, raco setup gets parallelism by using places, not by spawning new processes. If it were spawning new processes, I wouldn't expect gensym or hash-table traversals to cause non-reproducibility, but I'm not really sure how those things work in places where some things are shared. Even with places I find this slightly confusing, since each place has its own garbage collector and everything. But I've never really used places and I'm not sure what is shared besides the ability to send things over channels.
Both gensym and hash traversal order are affected by what order things happen in globally (for example, memory addresses or how many gensyms have been created). So lots of things can affect them.
Looking at the expansion of a simple rash module, here are a few things that might be an issue.
- It looks like there's an options hash that gets put into the module -- is the order that those keys appear deterministic?
- It looks like there are some gensyms in something about
current-paramter-environment
I also noticed that a trivial #lang rash program expands to 360+ lines of code. Could some of that be moved into functions where you expand to a reference to that? That makes it less likely that the expansion could change non-deterministically while also making zo sizes smaller.
When having non-deterministic hashes in other languages, we usually sort when iterating over them. https://github.com/bmwiedemann/theunreproduciblepackage/tree/master/hash has 2 examples.
- It looks like there's an options hash that gets put into the module -- is the order that those keys appear deterministic?
That's actually a run-time hash. But I've removed it anyway because it was stupid.
- It looks like there are some gensyms in something about
current-paramter-environment
That's it. Today I traced it down to my use of splicing-syntax-parameterize. This program does not build reproducibly:
#lang racket/base
(require racket/stxparam racket/splicing (for-syntax racket/base))
(define-syntax-parameter param #'something)
(splicing-syntax-parameterize ([param (make-rename-transformer #'some-identifier)])
(begin-for-syntax (void)))
Looking into the implementation of splicing-syntax-parameterize, it uses a do-local function that uses gensym. I'll look into that more tomorrow or early next week.
I also noticed that a trivial
#lang rashprogram expands to 360+ lines of code. Could some of that be moved into functions where you expand to a reference to that? That makes it less likely that the expansion could change non-deterministically while also making zo sizes smaller.
I did a little bit of that today while poking around it. I should certainly do some more.
This is still a problem with racket-8.11.1 rash-0.2
+++ new//usr/share/racket/pkgs/rash/rash/compiled/experimental_rkt.dep 2022-12-27 00:00:00.000000000 +0000
@@ -1 +1 @@
-("8.11.1" ta6le ("c766a8edc6824eb84469005591ebecfd6a74207e" . "010b0d6788990d0e951970461b12bdfa90d34464") (collects #"racket" #"base.rkt") (collects #"racket" #"runtime-config.rkt") (collects #"rash" #"private" #"escapable-template.rkt") (collects #"rash" #"private" #"lang-funcs.rkt"))
+("8.11.1" ta6le ("c766a8edc6824eb84469005591ebecfd6a74207e" . "0d7d0978458e495d931d91cc1a50fb926f7dd956") (collects #"racket" #"base.rkt") (collects #"racket" #"runtime-config.rkt") (collects #"rash" #"private" #"escapable-template.rkt") (collects #"rash" #"private" #"lang-funcs.rkt"))