succinctlabs
Panic for `cuda` prover running in docker container
Component
Other (please describe), sp1-sdk
Have you ensured that all of these are up to date?
- [x] SP1 SDK
- [x] cargo prove CLI/sp1up
What version of SP1 SDK are you on?
4.0.1
What version of the cargo prove CLI are you on?
cargo-prove sp1 (a80c17c 2025-04-01T00:41:38.259051986Z)
Operating System
Linux (Other)
Describe the bug
I am struggling quite a bit to get a dockerized prover working, I think there is only one issue that blocks me:
[sp1] groth16 circuit artifacts already seem to exist at /root/.sp1/circuits/groth16/v4.0.0-rc.3. if you want to re-download them, delete the directory
thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sp1-recursion-gnark-ffi-4.1.7/src/ffi/docker.rs:92:61:
failed to prove with docker: Docker command failed
stderr: "panic: open /circuit/groth16_circuit.bin: no such file or directory\n\ngoroutine 17 [running, locked to thread]:\ngithub.com/succinctlabs/sp1-recursion-gnark/sp1.ProveGroth16({0xc0002dee38, 0x8}, {0xc0002dee40, 0x8})\n\t/sp1/crates/recursion/gnark-ffi/go/sp1/prove.go:115 +0xce9\nmain.ProveGroth16Bn254(0x0?, 0x5baf5d79f4b0)\n\t/sp1/crates/recursion/gnark-ffi/go/main.go:115 +0x5a\n"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
^Croot@nuke-gpu-l4:/# ll /circuits/
total 6946624
drwxrwxr-x 2 ubuntu ubuntu 4096 Dec 25 19:56 ./
drwxr-xr-x 1 root root 4096 Apr 22 16:08 ../
-rw-rw-r-- 1 ubuntu ubuntu 24978 Dec 26 05:16 Groth16Verifier.sol
-rw-rw-r-- 1 ubuntu ubuntu 2479 Dec 26 05:16 SP1VerifierGroth16.sol
-rw-rw-r-- 1 ubuntu ubuntu 172612971 Dec 25 19:53 constraints.json
-rw-rw-r-- 1 ubuntu ubuntu 1152193000 Dec 25 19:56 groth16_circuit.bin
-rw-rw-r-- 1 ubuntu ubuntu 2831171769 Dec 26 05:15 groth16_pk.bin
-rw-rw-r-- 1 ubuntu ubuntu 396 Dec 26 05:16 groth16_vk.bin
-rw-rw-r-- 1 ubuntu ubuntu 823158 Dec 25 19:53 groth16_witness.json
-rw-r--r-- 1 root root 2956476403 Apr 22 16:05 pda-proxy.tar.gz
root@nuke-gpu-l4:~/.sp1/circuits/groth16/v4.0.0-rc.3# ls
Groth16Verifier.sol SP1VerifierGroth16.sol constraints.json groth16_circuit.bin groth16_pk.bin groth16_vk.bin groth16_witness.json
thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sp1-recursion-gnark-ffi-4.1.7/src/ffi/docker.rs:92:61:
failed to prove with docker: Docker command failed
stderr: "panic: open /circuit/groth16_circuit.bin: no such file or directory\n\ngoroutine 17 [running, locked to thread]:\ngithub.com/succinctlabs/sp1-recursion-gnark/sp1.ProveGroth16({0xc00025ee38, 0x8}, {0xc00025ee40, 0x8})\n\t/sp1/crates/recursion/gnark-ffi/go/sp1/prove.go:115 +0xce9\nmain.ProveGroth16Bn254(0x0?, 0x653728ad64b0)\n\t/sp1/crates/recursion/gnark-ffi/go/main.go:115 +0x5a\n"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Note I build the container for my app, upload it to a cloud instancve, run the container after manual modifications to ensure permissions and paths are correct etc.
The panic: open /circuit/groth16_circuit.bin: no such file or directory seems to indicate that internally something cannot resolve even with ~.sp1 and /circuits both containing the correct files.
relatedly I really would like a way to trigger https://github.com/succinctlabs/sp1/blob/51e3964b479953e866dc87bd472bf3525a3bcf2d/crates/sdk/src/install.rs#L24 manually and perhaps choose a location so I can do this inside a container (at build time perhaps) to ensure the artifacts I need are present & located where the must be
For reference:
-
Base= Stock ubuntu + nvidia drivers, .env files,staticfiles needed forProxyoperation but not embedded inProxy -
Proxy= pda-proxy app container built with https://github.com/celestiaorg/pda-proxy/blob/main/Dockerfile
So in Base I start Proxy with:
rm -rf ~/.sp1
mkdir ~/.sp1 # so we can link a volume & write to it
docker run -it\
-v /app/static:/app/static\
-v $PDA_DB_PATH:$PDA_DB_PATH\
-v /var/run/docker.sock:/var/run/docker.sock\ # required for docker in docker
-v /root/.sp1:/root/.sp1\ # required to persist
--env-file .env\
--env RUST_LOG=pda_proxy=debug\
--network=host\
--entrypoint /bin/bash\
pda-proxy
I get what looks to be a good download (persisted in Base) but a new panic:
[sp1] groth16 circuit artifacts for version v4.0.0-rc.3 do not exist at /root/.sp1/circuits/groth16/v4.0.0-rc.3. downloading...
[00:01:15] [############] 2.75 GiB/2.75 GiB (37.12 MiB/s, 0s)[sp1] downloaded https://sp1-circuits.s3-us-east-2.amazonaws.com/v4.0.0-rc.3-groth16.tar.gz to "/root/.sp1/circuits/groth16/v4.0.0-rc.3"
thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sp1-recursion-gnark-ffi-4.1.7/src/ffi/docker.rs:92:61:
failed to prove with docker: Docker command failed
stderr: "panic: read /witness: is a directory\n\ngoroutine 17 [running, locked to thread]:\ngithub.com/succinctlabs/sp1-recursion-gnark/sp1.ProveGroth16({0xc000224e68, 0x8}, {0xc000224e70, 0x8})\n\t/sp1/crates/recursion/gnark-ffi/go/sp1/prove.go:145 +0xccb\nmain.ProveGroth16Bn254(0x0?, 0x5b60574864b0)\n\t/sp1/crates/recursion/gnark-ffi/go/main.go:115 +0x5a\n"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
same panic even with artifacts in place on subsequent calls
Looks like the key issue here was needing to use the native-gnark feature, which requires:
- intalling go
- installing clang tooling
here is the gist of the changes:
- https://github.com/celestiaorg/pda-proxy/pull/3/commits/b5319c950a08734ef181f7fa050f190eee392e7e
- https://github.com/celestiaorg/pda-proxy/pull/3/commits/3d2ed3f11170d4b81dccf5dcca810191b2e7e57f#diff-391cf9fb9377a413bbfc6ed99e02fcddce89e8474bd4b1c839cac8eb563d63f2R19
Here is my app's logs for reference:
[2025-04-24T01:42:15Z DEBUG pda_proxy::internal::runner] Starting proof: Job { anchor: Anchor { data: [212, 82, 120, 90, 44, 139, 71, 87, 183, 191, 100, 83, 237, 192, 22, 89, 28, 11, 38, 243, 88, 106, 143, 154, 124, 111, 48, 160, 93, 236, 81, 93] }, input: Input { data: [12, 64, 3, 7, 125] } }
[sp1] groth16 circuit artifacts already seem to exist at /root/.sp1/circuits/groth16/v4.0.0-rc.3. if you want to re-download them, delete the directory
Setting environment variables took 7.97µs
Reading witness file took 134.63µs
Deserializing JSON data took 8.271939ms
Generating witness took 308.424368ms
01:42:26 DBG constraint system solver done nbConstraints=8173052 took=2353.370621
01:42:45 DBG prover done acceleration=none backend=groth16 curve=bn254 nbConstraints=8173052 took=19011.933463
Generating proof took 21.365459855s
ignoring uninitialized slice: Vars []frontend.Variable
ignoring uninitialized slice: Vars []frontend.Variable
ignoring uninitialized slice: Vars []frontend.Variable
01:42:45 DBG verifier done backend=groth16 curve=bn254 took=0.8777
[2025-04-24T01:42:45Z DEBUG pda_proxy::internal::runner] COMPLETE proof: Job { anchor: Anchor { data: [212, 82, 120, 90, 44, 139, 71, 87, 183, 191, 100, 83, 237, 192, 22, 89, 28, 11, 38, 243, 88, 106, 143, 154, 124, 111, 48, 160, 93, 236, 81, 93] }, input: Input { data: [12, 64, 3, 7, 125] } }
[2025-04-24T01:42:45Z INFO pda_proxy::internal::runner] 🎉 Job { anchor: Anchor { data: [212, 82, 120, 90, 44, 139, 71, 87, 183, 191, 100, 83, 237, 192, 22, 89, 28, 11, 38, 243, 88, 106, 143, 154, 124, 111, 48, 160, 93, 236, 81, 93] }, input: Input { data: [12, 64, 3, 7, 125] } } Finished!
[2025-04-24T01:42:45Z DEBUG pda_proxy::internal::runner] ZK request sent
Note that I didn't need go in the final runtime, only at compile time. 😁
I will need to do some testing to see if this changes performance at all, as I can build & run fine without the feature flag and go tooling 🤔 . I hope this native feature enabled is actually more performant at runtime? Would that be a reasonable expectation?
