Lilypad-Tech
EOF Error when running jobs with dev private key
Describe the bug
Before the release of Lilypad v2.12, I was able to successfully run jobs on the CLI using my regular development private key. However, after updating to v2.12, attempting to run jobs resulted in an EOF error. I checked the RP logs but the job doesn't reach the RP.
I created a new development private key, retried the same command, and it worked fine. This suggests that something changed in how private keys are handled in v2.12, possibly affecting previously used keys.
Note: The Severity of this bug is somewhere between annoyance and blocking all Lilypad usage for the user (if that were their only private key)
Reproduction
Use an existing development private key (generated before v2.12). Run the following job command:
lilypad run cowsay:v0.0.4 -i Message="moo" --web3-private-key $WEB3_PRIVATE_KEY
Observe the following error:
2025-02-18T11:28:27-05:00 ERR ../runner/work/lilypad/lilypad/pkg/jobcreator/controller.go:338 > ๐ข JC failed to download results="error downloading results for deal: unexpected EOF"
Generate a new development private key. Run the same command with the new keyโjob runs successfully.
Logs
lilypad run cowsay:v0.0.4 -i Message="moo" --web3-private-key $WEB3_PRIVATE_KEY
โ โ โ โ โ โ โฃโฃคโฃคโข โฃคโฃโ โ โ โ โ
โ โ โ โ โขดโฃฟโฃฟโฃฟโฃฟโขธโฃฟโกโ โ โ โ โ โโโ โโโโโโ โโโ โโโโโโโโโโ โโโโโโ โโโโโโโ
โ โ โฃฐโฃฟโฃฆโกโขฟโฃฟโฃฟโขธโกฟโ โ โ โ โขโ โโโ โโโโโโ โโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ โขฐโฃฟโฃฟโฃฟโฃฟโฃฆโกโฃฟโขธโ โขโฃ โฃดโฃพโฃฟโก โโโ โโโโโโ โโโโโโโ โโโโโโโโโโโโโโโโโโโ โโโ
โ โฃโฃโฃโฃโฃโฃโฃโ โ โฃโฃโฃโฃโฃโฃโฃ โโโ โโโโโโ โโโโโ โโโโโโโ โโโโโโโโโโโ โโโ
โ โขนโฃฟโฃฟโฃฟโฃฟโ โฃกโฃฟโขธโฃฎโกปโฃฟโฃฟโฃฟโฃฟโก โโโโโโโโโโโโโโโโโโโโโโ โโโ โโโ โโโโโโโโโโโ
โ โ โขปโฃฟโกโฃฉโฃพโฃฟโฃฟโขธโฃฟโฃฟโฃโ ปโฃฟโกโ โโโโโโโโโโโโโโโโโโโโโโ โโโ โโโ โโโโโโโโโโ v2.12.0
โ โ โ โ โขพโฃฟโฃฟโฃฟโฃฟโขธโฃฟโฃฟโฃฟโกทโ โ โ
โ โ โ โ โ โ โ โ โ โ โ โ โ โ โ โ โ Decentralized Compute Network https://lilypad.tech
๐ Lilypad submitting job
2025-02-18T11:28:26-05:00 WRN ../runner/work/lilypad/lilypad/cmd/lilypad/utils.go:63 > failed to get GPU info: gpuFillInfo not implemented on darwin
2025-02-18T11:28:26-05:00 INF ../runner/work/lilypad/lilypad/pkg/web3/sdk.go:209 > Connected to arbitrum-sepolia-rpc.publicnode.com
2025-02-18T11:28:26-05:00 INF ../runner/work/lilypad/lilypad/pkg/jobcreator/run.go:27 > Public Address: 0x765fEB3FB358867453B26c715a29BDbbC10Be772
2025-02-18T11:28:27-05:00 ERR ../runner/work/lilypad/lilypad/pkg/jobcreator/controller.go:338 > ๐ข JC failed to download results="error downloading results for deal: unexpected EOF"
Screenshots
System Info
- Apple M1 Pro
- arm64 CPU binary
- Funded wallet (for original dev and new dev wallet)
Severity
Annoyance
I had a similar issue when running jobs. Used my old Job Creator wallet
lilypad run github.com/noryev/module-sdxl-ipfs:ae17e969cadab1c53d7cabab1927bb403f02fd2a -i prompt="a technology decentralized network"
โ โ โ โ โ โ โฃโฃคโฃคโข โฃคโฃโ โ โ โ โ
โ โ โ โ โขดโฃฟโฃฟโฃฟโฃฟโขธโฃฟโกโ โ โ โ โ โโโ โโโโโโ โโโ โโโโโโโโโโ โโโโโโ โโโโโโโ
โ โ โฃฐโฃฟโฃฆโกโขฟโฃฟโฃฟโขธโกฟโ โ โ โ โขโ โโโ โโโโโโ โโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ โขฐโฃฟโฃฟโฃฟโฃฟโฃฆโกโฃฟโขธโ โขโฃ โฃดโฃพโฃฟโก โโโ โโโโโโ โโโโโโโ โโโโโโโโโโโโโโโโโโโ โโโ
โ โฃโฃโฃโฃโฃโฃโฃโ โ โฃโฃโฃโฃโฃโฃโฃ โโโ โโโโโโ โโโโโ โโโโโโโ โโโโโโโโโโโ โโโ
โ โขนโฃฟโฃฟโฃฟโฃฟโ โฃกโฃฟโขธโฃฎโกปโฃฟโฃฟโฃฟโฃฟโก โโโโโโโโโโโโโโโโโโโโโโ โโโ โโโ โโโโโโโโโโโ
โ โ โขปโฃฟโกโฃฉโฃพโฃฟโฃฟโขธโฃฟโฃฟโฃโ ปโฃฟโกโ โโโโโโโโโโโโโโโโโโโโโโ โโโ โโโ โโโโโโโโโโ v2.12.0
โ โ โ โ โขพโฃฟโฃฟโฃฟโฃฟโขธโฃฟโฃฟโฃฟโกทโ โ โ
โ โ โ โ โ โ โ โ โ โ โ โ โ โ โ โ โ Decentralized Compute Network https://lilypad.tech
๐ Lilypad submitting job
2025-02-18T15:18:08-06:00 WRN ../runner/work/lilypad/lilypad/cmd/lilypad/utils.go:63 > failed to get GPU info: gpuFillInfo not implemented on darwin
2025-02-18T15:18:08-06:00 INF ../runner/work/lilypad/lilypad/pkg/web3/sdk.go:209 > Connected to arbitrum-sepolia-rpc.publicnode.com
2025-02-18T15:18:08-06:00 INF ../runner/work/lilypad/lilypad/pkg/jobcreator/run.go:27 > Public Address: 0xEC26a7C56e05D208ad93586D108981200406Eca5
2025-02-18T15:18:09-06:00 INF ../runner/work/lilypad/lilypad/pkg/module/utils.go:149 > updating cached git repo=/tmp/lilypad/data/repos/noryev/module-sdxl-ipfs
โโโ Deal agreed. Running job...2025-02-18T15:20:13-06:00 ERR ../runner/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 > websocket error error="websocket: close 1006 (abnormal closure): unexpected EOF"
๐ Deal agreed. Running job...
๐ค Results submitted. Awaiting verification...
๐ค Results submitted. Awaiting verification...
โ
Results accepted. Downloading result...
๐ Data ID: QmQFQfnKD8xU17ZjsUtqN9VT16kHmKqzEDR7untADuExBU
๐ Lilypad job completed, try ๐
open /tmp/lilypad/data/downloaded-files/Qmb2tpLYX3xhWjhLHTzYzB2pS1iVj1jvZcShEBNeuvZqkx
cat /tmp/lilypad/data/downloaded-files/Qmb2tpLYX3xhWjhLHTzYzB2pS1iVj1jvZcShEBNeuvZqkx/stdout
cat /tmp/lilypad/data/downloaded-files/Qmb2tpLYX3xhWjhLHTzYzB2pS1iVj1jvZcShEBNeuvZqkx/stderr
โ ~ open /tmp/lilypad/data/downloaded-files/Qmb2tpLYX3xhWjhLHTzYzB2pS1iVj1jvZcShEBNeuvZqkx
It ran anyway, but I saw a similar error websocket error error="websocket: close 1006 (abnormal closure): unexpected EOF"
We discussed this issue earlier today and have a theory about it.
The CLI run command receives and handles messages from previous job runs that were abandoned. The intent is likely to finish the work and download results from jobs that may have not completed when the run process was not active.
It may be that these private keys were involved with a job that went bad somehow. When we run another job with them, they attempt to continue the previous job, but run into the bad state and are unable to proceed with the next job. We have some investigation to determine the failure case, why did the job reach a bad state?
We may also want to remove the "pick up old jobs" functionality. The UX has for the most part been confusing, and it might be better for the CLI run command to only focus on one job at a time.
