BUG: `oxen clone` fails with `UnexpectedEof` on large files pushed successfully — causes server thread panic

[type-checker]

Hi @jcelliott, I encountered a oxen clone failure with the Oxen CLI v0.36.1 after successfully pushing large binary files (3GB–8GB) from one machine to another machine where Oxen server was running on K3s. The oxen clone attempt on a second machine leads to repeated server thread panics with UnexpectedEof errors while reading chunked files.

Environment:

Machine-1, Client + Server Test (Single Machine Initial Push):

OS: Ubuntu Machine Type: GCP n2-standard-4 (4vCPU, 16GB RAM, 30GB Disk) Oxen Version: 0.36.1 (built locally from the latest cloned source) Storage available: Initially insufficient during testing (noted below) Remote Server:

• Hosted on same machine in initial test and it worked fine
• Later tested with second GCP machine with Oxen server on K3s and Traefik reverse proxy

Machine-2 (Server running on K3s cluster via Traefik):

OS: Ubuntu Machine Type: GCP n2-standard-4 (4vCPU, 16GB RAM, 30GB Disk) Oxen Version: 0.36.1 (built locally from the latest cloned source) K3s: Oxen server running as a container Proxy: Traefik Wildcard DNS: traefik.me

Reproduction Steps & Log Summary

1. Step 1: Created local repo and pushed a 3GB file

• ✅ Successful push from Machine-1 to Machine-2

2. Step 2: Created new test repo with 5GB + 4GB files

• ❌ Initial push failed due to disk space (No space left on device from add.rs line 362)
• ✅ After cleanup, push succeeded

3. Step 3: Cloned from second machine (Machine-2)

• ❌ oxen clone failed during chunk download — logs show:

Error reading file "mockfile5" chunk 363/513 chunk size 10485760 total_bytes_read: 3806330880 total_bytes: 5368709120
Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }

4. Step 4: Server log (Oxen server running in K3s) showed:

thread 'actix-rt|system:0|arbiter:*' panicked at src/server/src/controllers/entries.rs:128:31:
called `Result::unwrap()` on an `Err` value: Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }

5. Step 5: Pushed 8.6 GB total (3GB + 5GB) in a clean repo

• ✅ Push completed successfully
• ❌ Clone still fails with same EOF errors

🐞 Key Issues Identified

• oxen push appears successful even when some large file chunks might be incompletely uploaded or corrupted
• During oxen clone, Oxen server attempts to serve incomplete chunks, leading to:
  • UnexpectedEof errors
  • Server thread panics on unwrap() in entries.rs:128
• Server-side panic is non-resilient to corrupt/incomplete file chunks, crashing threads instead of handling the error gracefully

Full Logs from Machine-1

gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen$ export PATH="/home/gcpuser/oxen/target/release:$PATH"
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen$ oxen --version
oxen 0.36.1
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen$ mkdir oxtestrepo
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen$ cd oxtestrepo
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ fallocate -l 3G mockfile
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen add .
No oxen repository exists, looking for directory: .oxen
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen init
🐂 created a new config file in "$HOME/.config/oxen/auth_config.toml
🐂 repository initialized at: "/home/gcpuser/oxen/oxtestrepo"

    📖 If this is your first time using Oxen, check out the CLI docs at:
            https://docs.oxen.ai/getting-started/cli

    💬 For more support, or to chat with the Oxen team, join our Discord:
            https://discord.gg/s3tBEn7Ptg

gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen add .
🐂 oxen added 1 files (3.2 GB) in 9s 985ms
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen commit -m 3gb
Committing with message: 3gb
UserConfigNotFound(oxen not configured, set email and name with:

oxen config --name YOUR_NAME --email YOUR_EMAIL
)
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen config --name ox --email [email protected]
🐂 created a new config file in "$HOME/.config/oxen/user_config.toml
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen commit -m 3gb
Committing with message: 3gb
🐂 commit 7284d99ea890ca082a0788c7d4fb7548 -> '3gb' in 37ms

gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen create-remote --name ox/oxtestrepo --host oxtestserver123.traefik.me --scheme https
🎉 Remote successfully created for 'ox/oxtestrepo'

If this is a brand new repository:

  oxen clone https://oxtestserver123.traefik.me/ox/oxtestrepo

To push an existing local repository to a new remote:

  oxen config --set-remote origin https://oxtestserver123.traefik.me/ox/oxtestrepo

gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen config --set-remote origin https://oxtestserver123.traefik.me/ox/oxtestrepo
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen push
🐂 oxen push origin main -> 7284d99ea890ca082a0788c7d4fb7548
🐂 push complete 🎉 took 1m 11s 878ms
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ cd ..
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen$ mkdir oxlocal
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen$ cd oxlocal
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxlocal$ oxen create-remote --name ox/oxtestrepo --host localhost:3000 --scheme http
🎉 Remote successfully created for 'ox/oxtestrepo'

If this is a brand new repository:

  oxen clone http://localhost:3000/ox/oxtestrepo

To push an existing local repository to a new remote:

  oxen config --set-remote origin http://localhost:3000/ox/oxtestrepo

gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxlocal$ oxen config --set-remote origin http://localhost:3000/ox/oxtestrepo
No oxen repository exists, looking for directory: .oxen
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxlocal$ oxen init
🐂 repository initialized at: "/home/gcpuser/oxen/oxlocal"

    📖 If this is your first time using Oxen, check out the CLI docs at:
            https://docs.oxen.ai/getting-started/cli

    💬 For more support, or to chat with the Oxen team, join our Discord:
            https://discord.gg/s3tBEn7Ptg

gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxlocal$ oxen config --set-remote origin http://localhost:3000/ox/oxtestrepo
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxlocal$ fallocate -l 3G mockfile
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxlocal$ oxen add .
🐂 oxen added 1 files (3.2 GB) in 9s 752ms
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxlocal$ oxen commit -m 3gb
Committing with message: 3gb
🐂 commit 64d7ba87f96a5f10a9820448c0a0f54d -> '3gb' in 1s 568ms
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxlocal$ oxen push
🐂 oxen push origin main -> 64d7ba87f96a5f10a9820448c0a0f54d
🐂 push complete 🎉 took 34s 786ms

gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtest1$ fallocate -l 5G mockfile5
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtest1$ fallocate -l 4G mockfile4
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtest1$ oxen add .
⠓ 🐂 add 0 files, 0 unchanged (0 B) 0.00 MB/s                                                                                                                                          
thread '<unnamed>' panicked at src/lib/src/core/v_latest/add.rs:362:30:
called `Result::unwrap()` on an `Err` value: IO(Os { code: 28, kind: StorageFull, message: "No space left on device" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtest1$ oxen add .
🐂 oxen added 1 files (5.4 GB) in 3s 463ms
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtest1$ oxen create-remote --name ox/oxtestrepo --host oxtestserver123.traefik.me --scheme https
🎉 Remote successfully created for 'ox/oxtestrepo'

If this is a brand new repository:

  oxen clone https://oxtestserver123.traefik.me/ox/oxtestrepo

To push an existing local repository to a new remote:

  oxen config --set-remote origin https://oxtestserver123.traefik.me/ox/oxtestrepo

gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtest1$ oxen config --set-remote origin https://oxtestserver123.traefik.me/ox/oxtestrepo
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtest1$ oxen commit -m 5g
Committing with message: 5g
🐂 commit 84978413b1411f010fac6a14607d433d -> '5g' in 38ms
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtest1$ oxen push
🐂 oxen push origin main -> 84978413b1411f010fac6a14607d433d
⠤ 🐂 push (1/2 files) [00:02:29] [🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🌾🐂➖➖➖➖➖➖➖➖➖➖] 7.51 GiB/9.00 GiB[ERROR] 2025-07-12T09:02:51.824 - core/v_latest/push.rs:580 upload_large_file_chunks Error reading file "mockfile5" chunk 363/513 chunk size 10485760 total_bytes_read: 3806330880 total_bytes: 5368709120 Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }
🐂 push complete 🎉 took 2m 32s 955ms
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtest1$ oxen push
🐂 oxen push origin main -> 84978413b1411f010fac6a14607d433d
Everything is up to date
🐂 push complete 🎉 took 889ms

gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen$ mkdir oxtestrepo
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen$ cd oxtestrepo
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen init
🐂 repository initialized at: "/home/gcpuser/oxen/oxtestrepo"

    📖 If this is your first time using Oxen, check out the CLI docs at:
            https://docs.oxen.ai/getting-started/cli

    💬 For more support, or to chat with the Oxen team, join our Discord:
            https://discord.gg/s3tBEn7Ptg

gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ fallocate -l 5G mockfile5
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen add .
🐂 oxen added 1 files (5.4 GB) in 25s 382ms
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen commit -m 5g
Committing with message: 5g
🐂 commit 2325c7a9739481b26671506f941f5bf9 -> '5g' in 44ms
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen create-remote --name ox/oxtestrepo --host oxtestserver123.traefik.me --scheme https
🎉 Remote successfully created for 'ox/oxtestrepo'

If this is a brand new repository:

  oxen clone https://oxtestserver123.traefik.me/ox/oxtestrepo

To push an existing local repository to a new remote:

  oxen config --set-remote origin https://oxtestserver123.traefik.me/ox/oxtestrepo

gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen config --set-remote origin https://oxtestserver123.traefik.me/ox/oxtestrepo
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen push
🐂 oxen push origin main -> 2325c7a9739481b26671506f941f5bf9
🐂 push complete 🎉 took 1m 53s 954ms

gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ cd ..
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen$ mkdir oxtestrepo
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen$ cd oxtestrepo
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ fallocate -l 5G mockfile5
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ fallocate -l 3G mockfile3
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ utpost delete-remote --name ox/oxtestrepo --host oxtestserver123.traefik.me --scheme https^C
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen delete-remote --name ox/oxtestrepo --host oxtestserver123.traefik.me --scheme https
Are you sure you want to delete the remote repository: ox/oxtestrepo? yes
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen delete-remote --name ox/oxtestrepo --host oxtestserver123.traefik.me --scheme https
Remote repository not found: ox/oxtestrepo
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen add .
No oxen repository exists, looking for directory: .oxen
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen init
🐂 repository initialized at: "/home/gcpuser/oxen/oxtestrepo"

    📖 If this is your first time using Oxen, check out the CLI docs at:
            https://docs.oxen.ai/getting-started/cli

    💬 For more support, or to chat with the Oxen team, join our Discord:
            https://discord.gg/s3tBEn7Ptg

gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen add .
🐂 oxen added 2 files (8.6 GB) in 44s 23ms
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen create-remote --name ox/oxtestrepo --host oxtestserver123.traefik.me --scheme https
🎉 Remote successfully created for 'ox/oxtestrepo'

If this is a brand new repository:

  oxen clone https://oxtestserver123.traefik.me/ox/oxtestrepo

To push an existing local repository to a new remote:

  oxen config --set-remote origin https://oxtestserver123.traefik.me/ox/oxtestrepo

gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen config --set-remote origin https://oxtestserver123.traefik.me/ox/oxtestrepo
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen commit -m 8g
Committing with message: 8g
🐂 commit ed9305a7044f7115a7acdf26c1ecde5a -> '8g' in 41ms
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen commit -m 8g
Committing with message: 8g
No changes to commit
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen push
🐂 oxen push origin main -> ed9305a7044f7115a7acdf26c1ecde5a
🐂 push complete 🎉 took 1m 54s 868ms
gcpuser@builde-3894e829-head-2n4pd2vs-compute:~/oxen/oxtestrepo$ oxen push
🐂 oxen push origin main -> ed9305a7044f7115a7acdf26c1ecde5a
Everything is up to date
🐂 push complete 🎉 took 884ms

Logs from Machine-2

[INFO] 2025-07-12T09:33:27.731 - actix_web/middleware/logger.rs:424 user agent is 10.42.0.8 Oxen/0.36.1 (Linux; CLI)                     │
│ [INFO] 2025-07-12T09:33:27.733 - actix_web/middleware/logger.rs:424 10.42.0.8 "GET /api/repos/ox/oxtestrepo/chunk/ed9305a7044f7115a7acdf26c1e │
│ cde5a/mockfile3?chunk_start=3219128320&chunk_size=2097152 HTTP/1.1" 200 2097152 "-" "Oxen/0.36.1 (Linux; CLI)" 0.044742                  │
│ [INFO] 2025-07-12T09:33:27.923 - actix_web/middleware/logger.rs:424 user agent is 10.42.0.8 Oxen/0.36.1 (Linux; CLI)                     │
│ [INFO] 2025-07-12T09:33:27.923 - actix_web/middleware/logger.rs:424 10.42.0.8 "GET /api/repos/ox/oxtestrepo/chunk/ed9305a7044f7115a7acdf26c1e │
│ cde5a/mockfile3?chunk_start=3208642560&chunk_size=10485760 HTTP/1.1" 200 10485760 "-" "Oxen/0.36.1 (Linux; CLI)" 0.202934                │
│                                                                                                                                             │
│ thread 'actix-rt|system:0|arbiter:1' panicked at src/server/src/controllers/entries.rs:128:31:                                              │
│ called `Result::unwrap()` on an `Err` value: Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }                          │
│                                                                                                                                             │
│ thread 'actix-rt|system:0|arbiter:2' panicked at src/server/src/controllers/entries.rs:128:31:                                              │
│ called `Result::unwrap()` on an `Err` value: Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }                          │
│                                                                                                                                             │
│ thread 'actix-rt|system:0|arbiter:3' panicked at src/server/src/controllers/entries.rs:128:31:                                              │
│ called `Result::unwrap()` on an `Err` value: Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }                          │
│                                                                                                                                             │
│ thread 'actix-rt|system:0|arbiter:1' panicked at src/server/src/controllers/entries.rs:128:31:                                              │
│ called `Result::unwrap()` on an `Err` value: Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }                          │
│                                                                                                                                             │
│ thread 'actix-rt|system:0|arbiter:2' panicked at src/server/src/controllers/entries.rs:128:31:                                              │
│ called `Result::unwrap()` on an `Err` value: Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }                          │
│                                                                                                                                             │
│ thread 'actix-rt|system:0|arbiter:0' panicked at src/server/src/controllers/entries.rs:128:31:                                              │
│ called `Result::unwrap()` on an `Err` value: Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }                          │
│                                                                                                                                             │
│ thread 'actix-rt|system:0|arbiter:3' panicked at src/server/src/controllers/entries.rs:128:31:                                              │
│ called `Result::unwrap()` on an `Err` value: Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }   

📌 Suggested Fixes

1. Server Side:

• Prevent panics by handling Result::Err for chunk reads gracefully instead of unwrap()
• Add checksum validation of uploaded chunks (or entire file) during oxen push
• Log more verbose error context during panic (e.g., which commit / file)

2. Client Side:

• During oxen push, verify that all chunks are fully uploaded and readable before finalizing the commit
• Optionally add retries for chunk upload failure with validation

3. Enhancement:

• Add integrity check post-commit or during clone to detect incomplete/corrupted chunked uploads
• Add a CLI command like oxen verify

[subygan]

Hi @type-checker !! thanks for this detailed issue. I am looking at this issue with clone and am looking to reproduce this locally today. I'll keep you updated on the status.

[type-checker]

Hi @subygan, did you able to reproduce this issue on your side?

[subygan]

Hi!! @type-checker I couldn't reproduce this issue, even from hub or locally. basically the files are getting pushed and cloned properly. is it possible that it could be because of file type?

[type-checker]

Hi!! @type-checker I couldn't reproduce this issue, even from hub or locally. basically the files are getting pushed and cloned properly. is it possible that it could be because of file type?

@subygan Maybe, I would suggest you try with mock file using fallocate -l 3G mockfile