Oxen BUG with `oxen add`, `oxen push`, and `oxen delete-remote`

Hi @jcelliott @subygan, I thought of creating this issue I encountered while using oxen cli as I noticed team is working on the same Oxen-AI/oxen-archive#616

BUG Lists

1. Bug: oxen add . checks all files regardless of changes Currently, running oxen add . scans all files in the repository, even if only a single file has changed or all changes are already staged. This behavior persists regardless of the number of files modified.

Expected Behavior: Only unstaged or changed files should be checked and added to the staging area. If there are no changes to add, the command should not scan all files and should simply output nothing to add.

The previous issue with oxen add . handling repositories containing many nested directories appears to have been resolved in recent oxen versions. However, I'm still facing issues, please refer to the logs shared below.

2. Bug: oxen commit -m attempts to commit with no changes and hangs When running oxen commit -m, the command tries to commit even if there are no staged changes, causing it to hang. For example, after making and staging changes with oxen add ., running oxen commit -m "test" works as expected. However, if the same commands are repeated without any new changes, oxen add . again checks all files (due to the Bug 1), and oxen commit -m "test" hangs.

Expected Behavior: The commit command should only attempt to commit staged changes. If there are no new changes to commit, it should output nothing to commit and exit gracefully.

3. Bug: oxen push repacks all files and is slow The oxen push command repacks all files before pushing, even after the files have already been added and committed, which leads to significant delays. Coming from a Git background, I expected oxen push to immediately start uploading committed files to the server. The push process is especially slow when commands like oxen add and oxen commit are repeated, and it is noticeably sluggish with single large binary files (as reported earlier).

**Expected Behavior: **oxen push should efficiently upload only the committed changes without unnecessary repacking, and its performance should not degrade with repeated commands or large files.

Logs:

➜  next.js-canary git:(main) ✗ oxen add .
⠉ 🐂 add 1419 files, 0 unchanged (7.5 MB) 37.22 MB/s                                                                                                                                             
thread '<unnamed>' panicked at src/lib/src/core/v_latest/add.rs:347:26:
called `Result::unwrap()` on an `Err` value: Basic(Could not get file metadata: "/Users/ubuntu/next.js-canary/test/development/app-dir/ssr-in-rsc/node_modules/random-react-library" error Os { code: 2, kind: NotFound, message: "No such file or directory" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
⠲ 🐂 add 23496 files, 0 unchanged (137.0 MB) 36.04 MB/s                                                                                                                                          %
➜  next.js-canary git:(main) ✗ oxen add .
⠉ 🐂 add 1463 files, 0 unchanged (6.1 MB) 40.34 MB/s                                                                                                                                             
thread '<unnamed>' panicked at src/lib/src/core/v_latest/add.rs:347:26:
called `Result::unwrap()` on an `Err` value: Basic(Could not get file metadata: "/Users/ubuntu/next.js-canary/test/development/app-dir/ssr-in-rsc/node_modules/random-react-library" error Os { code: 2, kind: NotFound, message: "No such file or directory" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
⠒ 🐂 add 23584 files, 0 unchanged (137.2 MB) 58.37 MB/s                                                                                                                                          %
➜  next.js-canary git:(main) ✗ oxen add .
⠉ 🐂 add 1588 files, 0 unchanged (9.1 MB) 60.40 MB/s                                                                                                                                             
thread '<unnamed>' panicked at src/lib/src/core/v_latest/add.rs:347:26:
called `Result::unwrap()` on an `Err` value: Basic(Could not get file metadata: "/Users/ubuntu/next.js-canary/test/development/app-dir/ssr-in-rsc/node_modules/random-react-library" error Os { code: 2, kind: NotFound, message: "No such file or directory" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
⠒ 🐂 add 23383 files, 0 unchanged (124.1 MB) 60.54 MB/s                                                                                                                                          %
➜  next.js-canary git:(main) ✗ oxen add .
⠁ 🐂 add 1364 files, 0 unchanged (4.5 MB) 44.61 MB/s                                                                                                                                             
thread '<unnamed>' panicked at src/lib/src/core/v_latest/add.rs:347:26:
called `Result::unwrap()` on an `Err` value: Basic(Could not get file metadata: "/Users/ubuntu/next.js-canary/test/development/app-dir/ssr-in-rsc/node_modules/random-react-library" error Os { code: 2, kind: NotFound, message: "No such file or directory" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
⠤ 🐂 add 23479 files, 0 unchanged (131.1 MB) 74.91 MB/s                                                                                                                                          %
➜  next.js-canary git:(main) ✗ oxen commit -m hello
Committing with message: hello
🐂 commit bc716b1c712d7926d90c6a417196e371 -> 'hello' in 24s 817ms
➜  next.js-canary git:(main) ✗ oxen commit -m hello
Committing with message: hello
No changes to commit
➜  next.js-canary git:(main) ✗ oxen commit -m hello
Committing with message: hello
No changes to commit
➜  next.js-canary git:(main) ✗ oxen add .          
⠙ 🐂 add 0 files, 1739 unchanged (0 B) 0.00 MB/s                                                                                                                                                 
thread '<unnamed>' panicked at src/lib/src/core/v_latest/add.rs:347:26:
called `Result::unwrap()` on an `Err` value: Basic(Could not get file metadata: "/Users/ubuntu/next.js-canary/test/development/app-dir/ssr-in-rsc/node_modules/random-react-library" error Os { code: 2, kind: NotFound, message: "No such file or directory" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
⠂ 🐂 add 0 files, 7636 unchanged (0 B) 0.00 MB/s                                                                                                                                                 ^C
➜  next.js-canary git:(main) ✗ oxen add .
⠤ 🐂 add 0 files, 1736 unchanged (0 B) 0.00 MB/s                                                                                                                                                 
thread '<unnamed>' panicked at src/lib/src/core/v_latest/add.rs:347:26:
called `Result::unwrap()` on an `Err` value: Basic(Could not get file metadata: "/Users/ubuntu/next.js-canary/test/development/app-dir/ssr-in-rsc/node_modules/random-react-library" error Os { code: 2, kind: NotFound, message: "No such file or directory" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
⠒ 🐂 add 0 files, 23616 unchanged (0 B) 0.00 MB/s

Jun 28 '25 10:06 type-checker

Hey @type-checker !! Thank you so much for the detailed issue. Appreciate that very much.

Bug 1 and 2. We just merged a fix yesterday. We'll be doing a release on Monday. Post which the issues should be gone.

Push packing everything is new to me. I'll dig deeper.

But, we've merged a lot of optimization for push as well in the last couple of weeks. My guess is the push issue should be fixed as well. I'll update here, once we do the release.

Jun 28 '25 14:06 subygan

Hey @type-checker !! Thank you so much for the detailed issue. Appreciate that very much.

Bug 1 and 2. We just merged a fix yesterday. We'll be doing a release on Monday. Post which the issues should be gone.

Push packing everything is new to me. I'll dig deeper.

But, we've merged a lot of optimization for push as well in the last couple of weeks. My guess is the push issue should be fixed as well. I'll update here, once we do the release.

@subygan Thank you for your response! FYI, the log I shared was generated from the most recent pull of the main branch that I tested locally, so it seems the issue is still present. If you could please confirm this, I’ll be happy to retest and provide you with the latest log again.

Jun 28 '25 17:06 type-checker

Wow! that is surprising.

This is what I was seeing in the nextjs repo, when initializing.

next.js (oxen)  oxen add .

🐂 oxen added 23340 files (135.0 MB) in 2s 592ms

next.js (oxen)  oxen commit -m "init"

Committing with message: init

🐂 commit fcd3383f8744289504631726a6683d11 -> 'init' in 7s 962ms

--------------------------------------------------------------------------------------------------

after changing one file,

next.js (oxen:main)  echo "something3" >> [readme.md](http://readme.md/)

next.js (oxen:main)  oxen add .

🐂 oxen added 2 files (6.7 KB) in 16s 476ms

next.js (oxen:main)  oxen commit -m "update readme"

Committing with message: update readme

🐂 commit 40a891f2f9ed31530df2e8ad3e402d24 -> 'update readme' in 1m 31s 944ms

next.js (oxen:main)  oxen push

🐂 oxen push origin main -> 40a891f2f9ed31530df2e8ad3e402d24

🐂 push complete 🎉 took 26s 980ms

Which is significantly faster than before.

is it possible to share what system you're using?

Jun 28 '25 18:06 subygan

You were able to partially reproduce the issue.

23340 files (135.0 MB) in 2s 592ms 🐂 commit fcd3383f8744289504631726a6683d11 -> 'init' in 7s 962ms

When you first staged 23,340 files (135.0 MB), the staging process took 2.6 seconds, and the commit took about 7s 962ms.

🐂 oxen added 2 files (6.7 KB) in 16s 476ms 🐂 commit 40a891f2f9ed31530df2e8ad3e402d24 -> 'update readme' in 1m 31s 944ms

However, when staging just 2 files (6.7 KB), the process took 16s 476ms—about 6.36 times longer than the previous, much larger staging. The commit for these two files took 1m 31s 944ms, a 11.55 times increase compared to the initial commit.

We can’t compare push times since there’s only one push log above. But the problem is clear: committing a small change with just two files is much slower than the initial large commit, which is unexpected.

I’ll share my logs, steps, and OS details shortly for comparison. Meanwhile, I suggest repeating the steps: run oxen add ., then again oxen add ., or try combinations like oxen add . followed by oxen commit -m hello, but don’t push yet. Repeat oxen add . and oxen commit -m hello a few times; you’ll likely notice the same issue I described. Afterward, try oxen push as well—you’ll probably encounter more strange behavior.

Jun 29 '25 02:06 type-checker

Environment

OS: macOS 15.3.1 Oxen version: 0.35.0 (built locally from the latest cloned source)

Issue-1

When running oxen add . on a repository downloaded as a ZIP from GitHub, the command fails with a panic if a submodule path exists. The error message indicates it cannot find the directory, which is expected since GitHub ZIP downloads do not include submodules or their contents.

Steps to Reproduce

Download the next.js repository canary branch as a ZIP from GitHub (do not clone with git).
Unzip the repository locally.
Run oxen init, then oxen add ., inside the extracted directory.
Observe the error:

thread '<unnamed>' panicked at src/lib/src/core/v_latest/add.rs:347:26:
called `Result::unwrap()` on an `Err` value: Basic(Could not get file metadata: ".../node_modules/random-react-library" error Os { code: 2, kind: NotFound, message: "No such file or directory" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

next.js-canary oxen add .
⠉ 🐂 add 1410 files, 0 unchanged (10.4 MB) 52.16 MB/s                                                                                                                                                                                                
thread '<unnamed>' panicked at src/lib/src/core/v_latest/add.rs:347:26:
called `Result::unwrap()` on an `Err` value: Basic(Could not get file metadata: "/Users/type-checker/Downloads/next.js-canary/test/development/app-dir/ssr-in-rsc/node_modules/random-react-library" error Os { code: 2, kind: NotFound, message: "No such file or directory" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
⠙ 🐂 add 23426 files, 0 unchanged (137.0 MB) 42.14 MB/s

The process panics and stops when it encounters a missing directory that was a submodule in the original repository.

https://github.com/vercel/next.js/blob/canary/test/development/app-dir/ssr-in-rsc/node_modules/random-react-library

Expected Behavior

The command should handle missing submodule directories and continue without panicking. At minimum, it should show a warning rather than crash.

Other Issues

As I’ve mentioned, repeating these actions consistently results in unusual behavior and various errors. The file count and staging size change each time, and the staging process becomes progressively slower. Please review the logs below.

➜  next.js-canary oxen add .
⠁ 🐂 add 1620 files, 0 unchanged (10.6 MB) 105.78 MB/s                                                                                                                                                                                               
thread '<unnamed>' panicked at src/lib/src/core/v_latest/add.rs:347:26:
called `Result::unwrap()` on an `Err` value: Basic(Could not get file metadata: "/Users/type-checker/Downloads/next.js-canary/test/development/app-dir/ssr-in-rsc/node_modules/random-react-library" error Os { code: 2, kind: NotFound, message: "No such file or directory" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
⠤ 🐂 add 33232 files, 0 unchanged (4.6 GB) 975.96 MB/s                                                                                                                                                                                               

➜  next.js-canary oxen add .
⠁ 🐂 add 921 files, 0 unchanged (2.1 MB) 42.64 MB/s                                                                                                                                                                                                  
thread '<unnamed>' panicked at src/lib/src/core/v_latest/add.rs:347:26:
called `Result::unwrap()` on an `Err` value: Basic(Could not get file metadata: "/Users/type-checker/Downloads/next.js-canary/test/development/app-dir/ssr-in-rsc/node_modules/random-react-library" error Os { code: 2, kind: NotFound, message: "No such file or directory" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
⠁ 🐂 add 33141 files, 0 unchanged (4.6 GB) 1581.71 MB/s                                                                                                                                                                                              

➜  next.js-canary oxen add .
⠁ 🐂 add 932 files, 0 unchanged (2.1 MB) 42.38 MB/s                                                                                                                                                                                                  
thread '<unnamed>' panicked at src/lib/src/core/v_latest/add.rs:347:26:
called `Result::unwrap()` on an `Err` value: Basic(Could not get file metadata: "/Users/type-checker/Downloads/next.js-canary/test/development/app-dir/ssr-in-rsc/node_modules/random-react-library" error Os { code: 2, kind: NotFound, message: "No such file or directory" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
⠁ 🐂 add 33461 files, 0 unchanged (4.6 GB) 1581.61 MB/s

➜  next.js-canary oxen commit -m hello
Committing with message: hello
🐂 commit c8cc587665833334724962fd636a6779 -> 'hello' in 22s 401ms

➜  next.js-canary oxen commit -m hello
Committing with message: hello
No changes to commit

➜  next.js-canary oxen commit -m hello
Committing with message: hello
No changes to commit

➜  next.js-canary oxen add .                                               
⠈ 🐂 add 0 files, 1737 unchanged (0 B) 0.00 MB/s                                                                                                                                                                                                     
thread '<unnamed>' panicked at src/lib/src/core/v_latest/add.rs:347:26:
called `Result::unwrap()` on an `Err` value: Basic(Could not get file metadata: "/Users/type-checker/Downloads/next.js-canary/test/development/app-dir/ssr-in-rsc/node_modules/random-react-library" error Os { code: 2, kind: NotFound, message: "No such file or directory" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
⠐ 🐂 add 0 files, 33879 unchanged (0 B) 0.00 MB/s                                                                                                                                                                                                    

➜  next.js-canary oxen commit -m hello
Committing with message: hello
No changes to commit

Expected Behavior:

When there are no changes in the working directory, the tool should not re-scan or attempt to stage all files again. It should efficiently detect that nothing has changed and display a message like No new changes to stage, instead of repeatedly increasing the staging time and file count.

Update

I’ve also tested this with a cloned repository and am encountering the same issue. I’m not sure why you aren’t experiencing this submodule problem.

Jun 29 '25 02:06 type-checker

Issue-2

When running oxen add . on a repository cloned from GitHub without submodules (e.g., pytorch/pytorch), I noticed that each time I stage files with oxen add ., it appears to re-add the same files repeatedly, even when there are no changes. This suggests either a misleading print statement or that the Oxen CLI is actually restaging files unnecessarily.

Similarly, oxen push exhibits the same behavior. Even when there are no new changes to push, the CLI still prints messages like 🐂 push complete 🎉 took 5s 420ms, which implies that something was pushed, when in reality there were no updates. This can be confusing and makes it unclear whether any actual changes were sent to the remote.

➜  pytorch git:(main) ✗ oxen add .
🐂 oxen added 19467 files (1.3 GB) in 3s 384ms
➜  pytorch git:(main) ✗ oxen add .
🐂 oxen added 19467 files (1.3 GB) in 1s 177ms
➜  pytorch git:(main) ✗ oxen add .
🐂 oxen added 19467 files (1.3 GB) in 755ms
➜  pytorch git:(main) ✗ oxen add .
🐂 oxen added 19467 files (1.3 GB) in 768ms
➜  pytorch git:(main) ✗ oxen add .
🐂 oxen added 19467 files (1.3 GB) in 748ms
➜  pytorch git:(main) ✗ oxen commit -m pytorch
Committing with message: pytorch
🐂 commit 81050bea529247158379984816cc33f -> 'pytorch' in 980ms
➜  pytorch git:(main) ✗ oxen commit -m pytorch
Committing with message: pytorch
No changes to commit
➜  pytorch git:(main) ✗ oxen add .            
🐂 oxen added 0 files (0 B) in 615ms
➜  pytorch git:(main) ✗ oxen add .            
🐂 oxen added 0 files (0 B) in 585ms
➜  pytorch git:(main) ✗ oxen commit -m pytorch
Committing with message: pytorch
No changes to commit
➜  pytorch git:(main) ✗ echo "hello world" > hello.txt   

➜  pytorch git:(main) ✗ oxen add .                 
🐂 oxen added 1 files (12 B) in 620ms
➜  pytorch git:(main) ✗ oxen add .
🐂 oxen added 1 files (12 B) in 642ms
➜  pytorch git:(main) ✗ oxen add .
🐂 oxen added 1 files (12 B) in 563ms
➜  pytorch git:(main) ✗ oxen add .
🐂 oxen added 1 files (12 B) in 578ms
➜  pytorch git:(main) ✗ oxen commit -m hello.txt   
Committing with message: hello.txt
🐂 commit 5972edd890a8dd24e46b72afffac4e30 -> 'hello.txt' in 26ms
➜  pytorch git:(main) ✗ oxen commit -m hello.txt
Committing with message: hello.txt
No changes to commit
➜  pytorch git:(main) ✗ oxen add .              
🐂 oxen added 0 files (0 B) in 647ms
➜  pytorch git:(main) ✗ oxen commit -m hello.txt
Committing with message: hello.txt
No changes to commit
➜  pytorch git:(main) ✗ oxen log
commit 5972edd890a8dd24e46b72afffac4e30

Author: Ox
Date:   Sunday, 29 June 2025 04:04:47 +00

    hello.txt

commit 81050bea529247158379984816cc33f

Author: Ox
Date:   Sunday, 29 June 2025 03:59:32 +00

    pytorch
➜  pytorch git:(main) ✗ oxen push origin main                                                                 
🐂 oxen push origin main -> 5972edd890a8dd24e46b72afffac4e30
🐂 push complete 🎉 took 12m 26s 926ms
➜  pytorch git:(main) ✗ oxen push origin main
🐂 oxen push origin main -> 5972edd890a8dd24e46b72afffac4e30
Everything is up to date
🐂 push complete 🎉 took 5s 420ms
➜  pytorch git:(main) ✗ oxen push origin main
🐂 oxen push origin main -> 5972edd890a8dd24e46b72afffac4e30
Everything is up to date
🐂 push complete 🎉 took 4s 81ms
➜  pytorch git:(main) ✗ oxen push origin main
🐂 oxen push origin main -> 5972edd890a8dd24e46b72afffac4e30
Everything is up to date
🐂 push complete 🎉 took 4s 583ms
➜  pytorch git:(main) ✗ oxen add .
🐂 oxen added 0 files (0 B) in 651ms
➜  pytorch git:(main) ✗ oxen commit -m hello.txt                                                                
Committing with message: hello.txt
No changes to commit
➜  pytorch git:(main) ✗ oxen push origin main   
🐂 oxen push origin main -> 5972edd890a8dd24e46b72afffac4e30
Everything is up to date
🐂 push complete 🎉 took 4s 483ms
➜  pytorch git:(main) ✗ oxen log
commit 5972edd890a8dd24e46b72afffac4e30

Author: Ox
Date:   Sunday, 29 June 2025 04:04:47 +00

    hello.txt

commit 81050bea529247158379984816cc33f

Author: Ox
Date:   Sunday, 29 June 2025 03:59:32 +00

    pytorch

Expected Behavior

For oxen add .: When there are no new or modified files, the CLI should clearly state "No new changes to add" and should not attempt to restage files that have already been staged or committed.

For oxen push: When there are no new commits to push, the CLI should immediately indicate that everything is up to date, without printing misleading success messages or implying that files were pushed when nothing actually changed.

Additionally, as previously reported and seen in your own tests with the next.js repository, adding a new file or making a minor change often takes significantly longer than the initial staging of a large number of files, which is unexpected. A similar issue occurs with oxen push, where pushing small changes also takes disproportionately long.

Update

oxen commit is now working as expected.

Jun 29 '25 05:06 type-checker

Issue-3

Sometimes, when running oxen push or oxen delete-remote, I encounter errors stating there is not enough space to push a new repository or even delete an existing one. This is misleading, as I am testing locally with the Oxen server running inside Docker, and the mounted volume has sufficient free space. Despite this, the CLI reports "No space left on device."

Additionally, on occasions when the operation does succeed, no success message is displayed, so I have to rerun the command to check if it completed—only then do I see the Remote repository not found message confirming the deletion.

➜  pytorch git:(main) ✗ oxen delete-remote --name ox/pytorch --host localhost:3000 --scheme http
Failed to open refs database: IO error: No space left on device: While open a file for appending: /var/oxen/data/ox/pytorch/.oxen/refs/LOG: No space left on device

➜  react git:(main) ✗ oxen delete-remote --name ox/react --host localhost:3000 --scheme http
Are you sure you want to delete the remote repository: ox/react? yes

Expected Behavior

The CLI should accurately detect available disk space and only display "No space left on device" errors when the volume is genuinely full.
If there is sufficient space, both oxen push and oxen delete-remote should complete successfully and always display a clear success message upon completion.
Users should not have to rerun commands to verify the outcome; the CLI should provide immediate and reliable feedback for both success and failure scenarios.

Jun 29 '25 06:06 type-checker

Hi @subygan, does it help with identifying the issues?

Jul 05 '25 07:07 type-checker

Hi @type-checker This helped with identifying the issue. we have done a lot of optimisations for both add and push. So these issues should be cleared in the upcoming release.

Jul 07 '25 15:07 subygan

Hi @type-checker This helped with identifying the issue. we have done a lot of optimisations for both add and push. So these issues should be cleared in the upcoming release.

Hi @subygan, Great, I’m looking forward to testing it. I’ve also documented additional issues I encountered while working with Oxen. If you need the complete logs to help reproduce and resolve these issues, just let me know. Thanks!

Jul 09 '25 10:07 type-checker

Hi! @type-checker We've made a new release with the updates. This should fix add and push being slow in the nextjs repo issue.

Jul 09 '25 23:07 subygan

Hi @subygan, I’ve retested, and the speed of oxen add . has improved significantly (It is faster than git add ., happy with the performance, Congrats 🚀). However, I haven’t observed any speed improvement with oxen push—pushing a single mock 5GB binary file still took 🐂 push complete 🎉 took 24m 56s 429ms.

Additionally, all of the previously mentioned issues persist, and I encountered another issue during the oxen add . operation. Please see the log below:

Issue-4

oxen add . Fails with "Resource temporarily unavailable" Lock File Error

When attempting to add large files (6GB and 7GB) using oxen add ., I repeatedly encountered the following error:

[ERROR] YYYY-MM-DDTHH:MM:SS.sss - core/refs/ref_manager.rs:77 Failed to open refs database: IO error: While lock file: /Users/type-checker/astro/.oxen/refs/LOCK: Resource temporarily unavailable
Failed to open refs database: IO error: While lock file: /Users/type-checker/astro/.oxen/refs/LOCK: Resource temporarily unavailable

After several attempts, the terminal also started showing:

zsh: fork failed: resource temporarily unavailable
_omz_async_request:38: fork failed: resource temporarily unavailable

Eventually, oxen add . succeeded and added all files, but the above errors persisted during multiple runs.

No other oxen or related processes were running in parallel. Please see the full logs shared below for more details:

➜  astro git:(main) ✗ mkfile -n 6g 6GB_test_file3                                

➜  astro git:(main) ✗ mkfile -n 7g 7GB_test_file3

➜  astro git:(main) ✗ oxen add .
[ERROR] 2025-07-10T07:49:28.324 - core/refs/ref_manager.rs:77 Failed to open refs database: IO error: While lock file: /Users/type-checker/astro/.oxen/refs/LOCK: Resource temporarily unavailable
Failed to open refs database: IO error: While lock file: /Users/type-checker/astro/.oxen/refs/LOCK: Resource temporarily unavailable
➜  astro git:(main) ✗ oxen add .
[ERROR] 2025-07-10T07:49:34.189 - core/refs/ref_manager.rs:77 Failed to open refs database: IO error: While lock file: /Users/type-checker/astro/.oxen/refs/LOCK: Resource temporarily unavailable
Failed to open refs database: IO error: While lock file: /Users/type-checker/astro/.oxen/refs/LOCK: Resource temporarily unavailable
➜  astro git:(main) ✗ oxen add .
[ERROR] 2025-07-10T07:49:35.709 - core/refs/ref_manager.rs:77 Failed to open refs database: IO error: While lock file: /Users/type-checker/astro/.oxen/refs/LOCK: Resource temporarily unavailable
Failed to open refs database: IO error: While lock file: /Users/type-checker/astro/.oxen/refs/LOCK: Resource temporarily unavailable
➜  astro git:(main) ✗ oxen add .
[ERROR] 2025-07-10T07:49:38.152 - core/refs/ref_manager.rs:77 Failed to open refs database: IO error: While lock file: /Users/type-checker/astro/.oxen/refs/LOCK: Resource temporarily unavailable
Failed to open refs database: IO error: While lock file: /Users/type-checker/astro/.oxen/refs/LOCK: Resource temporarily unavailable
➜  astro git:(main) ✗ oxen add .
zsh: fork failed: resource temporarily unavailable
_omz_async_request:38: fork failed: resource temporarily unavailable                                                                                                                                                                                 
zsh: fork failed: resource temporarily unavailable
➜  astro oxen add .
zsh: fork failed: resource temporarily unavailable
_omz_async_request:38: fork failed: resource temporarily unavailable                                                                                                                                                                                 
zsh: fork failed: resource temporarily unavailable
➜  astro oxen add .
[ERROR] 2025-07-10T07:50:01.312 - core/refs/ref_manager.rs:77 Failed to open refs database: IO error: While lock file: /Users/type-checker/astro/.oxen/refs/LOCK: Resource temporarily unavailable
Failed to open refs database: IO error: While lock file: /Users/type-checker/astro/.oxen/refs/LOCK: Resource temporarily unavailable
➜  astro git:(main) ✗ oxen add .
🐂 oxen added 16 files (14.0 GB) in 7s 259ms
➜  astro git:(main) ✗ oxen add .
🐂 oxen added 16 files (14.0 GB) in 7s 381ms
➜  astro git:(main) ✗ oxen add .
🐂 oxen added 16 files (14.0 GB) in 8s 75ms

Observed Behavior

Frequent lock file errors when accessing .oxen/refs/LOCK.
Fork failures in zsh after repeated errors.
Eventually, the operation succeeds, but the errors are concerning and may indicate a resource or concurrency issue.

Jul 10 '25 02:07 type-checker

@type-checker thanks for following up! Glad we're making progress on this front.

I'm actually quite surprised by the time for that 5GB file. We've been running a test suite that adds 5M files, then we add 800k more files, then we add a 3GB file. The total time for add, commit, and push of that 3GB file was 36 seconds on our last run.

What kind of system are you running on? Are you pushing to Oxen Hub or a local server? And were you trying to add that 5GB file in the nextjs repo still, or somewhere else?

Jul 10 '25 15:07 jcelliott

@jcelliott

The total time for add, commit, and push of that 3GB file was 36 seconds on our last run.

Is this benchmark was with your local oxen server or with oxen.ai cloud server?

My tests result are for 3GB file:

Environment-1

OS: macOS 15.3.1 Oxen version: 0.36.1 both Server and CLI (built locally from the latest cloned source) Oxen Server: Running locally on http://localhost:300 Oxen CLI: Running locally

🐂 oxen added 1 files (3.2 GB) in 2s 48ms
🐂 commit 85e38fb63f13e3abbfe0dd996e79178f -> '3g' in 11ms
🐂 push complete 🎉 took 12s 204ms

# Total: 14s 263ms

Environment-2

OS: macOS 15.3.1 Oxen version: 0.36.1 both Server and CLI (built locally from the latest cloned source) Oxen Server: Running on K8s (4 vCPUs, 16 GB Memory) for testing Oxen CLI: Running locally

🐂 oxen added 1 files (3.2 GB) in 2s 388ms
🐂 commit 7f26d7fd25d0f3b88e0c803dcc93d56c -> '3g' in 26ms
🐂 push complete 🎉 took 21m 16s 907ms

# Total: 21m 17s 321ms

And were you trying to add that 5GB file in the nextjs repo still, or somewhere else?

I created a new repo to test both 5GB file and the current 3GB file push.

Jul 10 '25 16:07 type-checker

Is this benchmark was with your local oxen server or with oxen.ai cloud server?

That was running on a Linux machine in AWS, but both client and server were running on the same machine. We ran similar tests before where they were on separate machines but didn't see much of a difference.

Total: 21m 17s 321ms

Wow, that's a massive difference. While that was running did you see which steps were taking the longest? You can also set RUST_LOG=info in your shell to get logging output from oxen.

We'll keep looking into this. Performance is definitely a big priority for us right now.

Jul 10 '25 17:07 jcelliott

@jcelliott

Even Oxen cloud is taking nearly the same amount of time— Total: 24m 20s 482ms —to upload a 3GB file. Please review the full logs shared below:

Environment-3

OS: macOS 15.3.1 Oxen CLI version: 0.36.1 (built locally from the latest cloned source) Oxen Server: Oxen Cloud Oxen CLI: Running locally

➜  ~ mkdir oxtest
➜  ~ cd oxtest
➜  oxtest mkfile -n 3g 3GB_test_file3
➜  oxtest oxen init 
🐂 repository initialized at: "/Users/type-checker/oxtest"

    📖 If this is your first time using Oxen, check out the CLI docs at:
            https://docs.oxen.ai/getting-started/cli

    💬 For more support, or to chat with the Oxen team, join our Discord:
            https://discord.gg/s3tBEn7Ptg

➜  oxtest oxen --version
oxen 0.36.1
➜  oxtest oxen add .
🐂 oxen added 1 files (3.2 GB) in 9s 322ms
➜  oxtest oxen commit -m 3gb
Committing with message: 3gb
🐂 commit 680e462604ac4ce6c1d301a8eef379cc -> '3gb' in 58ms
➜  oxtest oxen create-remote --name abhi/oxtest --host hub.oxen.ai --scheme https
🎉 Remote successfully created for 'abhi/oxtest'

If this is a brand new repository:

  oxen clone https://hub.oxen.ai/abhi/oxtest

To push an existing local repository to a new remote:

  oxen config --set-remote origin https://hub.oxen.ai/abhi/oxtest

➜  oxtest oxen config --set-remote origin https://hub.oxen.ai/abhi/oxtest
➜  oxtest oxen --version
oxen 0.36.1
➜  oxtest oxen push
🐂 oxen push origin main -> 680e462604ac4ce6c1d301a8eef379cc
🐂 push complete 🎉 took 24m 11s 102ms

Total: 24m 20s 482ms

That was running on a Linux machine in AWS, but both client and server were running on the same machine. We ran similar tests before where they were on separate machines but didn't see much of a difference.

We definitely can't compare the high-bandwidth internal AWS network performance to transfers from local to server. We should benchmark using two different methods:

Local-to-cloud server
Cloud server-to-cloud server

Please note that in my tests, pushing from local to cloud server using Oxen is slower than HF, DVC, and GitHub.

Also, is the repository you're benchmarking with open source? If so, I can try testing with the same repo and experiment with different repository combinations to help improve performance.

Jul 10 '25 17:07 type-checker

I am able to trigger "Resource temporarily unavailable" in a test - https://github.com/Oxen-AI/Oxen/pull/651/commits/cf9fb6cbdf5c80f1fefe39c396616f73bf83699a, though I grant you it could be unrelated

Jul 18 '25 14:07 paul-hammant

I am able to trigger "Resource temporarily unavailable" in a test - cf9fb6c, though I grant you it could be unrelated

Do you know the root cause of this?

Jul 21 '25 07:07 type-checker

I don't. Also, the test I was able to reproduce it in is gone now, so ignore me

Jul 21 '25 07:07 paul-hammant

BUG with `oxen add`, `oxen push`, and `oxen delete-remote`

BUG Lists

Environment

Issue-1

Steps to Reproduce

Expected Behavior

Suggested Solution

Other Issues

Expected Behavior:

Update

Issue-2

Expected Behavior

Update

Issue-3

Expected Behavior

Issue-4

Observed Behavior

Environment-1

Environment-2

Environment-3