flaky test: `python/tests/test_optimize.py::test_compact_with_write`

[chebbyChefNEQ]

FAILED python/tests/test_optimize.py::test_compact_with_write - OSError: Dataset at path tmp/pytest-of-runner/pytest-0/test_compact_with_write0/dataset was not found: LanceError(IO): Generic LocalFileSystem error: Unable to access metadata for tmp/pytest-of-runner/pytest-0/test_compact_with_write0/dataset/_versions/10.manifest-e86142ed-10fd-41e6-9dea-1d06832ae144: IO error for operation on /tmp/pytest-of-runner/pytest-0/test_compact_with_write0/dataset/_versions/10.manifest-e86142ed-10fd-41e6-9dea-1d06832ae144: No such file or directory (os error 2), /home/runner/work/lance/lance/rust/lance-table/src/io/commit.rs:89:26, 

https://github.com/lancedb/lance/actions/runs/8824077950/job/24225845820