immich
immich copied to clipboard
immich-app
feat(cli): use a queue for duplicate and upload
Using a queue to process the files makes the file duplicate detection and asset upload more stable and tolerant of network errors. If an error occurs, the whole command will not stop; the task will be retried (3 times) before logging the error and moving to the next step.
The new queue abstraction is using fastq internally.
Fixes: #5839
I tested those changes with a 62.7 GB upload (12k files), but I did not write any tests; let me know if you would like some tests. 🙂
I tested those changes with a 62.7 GB upload (12k files), but I did not write any tests; let me know if you would like some tests. 🙂
If you would be willing to write some tests, that would be awesome! We're always trying to increase our coverage to decrease the chance of regressions in future releases :slightly_smiling_face:
I tested those changes with a 62.7 GB upload (12k files), but I did not write any tests; let me know if you would like some tests. 🙂
If you would be willing to write some tests, that would be awesome! We're always trying to increase our coverage to decrease the chance of regressions in future releases 🙂
I see no setup for e2e/integration test (testing a command). Would you prefer unit tests or an e2e/integration test?
@zackpollard @jrasm91 I'm not sure what the best strategy for testing the upload retry could be. This test may not be an e2e test. I thought of mocking the HTTP request in the e2e tests with something like vitest-fetch-mock, but since we are executing a shell command to run the CLI, it's not possible.
What do you suggest?
Yeah, good point. I don't see any great way to accomplish that in our actual e2e environment. The only thing i can think of is doing something like executing docker restart immich_server or something. Not sure if that is a good, reliable option or not.
Yeah, good point. I don't see any great way to accomplish that in our actual e2e environment. The only thing i can think of is doing something like executing
docker restart immich_serveror something. Not sure if that is a good, reliable option or not.
It feels like it could introduce flaky tests as the request retry will exhaust until the server has restarted.
Yeah, probably. You could always add some unit tests that run with vitest where you mock the sdk methods and have one reject with a network error.
@zackpollard @jrasm91 This is ready for review.
I did my best to mock with accuracy, inspecting the requests made in the e2e tests.
I'm waiting for your feedback! 🙂