Lychee icon indicating copy to clipboard operation
Lychee copied to clipboard

:construction: Upload zip extract import

Open ildyria opened this issue 1 year ago • 1 comments

Add support of upload of single zip files.

:warning: this is quite experimental, I tested locally that it works, however there are some caveats to know about.

  1. The uploaded zip file will stay on your server unless it is properly extracted without faults (after which it is removed).
  2. Similarly the extracted files from the zip will stay on your server unless they are properly imported without fault.
  3. On import success, you will see in /path-to-lychee/storage/extract-jobs an empty folder prefixed with YmD so you know when it was created, making easier to clean up later.
  4. ONLY THE FIRST LEVEL OF THE ZIP IS IMPORTED. See explanation below.

In other words, if you have

Something_Stupid.zip
├── img1.jpg
├── img2.jpg
├── img3.jpg
└── SubFolder
    ├── img4.jpg
    ├── img5.jpg
    └── img6.jpg

Only img1.jpg, img2.jpg and img3.jpg will be imported in Something Stupid album (note that _ is automatically replaced by space) and deleted from /path-to-lychee/storage/extract-jobs/yyyymmddSomething_Stupid. The folder SubFolder will still be in /path-to-lychee/storage/extract-jobs/yyyymmddSomething_Stupid/SubFolder

ildyria avatar Oct 17 '24 13:10 ildyria

Codecov Report

:x: Patch coverage is 98.55072% with 2 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 90.27%. Comparing base (f24e4d3) to head (6fbd19e). :warning: Report is 2 commits behind head on master.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov[bot] avatar Oct 17 '24 13:10 codecov[bot]

📝 Walkthrough

Walkthrough

Adds ZIP upload extraction and import jobs, a job-bus batch executor, removes the use_job_queues option, introduces extract_zip_on_upload config/migration, changes photo processing to always dispatch jobs, adds tests and small refactors.

Changes

Cohort / File(s) Summary of changes
Import batching pipeline
app/Actions/Import/Exec.php, app/Actions/Import/Pipes/ExecuteBatch.php, app/Actions/Import/Pipes/ImportPhotos.php, app/DTO/ImportDTO.php, app/Console/Commands/Sync.php
Adds should_execute_jobs flag to Exec and ImportDTO; Exec::do() now returns queued jobs (ImportImageJob[]); introduces ExecuteBatch pipe that dispatches jobs from ImportDTO->job_bus; ImportPhotos enqueues import jobs onto job_bus; Sync passes should_execute_jobs: true.
ZIP upload and extraction
app/Http/Controllers/Gallery/PhotoController.php, app/Jobs/ExtractZip.php, app/Jobs/CleanUpExtraction.php, app/Exceptions/Internal/ZipExtractionException.php, app/Exceptions/ZipInvalidException.php
Detects ZIP uploads and, when enabled/allowed, dispatches ExtractZip; adds ExtractZip job to validate/extract/import ZIP contents and to dispatch resulting import jobs; adds CleanUpExtraction job and ZIP-related exceptions.
Config and migrations
app/Http/Middleware/ConfigIntegrity.php, database/migrations/2025_08_23_204700_extract_zip_on_upload.php, database/migrations/2025_08_24_153735_remove_job_queue_option.php
Adds extract_zip_on_upload to integrity checks; migration to add that config; migration to remove the use_job_queues option.
Queue simplification (always async)
app/Actions/Photo/Pipes/Shared/ExtractColourPalette.php, app/Actions/Photo/Pipes/Shared/UploadSizeVariantsToS3.php, docs/backend/Lifecycle-of-a-request-photo-upload.md
Removes conditional sync/dispatch branches and always dispatches jobs asynchronously; docs updated to reflect unified dispatch behavior and stage decision based on queue.default.
Tests and sample handling
tests/Feature_v2/Photo/PhotoZipUploadTest.php, tests/Constants/TestConstants.php, .gitignore
Adds feature tests for ZIP upload and malformed ZIP; new ZIP test constants and sample path; removes use_job_queues test constant; adds tests/Samples/test_photos.zip to .gitignore.
Minor refactors / UI
app/Actions/HoneyPot/EnvAccessTentative.php, app/Image/Files/ProcessableJobFile.php, resources/js/views/Jobs.vue
Replaced Str::endsWith with PHP str_ends_with and removed unused import; renamed internal property fakeBaseNamefake_base_name; adjusted a meter color token in Jobs.vue.
Removals
app/Constants/UsersUserGroupsConstants.php, app/Exceptions/TagAlreadyExistException.php
Deleted UsersUserGroupsConstants class and TagAlreadyExistException class/files.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Poem

I twitch my whiskers at the zip,
I dig the threads and stack each trip.
Jobs march out in tidy rows,
I sweep the crumbs where extractor goes.
Hooray — the pipeline hops and skips. 🐰✨

✨ Finishing Touches
  • [ ] 📝 Generate Docstrings
🧪 Generate unit tests
  • [ ] Create PR with unit tests
  • [ ] Post copyable unit tests in a comment
  • [ ] Commit unit tests in branch upload-zip-extract-import

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

coderabbitai[bot] avatar Aug 18 '25 16:08 coderabbitai[bot]