snipe-it icon indicating copy to clipboard operation
snipe-it copied to clipboard

Improve import performance

Open bryanlopezinc opened this issue 1 year ago • 1 comments

Description

This PR introduces improvements to the importer, which previously stored imported data in memory, potentially consuming excessive resources for larger files. This update leverages the League/Csv package memory efficient technique. This PR also reduces the overhead of performing two iterations. The only caveat is the use of a progress indicator when using the import command instead of a progress bar, which, in my opinion, is more appropriate for long-running commands and tasks that cannot be easily quantified.

Checklist:

  • [x] I have read the Contributing documentation available here: https://snipe-it.readme.io/docs/contributing-overview
  • [x] I have formatted this PR according to the project guidelines: https://snipe-it.readme.io/docs/contributing-overview#pull-request-guidelines
  • [x] My code follows the style guidelines of this project
  • [x] I have performed a self-review of my own code
  • [x] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [x] My changes generate no new warnings
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • [x] New and existing unit tests pass locally with my changes

bryanlopezinc avatar Oct 10 '24 23:10 bryanlopezinc

PR Summary

  • Enhanced Command Line Feedback We now have a new ProgressIndicator class that will improve visual feedback during the import process which replaces the old comment-based feedback mechanism. It gives a more dynamic display of the ongoing status and completion stage of the import.

  • Code Streamlining and Clarity Enhancement This update includes a cleanup of the code by removing unused variables like the bar variable and error tracking arrays from the ObjectImportCommand class. This helps to keep the code clean and easier to understand.

  • Simplified Data Handling There is an improvement in the way we were handling CSV data. The normalizeInputArray method in the Importer class has been removed and replaced with a direct case normalization process in the import loop. This makes our data handling process more straightforward.

  • Improved Usage of Parameters The manner in which we track the count of imported items within transactions has been adjusted to provide better clarity on the handling of callbacks.

  • Better Error Handling When encountering errors with invalid items during the import process, you will notice a cleaner and easily readable format for displays which makes rectifying such issues much effortless.

what-the-diff[bot] avatar Oct 10 '24 23:10 what-the-diff[bot]

This looks great, thanks!

snipe avatar Oct 22 '24 15:10 snipe