Improve import performance
Description
This PR introduces improvements to the importer, which previously stored imported data in memory, potentially consuming excessive resources for larger files. This update leverages the League/Csv package memory efficient technique. This PR also reduces the overhead of performing two iterations. The only caveat is the use of a progress indicator when using the import command instead of a progress bar, which, in my opinion, is more appropriate for long-running commands and tasks that cannot be easily quantified.
Checklist:
- [x] I have read the Contributing documentation available here: https://snipe-it.readme.io/docs/contributing-overview
- [x] I have formatted this PR according to the project guidelines: https://snipe-it.readme.io/docs/contributing-overview#pull-request-guidelines
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
PR Summary
-
Enhanced Command Line Feedback We now have a new
ProgressIndicatorclass that will improve visual feedback during the import process which replaces the old comment-based feedback mechanism. It gives a more dynamic display of the ongoing status and completion stage of the import. -
Code Streamlining and Clarity Enhancement This update includes a cleanup of the code by removing unused variables like the
barvariable and error tracking arrays from theObjectImportCommandclass. This helps to keep the code clean and easier to understand. -
Simplified Data Handling There is an improvement in the way we were handling CSV data. The
normalizeInputArraymethod in theImporterclass has been removed and replaced with a direct case normalization process in the import loop. This makes our data handling process more straightforward. -
Improved Usage of Parameters The manner in which we track the count of imported items within transactions has been adjusted to provide better clarity on the handling of callbacks.
-
Better Error Handling When encountering errors with invalid items during the import process, you will notice a cleaner and easily readable format for displays which makes rectifying such issues much effortless.
This looks great, thanks!