influx-cli
influx-cli copied to clipboard
UX improvements for influx write
Proposal:
To support large data imports, influx write
should have the following improvements:
- [ ] Error handling - failing lines are written out to a log file with the error message, for easy correction and re-importation
- [ ] Error threshold - an import will halt when it reaches a given error-threshold
- [ ] Restart-ability - record its progress through each file and record when it failed or was halted
- [ ] Progress reporting - report on the status of import periodically, with lines imported, lines failed, est. percentage completion, etc.
- [x] Handles compressed files - compressed line protocol or CSV files should be able to be read
- [ ] Respond to standard HTTP errors to throttle uploads dynamically
Use case: These features will make importing very large data sets into Influx much simpler and more robust, helping new customers on-board with less pain.
Throttling on a known allowed ingest rate as to not cause disruption for the receiver would be helpful as well. Will this support multi-threading?
@Joe-Influx - I believe influx write
already supports throttling to a fixed rate with the rate-limit
argument. I wasn't planning on changing the threading model, just to keep the scope within the time I have.
Converting this to an epic so we can track the improvements individually.
It looks like there might already be code for the "Error handling" idea, I see an --errors-file
option in the CLI. Need to investigate room for improvement there if the current behavior isn't useful.