influx-cli icon indicating copy to clipboard operation
influx-cli copied to clipboard

UX improvements for influx write

Open davidby-influx opened this issue 4 years ago • 4 comments

Proposal: To support large data imports, influx write should have the following improvements:

  • [ ] Error handling - failing lines are written out to a log file with the error message, for easy correction and re-importation
  • [ ] Error threshold - an import will halt when it reaches a given error-threshold
  • [ ] Restart-ability - record its progress through each file and record when it failed or was halted
  • [ ] Progress reporting - report on the status of import periodically, with lines imported, lines failed, est. percentage completion, etc.
  • [x] Handles compressed files - compressed line protocol or CSV files should be able to be read
  • [ ] Respond to standard HTTP errors to throttle uploads dynamically

Use case: These features will make importing very large data sets into Influx much simpler and more robust, helping new customers on-board with less pain.

davidby-influx avatar Jan 06 '21 21:01 davidby-influx

Throttling on a known allowed ingest rate as to not cause disruption for the receiver would be helpful as well. Will this support multi-threading?

Joe-Influx avatar Jan 08 '21 20:01 Joe-Influx

@Joe-Influx - I believe influx write already supports throttling to a fixed rate with the rate-limit argument. I wasn't planning on changing the threading model, just to keep the scope within the time I have.

davidby-influx avatar Jan 08 '21 21:01 davidby-influx

Converting this to an epic so we can track the improvements individually.

danxmoran avatar Jan 15 '21 20:01 danxmoran

It looks like there might already be code for the "Error handling" idea, I see an --errors-file option in the CLI. Need to investigate room for improvement there if the current behavior isn't useful.

danxmoran avatar Jan 15 '21 20:01 danxmoran