Akumuli icon indicating copy to clipboard operation
Akumuli copied to clipboard

parallel load data

Open a615549958 opened this issue 5 years ago • 2 comments

Hi, I use influxdb-comparisons which tool you developed. In load_data.sh file ,I set workers>1, where is "broken pipe" error running. I find "late write" in log. Because I loading data's timestamp is smaller than database data? If use parallel loading ,this error will still exist. Do you have good idea for parallel loading.

a615549958 avatar Apr 11 '19 07:04 a615549958

This can happen if you haven’t cleared the database between the runs of the ingestion tool from the influxdb-comparisons. It'll try to write the same data twice. And, I believe that the 'influxdb-comparisons' doesn't work in parallel. I have this set of scripts that can be used to write data in parallel, if you want to benchmark the top ingestion performance - https://github.com/Lazin/roundtrip_test. I think it's easier to use than the tool you mentioned.

BTW, I'm planning to add Akumuli support to the https://github.com/timescale/tsbs test suite. It will be easier to use and it will work in parallel.

Lazin avatar Apr 11 '19 09:04 Lazin

Thanks, I'll have a try.

a615549958 avatar Apr 13 '19 05:04 a615549958