conveyor
conveyor copied to clipboard
samtecspg
Unable to upload huge csv file
Hi All,
We have a requirement to upload a billing CSV data into elasticsearch which is close to 30 GB.
I used conveyor for this process and created a channel and upload the file. All the progress bar shows the file upload is completed, but when I access the index I get an error saying there are no data to show for the specified index.
I believe though the progress bar shows the upload is completed, still the files are not loading to the elasticsearch.
I tried with both latest version and the previous version, I end up still the same.
no errors but no data have been a load to the elasticsearch.
Need help in troubleshooting this problem.
I tried with simple 700mB file and that works just fine, so I am guessing the conveyor is not handling huge data files.
Yeah we never loaded anything over probably 1G. Are you using our compose for Elasticsearch or is this in your own ES cluster? Might need to increase the RAM allocated to Docker and/or ES.
It's also possible that there is some sort of a mapping problem preventing indexing. Are there any console errors?
Yes I am using the complete rocker set up to test it (not my own ES).
I was gettung, payload is greater than X value once the upload was completed but apart from that I din see any other error message.
I tired to tweak the server payload settings in kibana config file, but it doesn't help.
I would really appreciate if we can have this setup which supports huge upload as well.
So if I remember correctly you have to change the payload settings in Kibana, but also Node-RED, which is what Conveyor uses under the hood for moving stuff around. We don't expose this as a parameter, but I will try to increase it and test on a larger file sometime today.
https://github.com/samtecspg/conveyor/blob/master/api/plugins/startup-install.plugin.js#L77