cyclop icon indicating copy to clipboard operation
cyclop copied to clipboard

Java 8 -> use streaming API for import

Open maciejmiklas opened this issue 10 years ago • 0 comments

Currently we upload whole import file to server, divide it into chunks, and each chunk is being submitted to execution service, so that it can be processed in parallel.

New implementation will create Selector for input stream. Each parsed query will be submitted to blocking queue. We will define xx threads that can process submitted queries. This has some advantages: we do not have load whole import file into memory, import starts immediately after receiving first query, load spreads equally over available threads.

maciejmiklas avatar May 13 '14 05:05 maciejmiklas