cyclop
cyclop copied to clipboard
Java 8 -> use streaming API for import
Currently we upload whole import file to server, divide it into chunks, and each chunk is being submitted to execution service, so that it can be processed in parallel.
New implementation will create Selector for input stream. Each parsed query will be submitted to blocking queue. We will define xx threads that can process submitted queries. This has some advantages: we do not have load whole import file into memory, import starts immediately after receiving first query, load spreads equally over available threads.