Make it possible to index a large file with many documents using multiple threads

[jan-niestadt]

Right now, multithreading works per-file. This means that a single large file that contains many documents cannot use more than one CPU core.

We could try reading a few input documents into memory in a single thread, then process those documents in multiple threads. This would allow better CPU usage in this case.