Limit the number of documents processed at a single time.

[ebolyen]

Proposal

Currently it appears that all documents are processed at the same time in parallel. When using an executable plugin, this results in a process for each document. In the event that this executable plugin does something computationally expensive (like execute a code example), this quickly thrashes your machine.

I'm not sure if it makes sense to have a process pool for executable plugins, so that only X number run at a time (leaving promises on the event loop to wait around for a worker). Or, if the entire mdAST transform (getFileContent) should be batched, so that only X documents are processed at the same time.

We would probably want something like a single transform in our executable-plugin to run per CPU available.