batch_api
batch_api copied to clipboard
Add naive semiparallel processor
A naive semiparallel processor.
It chunks consecutive GET requests and executes each chunk in a map of threads.
To use it, simply pass sequential: false
in the request parameters.
After writing a spec for the Batch API, I'm thinking this should be the default. There's really no need for fully sequential. I'll poke around with this later this week.
Any progress on this?
I would love to see some movement on this as well.
@lukelex @DWvanGeest I haven't worked on this project in a while, but hope to catch it up this weekend. Did you ever test @txus' pull request? Does it work for you? If so, I'll merge it in.
Actually, there's a later pull request that does full parallel processing with Celluloid: https://github.com/arsduo/batch_api/pull/33 -- if you're still interested in parallel processing and could try that out and see if that works for you, that's probably the direction this project should go.
So should this commit be closed in favor of #33?