batch_api icon indicating copy to clipboard operation
batch_api copied to clipboard

Add naive semiparallel processor

Open txus opened this issue 12 years ago • 6 comments

A naive semiparallel processor.

It chunks consecutive GET requests and executes each chunk in a map of threads.

To use it, simply pass sequential: false in the request parameters.

txus avatar Dec 04 '12 10:12 txus

After writing a spec for the Batch API, I'm thinking this should be the default. There's really no need for fully sequential. I'll poke around with this later this week.

arsduo avatar Jan 14 '13 02:01 arsduo

Any progress on this?

lukelex avatar Oct 04 '14 21:10 lukelex

I would love to see some movement on this as well.

dvgica avatar Nov 11 '14 21:11 dvgica

@lukelex @DWvanGeest I haven't worked on this project in a while, but hope to catch it up this weekend. Did you ever test @txus' pull request? Does it work for you? If so, I'll merge it in.

arsduo avatar Apr 04 '15 18:04 arsduo

Actually, there's a later pull request that does full parallel processing with Celluloid: https://github.com/arsduo/batch_api/pull/33 -- if you're still interested in parallel processing and could try that out and see if that works for you, that's probably the direction this project should go.

arsduo avatar Apr 04 '15 18:04 arsduo

So should this commit be closed in favor of #33?

jeremyhaile avatar Jun 10 '15 15:06 jeremyhaile