feed-to-sqlite Fetch feeds in parallel

Is there an easy way to do this within the standard library? Don't want to add a giant dependency if I can avoid it. Haven't done much with asyncio.

Dec 12 '19 13:12 eyeseast

Possibly useful: https://github.com/pallets/click/issues/85#issuecomment-503464628

Dec 23 '19 20:12 eyeseast

Thinking through this further, it would mean turning the whole thing async. Which is maybe worth trying, just for kicks.

Dec 28 '19 02:12 eyeseast

Also possibly useful: https://github.com/encode/httpx

The sqlite work will probably have to be in a thread, or at the very end, since concurrent writes won't work.

Jan 02 '20 00:01 eyeseast

Switched to HTTPX. Proper async report is likely dependent on upstream help here: https://github.com/simonw/sqlite-utils/issues/242

Feb 27 '21 20:02 eyeseast

Realizing I can probably do this without waiting on full async support from sqlite-utils. The key is making normalize an async function, and then using loop.run_in_executor.

Mar 01 '21 14:03 eyeseast