fbms icon indicating copy to clipboard operation
fbms copied to clipboard

Published 20 hours ago • verified publisher icon jxnl

Systems.

Open jxnl opened this issue 10 years ago • 3 comments

A bit hand wavy but the guys working on the datastore and systems should talk together and think of smart ways to automate the systems involved and to connect all the pieces.

jxnl avatar Oct 02 '14 22:10 jxnl

the crawler is in place, we just need to consistently update the stores

jxnl avatar Nov 13 '14 20:11 jxnl

Can you add a little more detail to this issue?

Do you just want a background process to run the crawler every minute? You can do that pretty easily with celery or rq which I have experience with. Huey is another good contender.

AlJohri avatar Nov 27 '14 23:11 AlJohri

So there are the current problems.

  1. tokens expiring and refreshing tokens
  2. robust way to avoid rate limiting

Right now I have to generate a api key, and then run the crawl job (crawling everything takes about 20 minutes with sleeps for rate limiting). next time I need to run it, the token has expired.

I think there is a way to request a longer term token but I couldn't seem to figure out how to do it...

also: Huey looks like exactly the job for the scheduling part!

jxnl avatar Nov 28 '14 00:11 jxnl