Aneel Nazareth

Comments? Suggestions?

The purpose of using Docker is to encourage folks to properly encapsulate any dependencies (Texract has a bunch of non-Python dependencies, for example) so that when the time comes to...

Yes, exactly. The idea would be that we would have a table listing scraping requests and their statuses. The scraper would look for any requests with a NEW status or...

Incidentally, this is a fairly common "work queue" pattern.

I strongly recommend using Data Access Objects to talk to the DB, rather than writing SELECT and UPDATE statements. I'm planning to use SQLAlchemy on the Python side. You can...

And to answer your question about the scraper, yes, we'd want to have a scraper periodically checking the DB for URLs that need scraping. But, of course, if you're running...

Gotcha. #77 was the "get the AWS RDS DB working just like the SQLite DB" PR. There will be a PR forthcoming off of the `schema` branch that uses SQLAlchemy....

Sort of. SQLAlchemy uses the low-level drivers like sqlite3 and psycopg2 to talk to SQLite and PostgreSQL databases. You won't have to interact with anything but the model classes in...

Great, thanks!

@aminebouzaghrane, it would not be productive to work on this. This repo is on hiatus while we work with the IDMC on continuing this project.