David Eads

Results 31 issues of David Eads

better testing mode

thinking there can be a configurable target % of urls and then some way of doing modulo math or taking a random sample of urls

need to mock environment variables, not actually set them, in tests

does "target" even make sense as configuration option?

priority: normal

scrape.sh should respect target

priority: low

more tests

Things to test: - [ ] Cleanup scripts - [ ] Spider use of config - [ ] Mirroring / saving bits - [ ] URL generation - [ ]...

develop basic ingest script from CSV files

3 comment

depends on #32

priority: high
data processing
feature

decouple s3 storage for scraped pages and csv snapshots

priority: normal
scraping
feature

clean up requirements

2 comment

priority: low

develop and document new data model

priority: high
data processing
documentation

document s3 dependencies

priority: normal
documentation