aws_pricing_data_ingestor
aws_pricing_data_ingestor copied to clipboard
Migrate to cockroach DB
First we should choose an execution environment, the candidates are:
- Github Actions (leaning towards this, as I've had nothing but good experiences with it)
- AWS Lambda (the existing infra, but I'm only using it because it's free...no real reason to stay)
The high level requirements of the ingestor are
- [ ] dynamic schema generation from CSVs (existing python already does this, will need to be modified to work with cockroach schemas)
- [ ] dynamic service ingestion (as new services are added by AWS the ingestor automatically picks them up and imports them, this already happens with the python scripts)
- [ ] drop any table we are about to import
- [ ] trigger an async cockroach import of a delimited data file, this has been tested in a local POC and works wonderfully
- [ ] create an index on the EC2 table after it has been generated