aws_pricing_data_ingestor icon indicating copy to clipboard operation
aws_pricing_data_ingestor copied to clipboard

Migrate to cockroach DB

Open Bjorn248 opened this issue 3 years ago • 0 comments

First we should choose an execution environment, the candidates are:

  • Github Actions (leaning towards this, as I've had nothing but good experiences with it)
  • AWS Lambda (the existing infra, but I'm only using it because it's free...no real reason to stay)

The high level requirements of the ingestor are

  • [ ] dynamic schema generation from CSVs (existing python already does this, will need to be modified to work with cockroach schemas)
  • [ ] dynamic service ingestion (as new services are added by AWS the ingestor automatically picks them up and imports them, this already happens with the python scripts)
  • [ ] drop any table we are about to import
  • [ ] trigger an async cockroach import of a delimited data file, this has been tested in a local POC and works wonderfully
  • [ ] create an index on the EC2 table after it has been generated

Bjorn248 avatar Jan 08 '22 20:01 Bjorn248