list
list copied to clipboard
Poor performance of ADI
Automated data ingestion is around an order of magnitude slower than it could be: importing all the US data into mongodb atlas should take ~12h but ingesting takes nearly two weeks. Some rough testing shows that the solution is to not use mongoose in the data-service, but we need to address two things:
- reporting validation errors currently done by
batchValidate
using mongoose's schema validation (probably fixed by updating to mongodb 5) - keeping the confirmationDate and the confirmation event's date in sync is currently done by a mongoose trigger.