stripe-archive
Reimport takes several hours
Great project, it's been super effective for us when exporting small amounts of data from mongo, but as we've added more data we're seeing it take several hours rather than several minutes. Has anyone hit a similar wall? It is a substantial amount of data, but maybe we're doing something wrong?
| Fields | Objects |
|---|---|
| 3 | 42K |
| 8 | 1.4M |
| 5 | 18K |
| 10 | 180K |
| 4 | 1.5K |
| 5 | 13K |
If you run an import with -v, it should show some basic stats about where the time is spent -- in SQL vs. reading/transforming the data. What does that show? Approximately what rate of import (documents/s) are you getting?
I saw performance problems when I had big latency between MongoDB & PostGre databases. I tried different configurations on the cloud, using different MongoDb providers (MongoDB, EC2 instance, etc.) & PostGre providers (Amazon RDS, Heroku PostGre, etc.) to had the better performance. Hosting everything on EC2 was the best solution!
Hope it helps!
