mempool-dumpster
mempool-dumpster copied to clipboard
Dockerize production setup
Goal:
- Run mempool dumpster instances in Docker
- Uploading hourly files to R2/S3 (with a bit of delay)
- Nightly processing should pull all these files from S3, and process them from there
Notes:
- collector and processor need access to reliable EL nodes, for checking of inclusion status
- nightly processing is using a lot of memory to process all the transactions (up to ~6.5GB)
Links:
- https://github.com/aws/aws-sdk-go
- https://developers.cloudflare.com/r2/examples/aws/aws-sdk-go/