clean-and-green-philly
clean-and-green-philly copied to clipboard
Task: Figure out best practice for caching data in GC
We still need to make decisions regarding our plans for caching data in between runs on GC, and how that will interact between the VM running our pipeline and eventually the bucket serving our frontend data.
This is a bit nebulous as a ticket, but generally we need a plan for the infrastructure to store all the other auxiliary data involved in data-diffing and long term tracking of source data changes for pipeline and where that's going.