netlify-plugin-gatsby
netlify-plugin-gatsby copied to clipboard
netlify
Optionally load Gatsby datastore into lambdas after deploy
We currently include the Gatsby lmdb datastore (data.mdb) in the lambda bundle, then move it into the tmp dir at first run, because it needs to be in a writeable filesystem. The problem with this is that for large sites the database can be larger than the maximum lambda deploy size. We could avoid this by instead loading the db from the CDN at first run. This would mean its size would be limited only by the max tmp dir size (512MB), but it would have a cold start cost. It would also mean the db was technically exposed to the internet. However if we create a filename that is very long and random then that is equivalent to password protection. The filename could be bundled inside the lambda instead, along with a hash of the file.
Because of the tradeoff, it would make sense for this to be optional. We could either do it automatically if we see the datastore is larger than a certain size, or we could use an env var and give an error message that suggests enabling it if the db is too large.
Some notes from our discussion:
- we could set headers config to add basic auth to the db file too
- we could pre-warm the lambda in an
onSuccesscall, which would ensure the db was loaded before the first request
Enterprise customer excited for the fix: https://netlify.zendesk.com/agent/tickets/92557
As a slight widening of scope here, if changes to the DB can be made independently to the rest of the build that would be a huge value add to our customers. It might require some work from Workflow folks but please include this in your conversations.
For more context reach out to me (or technically) @ascorbic .
@krider2010 I've created a separate issue for the work that you mentioned here and assigned @ascorbic so he can fill out the details when some thought has been given to how we want to approach that
Just to give an update here - main functionality has been released, currently working on the documentation for this
Looks like the documentation is done. Is there anything else left for this issue @ericapisani or is it OK to close this?
