gatsby-source-airtable icon indicating copy to clipboard operation
gatsby-source-airtable copied to clipboard

Published 20 hours ago • verified publisher icon jbolda

faster build when sourcing a lot of data

Open kimberrypi opened this issue 4 years ago • 1 comments

We are using this library to load a lot of records from Airtable and we noticed the significant increase in time when sourcing and transforming nodes. At around 30,000 records, we already for 11 minutes every run. Is it possible to only source and transform updated records?

Thank you.

kimberrypi avatar Jun 16 '21 02:06 kimberrypi

As far as I am aware, Airtable doesn't have any real way to fetch in a way that we could incrementally build. This plugin outputs the time it takes to fetch your data. How long does that step take? From there, it is mostly a handoff to gatsby, and I imagine much of the slow down is happening through normal / core gatsby graphql node processing: 30k nodes / queries / pages is a pretty large site.

Also at some point, I imagine there is a level where the scope and expectation of this plugin is going to be underwhelming for a large enough site, and warrant a true CMS that can support the incremental builds within gatsby.

jbolda avatar Jun 18 '21 03:06 jbolda

Don't believe there is anything further to add here. Going ahead and closing it.

jbolda avatar Jan 28 '23 07:01 jbolda