gatsby-source-mysql icon indicating copy to clipboard operation
gatsby-source-mysql copied to clipboard

Taking long time in source and transform steps even in incremental build. Does this plugin using cache api aptly ?

Open yashvekaria opened this issue 3 years ago • 0 comments

I built a blog site of about 4000 articles and 12000 images, let's say 1 article has 3 images. My data source is MySQL and I am using this plugin to fetch data and also pass down my URLs to remoteImageFieldNames for image processing. So here are some stats give below.

FYI - we were using a T3 medium EC2 instance with 4GB RAM and 2 cores machine. CPU utilization was about 3-5% all the time.

For Fresh build (which is acceptable for 1st time)

Now for actual data which is 4000 articles and 12000 images (1 article has 3 images) it took 2.46 Hrs for the source and transform process and 2.49 Hrs for the whole build.

4000_1

I.e 2.49 Hrs and source and transform plugin took 2.46 Hrs.

Now I had changed the author name in the article table from the database and build it again. So here I expect less time as it will undergo incremental build, but it took

4000_2

I.e 2.56 Hrs (changed pages 3) and again maximum time is consumed by source and transform plugin around 2.55 Hrs.

So I believe that it is carrying out the whole source and transform process again even though it's an incremental build or is there any caching mechanism implemented within the plugin?

Can you please help me out to resolve this issue?

yashvekaria avatar May 27 '21 11:05 yashvekaria