feat(source-nodes): optimize node sourcing and transformation for improved performance

[demirgazetic]

Changes made:

• Implement concurrent fetching of pages using Promise.all to speed up data retrieval.
• Refactor variable names for better readability and maintainability.
• Enhance node processing logic to handle content fields more efficiently.
• Ensure robust handling of datasource entries and their dimensions.
• Maintain support for local assets with improved caching mechanism.
• Remove unused code.
• Fix linting issues.
• Update README.md.

Reason for the Change

Our goal was to enable incremental builds in our Gatsby using the official gatsby-source-storyblok plugin. During implementation, we noticed that the source and transform nodes step was significantly slow, leading us to reconsider using this plugin.

After reviewing the source code of the Storyblok plugin, we identified several potential improvements. This pull request (PR) details the enhancements we've made.

Key Changes and Benefits

1. Sequential Fetching to Concurrent Fetching:

   • Improved Performance: Parallel requests significantly boost performance, especially for large datasets.
   • Rate Limit Handling: Concurrent fetching allows for more careful management of rate limits and server load.
   • High-Performance Applications: Though more complex, this approach is necessary for optimizing high-performance applications.

   By implementing concurrent fetching, we observed a 20%-30% speed increase for large spaces with over 5000 pages and more than 20 datasources, some containing over 10,000 values.

2. Optimizing Datasource Fetching:

   • Focused Fetching: Some datasources, in our case such as icons are used internally in Storyblok, are not required on the frontend. By selectively fetching only the needed datasources, we further optimized performance.
   • New Option includeDatasources: This option allows specifying which datasources to fetch, avoiding unnecessary data retrieval.

   Example configuration in gatsby-config.js:

   {
     resolve: 'gatsby-source-storyblok',
     options: {
       accessToken: 'YOUR_TOKEN',
       version: 'draft',
       resolveRelations: [''],
       includeLinks: false,
       includeDatasources: ['datasource1', 'datasource2', 'datasource3']
     }
   }
   

   Implementation logic:

   if (options.includeDatasources === undefined) {
     datasources = await fetchAllDatasources();
   } else if (options.includeDatasources.length > 0) {
     datasources = options.includeDatasources;
   }
   

3. Fetching Tags Since not everyone uses Storyblok Tags, having the option to disable this feature would also save time.

Summary

These changes result in significant performance improvements, making the plugin more suitable for large projects. By implementing concurrent fetching and selective datasource fetching, we have optimized the source and transform nodes step, making the Gatsby build process more efficient.