Graph fails to query images

[mosesoak]

Preliminary Checks

• [X] This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• [X] This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

A project that uses Netlify CMS with gatsby-transformer-remark and the community plugin gatsby-remark-relative-images which worked perfectly in Gatsby 3 now gets null on Gatsby 4 after a cache has been built. Netlify builds exhibit this same behavior: they work on fresh build but fail on cached builds.

Sister issue filed with the relative image plugin here https://github.com/danielmahon/gatsby-remark-relative-images/issues/61

We've been trying to upgrade to latest Gatsby (4.21) and React 18.2 for some time now and this is blocking us.

Reproduction Link

https://github.com/yallllc/gatsby-image-query-bug

Steps to Reproduce

• yarn
• yarn clean && yarn build
  • succeeds with image
• yarn build (again, this uses the cache)
  • fails on null image (TypeError: Cannot read properties of null (reading 'childImageSharp'))
• yarn clean && yarn develop
  • image succeeds in frontend app
  • image is null in GraphiQL
  • changing text in home.md makes GraphiQL work
• Stop development server
• yarn develop (again, this uses the cache)
  • fails on null image in frontend & GraphiQL
  • changing text in home.md makes frontend & GraphiQL work

Expected Result

Graph queries images properly

Actual Result

Graph gets null images. Changing the md file results in finding the images. Cached builds fail on Netlify have missing images as well.

Environment

System:
    OS: Linux 5.4 Ubuntu 22.04 LTS 22.04 (Jammy Jellyfish)
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    Shell: 5.1.16 - /bin/bash
  Binaries:
    Node: 16.14.2 - ~/.nvm/versions/node/v16.14.2/bin/node
    Yarn: 1.22.18 - ~/.nvm/versions/node/v16.14.2/bin/yarn
    npm: 8.5.0 - ~/.nvm/versions/node/v16.14.2/bin/npm
  npmPackages:
    gatsby: ^4.21.1 => 4.21.1 
    gatsby-plugin-image: ^2.21.0 => 2.21.0 
    gatsby-plugin-netlify-cms: ^6.21.0 => 6.21.0 
    gatsby-plugin-sharp: ^4.21.0 => 4.21.0 
    gatsby-remark-images: ^6.21.0 => 6.21.0 
    gatsby-remark-relative-images: ^2.0.2 => 2.0.2 
    gatsby-source-filesystem: ^4.21.1 => 4.21.1 
    gatsby-transformer-remark: ^5.21.0 => 5.21.0 
    gatsby-transformer-sharp: ^4.21.0 => 4.21.0

Config Flags

We use FAST_DEV in our production project but the repro does not and exhibits the same bug behavior.

[i-bsd]

Can confirm - having the exact same problem with gatsby-remark-images.

Editing and saving a markdown file throws TypeError: Cannot read property 'dir' of null for the image. Only way forward is to clear cache and run gatsby develop again, which works until I make another markdown edit.

[mosesoak]

CC @KyleAMathews -- Sidebar: we're now experiencing the dev-ssr "split" error reported in https://github.com/gatsbyjs/gatsby/issues/30780 on dev after adding the super-awesome @loadable/component / gatsby-plugin-loadable-components-ssr plugin, which code-splits our heavy CMS project correctly until we can get on React 18 and Gatsby 4.

Unfortunately your fix for the 'split' bug https://github.com/gatsbyjs/gatsby/pull/30801 made it into Gatsby 4 but we're stuck on 3 due to the above issue. Turning off DEV_SSR for now! 🤷

Thanks for any help you can provide on this issue!

[pieh]

gatsby-remark-relative-images (at least part that "rewrite" frontmatter paths) is not compatible with gatsby@4. In particular https://www.gatsbyjs.com/docs/reference/release-notes/migrating-from-v3-to-v4/#dont-mutate-nodes-outside-of-expected-apis is not handled

[mosesoak]

Thanks @pieh that might be the underlying reason...

I noticed that in the repo https://github.com/netlify-templates/gatsby-starter-netlify-cms/ the author of gatsby-remark-relative-images, @danielmahon added the plugin on Gatsby 4 and it did not produce the above bug.

After a bunch of sleuthing I narrowed it down to the way the graphql query is formatted (again I don't know if this is a quirk of the plugin or the query itself) --

This one fails on cached build/develop:

   {
      pages: allMarkdownRemark(
        filter: {
          fileAbsolutePath: { regex: "/.*/cms/content/pages/home.md/" }
        }
      ) {
        edges {
          node {
            frontmatter {
              image {

While this one that doesn't call allMarkdownRemark always succeeds, without the bug:

    {
      markdownRemark(frontmatter: { slug: { eq: "home" } }) {
        frontmatter {
          image {

I've added a proof of this in my repro repo, on main, open the gatsby-node.ts file to test. Diff: https://github.com/yallllc/gatsby-image-query-bug/commit/8b76c8d4994acaad9d0b8f859001a23b395517bd

[pieh]

When I enabled debug mode made exactly to detect those problems I mentioned in previous comment (mode described in https://www.gatsbyjs.com/docs/how-to/local-development/debugging-missing-data/) I get:

success createSchemaCustomization - 0.014s
warn Node mutation detected

File /home/misiek/test/i36432/node_modules/traverse/index.js:134:50
  132 |             update : function (x, stopHere) {
  133 |                 if (!state.isRoot) {
> 134 |                     state.parent.node[state.key] = x;
      |                                                  ^
  135 |                 }
  136 |                 state.node = x;
  137 |                 if (stopHere) keepGoing = false;

Stack trace:
    at Object.update (/home/misiek/test/i36432/node_modules/traverse/index.js:134:50)
    at Object.<anonymous> (/home/misiek/test/i36432/node_modules/gatsby-remark-relative-images/dist/on-create-node.js:37:18)
    at walker (/home/misiek/test/i36432/node_modules/traverse/index.js:190:22)
    at /home/misiek/test/i36432/node_modules/traverse/index.js:208:29
    at Array.forEach (<anonymous>)
    at forEach (/home/misiek/test/i36432/node_modules/traverse/index.js:298:31)
    at walker (/home/misiek/test/i36432/node_modules/traverse/index.js:203:13)
    at walk (/home/misiek/test/i36432/node_modules/traverse/index.js:226:7)
    at Traverse.forEach (/home/misiek/test/i36432/node_modules/traverse/index.js:50:18)
    at Object.exports.onCreateNode (/home/misiek/test/i36432/node_modules/gatsby-remark-relative-images/dist/on-create-node.js:17:46)
    at runAPI (/home/misiek/test/i36432/node_modules/gatsby/src/utils/api-runner-node.js:487:22)
    at Promise.catch.decorateEvent.pluginName (/home/misiek/test/i36432/node_modules/gatsby/src/utils/api-runner-node.js:638:13)
    at Promise._execute (/home/misiek/test/i36432/node_modules/bluebird/js/release/debuggability.js:384:9)
    at Promise._resolveFromExecutor (/home/misiek/test/i36432/node_modules/bluebird/js/release/promise.js:518:18)
    at new Promise (/home/misiek/test/i36432/node_modules/bluebird/js/release/promise.js:103:10)
    at /home/misiek/test/i36432/node_modules/gatsby/src/utils/api-runner-node.js:636:16
success Checking for changed pages - 0.002s

which means something is updating object in-memory.

/home/misiek/test/i36432/node_modules/gatsby-remark-relative-images/dist/on-create-node.js:37:18)

In particular point to a code in gatsby-remark-relative-images that mutates frontmatter value.

It might "work" sometimes, because there is small in-memory lru cache (for performance reason), so those mutations might "stick" long enough to work in current build, but after that they are lost. If you don't edit the markdown file and you build again, gatsby won't run the code that mutates frontmatter (because file didn't change, so nothing created out of it - like markdown's frontmatter could change) and will use data it has stored in database (which won't contain those mutations), so if we will have to run query again it will fail with missing data kind of errors. Page and static quries are cachable between builds, so they might not "fail" if gatsby decides to reuse results from previous builds, but queries in gatsby-node currently are not cached and they just run on every build

There's no more debugging needed here. gatsby-remark-relative-images is mutating nodes in memory without updating them in database (source of truth for our data) and is as long as that's the case it won't be really compatible with gatsby@4

[danielmahon]

Hello, I just wanted chime in and say that I finally put up a "needs maintainer" notice on the gatsby-remark-relative-images repo. I have no plans to continue working on it so feel free to reach out if this functionality is something that could be bundled with Gatsby or if anyone would like to maintain it. Thanks.

[mosesoak]

Gatsby offers a Netlify CMS plugin that its team maintains, https://github.com/gatsbyjs/gatsby/tree/master/packages/gatsby-plugin-netlify-cms.

My question to @pieh or someone else who understands this stuff a bit more than I do would be, can't the main plugin just be updated slightly to target /static images? Like maybe here in the Webpack settings or node traversal? https://github.com/gatsbyjs/gatsby/blame/master/packages/gatsby-plugin-netlify-cms/src/gatsby-node.js

It just needs to be able to convert markdown image paths (e.g. /assets/image.jpg) during Gatsby build... that doesn't seem like a huge ask.. or is it? Thanks again, appreciate the help

[mosesoak]

Actually quick question for the Gatsby team: maybe we could just get someone familiar with gatsby-transformer-remark to help fix the image sourcing?

Seems like that transformer should be able to consume image paths that start with /assets/ but source them from static/assets/? That's the same way most static paths work I'd think? Thanks

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here. If we missed this issue or if you want to keep it open, please reply here. As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[mosesoak]

@pieh Ping on the above question, asking you because you and LekoArts seem to be the only ones around with any insight into gatsby-transformer-remark.

If the root of the problem is simply that markdown paths need to resolve without including static/ in their paths, would it be a feasible ask to add such an option to the transformer config?

Thank you and again sorry to bother you. I'm trying to make it possible to upgrade Gatsby in this live Netlify CMS site without it being left behind! 😬 😅

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here. If we missed this issue or if you want to keep it open, please reply here. As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[github-actions[bot]]

Hey again!

It’s been 60 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it. Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to comment on this issue or create a new one if you need anything else. As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community! 💪💜

[willianantunes]

Solution can be checked here.