[App Router -> Caching] Fetch cache is not revalidating for pages that have turned into 404 errors.

[rashidul0405]

Link to the code that reproduces this issue

https://codesandbox.io/p/devbox/fetch-cache-issue-gvq3xl

To Reproduce

1. pnpm start / next start
2. Visit /products/2 - responding 200
3. Underlying API responds 404: https://api.mockfly.dev/mocks/73685d60-5e65-4a33-8750-baffe7f3bcf9/2

Current vs. Expected behavior

Context

~/app/products/[id]/page.tsx { next: { revalidate: 60 } }

Current

while user first visits the URL /products/2, nextjs fetches the data from api server and cache the data which we can see from ~/.next/cache/fetch-cache/5825d44a41dd7ba5ba030df9c6bc168b287e287cca7fd1915d2088fed662a5e3. Then I deleted the data from the API end. Now it still getting from cache and not revalidating after 60secs.

Expected

It should return 404 page instead of 200.

Provide environment information

Operating System:
  Platform: linux
  Arch: x64
  Version: #1 SMP PREEMPT_DYNAMIC Sun Aug  6 20:05:33 UTC 2023
  Available memory (MB): 4102
  Available CPU cores: 2
Binaries:
  Node: 20.11.0
  npm: 10.2.4
  Yarn: 1.22.19
  pnpm: 8.15.1
Relevant Packages:
  next: 14.2.0-canary.2 // Latest available version is detected (14.2.0-canary.2).
  eslint-config-next: 14.1.0
  react: 18.2.0
  react-dom: 18.2.0
  typescript: 5.3.3
Next.js Config:
  output: N/A

Which area(s) are affected? (Select all that apply)

App Router, Data fetching (gS(S)P, getInitialProps)

Which stage(s) are affected? (Select all that apply)

next start (local), Other (Deployed)

Additional context

I tested against latest stable and latest canary

[DanStuartDept]

This is also bugging us too! We are deleting pages in the CMS but they are still available in Next.js.

We even have a webhook that gets called when pages are deleted in our backend, that calls an on-demand revalidation route in Next.js. But then we encountered a separate problem: our api has its own cache control, so when the next.js revalidate the path, our API still gives page data.

So now we have to purge our API's CDN cache when pages are deleted, then call our revalidation route! What a mess!

[goguda]

Noticing this as well for us :(

[goguda]

Thankfully we have control over our API, so for now we've added a query parameter to the API endpoint status200OnNotFound=true, where we send back a 200 response with an empty body instead of a 404. The 200 response code seems to get Next.js to revalidate again.

Very hacky, I don't like it, but it works for us for now. Although this should get looked at since 404 from an API should be considered a valid response that triggers revalidation, and certainly not everyone has the ability to modify the response code of the API they're using.

I might poke around the source code later to see if I can figure out what's causing this.

[florianliebig]

Exactly same here. Debugged with proxy and there is even a HTTP call to the endpoint returning 404. Logging the fetch itself then seems to return the stale data from the fetch cache (e.g. cache hit 200). Our api endpoint cache headers are "public, stale-while-revalidate=5, max-age=10" using eTags

Tested in 14.2.11

Also reproduceable in next 15 RC (15.0.0-rc0)

[florianliebig]

Traced it down more or less:

https://github.com/vercel/next.js/blob/6d7ced47babace09f5ab1cf5a43ba26d88d984ac/packages/next/src/server/lib/patch-fetch.ts#L601

I think for 404 this should delete / revalidate the cache entry somehow or force it not to take it from cache. Because it just takes the existing one in that case.

Seems like error here is only logged, not handled somehow: https://github.com/vercel/next.js/blob/6d7ced47babace09f5ab1cf5a43ba26d88d984ac/packages/next/src/server/lib/patch-fetch.ts#L628 as data is stale

[MVaik]

Still reproducible in 15.3.1.

[dealer-solutions-gene]

I'm here because a valid 404/410 gone is being ignored and an invalid cache is being reused.