[Bug]: Sitemap-fetching use case of Middleware not working as expected

[janakitti]

Steps to reproduce

This particular use of Middleware involves fetching the sitemap of an external website (siteUrl) and rewriting to a page on that site corresponding to the URL path.

When run locally, visiting the root path (localhost:3000) takes you to the root path of of the external site (siteUrl). On Netlify, the root path takes you to index.tsx, even though _middleware should run before the filesystem lookup. Also, when running locally, visiting other routes on the sitemap takes you to the corresponding page of siteUrl. On Netlify, visiting any path from the sitemap leads to a 404.

// pages/_middleware.ts
import { NextFetchEvent, NextRequest, NextResponse } from "next/server";
import xmlParser from "fast-xml-parser";

const siteUrl = "https://www.sitemaps.org";
const sitemapLink = siteUrl + `/sitemap.xml`;

interface Sitemap {
  urlset: {
    url: Array<{
      loc: string;
    }>;
  };
}

const buildSitemapTable = async () => {
  const sitemapRes = await fetch(sitemapLink);
  const sitemapText = await sitemapRes.text();
  const xml = xmlParser.parse(sitemapText) as Sitemap;
  const pathTable = new Map<string, string>();
  for (const { loc } of xml.urlset.url) {
    const path = loc.slice(siteUrl.length);
    pathTable.set(path, loc);
  }
  return pathTable;
};

export async function middleware(req: NextRequest, ev: NextFetchEvent) {
  const pathTable = await buildSitemapTable();
  if (pathTable.has(req.url)) {
    const url = pathTable.get(req.url)!;
    return NextResponse.rewrite(url);
  }
  return NextResponse.next();
}

A link to a reproduction repository

https://github.com/janakitti/next-test/tree/middleware-2

More information about your build

• [ ] I am building using the CLI
• [ ] I am building using file-based configuration (netlify.toml)

What OS are you using?

No response

Your netlify.toml file

No response

Relevant log output (or link to your logs)

log-2.txt

[ascorbic]

Hi @janakitti Thanks for the report. I'm wokring on the specifics of middleware handling this week, and I'll use your site as a test case. I'll let you know how I get on.

[ascorbic]

Hi @janakitti The problem here is that currently on Netlify, middleware is run after redirect shadowing, i.e. it looks for static files first. This will be something that we'll fix, but in the meantime you can work around it by adding a forced rewrite for the root. Try this inside public/_redirects:

/  /.netlify/builders/___netlify-odb-handler  200!

[janakitti]

Hi @janakitti The problem here is that currently on Netlify, middleware is run after redirect shadowing, i.e. it looks for static files first. This will be something that we'll fix, but in the meantime you can work around it by adding a forced rewrite for the root. Try this inside public/_redirects:

/  /.netlify/builders/___netlify-odb-handler  200!

Hi @ascorbic, I tried adding the public/_redirects file as shown above, but am still running into the same issue.

[ascorbic]

Hi @janakitti Can you confirm whether this issue is still present in the latest beta?

[janakitti]

The issue is still present on the latest beta. I also tried using the forced rewrite as you mentioned, but I'm still getting the same behaviour. https://github.com/janakitti/next-test/tree/middleware-2

[sarahetter]

@janakitti Next's middleware has changed quite a bit since this issue was created (for example, nested middleware is no longer supported, in favour of root level middleware) https://nextjs.org/docs/advanced-features/middleware

Are you still seeing an issue with Next v12.2.*?

[MarcL]

Closing due to inactivity. Please reopen if this is still an issue. Thanks!