Node 18+: make `nock` work native `fetch`

[johnb8005]

Please avoid duplicates

• [X] I checked all open feature requests and none of them matched my request.

Context

Since node v18, fetch is now natively available and one does not require to install node-fetch or the like anymore. I have been using the native fetch quite successfully however I can't use it with nock

Alternatives

No response

If the feature request is accepted, would you be willing to submit a PR?

• [ ] yes

[mastermatt]

Dup of https://github.com/nock/nock/issues/2183 and https://github.com/nock/nock/issues/2336

For now, set --no-experiemental-fetch flag

[Lewiscowles1986]

TBH neither of those directly mention Nock. Would it be possible to add fetch, node18+, and undici as labels?

I Think it might help prevent the need to mark things as duplicate.

[dzek69]

If upgrading nock to work with native fetch requires some time then at least add something about that to README.

Common issues mentions axios for example and it should mention fetch as its popularity will probably only grow.

[mattccrampton]

Do we have any updates to this? Would love to keep using nock but this is breaking our flow now.

[gr2m]

If upgrading nock to work with native fetch requires some time then at least add something about that to README.

That's a good idea. Thoughts @mastermatt? If someone could get a pull request started that would be great.

[mastermatt]

It should absolutely be documented in the Readme.

[gr2m]

For now we can recommend to set the --no-experiemental-fetch flag to make it work again.

For what it's worth, my net interceptor library (https://github.com/gr2m/node-net-interceptor) does intercept native fetch as it intercepts at the net/tls level (while nock is intercepting at the http module level which fetch does not use), so there is hope.

If anyone would like to dig into https://github.com/gr2m/node-net-interceptor and native fetch and see how to successfully intercept a simple GET request, that would be a great start.

Here is a starting point. Intercepting works, but I haven't digged into how the response needs to look like for it to successfully mock a full request lifecycle for the native fetch

import netInterceptor from "@gr2m/net-interceptor";

netInterceptor.start();
netInterceptor.on("connect", (socket, options, bypass) => {
  console.log("intercepted!");
  // bypass();
});

netInterceptor.on("connection", (socket) => {
  socket.write(
    `HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Content-Encoding: UTF-8
Accept-Ranges: bytes
Connection: keep-alive

works`
  );

  socket.end();
});

const response = await fetch("http://example.com");
console.log(await response.text());

Update: see working example below.

[dzek69]

For now we can recommend to set the --no-experiemental-fetch flag to make it work again.

This could only help if one is not using native fetch by desire. Disabling native fetch just makes my code not working at all :)

[gr2m]

good point. I mostly work in projects where node-fetch is used which uses the native fetch when available and falls back to a custom implementation.

I know it's not ideal, but as a temporary workaround, could you for testing use node-fetch and make it the global fetch method as described here? https://github.com/node-fetch/node-fetch#providing-global-access

[wujekbogdan]

The best solution I found so far is the msw library. It works with any HTTP library, including the native fetch. Here's what I came up with:

import { SetupServer, setupServer } from 'msw/node';
import { rest } from 'msw';
import { myApiClient } from './my-api-client';

type ServerOptions = Parameters<typeof setupServer>;

const withRequestInterception =
  (handlers: ServerOptions, test: (server: SetupServer) => any) => async () => {
    const server = setupServer(...handlers);
    server.listen();

    return Promise.resolve(test(server)).finally(() => {
      server.resetHandlers();
      server.close();
    });
  };

describe('myApiClient', () => {
  it(
    'should work!',
    withRequestInterception(
      [
        rest.get('https://my-mocked.url', (req, res, ctx) =>
          res(ctx.status(200, 'Mocked status'))
        ),
      ],
      async () => {
        const response = await myApiClient('https://my-mocked.url');
        expect(response.status).toEqual(200);
        expect(response.statusText).toEqual('Mocked status');
      }
    )
  );
});

[RichardJECooke]

Please state that nock does not work with fetch right at the top of your readme.md. Even the most basic nock example fails with the most common way of calling sites - fetch. I've wasted an hour before I realised this expected feature doesn't work:

async function fetchExample() {
  nock("http://example.com").get("/").reply(200, "Mocked response");
  try {
    const response = await fetch("http://example.com");
    const data = await response.text();
    console.log(data);
  } catch (error) {
    console.error("An error occurred during the fetch request:", error);
  }
}

fetchExample();

[gr2m]

I added a warning in dd15ba597f1912b3d2ec4249244a69d464e061e3

[isaaxite]

Please state that nock does not work with fetch right at the top of your readme.md. Even the most basic nock example fails with the most common way of calling sites - fetch. I've wasted an hour before I realised this expected feature doesn't work:

async function fetchExample() {
  nock("http://example.com").get("/").reply(200, "Mocked response");
  try {
    const response = await fetch("http://example.com");
    const data = await response.text();
    console.log(data);
  } catch (error) {
    console.error("An error occurred during the fetch request:", error);
  }
}

fetchExample();

+1

[kddsultan]

@gr2m, are there any plans to support this in near feature? Nock is a great testing library...

[gr2m]

I'm not able to work on it by myself right now, I just don't have the time. See my comment at https://github.com/nock/nock/issues/2397#issuecomment-1568646137. I'd be happy to onboard a new co-maintainer if someone wants to take this on

[kddsultan]

I'm not able to work on it by myself right now, I just don't have the time. See my comment at #2397 (comment). I'd be happy to onboard a new co-maintainer if someone wants to take this on

OK. Let me at least try... How can we do the onboarding?

[gr2m]

Please do the work first, I'll make sure you are not blocked. If it works out and you'd like to help maintain the project moving forward, I'd be happy to onboard you

[SDBowen]

Expanding on the example from #2397 (comment). This results in a successfully mocked response:

import netInterceptor from "@gr2m/net-interceptor";

netInterceptor.start();
netInterceptor.on("connect", (socket, options, bypass) => {
  console.log("intercepted!");
  // bypass();
});

netInterceptor.on("connection", (socket) => {
  const data = "hello world";
  const contentLength = Buffer.byteLength(data);

  socket.write(
    "HTTP/1.1 200 OK\r\n" +
      "Content-Type: text/html; charset=UTF-8\r\n" +
      "Content-Encoding: UTF-8\r\n" +
      "Accept-Ranges: bytes\r\n" +
      "Connection: keep-alive\r\n" +
      "Content-Length: " +
      contentLength +
      "\r\n\r\n" +
      data
  );

  socket.end();
});

const response = await fetch("http://example.com");
console.log(await response.text());

[gr2m]

confirmed, thanks a lot! The \r\n seems to make the difference 👍🏼

[dgellow]

For people looking for a way to pass --no-experimental-fetch to mocha and setup a polyfill, you can see @nikitaeverywhere's snippet found here.

package.json

{
  "scripts": {
    "test": "mocha --node-option no-experimental-fetch -r tests/_fetch-polyfill.ts tests/**/*.test.ts ...",
  },
  ...
}

tests/_fetch-polyfill.ts

// nock doesn't support native fetch, and hence we need this polyfill.

import fetch, { Headers, Request, Response } from 'node-fetch';

if (!globalThis.fetch) {
  (globalThis as any).fetch = fetch;
  (globalThis as any).Headers = Headers;
  (globalThis as any).Request = Request;
  (globalThis as any).Response = Response;
}

That works as a temporary workaround.

[lox]

After trying polly, nock, @gr2m/net-interceptor and friends, etc, seems like https://github.com/mswjs/interceptors works reliably for native fetch and node-http.

[mikicho]

@gr2m maybe can we use mswjs interceptors as the mocking mechanism and just expose the delightful Nock's API and behavior on top of it? Do you think it will reduce the required effort?

[gr2m]

That's a good idea, but it will be quite a big effort, at least if we want to transition nock itself to it. It might be more straight forward to start a separate project build on MSW's interceptors with a nock-compatible API. But then we introduce yet another mocking library 🤷🏼

I started the big effort of decomposing nock that would make this possible, that's why I created this low-level mocking and recording libraries in the first place: https://github.com/nock/nock/pull/2252. My efforts stagnated because of multiple personal impacts, but I do hope to pick them up one day and conclude them. I sure would love some help if anyone is interested. I even streamed most of my coding for that effort: https://github.com/gr2m/helpdesk#past-shows

So if you or anyone is interested, I'm happy to setup a call and chat about it. It is a big effort, but I also think it has tremendous value to transition a widely used project such as nock to a modern, maintainable architecture with as little friction to its users as possible.

[mikicho]

@gr2m I came up with a more practical solution that lets us do things more gradually. I'm trying to implement this now. At first, we need to add the missing fetch support via msw/interceptors. I THINK it should be surprisingly easy.

About the decomposition PR. IMHO, msw is probably doing well already, so we don't need to reinvent the wheel. What I like so much about Nock is its stack-like behavior, the fact you can easily insert new nocks inside the tests, and the API.

[gr2m]

sounds good, looking forward to see what you come up with 👍🏼

[kettanaito]

Hey, folks. Got this discussion referenced by @mikicho, and wanted to offer my assistance if you need anything when it comes to supporting fetch in Node, regardless if you decide to use Interceptors or not. I'm certain my knowledge will be useful in bringing the Fetch API support to Nock!

My initial thought was that you can bring only the FetchInterceptor from @mswjs/interceptors and use it as the source of requests in Nock. It won't conflict with your request interception algorithm, which is http.ClientRequest-based (fetch in Node doesn't even use ClientRequest). I'm afraid I don't know much about the inner structure of Nock to estimate how major of a rework that'd be, but I imagine there's a "given a request, lookup the mock and produce a response" logic where the FetchInterceptor would fit nicely.

[mikicho]

Thanks @kettanaito!

My initial thought was that you can...

This was mine, too. Unfortunately, it seems like the mocking parts and the "nock-related" features code are jumbled in hard-to-separate code. @gr2m started a work to unravel this.

My current path, and I'm not sure if it will work, is to convert the Fetch request (from msw) to ClientRequest. WDYT?

[kettanaito]

My current path, and I'm not sure if it will work, is to convert the Fetch request (from msw) to ClientRequest.

You mean from @mswjs/interceptors, not msw, right?

I think you should be able to represent Request as http.ClientRequest as a temporary measure. From the top of my head, most Fetch features can be represented in ClientRequest:

• Request's body stream as regular WritableStream. I think node:stream may even have a utility to convert Fetch API ReadableStream into a regular Node stream to then pipe into the ClientRequest.
• Request's Headers as OutgoingHeaders. You can use transformer functions from the headers-polyfill to transform one into another.
• Request's signal and AbortController via the native options.signal on the http.ClientRequest options object.

My concern here is that, as far as I recall, Nock doesn't only lookup the right mock for the request but will also tap into the ClientRequest to respond to it. Since the ClientRequest instance in this scenario is just a compatibility detail, responding to it won't do anything. This is where you may meet the most friction since the Interceptors expects a Fetch API Response instance to be sent from the "request" event listener to respond to an intercepted request.

I may be wrong on this, but it looks like you'd have to use the Request -> ClientRequest -> Response transformation, which is technically still possible but sounds inefficient.

interceptor.on('request', async ({ request }) => {
  // Convert the Request instance to http.ClientRequest.
  const clientRequest = toClientRequest(request)
  // Pass it to Nock to find a mock and "respond" to
  // the intermediary ClientRequest instance.
  getResponseFromNock(clientRequest)
  
  // Create a Fetch API Response that Interceptors expect.
  const response = new Response()

  // Here would be a good place to check somehow if Nock
  // has found any mocks and return early if it didn't.

  // Convert the response from Nock to the Fetch API response.
  clientRequest.on('response', (clientResponse) => {
    // write to the "response"...

    clientResponse.on('end', () => {
      request.respondWith(response)
    })
  })  
})

We actually have a utility that does IncomingMessage -> Response transformation in Interceptors. We can expose it as a part of our public API and you can utilize it here instead of implementing it by yourself.

My concern here may be irrelevant depending on how exactly Nock applies the mock response definition onto the intercepted request (maybe there's an abstract representation of that response that you can get and convert it to a Fetch API Response directly instead of hooking into the ClientRequest's response event).

[mikicho]

@kettanaito Thanks for the insights and details! I ended up doing something very similar, and it's working (woohoo). https://github.com/nock/nock/pull/2517 WDYT?

PS: for some reason, got gets stuck, but fetch and axios don't. Would you happen to have an idea of what can cause this?

[kettanaito]

I let some initial thoughts on the PR but overall looks like a great start. We will get to the got issue, I've posted some suggestions to try in the Interceptors repo.