[BUG] [Chromium/WebKit] Request object does not contain postData for file/blob

[Meir017]

Context:

System:

• OS: Windows 10 10.0.19043

Binaries:

• Node: 12.13.1 - C:\Program Files\nodejs\node.EXE
• Yarn: 1.19.1 - ~\AppData\Roaming\npm\yarn.CMD
• npm: 6.14.2 - C:\Program Files\nodejs\npm.CMD

Languages:

• Bash: 4.4.20 - C:\WINDOWS\system32\bash.EXE

npmPackages:

• playwright: ^1.10.0 => 1.10.0 / tip-of-tree

Code Snippet

Help us help you! Put down a short code snippet that illustrates your bug and that we can run and debug locally. For example:

const playwright = require('.');

(async () => {
  const browser = await playwright.webkit.launch();
  const page = await browser.newPage();

  await page.goto('http://example.com');

  const requestPromise = page.waitForRequest(() => true);
  const [request] = await Promise.all([
    page.waitForRequest("**/*"),
    page.evaluate(() => {
      var file = new File(['file-contents'], 'filename.txt');

      fetch('/data', {
        method: 'POST',
        headers: {
          'content-type': 'application/octet-stream'
        },
        body: file
      });
    })
  ])
  console.info('request.method()', request.method());
  console.info('request.url()', request.url());
  console.info('request.headers()', request.headers());
  console.info('request.postData()', request.postData());
})();

Describe the bug

the request.postData() contains null in Chromium and WebKit. In Firefox it's filled.

[Meir017]

@mxschmitt is there a workaround for this? I was thinking of using something like

const cdp = await context.newCDPSession(page);
let requestId;
cdp.on('Network.requestWillBeSent', req => {
    requestId = req.requestId;
});
const result = await cdp.send('Network.getRequestPostData', { requestId: requestId });
console.info('result.postData', result.postData);

but this doesn't work for me

[JoelEinbinder]

Thanks for filing! Short of using a proxy, I don't think there are any good workarounds at this time. These look like bugs in chromium and our webkit. It might take some time to fix, but I'm on the case!

[Meir017]

not sure if this would affect the fix but this reproduces for both XMLHttpRequest and fetch

[JoelEinbinder]

Yep. It's apparently a known bug in the chrome DevTools protocol that we lack post data for mulitpart/files.

[nanaceba]

https://bugs.chromium.org/p/chromium/issues/detail?id=1019710

[karamfilovs]

Facing the same issue. The postData is always null but in network tab its clear that the data exists. In my case it is normal json payload post request. I am using the java binding at the moment.

[sch0057k]

Same for the JS version with Chromium and contentType application/json on PUT and POST. Works with Firefox for the same requests.

[yannsartori]

I know that the cause is somewhat known, but I encountered this issue using the Python binding as well.

[humbienri]

The link above seems to indicate this is a problem in Chromium, not Playwright. However, if you follow the discussion details it does not seem like the Chromium folks "are on it".

[dgozman]

See also https://github.com/microsoft/playwright/issues/9648#issuecomment-953400441 for more cases where Chromium lacks post data.

[dgozman]

See https://github.com/microsoft/playwright/issues/15853 for another repro.

[purepear]

I also hit similar issue because of the CDP bug in Chromium. I'm using Ky which utilises request.clone() for immutability and hooks. Are you aware if there's any way around this issue while still using request.clone()(without abandoning Ky)?

Here's a test: npx playwright test --browser all

import { test, expect } from '@playwright/test'

test('postDataJSON() works with cloned request', async ({ page }) => {
  const [request] = await Promise.all([
    page.waitForRequest('**/*'),
    page.evaluate(async () => {
      const request = new Request('https://test-post-data-json.com', {
        method: 'POST',
        headers: { 'content-type': 'application/json' },
        body: JSON.stringify({ test: 'test' })
      })
      
      // Chromium, Firefox and Webkit work with the original request: fetch(request).
      // Firefox and Webkit also work with cloned request but Chromium fails:
      fetch(request.clone())
    })
  ])
  
  expect(request.postDataJSON()).toEqual({ test: 'test' })
})

[piercefreeman]

Seeing the same issue with ping requests. My workaround was to rewrite my handlers to use a nodejs-based proxy. I'm using http-mitm-proxy since it can coexist in a node.js codebase with the playwright crawler. Route interceptions themselves port more-or-less 1:1 into the proxy callback functions for requests and responses.

The only drawback is this proxy is certainly slower than .route based request interception, since the proxy has to do the additional legwork of issuing a local certificate for ssl connections and spawning two roundtrip network requests.

Seems like there's nothing to do on the playwright side until the chromium ticket is merged upstream. Hoping for a fast solution however since this issue affects a wide swath of request types.

[cameronbraid]

A bit of a hack for ky users, you can use a hook to prevent ky from cloning responses :

    hooks : {
      beforeRequest: [
        (request: Request) => {
          request.clone = () => request
          return request
        }
      ],
    },

[romaleev]

A pity that this critical bug is still not fixed for 17 months.

It stops me from using Chromium in Playwright tests with post data requests (as data is missing on recipient side). Firefox works well.

Any idea how to ping Chromium folks to finally fix this issue?

[MikeFear]

It seems that they are finally working on it: https://bugs.chromium.org/p/chromium/issues/detail?id=1019710#c23 https://chromium.googlesource.com/chromium/src/+/2f4d40dfc9fbd14e0f105551fce94f428566a796

[romaleev]

Seems to be fixed, at least my tests are passing now using 'chromium' (previously passed with 'firefox' only)

[halilemreozen]

I can confirm the request.clone() still causes this issue with playwright request.postData and request.postDataJson on

• Chromium - Version 112.0.5615.29 (Developer Build) (x86_64)
• Google Chrome - Version 112.0.5615.39 beta
• Google Chrome - Version 114.0.5677.0 (Official Build) canary (x86_64)

While firefox and webkit can pass the test shared above ( https://github.com/microsoft/playwright/issues/6479#issuecomment-1193105330 )

With the

• [email protected]
• @playwright/[email protected]

[groinder]

I can confirm that the issue still exists for me on the latest stable versions of Chromium, just as halilemreozen mentioned.

I think this chromium bug is more adequate: https://crbug.com/1058404. Please "Star" it so it gets more traction.

[apeltz]

@.mxschmitt is there a workaround for this? I was thinking of using something like

const cdp = await context.newCDPSession(page);
let requestId;
cdp.on('Network.requestWillBeSent', req => {
    requestId = req.requestId;
});
const result = await cdp.send('Network.getRequestPostData', { requestId: requestId });
console.info('result.postData', result.postData);

but this doesn't work for me

@Meir017 I think you were painfully close to having a workaround identified 2 years ago! I think you were just missing the cdp call to enable the Network domain? I was able to successfully get postData using cdp with something like this:

import { Protocol } from "playwright-core/types/protocol";

const client = await page.context().newCDPSession(page);

// https://chromedevtools.github.io/devtools-protocol/tot/Network/#method-enable
await client.send("Network.enable");

const handleNetworkRequest = async ({ requestId }: Protocol.Network.requestWillBeSentPayload) => {
  const res = await client.send("Network.getRequestPostData", { requestId });
  console.log("postData", res.postData);
};

// https://chromedevtools.github.io/devtools-protocol/tot/Network/#event-requestWillBeSent
client.on("Network.requestWillBeSent", handleNetworkRequest);

// remove listener when no longer needed; avoids errors from attempting to get postData
// for requests in a browser that has been closed
client.off("Network.requestWillBeSent", handleNetworkRequest);

Notably, the request passed to the Network.requestWillBeSent callback didn't have postData (hasPostData was true but postData was undefined) even when providing large byte values for maxPostDataSize, maxTotalBufferSize and maxResourceBufferSize to the Network.enable call.

[Meir017]

@apeltz I'm rather hesitant to using the CDP directly as it's only for chromium, if this is supported now then I think playwright should be able to support this (and fix the failing tests)

[apeltz]

@Meir017 I totally agree with preferring the playwright api(s) 👍. The CDP use is just a workaround for now.

[Horsty80]

I'm facing this issue, but not on all my playwright project.

I've 2 projects with e2e, both use chromium, and i mock request to get postDataJson() On the first project is working fine, i get the value that i need. On the second project i get undifined, like the bug you describe here.

My workaround for the second project is to use firefox and it's working fine.

But i don't understand why is working on the first and not on the second project.... The depenedencies are the same, the mock request to get postDataJson is the same.

I'm surprise this issue have 2 years old !

[paradox37]

There must be some difference between your projects. From what I understood, problem is when you or libraries are using request.clone(). In that case, postData is empty and it's a Chromium bug, not playwright. But Chromium has that bug since 2020, and who knows if they'll ever fix it, so maybe PlayWright could introduce some workaround.

[ShlomoVinny]

I'm facing this issue, but not on all my playwright project.

I've 2 projects with e2e, both use chromium, and i mock request to get postDataJson() On the first project is working fine, i get the value that i need. On the second project i get undifined, like the bug you describe here.

My workaround for the second project is to use firefox and it's working fine.

But i don't understand why is working on the first and not on the second project.... The depenedencies are the same, the mock request to get postDataJson is the same.

I'm surprise this issue have 2 years old !

Exactly what Im experiencing right now. 2 project that are virtually the same process just on 2 different domains. Both domains use the multipart/form-data type with the WebKitBoundary and all that jazz. On the one I get successful intercepts and on the other one I get postData: undefined. Funny thing is, it only happens when I let Puppeteer click the submit button during the automation. If I set-up a request interceptor (basically a waitForRequest with timeout: 0) and click the submit button manually, it does get the postData correctly...

So Im not sure what the hell is going on, but it seems that people solved this issues by using firefox instead of chromium.

This is DEFINITELY a CDP issue, since i used CDP's own Fetch API to get the requests directly from chromium, instead of going through puppeteer and still it did not work. Reference: https://chromedevtools.github.io/devtools-protocol/tot/Fetch/ Heres an example in TS:

    let requestId: string | undefined;
    let postData: string | undefined;
    const session = await page.target().createCDPSession();
    await session.send("Fetch.enable", { patterns: [{ urlPattern: URL }] });
    session.once("Fetch.requestPaused", async (request: Protocol.Fetch.RequestPausedEvent) => {
      console.log(`Request paused!`);
      console.log(`Id: ${request.requestId}`);
      console.log(`Url: ${request.request.url}`);
      console.log(`Method: ${request.request.method}`);
      console.log(`postData: ${request.request.postData}`);
      requestId = request.requestId;
      postData = request.request.postData;
    });
session.send("Fetch.continueRequest", { requestId: requestId as string, postData: postData });

[shinijimmy3]

chromium bug is closed as Wontfix, https://bugs.chromium.org/p/chromium/issues/detail?id=1058404. Is there any way to route multipart/form-data requests and get not null postData? (using chrome/chromium)

[genie-youn]

There must be some difference between your projects. From what I understood, problem is when you or libraries are using request.clone(). In that case, postData is empty and it's a Chromium bug, not playwright. But Chromium has that bug since 2020, and who knows if they'll ever fix it, so maybe PlayWright could introduce some workaround.

I also encountered the exact same situation with this issue. request.postData always returned null, and the reason was that the HTTP client library I was using, which was ky, executed a clone before sending the request.

I checked that using native fetch instead of ky returned the postData with the correct payload.

It's my workaround

global.setup.ts

import { test as setup } from '@playwright/test';

setup('mock Request.prototype.clone', async ({ page }) => {
  await page.evaluate(() => {
      Request.prototype.clone = function() {
        return this;
      }
    });
});

playwright.config.ts

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  // ...
  projects: [
    {
      name: 'setup',
      testMatch: /global\\.setup\\.ts/,
    },
    {
      name: 'my test',
      use: { ...devices['Desktop Chrome'] },
      dependencies: ['setup'],
    },
  ]
});