crawlee-python icon indicating copy to clipboard operation
crawlee-python copied to clipboard

bug: Request headers when using fingerprint generator or header generator are not saved in `context.request.headers`

Open Pijukatel opened this issue 9 months ago • 1 comments

Request headers are correctly present when making the request, but context is not aware of them. This is due to the fact that the request headers are set directly on Playwright in browserforge and this bypasses our common header setting approach

Make sure that regardless of how the headers are set, they are present in the context.request.headers

This is same situation as when setting headers through header generator - also missing in the context.

Pijukatel avatar Mar 06 '25 10:03 Pijukatel

Either we do some internal refactoring to make it available or we wait for Playwright support.

Feature is tagged with 1.52 https://github.com/microsoft/playwright/issues/16239 (upcoming release)

We have to wait for such functionality to make it to the Python release as well, but maybe that would give as option how to handled this on Playwright level in straight forward way. Jeust get the headers from Playwright and save them to context.

Pijukatel avatar Mar 07 '25 13:03 Pijukatel