apify
AddRequest should support binary Buffer payload
Which package is the feature request for? If unsure which one to select, leave blank
@crawlee/core
Feature
I'm trying to submit HTTP POST requests and add them directly to the queue, but when adding a request like this: let response = await requestQueue.addRequest({ url: interceptedUrl, method: method, payload: postData, ...
The payload parameter accepts only string postData, and not a Buffer, while in multipart file upload scenarios, the uploaded file can be a binary one, requiring the payload to be either a string or a Buffer.
Motivation
I'm using Crawlee to write a web spider, including file upload and it is not supported.
Ideal solution or implementation, and any additional constraints
Just change the type of payload to string, undefined or Buffer, and support Buffer along the way. At least in Playwright who also supports it.
Alternative solutions or implementations
No response
Other context
No response
Note that implementing this wouldn't be as easy as widening the type annotation, as the RequestQueue stores the requests serialized in JSON files. The only idea I have here is to base64-encode the payload during the serialization, which might in turn result in (de-)serializing large files, which is a potential performance issue.
Despite these problems, I can definitely imagine the use cases for this. I'll pass this to our internal list of feature candidates for the next major version of Crawlee.
