rod
rod copied to clipboard
Support connecting to BrightData Scraping Browsers
I'd like to use scalable browser infrastructure services, such as the BrightData Scraping Browser, which integrate well with Puppeteer. But, I have encountered some issues when trying to use these services with go-rod. I would like to request the following enhancements to improve compatibility:
1. WebSocket Authentication:
The connection to these services’ WebSocket requires authentication (e.g., wss://user:pass@host:9222). However, go-rod does not currently send the necessary authentication headers, which I think are not defined on any WebSocket standard.
Through my research, I discovered that authentication is performed using Basic tokens. I have implemented a working solution to inject the Authorization header. However, I am unsure if this is the optimal place to inject it. If this solution aligns with the project's direction, I am willing to submit a PR with my implementation.
2. Less Restrictive WebSocket Response Handling:
The services sometimes send responses that deviate from the expected go-rod Response structure, causing panics due to unmarshalling failures. Specifically, the Error struct expects an integer Code, but some responses include a string (e.g., "navigate_limit").
The Response structure is defined this way:
type Response struct {
ID int `json:"id"`
Result json.RawMessage `json:"result,omitempty"`
Error *Error `json:"error,omitempty"`
}
type Error struct {
Code int `json:"code"`
Message string `json:"message"`
Data string `json:"data"`
}
And Brightdata is sending to me struct that panic, like this
{
"id": 27,
"sessionId": "BRD_461626884EEF95862B6188C2DBB766D1",
"error": {
"message": "Page.navigate limit reached",
"code": "navigate_limit" // This is expected as an Int.
},
"duration": 1.2261550000112038
}
To make go-rod more compatible, it may be necessary to relax the strictness of the standard for the Error struct. I would appreciate guidance on the best approach to achieve this flexibility. If you agree with this, I am happy to work on the implementation with a bit of guidance.