request-boost
request-boost copied to clipboard
singhsidhukuldeep
Added param response_process (callback) to process response object
Requests to some urls return binary data which cannot be decoded (eg. image data).
In these cases, param parse_bytes can be set to False.
@singhsidhukuldeep Please review my pull request, and let me know if something can be improved. Thanks in advance 🙂
@singhsidhukuldeep Resolved the conflicts - they were because of the commits in main branch since February.
I don't think there is any need for this, you can set parse_json as False
This will give you the raw response and later you can process it as you want
you can set parse_json as False
@singhsidhukuldeep Setting parse_json to False does not help, because the request bytes are always decoded irrespective of parse_json. In the following code from __init__.py:
data = response.read()
encoding = response.info().get_content_charset("utf-8")
decoded_data = data.decode(encoding) # THE BYTES ARE ALWAYS DECODED TO STRING FIRST
self.results[loc] = (
json.loads(
decoded_data) if self.parse_json else decoded_data
)
self.queue.task_done()
Decoding the bytes is always done (data.decode(encoding)). This is not desirable for binary files like images, videos, etc.
Makes sense, let's make it parse JSON or parse nothing and return the Request object itself
Makes sense, let's make it parse JSON or parse nothing and return the Request object itself
That sounds good. But this will break compatibility for existing users - when parse_json=False, they would expect the response text, but would instead get the response object.
@sohang3112 need to rethink this; people will start adding specific post processes for their own use cases
We need to keep it light-weight and extendable
one idea is to allow the user to pass their own post-processing logic as a callable function (post_process_func), We will pass the request object through that, this can overwrite parse_json when not None
@singhsidhukuldeep Yes this sounds like a good idea - if a callable is passed, then pass it the request object and use its result. Otherwise do existing processing.
if a callable is passed, then pass it the request object and use its result. Otherwise do existing processing.
@singhsidhukuldeep Should I modify my PR to do this??
Yes please
On Tue, 12 Sept 2023, 4:11 pm Sohang Chopra, @.***> wrote:
if a callable is passed, then pass it the request object and use its result. Otherwise do existing processing.
@singhsidhukuldeep https://github.com/singhsidhukuldeep Should I modify my PR to do this??
— Reply to this email directly, view it on GitHub https://github.com/singhsidhukuldeep/request-boost/pull/10#issuecomment-1715480680, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBEA4PXPSSAGTLFVJWJOLX2A36PANCNFSM6AAAAAAU4Q2YRI . You are receiving this because you were mentioned.Message ID: @.***>
@singhsidhukuldeep I updated the PR to accept optional callback response_process which is used like this:
response_ok, data = response_process(response)
data is used if response_ok is true, otherwise request will be retried.
@singhsidhukuldeep Please review and merge my PR. Let me know if any further changes are needed.
@singhsidhukuldeep Please merge my PR. Let me know if any further changes are required from my side.
@singhsidhukuldeep Did you have a chance to review the PR yet? As I mentioned previously, I had modified the PR to include the changes we discussed.
@singhsidhukuldeep Hi. Did you have a look at the PR yet? As I mentioned before, I modified the PR to include the changes we discussed.
Hi @sohang3112 I am occupied with few things, will close this shortly