Integration with httpx

[vtbassmatt]

Loving this library, thank you for building it!

I'm using httpx most everywhere in my project. With a little hackery and copying what you did to gain compatibility with requests, I was able to make it work. Before I spend time preparing a PR, is this something you'd be interested in accepting?

The code is approximately two lines different from json_stream/requests/__init__.py:

import io

import json_stream
from json_stream.select_tokenizer import default_tokenizer


class IteratorStream(io.RawIOBase):
    def __init__(self, iterator):
        self.iterator = iterator  # we get an iterator, not an iterable, from httpx.stream()
        self.remainder = None

    def readinto(self, buffer):
        try:
            chunk = self.remainder or next(self.iterator)
            length = min(len(buffer), len(chunk))
            buffer[:length], self.remainder = chunk[:length], chunk[length:]
            return length
        except StopIteration:
            return 0    # indicate EOF

    def readable(self):
        return True


def _to_file(response, chunk_size):
    # httpx has an `iter_bytes`, not an `iter_content`
    return io.BufferedReader(IteratorStream(response.iter_bytes(chunk_size=chunk_size)))


def httpx_load(response, persistent=False, tokenizer=default_tokenizer, chunk_size=10240):
    return json_stream.load(_to_file(response, chunk_size), persistent=persistent, tokenizer=tokenizer)


# and to use it:
import httpx

with httpx.Client() as client:
    with client.stream('GET', 'http://httpbin.org/json') as r:
        data = httpx_load(r)
        for k, v in data.items():
            print(f"{k}: {v}")

[daggaz]

I have a few suggestions:

• the IterableStream could be moved to shared package (it doesn't matter if it's an iterable or an iterator you pass, because iter() will do the right thing anyway)
• keep the existing naming convention (function should just be load() not httpx_load())

are you able to make a PR for this?

[vtbassmatt]

Thanks!

• the IterableStream could be moved to shared package (it doesn't matter if it's an iterable or an iterator you pass, because iter() will do the right thing anyway)

Will do. I thought I tried it without removing the iter bit and it didn't work, but I may have had something else wrong with my code when I did that 😁 Either way, I'll work it out.

• keep the existing naming convention (function should just be load() not httpx_load())

💯 , this was just for demo purposes.

are you able to make a PR for this?

Yep! Didn't want to take the time if you weren't interested, but this should be pretty straightforward.

[daggaz]

@vtbassmatt this is now released under version 2.1.0

I thought you might be interested to see the changes I made to generally support iterables, and some related updates I made. You can see them in these commits.