amzn
Support stream reading/parsing
Given an Ion file of a series of records (e.g. an object in S3), we'd like to be able to stream read/parse this file, ultimately creating an async (Tokio) Stream of Ion records. Due to memory constraints (Lambda, Fargate, etc), we cannot read the entire file into memory and then iterate through the records.
To clarify a bit: the Reader is already capable of reading a file without pulling the entire thing into memory. However, if the input source it is reading from contains incomplete data, the Reader would either block (e.g. waiting on a socket) or fail (e.g. at the end of a Vec<u8>).
We'd like to offer an API that allows the Reader to be resilient to such a use case. For example, calls to next() or read_string() might return an IonError::InsufficientData (or IonError::WouldBlock?), indicating that the user should try again once more data is available.
We now have non-blocking text and binary readers, but they can't be constructed with the ReaderBuilder API yet. Once that's done (see #484), we can close this issue.