object-store-python icon indicating copy to clipboard operation
object-store-python copied to clipboard

Add `get_opts` and `get_ranges`

Open kylebarron opened this issue 1 year ago • 0 comments

  • The underlying get_opts returns a GetResult. So ideally it would be nice to return a Python object that wraps this GetResult so that in the future we could potentially stream a result into a Python AsyncIterator. However I don't think that would ever be possible because its bytes and into_stream consume self. And you can't have a Python method that consumes self.

    So I think the best approach is to currently return bytes and potentially in the future have a separate get_opts_stream method that returns an async iterator.

I had tried with

#[pyclass(name = "GetResult")]
#[derive(Debug)]
pub struct PyGetResult {
    rt: Arc<Runtime>,
    result: Arc<Mutex<GetResult>>,
}

#[pymethods]
impl PyGetResult {
    pub fn bytes(&self) -> PyResult<Cow<[u8]>> {
        let lock = self.rt.block_on(self.result.lock());
        let obj = self
            .rt
            .block_on(lock.bytes())
            .map_err(ObjectStoreError::from)?;
        Ok(Cow::Owned(obj.to_vec()))
    }

    pub fn bytes_async<'a>(&'a self, py: Python<'a>) -> PyResult<&PyAny> {
        pyo3_asyncio::tokio::future_into_py(py, async move {
            let obj = self
                .result
                .lock()
                .await
                .bytes()
                .await
                .map_err(ObjectStoreError::from)?;
            Ok(Cow::<[u8]>::Owned(obj.to_vec()))
        })
    }
}

This is implemented on top of #6, and is expected to be reviewed after that.

kylebarron avatar Feb 08 '24 21:02 kylebarron