Kyle Barron
Kyle Barron
Often my midnight brain is very productive 😛 > Yeah, may as well - the python tendency toward methods that take basically anything (as in lists of dicts OR dicts...
In https://github.com/kylebarron/arro3/pull/204 I figured out a way to have Python-backed Arrow `Array` objects. So we can interpret a numpy numeric array as a non-null Arrow array without any memory copies....
And in https://github.com/kylebarron/arro3/pull/208 I automatically handle multidimensional numpy arrays as input, converting them (zero-copy) to nested fixed size lists
Latest update removes a **lot** of code by reusing the `PyArrowBuffer` from `pyo3_arrow::buffer`. See https://github.com/kylebarron/arro3/pull/241 and https://github.com/kylebarron/arro3/pull/242. Work still to do: - Update to latest git version from https://github.com/kylebarron/arro3/pull/242 to...
> While looking at [from_url](https://github.com/developmentseed/obstore/blob/95e586fc6c8c9425fb682fdb9aedb16e0b0a2935/obstore/python/obstore/store.py#L656-L765) I did realize there is likely another deeper problem associated with our transition from `fsspec` to `obstore`. There are likely a large number of users...
From https://github.com/fsspec/filesystem_spec/issues/1446#issuecomment-1843698132 > The trouble that we are facing, is that not all implementations need to implement all functions. For instance, some filesystem might be read-only or (like HTTP) not...
(If obspec ever got traction and stabilized, we could swap it in to the zarr-obstore integration in https://github.com/zarr-developers/zarr-python/pull/1661 without any backwards incompatibilities)
> I think this is the right call. That functionality wasn't in obstore when I started ManifestStore IIRC but now that it's released we should definitely use it. FWIW we...
This now compiles on 3.8! I still need to figure out how to exclude this code from trying to compile on earlier GDAL versions, and I still need to test...
Yeah I was confused because I thought I was doing the same process as we have for reading