Francesc Alted

icon indicating a website link https://ironarray.io

icon company @FrancescAlted icon location Castelló de la Plana, Spain icon location CEO of https://ironarray.io Project leader of the Blosc project. Compression and efficient data I/O to efficient storage and computation.

Results 320 comments of Francesc Alted

Bcolz with c-blosc 1.7.0 and bitshuffle mangles data

We should resolve it of course :) PR's appreciated as always. 2015-08-08 23:37 GMT+02:00 Valentin Haenel [email protected]: > What should we do with this issue? I saw more emails about...

Reading selected columns into memory from an on-disc ctable

Yes, @gogoengie is probably right in that the `lastchunk` buffer is the responsible for all the allocated memory. A possible solution would be to make `lastchunk` lazy and create it...

index / slice return carray option

A self-contained example is always best to show your point.

index / slice return carray option

I would say that would be a useful addition, yes. A sensible parameter for the carray/ctable constructors could be `out_flavor` as it is already implemented in `bcolz.eval` (http://bcolz.blosc.org/reference.html?highlight=eval#bcolz.eval). @vapemaster would...

index / slice return carray option

Well, Cython is very close in syntax to Python, so my recommendation is that you do a quick read at this first: http://docs.cython.org/src/tutorial/cython_tutorial.html Then it would just a matter of...

introduce chunk._getitem_r(...), a thread-safe version of chunk._getitem(...)

This looks good, thanks. Unfortunately c-blosc only supports being called effectively from multi-threaded operations in version >= 1.5, and with lesser versions, c-blosc makes sure that a lock is put...

introduce chunk._getitem_r(...), a thread-safe version of chunk._getitem(...)

Yes, @ARF1 suggestion of using prange is a good one. And he is right, Blosc used to pose a lock itself (similar to GIL), but with c-blosc >= 1.5 there...

introduce chunk._getitem_r(...), a thread-safe version of chunk._getitem(...)

I don't think that prange allows you to ensure an order, but if you want that, then use regular range right?

Fancy indexing does not work with multidimensional carrays

@alimanfoo is correct in that zarr is much better positioned for handling multidimensional arrays than bcolz is. Also, bcolz is mainly in maintenance mode (it has been underfunded for too...

Problem with dtype in bcolz.carray constructor.

Yep. Perhaps forcing a view of the numpy array should be relatively easy to add. Interested in providing a PR? If not easy, perhaps fixing the documentation would be better.