hdf5storage icon indicating copy to clipboard operation
hdf5storage copied to clipboard

Add marshaller/s for scipy.sparse matrices

Open frejanordsiek opened this issue 8 years ago • 4 comments

Due to fixing Issue #49 , this can now be done without adding scipy as a package dependency/requirement.

Sparse matrices are useful and are additionally one of the ndarray/matrix types built into the MAT v7.3 format without resorting to classes that this package still does not support.

frejanordsiek avatar Sep 14 '16 00:09 frejanordsiek

I'm trying to load a struct array that contains a bunch of sparse matrices. It's taking a very long time, probably because it is converting each of them to a dense matrix? Would it be worth it to try to dig into this and open a PR or is this a tricky enhancement? Thanks for your effort on this package - it looks real nice!

ahwillia avatar Jan 06 '17 22:01 ahwillia

I can see it being tricky given that the Marshallers are in a bit of flux being that they are mid overhaul, but then again it may just be tricky to me and not a new set of eyes (a PR would be very welcome). I can look more in depth this week. Do know that a Marshaller for sparse matrices is one of the main priorities in order to work with the vast majority of v7.3 MAT files.

frejanordsiek avatar Feb 06 '17 23:02 frejanordsiek

Any update on this? I would still love to have this feature... (I just re-encountered this problem today for a new dataset)

ahwillia avatar Apr 26 '18 00:04 ahwillia

I haven't worked on it in a while. I actually ran into a major problem that I have been trying to think about that is in the way. That is, str(some_type) produces different output on different python version. For str(float), it is producing "<class 'float'>" on 3.5.3 and "<type 'float'>". This makes the current way that having marshallers can lazy load other modules brittle. I tried digging around more in the inspect module but still haven't found anything that works that well yet.

frejanordsiek avatar May 12 '18 09:05 frejanordsiek