Load psf waveforms on demand

[michaelnt]

Opening a large (700M) transient results file loads the complete file into memory, which is slow and uses lots of memory.

Would it be possible load the trace names and other meta data on open but then only load the waveforms on demand.

[henjo]

Actually I have been working on an improved psf reader which use memory mapped io to improve performance and memory consumption. I just found it in a git stash and I have now merged it with the latest changes on master and created a new branch for it. The new branch is called psfmmap.

I don't remember what state it is in, so it might not work at all. The reason I abandoned was that I chose to spend more time on libpsf instead which has far better performance than the mmap version of psf.py.

/Henrik

[michaelnt]

Thanks, i'll tale a look at that when I get back from my holiday.

Do you think its better to focus on libpsf rather than the pure python implementation? On Sep 18, 2011 7:05 a.m., "Henrik Johansson" < [email protected]> wrote:

Actually I have been working on an improved psf reader which use memory mapped io to improve performance and memory consumption. I just found it in a git

stash and I have now merged it with the latest changes on master and created a new branch for it. The new branch is called psfmmap.

I don't remember what state it is in, so it might not work at all. The reason I abandoned was that I chose to spend more time on libpsf instead which has far better performance than the mmap version of psf.py.

/Henrik

Reply to this email directly or view it on GitHub: https://github.com/henjo/pycircuit/issues/13#issuecomment-2125447

[henjo]

Yes, I think it's probably better to focus on libpsf because of its better performance. The disadvantage of libpsf is that it's not pure Python and right now more complicated to install.