JLD2.jl icon indicating copy to clipboard operation
JLD2.jl copied to clipboard

Published 20 hours ago • verified publisher icon JuliaIO

BufferedReader/writer for general IO?

Open grero opened this issue 4 years ago • 5 comments

I was wondering whether it would be feasible to have BufferedReader/Writer work for general io, rather than just for IOStream? The reason I'm asking is that I'm experimenting with reading parts of large jld2 files (several 10s of G) over sftp. I implemented an IO compliant interface for reading/writing and seeking, and now I would like to have BufferedReader be able to read from this IO. Looking at the source, it seems that if the io field of BufferedReader were of type IO rather than IOStream, I could use it without having to re-implement the BufferedReader interface. I'd be interesting in any thoughts on this approach. Thanks!

grero avatar Sep 17 '20 03:09 grero

Hi @grero ,

this sounds like an interesting idea that could be useful in some applications. I must admit I have not looked at that part of the code before and at this point cannot help with that but your suggestion sounds plausible.

I would recommend that you fork JLD2 and just try it out and you are very welcome to report your findings. If it works and doesn't break anything else in JLD2, we could then likely adapt the signatures to your needs.

JonasIsensee avatar Sep 17 '20 08:09 JonasIsensee

I'm close to having something that sort of works here: https://github.com/grero/RemoteFiles.jl/pull/4 Once I have something a bit more polished perhaps we can discuss whether to include the functionality in JLD2 itself. Thanks for your reply!

grero avatar Sep 17 '20 09:09 grero

Hi @grero,

I just wanted to ask whether you've made any progress on this

JonasIsensee avatar Oct 21 '20 07:10 JonasIsensee

Hey, I'm so sorry, I sort of got distracted with other projects. Regarding the tests, I was trying to figure out how best to structure them. I gather there aren't currently any tests that explicitly tests the BufferedReader? Would the best approach be to add those, or to basically run the current tests again, using a mocked up bytesavailble?

grero avatar Oct 22 '20 12:10 grero

Hi, no worries. This isn't urgent but I just didn't want the idea to be forgotten.

I think you're right. I haven't seen any explicit tests for that either. If you've had a look at the tests you'll realize that they are in a bad state in terms of organization. Anything is appreciated but I think it'll be best to just add a few small tests for your mockup bytesavailable. (Really just some tiny tests that reproduce the bug you found)

JonasIsensee avatar Oct 22 '20 13:10 JonasIsensee