asdf-standard icon indicating copy to clipboard operation
asdf-standard copied to clipboard

Streaming multiple arrays to file

Open kjmiller944 opened this issue 3 years ago • 1 comments

I would like to know how to stream multiple arrays to the same ASDF file. I modified the example to include two different streams in the tree dictionary. How do I write both to the same file?

from asdf import AsdfFile, Stream
import numpy as np

tree = {
    # Each "row" of data will have 128 entries.
    'my_stream_0': Stream([128], np.float64)
    'my_stream_1': Stream([128], np.float64)
}

ff = AsdfFile(tree)
with open('test.asdf', 'wb') as fd:
    ff.write_to(fd)
    # Write 100 rows of data, one row at a time.  ``write``
    # expects the raw binary bytes, not an array, so we use
    # ``tobytes()``.
    for i in range(100):
        fd.write(np.array([i] * 128, np.float64).tobytes())
        fd.write(np.array([i] * 128, np.float64).tobytes())

kjmiller944 avatar May 05 '22 23:05 kjmiller944

I'm curious about the motivations for this, but before delving into this, are you aware of other formats that support this kind of feature? That might help us understand the issues better.

perrygreenfield avatar May 06 '22 13:05 perrygreenfield