conduit
conduit copied to clipboard
support sub-tree fetch for conduit_bin protocol
- add support to parse schema, and then only load the proper subset of the tree from the binary file
To implement this generally, we will need something like the following:
load(ifstream &ifs, const Schema &s, const std::string &sub_path, Node &out)
In this method given s, we can fetch the sub-path schema and call a more general function:
load(ifstream &ifs, const Schema &s, Node &out)
We will assume the schema offsets are interpreted as byte offset into the bin file.
Ideally, we want a single I/O request.
To do this, we need the minimum offset in the passed schema.
We don't have a method that gives us this currently.
spanned_bytes() gives us the total spanned bytes, which includes the offset.
I believe what we need is to seek the bin file to the minimum offset, and then read spanned_bytes() - minimum offset bytes.
Finally, when constructing the node from this chunk of data, we will need to shift the schema used to subtract the minimum offset.
To do this, we need:
- a helper in
Schemato get the minimum offset:Schema::minimum_offset(). - a helper in
Schemato shift the offset of an entire schema treeSchema::shift_offset(index_t )
@bryujin What do you think about adding the above methods to Schema?
If we don't feel comfortable doing this, I can explore implementing them directly in the new load and see how things work.
@cyrush Schema::shift_offset(index_t ) would actually apply the offset to the entire Schema and change it? Is the plan to shift it, read the file and then shift it back? Is that really necessary? I'm probably OK with the Schema::minimum_offset() method.
@bryujin The plan would be to read the proper chunk of data, then make a shift-ed copy of the schema so it can be used to describe the chunk.
(the output node would get the chunk of data with the shifted schema)
I think its probably best to prototype this w/o changing the public interface of Schema, then see what we are open to supporting.
OK, that would help.
An idea like #286 could help