go-capnp
go-capnp copied to clipboard
Add Data list creation method to UInt8List fields
It would be useful for structs to have New* methods for fields with type Data, which would take a length and return a []byte, which is a view into their segment.
Background: as I was playing with the initial implementation of docker-spk, I noticed a limitation for that kind of application: I need to keep a message that is being constructed entirely in-memory. For docker-spk this is OK -- if an app developer is supplying a docker image that is too big to comfortably fit in memory, it is also too big to distribute to users. But for the more general case, I had been thinking it would be possible to implement an Arena that allocated out of a memory-mapped file. Unfortunately, Because there's no way afaik to to initialize a Data field allocated inside of the message and write directly into it, I would still have to read whole files into memory to call Set() on the field, copying the byte slice.
An oblique way to do this right now is by using data.ToPtr().Data()
, which returns the underlying byte slice. It would be good to have these view methods placed directly on UInt8List
, so I'll keep this issue open. I've edited the title to match.
Hm, I'm not sure I fully understand your suggested workaround. Suppose I have a schema:
struct Foo {
someField @0 :Data;
}
...and I want to allocate someField
from the arena. I could do:
list, _ := NewUInt8List(foo.Struct.Segment(), 1<<30)
I can get at the raw bytes as you describe. But how do I actually attach the list to foo
as the someField
field? It can be done via foo.Struct.SetPtr(?, list.ToPtr())
, but I can't figure out a way that doesn't require me to know the offset into the pointer section for that field and doesn't end up allocating more than the one slice, already inside the message.
My notion was that we'd generate a function:
func (s Foo) NewSomeField(length int32) ([]byte, error)
Which would do the above and then return the slice.
I see what you're saying now and updated the title to reflect. Yes, I think what you suggest is the right way to approach this. Happy to review a PR for this.