osmpbf icon indicating copy to clipboard operation
osmpbf copied to clipboard

Support PBF writing

Open nyurik opened this issue 3 years ago • 4 comments

I would like to implement PBF write support. My initial thoughts are to have two structs:

  • BlockBuilder struct equivalent to PrimitiveBlock, contains shared state and a list of GroupBuilders
  • GroupBuilder struct contains values that will go into PrimitiveGroup

The BlockBuilder owns zero or more GroupBuilder instances, possibly of different types, and has methods to add node/way/rels to them. When adding a single feature the block builder would also track the bounding box of all features, as well as the shared tags table. Once the block is full, it self-compresses into a stream. The actual interfaces are TBD.

nyurik avatar Feb 17 '22 20:02 nyurik

I like that idea!

And builder structs seem like a good way to avoid having to allocate a node with all of its tags, only to be destroyed again so that it fits the PBF memory model. So I guess there should be a NodeBuilder too with an add_tag method that mutates the current stringtable.

First I thought that GroupBuilder can be abstracted away and a BlockBuilder would just have add_node, add_way, etc. and take care of putting it into the right group. But then the order of the elements in reading and writing could be different. So for a low-level interface I think it's good to have an explicit GroupBuilder.

b-r-u avatar Feb 19 '22 23:02 b-r-u

Hey nyurik,

I wanted to let you know, that I started pushing commits to this branch:

https://github.com/b-r-u/osmpbf/tree/writer

Writing blobs and blocks is implemented, but there is still a lot of functionality (and ergonomics) missing.

b-r-u avatar Jul 02 '22 18:07 b-r-u

Thanks!! I have been hacking locally on it for a long time on and off, and will push my work in progress to my fork. What do you think about the delta crate? I think it will make the code simpler to read, and might help with writing too

nyurik avatar Jul 02 '22 19:07 nyurik

@b-r-u what's the status of the rewrite? I have pushed my own work in that direction, but not sure if that's how you want to proceed. See https://github.com/nyurik/osmpbf/tree/write

nyurik avatar Sep 07 '22 22:09 nyurik