Streaming regions over network

[chiguireitor]

I've been using this module for my project recently and have been fascinated with it's featureset.

However, thinking of use cases where networked players make edits to the terrain i wonder if there's a way to stream the blocks for a particular region? I don't see a way to serialize/deserialize VoxelBuffers from GDScript (although it shouldn't be too hard to do as it's just a byte array).

However, is there someone doing something in this regard? I'm thinking there are several features one would need to implement before going down this rabbit hole:

• Region state transfer: a way to get a "hash" or last modified time for a given region. So the client just queries "regions modified since X time" (or a data-hash where you could query if a specific region has been modified) and the engine could tell what regions need updating/adding/removing.
• Region data transfer: once the regions are know to be "dirty" a way to stream or pre-cache them locally would be interesting to allow them to be downloaded preemptively.

Am i thinking too far ahead? Should i drop user networked editability? i'm thinking i could even create an "edit-log" so the user just gets the "latest edits since time X" and the server sends the diff, but that would mean a state falling out of sync would be hard to detect.

[chiguireitor]

Thinking on the hash approach: you could have a merkle tree for the regions for fast queries of the data, so you could traverse the tree checking what nodes have changed hash till you reach the changed leaves, and then fetch them accordingly.

[Zylann]

So far there is nothing in this module to ease networking, so you're on your own for now. There is of course room to make a chunk-based synchronization system, with either an approach where everything is synced (like in Minecraft I guess) or where a delta is sent while reconstructing the unchanged parts using a generator (if any). If I were to add networking it would likely involve tinkering in C++. It might be possible in GDScript but it's going to miss a lot of possible optimizations.

[nic96]

I really like the idea of only storing and sending changes and have the generator do the rest. I might look into it this weekend and see what I can do. My skillset is quite limited so if I do create something it probably won't be very performant.

[chiguireitor]

One simple/naive way to do it is to have a modification log with timestamps/monotonically-increasing nonces, users just send their terrain modifications and they get stored in a transaction log. Users just ask modifications from a given nonce and they get them which in turn they apply to their local copy.

This assumes you have a way to sync the whole generated voxel map from a given "seed" and then apply modifications each time a new user is connected (which could be costly if the modification log is big). You could also sync the voxel data files for each new user, this however would need a mechanism to snapshot the voxel data from time to time.

[Zylann]

I had to think about a similar approach to solve something else, where there was an additional channel in voxel data called "Edits", storing exclusively user-made modifications, the rest being procedurally obtainable. Then syncing edits would be the same as syncing everything, just focused on that particular channel (note: empty channels are optimized out). That was for the SDF channel in the context of smooth terrain. I don't have time to work on that yet though.

[chiguireitor]

Makes sense, so you get an additional channel to mask over the generated/streamed data. Would still need a way to stream it but would optimize data throughput severely.

Wish i had time to understand the code and add this :cry:

[nonchip]

i think this could be rather easily done using e.g. on the server side a class derived from VoxelStreamRegionFiles, that just sends regions/blocks (or their seeds i guess, to offload clientside generation of unchanged blocks) over the network when connecting (or traveling near a new one), and then hooking into immerge_block to detect and stream realtime changes in the block you're in. on the client side i'd just derive from the base VoxelStream class and implement the caching of the "nearby" blocks/regions in memory, kicking out the ones you don't need anymore (because they're far away) while syncing in any changes you get from the server.

[Zylann]

That approach sounds like it make sense in the current way things are setup, and would surely work. But I believe it's not enough, and is going to lack some efficiency. This only takes care of a part of networking, and the following issues arise:

• It makes clients decide what blocks they should load and query them all, while the server could actually send relevant ones only, and be authoritative on it, avoiding queries for empty ones to even go over the pipe
• VoxelStream is a synchronous interface, network isn't. This forces a very inefficient request design where the query would block until it receives the data. That wasn't a big issue with files because the time spent actually waiting is so short (and file i/o doesnt have async interface). The fact queries are batched can ease this, but there is still a wait phase which I'm not entirely convinced with. Replacing VoxelDataLoader sounds like a better alternative.
• The thread dedicated to block I/O runs at a relatively slow rate, which might not be wanted in a multiplayer scenario when polling the incoming messages buffer
• I am not sure if Godot's low level networking API is thread-safe
• Save requests should not go over network, the server only takes care of that
• Servers need multiple streaming sources, not just one as is the case now
• Voxel edition behaviors are still left to be implemented, which are often tied with game-specific logic

Anyways, if you do it with the current VoxelStream then you don't need to modify the module, and can make a prototype. I believe integrating networking can be a relatively big task, and if made built-in, should be made efficiently. It's possible that it doesnt actually need to be integrated, and that it only requires a few helper features instead. There is need for a prototype of this to be made and a good way to do this is to actually make a game/demo using it, to find bottlenecks and what the needs are.

[chiguireitor]

I think what is needed is just a way to bundle a region into an opaque blob into a PoolByteArray, then have a way to timestamp modifications and base them on a given snapshot of the voxels. However, modifications need to be saved outside of the whole Voxel* classes, so it's the app responsability to keep account of these modifications for later pushing to the users.

In short, what i'm saying is:

• Have a way for the Voxel* classes to save a blob into a PoolByteArray
• Leave all networking handling to the specific App, it's not this module's responsability to handle that

[Zylann]

I just exposed the block serializer, which should make it easier to serialize voxel buffers in 4991b682d9c8e037c6c22333275c220f4c61b558

Here is the test code I used:

	var use_compression = true

	var vb = VoxelBuffer.new()
	vb.create(16, 16, 16)
	vb.set_voxel(42, 4, 4, 4, 0)
	vb.set_voxel(43, 5, 4, 4, 1)
	vb.set_voxel(44, 6, 4, 4, 2)
	
	var peer = StreamPeerBuffer.new()
	
	# Note: that serializer can be re-used,
	# no need to re-create it for every buffer
	var serializer = VoxelBlockSerializer.new()
	serializer.serialize(peer, vb, use_compression)
	print("Saved size ", peer.data_array.size())

	peer.seek(0)
	
	vb = VoxelBuffer.new()
	# Warning!
	# Dimensions and data size are not part of the serialized data,
	# you may have to handle them yourself, make sure they are correct
	vb.create(16, 16, 16)
	serializer.deserialize(peer, vb, peer.data_array.size(), use_compression)
	
	print("Deserialized ",
		vb.get_voxel(4, 4, 4, 0), ", ",
		vb.get_voxel(5, 4, 4, 1), ", ",
		vb.get_voxel(6, 4, 4, 2))

[nic96]

I've been trying to stream regions over the network, but my main problem is that in the VoxelGenerator I can't wait for data. Is it possible to somehow queue blocks to get generated once data becomes available. Maybe generate chunks when a certain signal gets emitted? I'm using Godot's StreamPeerTCP to send the regions. One way to get around this would be to just wait until all regions within a certain radius of a player are loaded. Then start the generator and have invisible walls that prevent a player from moving close enough to the edge of loaded regions for chunks to generate in regions that aren't loaded. That way the required changed chunk data would always be available for the generator.

I also just discovered coroutines with yield in Godot, but I'm not sure if that would work with the generate_block() function.

[Zylann]

Is it possible to somehow queue blocks to get generated once data becomes available.

Not yet.

I'm using Godot's StreamPeerTCP to send the regions

I would not focus on sending regions. This concept was engineered specifically for disk I/O, and isn't suited for network.

Then start the generator

Also VoxelGenerator is not intented for loading, it's for generating procedurally. If network features appear in the future, it won't be in that class.

I also just discovered coroutines with yield in Godot, but I'm not sure if that would work with the generate_block() function.

No it won't because it's designed synchronously, and yield would make it return a GDFunctionState, which isn't expected by the whole streaming stack. I likely won't make it accept that anyways because I want a language-agnostic API.

So as I said in this comment: https://github.com/Zylann/godot_voxel/issues/151#issuecomment-660666810 Implementing this requires a new API which I plan to work on in the future. At the moment there is no easy way to plug in an asynchronous data source in the terrain system, without blocking.

[nic96]

Alright, thanks for the info 👍

[WithinAmnesia]

Anything happen with networking progress for multiplayer? I opened a similar issue. I was just using some duedilegence to see what people might have found? https://github.com/Zylann/godot_voxel/issues/602