andrewvy
Progression
(copied from https://github.com/andrewvy/flux/issues/2)
Thinking of a way to recreate bungeecord for minetest. A proxy should be feasible, but I'm not sure if it would be useful without modifications to the base engine // server to allow for sharing data between contexts. (player data, message-passing API, etc.).
Goal is to allow for multiple dimensions through the use of multiple servers. Ultimate goal is to provide an API that will allow building global plugins. (global-chat across servers, shared inventories, shared economy, etc.)
For a POC, the proxy server will probably have to host all media resources unless out-of-band resource sending is capable for clients. Building this proxy application will also highlight areas where changes/contributes to minetest core could be proposed.
Milestones:
- Handshake Server
- Isolated-server state (Proxy-handoff)
- Shared-server state (Proxy-middleware)
<__vy> going through https://github.com/minetest/minetest/blob/master/src/clientiface.h
1:59:17 PM <__vy> it would be nice if I could just `TOCLIENT_ANNOUNCE_MEDIA` again and have the client download new resources
2:02:04 PM <__vy> I guess making the client go through that bootstrap phase again
2:02:58 PM <•sofar> do you just want to dynamically add new resources or are we talking bungeecord-like server-to-server hopping?
2:03:19 PM <__vy> ah yeah, this is for server hopping
2:03:27 PM <•sofar> I wouldn't do it that way
2:03:32 PM <•sofar> I would just reinitialize the client
2:03:53 PM <•sofar> avoid all the protocol rigidity that way
2:04:11 PM <•sofar> create a new client object, initialize it in a secondary thread and authenticate and download resources
2:04:19 PM <•sofar> then switch to it once it's ready
2:04:44 PM <Out`Of`Control> hi
2:04:58 PM <•sofar> seems much simpler that way
2:05:13 PM <__vy> ah, this is trying to avoid core changes if possible
2:05:52 PM <Out`Of`Control> what about adding new tool. that can dig only metal nodes
2:06:07 PM <•sofar> Out`Of`Control: make a mod?
2:07:24 PM <__vy> though, having the ability for the server to force a client to re-init would be a nicer experience
2:07:41 PM <•sofar> __vy: simplest change you could do is just add a new TO_CLIENT_REDIRECT packet
2:07:49 PM <•sofar> and add a lua API for it
2:07:56 PM <cal> Out`Of`Control: what free/libre program can i use to make a skin for my character?
2:07:57 PM <•sofar> then all the work is client-side
Related IRC chat, I think the idea of making a TO_CLIENT_REDIRECT packet has some merit. This would make the client just reconnect to a specified IP address. (going the bootstrapping and resource fetch flow again).
This would allow other people to take advantage of doing server redirects, but will also allow us to use the same packets for the knot proxy. I'm all for trying to make this proxy work without minetest-core changes, but a "simple" change like this has some benefit.
Heh, it's a rough UDP proxy at the moment, you can connect with a client and it'll proxy all UDP packets while parsing them. It does middle-man chat packets and server join packets: https://github.com/andrewvy/knot/commit/66d25a92f0e4d1fcd78f52bc907278952a70c7ef#diff-28fb7c5cd7d8c02a6e71edd07ca10d10R101
This could be expanded to provide chat integration across multiple servers, but it's really rough.
Middle-manning chunk packets is possible, but because we're the middleman proxy, there's more complexity when handling seqnums on packets. (Need to keep track of what packets we've sent to the client and server so that we can reconcile what the client expects in terms of packet ordering.)
^ This is part of the reason why I stopped working on this project, just a lot of work to do. :)
Yeah, that's the bit I was most apprehensive about when considering doing this myself. I wish that MT used some existing reliable layer like ENet.
Do you just use different sockets and thus ports to allow the server to differentiate between different users?
Yup, no smart multiplexing, just simple "create-a-proxy-socket-for-each-connected-user". There's also engine work to do to make clients refetch resources somehow, might be a hacky way to make it work through proxy-only, but that'd be tough.
A little look, may be able to hijack https://github.com/minetest/minetest/blob/bba4563d89b6708d75a4053c69873dff0d747538/src/network/networkprotocol.h#L254 to make the client reconnect.
So, a hacky way would be:
- Client wants to travel to another server.
- Save the user's association to a particular server on the proxy-side of things.
- Kick the client and ask it to reconnect.
- Send the client to the saved associated server.
I've been recently busy (and will still be heavily busy for time to come!) but if there's interest and I have free time, I'd be able to explore a proof-of-concept.
Why can't you implement authentication at the proxy layer, and either modify the Minetest client to trust the proxy or have proxy tokens used in the SRP?
ie: The user's password is only used to authenticate with the proxy, the proxy then uses a token stored in a database as the password or the minetest server just has no passwords.
Any blocks sent by Minetest will override existing blocks on the client, so the client will quickly have any old content gone.
You just then need to have the same content available, which can be done by the proxy multiplexing all the servers. Content loading works by the server telling the client the file names and hashes of media. The proxy could combine all of the names and hashes from all the servers into one to be sent on connect, meaning that the user doesn't need to disconnect and reconnect to load new media. This doesn't need to be done for an initial prototype however. The worst that would happen is missing and thus dummy textures
Good to know (re: block override)!
I think authentication via proxy is the right solution, especially in usual cases where the proxy is usually exposed to the public, but the internal game servers are behind the network. Very useful when building ban/whitelist/blacklist plugins at the proxy-level as well.
The media is also a difficult roadblock, proxy would need to know what media from all servers in order to be served to the client. A hacky way suggested above was re-using the reconnect functionality in order to bootstrap the media fetching process again. But as you mentioned, initial prototype doesn't need to solve for it.
Just one thing about multiplexing: I don't think this is possible, we will always need a socket for each client. The Problem is that we need to difference the packets for the clients sent by the minetest-server. If they all arrive on the same channel we cannot know which client should receive the packet. Or are there some ways to difference the packages?
You're correct, there's no actual multiplexing going on: https://github.com/andrewvy/knot/issues/1#issuecomment-451693330
Procedure is for:
-
client connects to proxy -
proxy creates a UDP connection tagged for that client, and can middleman packets -
upstream server gets connected to like any regular client would
this way, the upstream minetest server requires no additional code, and the work of distributing packets is up to the proxy-level.
I'm sure it's possible to do real multiplexing with a singular connection between proxy <-> server, but that'll require some work.
I've been extremely busy working on private things & I'm out of the loop as far as minetest development goes, so this project is pretty much abandoned. I think if MT networking becomes a bit easier to deal with, it should be pretty easy to bolt on a proxy layer. If there's still a lot of interest from the community, I'd be happy to help a proxy project, but don't expect much from me. :)