REST `/watch/gameserver` API fails to send updates until stream is closed

[austin-space]

What happened: When requests to the /watch/gameserver http endpoint have to touch a network interface(including going between containers in the same pod), the sdk server fails to return partial responses(the watch updates) until the sdk server is closed, at which point the server receives all of the updates that it should have received earlier.

What you expected to happen: Watch should send updates as they occur.

How to reproduce it (as minimally and precisely as possible):

The unity example should reproduce this when actually deployed into a cluster(or when built with the il2cpp scripting backend and run locally). Watch will appear to register correctly, but won't receive any updates.

Anything else we need to know?:

Unless I'm missing something(very possible) it seems like this is due to streaming RPCs not being supported when doing http bridging, see: https://github.com/googleforgames/agones/blob/e7ae8d80de6cfbda4da3bbcdddabda71d8b6c82a/pkg/sdk/sdk.pb.gw.go#L334

Environment:

• Agones version: 1.26
• Kubernetes version (use kubectl version): 1.23
• Cloud provider or hardware configuration:
• Install method (yaml/helm):
• Troubleshooting guide log(s):
• Others:

[markmandel]

Just so I'm clear - are you actually using the Unity SDK, or another REST client?

If you aren't using the Unity client (admittedly I haven't touched in a while), I did have to write some middleware to capture when data came in, as with a standard HTTP GET, usually you only get the results when the request closes.

See: https://github.com/googleforgames/agones/blob/f1994ea188b728e5f890ffa4ed3621cfc908389e/sdks/unity/AgonesSdk.cs#L360-L376

[austin-space]

I first saw it with the unity sdk, and verified with other rest clients and some packet captures.

The problem is that ReveiveData only ever gets called when running in editor, because otherwise the sdk server never actually sends the data. If you make a build, then it waits until the server is closed and sends its data.

[markmandel]

Huh that's so weird that it works in the Unity editor, but not in the build. It's not like the SDK Server has changed - it's sending data on each change. The middleware in Unity is behaving badly at that point? 🤔

[austin-space]

It's not unity specific it seems. I was able to observe this with a number of http clients, and verified with packet captures that no packets are actually returned until the stream is closed.

It seems that the only difference the matters between it working and not is if you ever hit a real network interface. It seems like the editor is short circuiting that somehow(I can't get a packet capture on it in editor on windows), but when I force it to run through any real network interface(including the trivial packet capture one, or the pod CNI) then it breaks.

[markmandel]

a real network interface

Can you expand what you mean by this?

[austin-space]

Speaking as not an expert and from my current understanding: loopback on windows is weird in that it shortcuts a large portion of the networking stack(more so than linux).

That being said, I've finally gotten an elusive packet capture showing the data being sent back, but unity not getting it, so the network interface stuff may have been a red herring.

[markmandel]

The easier options may be to just use the websocket interface like the Unreal SDK does.

[github-actions[bot]]

'This issue is marked as Stale due to inactivity for more than 30 days. To avoid being marked as 'stale' please add 'awaiting-maintainer' label or add a comment. Thank you for your contributions '

[github-actions[bot]]

This issue is marked as obsolete due to inactivity for last 60 days. To avoid issue getting closed in next 30 days, please add a comment or add 'awaiting-maintainer' label. Thank you for your contributions