Universalis
Universalis copied to clipboard
Universalis-FFXIV
WebSocket endpoint
See title. This is already in-progress on the feat/websocket branch.
Current thoughts:
- Should the format be JSON or binary? Sending binary data might be preferable on my end, to minimize bandwidth (this can send up to 110 messages/s/client sometimes). If it's binary data, I should write a JS client library to handle marshalling data from buffers. However, sending binary data introduces considerations of alignment etc.
Only thought on that is not everyone might be using the JS client, which means if you want to consume it you would need to write in your language (For example, I can imagine Python will be a common one and they generally use either the marshal or pickle module generally for deserializing binary data). JSON is quite portable and widely supported.
If bandwidth is a concern, it could be handled as a separate concern. Another way to control the amount of data going out per second would be to decouple the Publishing from the uploads and send the updates in controlled batches every second (or something like that).
From: X Uploads -> Y events per second -> Z Subscribers
To: X Uploads -> Constant Events per second -> Z Subscribers
Once you've done that, you could make the amount of events that are being sent out configurable, and you would probably be able to handle more clients at a time.
I considered that, but batching updates doesn't allow me to trim down that much data overall. For starters, I'm just doing listings (added/removed) and sales (added), and just about every update is unique - the only thing that batching would sometimes cut down are world and item IDs. Beyond that, I'm just sending fewer, significantly bulkier messages, which isn't much better.
The first thing I looked into was just sending changed data, rather than full updates, and that cut down a huge amount of what got sent. However, the rate at which updates occurs means that even with that, it still won't scale very well.
Ughh yeah, that makes sense. I mean binary + custom js client will probably get you hands down the best result, here's another thought (and correct me if I'm misunderstanding, I can't run the code for whatever reason I can't download prometheus-net but different problem). One way I've seen people cut down large json objects is by sending in arrays as opposed actualy json objects and communicate ahead of time what each index is. For example a json such as
{ "item": 1234, "world": "jenova", "listings": [ { "lastReviewTime": "2022-01-01 00:00:00" "pricePerUnit": "1" ... }, { "lastReviewTime": "2022-01-01 00:00:00" "pricePerUnit": "2" ... }, ] }
Becomes
{ "item": 1234, "world": "jenova", "listings": [ [ "2022-01-01 00:00:00", "1", ... ], [ "2022-01-01 00:00:00", "2", ... ], ] }
With that said, it reduces the amount of bytes required for each message pretty significantly, but still in a relatively constant way (specially in the ListingView object). I think this might be a middle-ground but it still won't do as well as just straight up sending it in binary as you're still constrained by utf-8.
That's a good idea 👀 I think that's a good middle-ground, will look into it.
You also have an inbuilt spec in json that is called bson which is a binary form of json.
Actually, I might go with that instead, to split the difference between minimizing user setup and minimizing payload sizes, I'll have to test it first, though.
