Cross-platform plugin on TCP

[ndelta0]

This version uses no external libraries. It's implemented with TCP sockets for each of the OSes (Windows and Unix). The server listens on 0.0.0.0:45454. The transmission format is as follows:

1. Send 4 byte (32 bit) integer (Int32) with the size of the data that will follow.
2. Send the telemetry data. The format is unchanged and is compatible with previous releases and clients.

On my computer (i7-4790k) there were no perfomance drops in the game with the telemetry being sent every simulation frame. From what I saw in IO Ninja, the bandwidth that it requires is applicable only for local network data transfer and fast networks (I think it was about 1.5-2 MB/s).

Known Issues:

• If the socket stops receiving data (blocking method call or something) the game can freeze, until the receiving client resumes.

Compatibility: The project is set to build on Linux x64, Windows x64 and Windows x86. I only tested Windows x64 and Windows x86. It's not confirmed nor said to work on MacOS, but it's possible to work with minor or no tweaks.

[RenCloud]

I will probably make some tests today/tomorrow to check the performance of the implementation.

On first view I had again some problems with TCP, but I think when the performance is fine it allows some feature to build much faster than with shared memory.

In addition, I think it makes the development of other clients simpler.

That means a lot of plus points for TCP if the performance is right.

[ndelta0]

I've also considered using UDP instead of TCP, but it would be too unreliable and you could lose important events

[RenCloud]

Performance seems to be ok. Also, I have some ideas for better performance, so that should not be such a problem.

2 other things are more of a problem:

• [x] I can not change the application when this SDK is loaded. ALT + Tab did not minimize the game.
• [x] when the SDK is unloaded, the game crashes.

[ndelta0]

I'll look into that

[ndelta0]

Fixed the issue with crash on sdk unload. I couldn't reproduce the Alt + Tab issue, it might be related to how the game is made since it doesn't really like to be alt tabbed from. If anyone reproduces it send the game log here.

[alphavector]

I was able to build a plugin for mac os with minor edits in the code. The game itself runs, but I don't really understand where I should look at the telemetry readings. When I access 0.0.0.0:45454 I get ERR_INVALID_HTTP_RESPONSE, in logs <WARNING> Error sending: 32. @ndelta0, I will send pr to your branch if you are still interested in developing it.

[ndelta0]

I was able to build a plugin for mac os with minor edits in the code. The game itself runs, but I don't really understand where I should look at the telemetry readings. When I access 0.0.0.0:45454 I get ERR_INVALID_HTTP_RESPONSE, in logs <WARNING> Error sending: 32. @ndelta0, I will send pr to your branch if you are still interested in developing it.

Well, the listening on 0.0.0.0:45454 means that it listens on every interface, i.e. you can access it on localhost:45454 and by using your local device IP from other device.

Regarding ERR_INVALID_HTTP_RESPONSE, it was not meant to be read by a browser (i.e. not JSON or any other plain text data). It's a raw stream of bytes exactly as you would read it from the memory mapped file. You need to open a raw/stream TCP socket to be able to read the data. Tell me what language you're using and I'll send you a sample code.

[alphavector]

If you don't mind, send a sample, in Python, for example

[ndelta0]

If you don't mind, send a sample, in Python, for example

The code for receiving data in Python would look something like this:

import socket # import the socket library


def main():
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: # create the socket object with the Internet address family (IPv4) and stream socket type
        s.connect(("127.0.0.1", 45454)) # connect to the plugin listener
        while True:
            data = s.recv(1024) # receive the bytes
            print(data) # print the bytes


if __name__ == '__main__':
    main()

I was able to build a plugin for mac os with minor edits in the code. [...] I will send pr to your branch if you are still interested in developing it.

I'd love to, I don't have any Apple device myself so it would be really useful for me

[ndelta0]

NOTE: Marked the PR as draft since it's not 100% finished and there's still room for improvement

[alphavector]

Yes, we almost synchronously managed to write the code to get the data :D I get a sequence of bytes, but how to deserialize it? I understand correctly that this sequence of bytes is a structure scsTelemetryMap_t?

[ndelta0]

I get a sequence of bytes, but how to deserialize it? I understand correctly that this sequence of bytes is a structure scsTelemetryMap_t?

Exactly, except that every "message" is prefixed with a 4 byte (Int32) value specifying the length of the following data. It can be ignored if you know the data size and have a buffer of the appropriate size (which is required to be pre-allocated before receiving the data in languages like C/C++/C#/etc, I don't this is required in Python). The actual data is exactly as in the telemetry map.

[ndelta0]

Also, since this is a draft pull request, anyone should be able to commit although I don't really know how that works.

[ndelta0]

TODO:

• [ ] Check performance on Linux
• [ ] Check performance on MacOS
• [ ] Check for any memory leaks
• [ ] Investigate freezing the game when receiving socket is blocking
• [x] Update Readme
• [x] Update any versions
• [ ] Remove unused/redundant code
• [x] Use better logging (error descriptions along with the error code)
• [x] ~~Refactor code~~ Nothing to refactor, it'd have to rewritten completely
• [x] ~~Improve documentation~~ Everything important seems to be documented
• [x] ~~Add example clients for various languages (at least C++, C#, Java, Python, JS/TS)~~ Will be moved to a separate repository
• [x] Add Github Actions for building release builds

[alphavector]

I was able to serialize the data into a structure using ctypes.Structure. It was quite time consuming to fully describe the whole scsTelemetryMap_t structure in Python, luckily I was able to partially automate it with pycparser. @ndelta0, have you thought about looking into gRPC? It should still be pretty fast for data transfer (HTTP 2.0, data compression), supports streaming. In that case we would need to describe only 1 proto file, and with a code generator we would get a client for the desired programming language

In any case I will try to sketch a minimal working client for Python and send pr.

[ndelta0]

[...] have you thought about looking into gRPC? It should still be pretty fast for data transfer (HTTP 2.0, data compression), supports streaming. In that case we would need to describe only 1 proto file, and with a code generator we would get a client for the desired programming language

gRPC is also an option. Initially I wanted to use ZeroMQ and use different topics (events) for things like telemetry, job start, end, crash, etc. But @RenCloud wanted this plugin to stay dependency-free, so I went with pure sockets implementation.

EDIT: It's always possible to create a separate branch/fork with a ZeroMQ/gRPC implementation

[RenCloud]

Yeah i would like to work without a library, but i would like it, that if someone want it, it is simple to do. Kinda like just change the imported header file or write a small class that exdends an interface. But this is maybe the next step.

I will take a look at the new changes and test it tomorrow. When it works i will test win 10 and linux ( ubuntu 21.10 i guess) . After that i would see what to do then to finish this here ( i think client was missing, but that should be than in a new repository or so)

To see that it works en all 3 major platforms would be nice, so i will try to get this pr done. However, i can not test mac os.

@ndelta0 I think you mentioned somewhere that you planned to make the events in their own calls or something like that? I would like that. My plan was to split the data in 2 sets and than the events their own calls. But i need to look how you make that with tcp with good practice.

2 sets are similar to the ones in the SCS files:

• more or less static data (refreshed barely just on specific game events like truck model)
• rapidly refreshed date( location, speed, ...)

This would reduce the packets that are send often, because then the big string fields are not send constantly and i think than the performance impact on the game is really small. Also for slower PC's.

Did you think that change( using 2 separated dada sets) make sense?

[ndelta0]

I think you mentioned somewhere that you planned to make the events in their own calls or something like that? I would like that. My plan was to split the data in 2 sets and than the events their own calls. But i need to look how you make that with tcp with good practice.

2 sets are similar to the ones in the SCS files:

• more or less static data (refreshed barely just on specific game events like truck model)
• rapidly refreshed date( location, speed, ...)

This would reduce the packets that are send often, because then the big string fields are not send constantly and i think than the performance impact on the game is really small. Also for slower PC's.

Did you think that change( using 2 separated dada sets) make sense?

Yes, something similar. I thought, that it could be possible (with the TCP server) for the client to specify what events/topics it would like to subscribe to and have events e.g. "telemetry" (for information like truck speed, location, rotation, etc), job related (i.e. "jobStart", "jobEnd"), config (i.e. truck/trailer info, "substances" or more) and other like "speeding", "crash" and other fines. The client could either send the subscribed topics at the connection time or anytime during the connection if need be.

[ndelta0]

Also one thing that could be done is instead of sending data as received from the game send it in a more optimised format (at least change the arrays from fixed length to length prefixed, which would cut the current data size at least by 3-5 times - the default is 10 trailers, of which 8 are always empty - and the strings to UTF-8 length prefixed strings.

[RenCloud]

That would be a way. I will have to read a bit about TCP to see what and how a good way is to do this.

Indeed the 10 possible trailers are only available through mods i guess, so for most of the users there are useless data. With prefix you mean something like 0trailer... That should do the job.

And for strings 4Test? I am not sure why a prefix should be used here instead of something like Test\0?

[ndelta0]

That would be a way. I will have to read a bit about TCP to see what and how a good way is to do this.

Indeed the 10 possible trailers are only available through mods i guess, so for most of the users there are useless data. With prefix you mean something like 0trailer... That should do the job.

And for strings 4Test? I am not sure why a prefix should be used here instead of something like Test\0?

I thought about it in a more binary sense, i.e. prefixing the count of items in array with a Int32 or less if it would be enough, that way only the required amount of items can be transmitted, omitting the unused entries (in vanilla ETS2/ATS that would save 8 trailer entries each time).

Regarding the strings they also can be null terminated, but usually it's easier to know the length beforehand, you can then allocate a big enough buffer before getting the actual UTF-8 bytes. It makes the receiving easier, since arrays cost a lot to expand and adding to a dynamic list is costly as well.

[N95JPL]

Is there any kind of "estimated date" for when this will drop? I am working on a new application which will tie in to the SDK which will make it easier for people to use in their programs/Arduino Connections/WebServers etc. Currently writing it in C# / ASP.NET with WinForms however, if this is going to be in the near future I will double-back and start working on a leaner, meaner React project for it instead...

Thanks, JPL!

[ndelta0]

Is there any kind of "estimated date" for when this will drop?

Technically it's ready, you can already use it when you download the artifact on my fork or build it yourself (keep in mind that artifacts are currently x64 only).

[N95JPL]

Is there any kind of "estimated date" for when this will drop?

Technically it's ready, you can already use it when you download the artifact on my fork or build it yourself (keep in mind that artifacts are currently x64 only).

I had actually downloaded the src and built with CMake (Wins x64) and then built the Solution and it appears to be working, I didn't notice the artifact!

I connected via a console and received data (albeit a bunch of scrambled chars/symbols)... i will hook up a JS script later and get some bytes!

[ndelta0]

I connected via a console and received data (albeit a bunch of scrambled chars/symbols)... i will hook up a JS script later and get some bytes!

Actually it's supposed to be what you received, these bytes are actually exactly what was used before, with addition of an Int32 (4 bytes) at the beggining, which indicate the size of the following payload. Current implementations that receive and parse the data can be adapted by basically receiving the data from a TCP socket instead of a Memory Mapped File and skipping the first 4 bytes when receiving the data.

EDIT: Unlike ETCARS which sends the data in a JSON format, this plugin basically sends the memory that's assigned to the telemetry map type

[RenCloud]

I finally managed to look at the changes in more detail. I'm sorry for delaying this pr so long, but it's a big change, so I need a bit of time which I did not manage to put into this project lately. Beside some smaller support, this big change was sadly too much.

Thanks for that much afford. Also fixing old mistakes and formatting mistakes.

Most parts look good. Here and there I will change code formatting, but that's more or less personal preferences. I will check for a linting GitHub action in the future.

I currently prefer the current event handling implemented with rev 11, but that was later introduced than the start of this pr. ... Maybe neither the new nor the old way will stay, when subscriber pattern is used? At least, I think it would be possible to broadcast the data direct at the gameplay event. Then it would be an event by itself and no extra flag/switch/state/... needed. But that's just an idea at the moment.

Some minor things I need to test, because I may have some questions on them. There are also some small changes I may do. Possibly I can get rid of some commentaries and old to-dos before creating the new dev state out of it.

Beside the other points, I think

Investigate freezing the game when receiving socket is blocking

is one of the important to fix, before merging.

For memory leaks, it will probably code check up and observing application memory while doing a longer test. Perhaps I find some tool that can help here.

Additional to fixing mentioned things, I'm currently thinking if it is useful to add the subscriber pattern in this pr or later with an additional pr same with string and trailer optimizations.

[N95JPL]

ETS2-TCP-Telemetry-Byte-Map.xlsx

This is a Excel document I made to quickly reference where each Value can be found. Saves having to count and use my calculator all the time!

It's here for anybody to use if they wish too!

You'll notice "End Byte" has 2 columns, this is because some languages want the byte after the range you want (non-inclusive), whereas others are happy with inclusive ranges!

Enjoy, Thanks JPL!

[RenCloud]

I felt free to directly pushed into your branch, I hope that's fine for you.

Most/all are format changes or changes on comments. And remove the logic from events bits, like in the current main branch.

I tested it a bit with the above python script and displayed in game speed. Seems to still work after my changes. Also, logging seems to work wonderful. The Excel sheet was already helpful for that. Thanks JPL. May I should include a Markdown table with that information.

Performance seems to be ok/good already (windows 10). However, I'm exited to work on the optimizations for strings, trailer and the pattern.

-- Currently, I think getting this stage into dev-branch and after that working on the next steps (creating client lib, subscriber pattern, optimizations on code and transmitted object) is the way to go here to bring this big change to a "production ready" state.

I will check this on Linux and go another time through the code if I overlooked something. If I have then no questions/changes, and you have no changes, I would like to merge it and work than one the next steps

[alphavector]

I would like to suggest a few additional changes:

• I propose to allow you to specify an arbitrary port via environment variables and not be tied strictly to port 45454. This is a small change, I will make it soon. The problem I encountered:
• any connection to the server (i.e. the game) blocks the game, for some reason even the asynchronous client in python does not solve the problem. I have to open the connection, quickly read the data and close the connection and reopen it again and so on. This works and has little effect on game performance, but seems like a crutch, ideally I'd like to keep the connection open all the time. I don't know if this is a problem on Windows. Perhaps the problem is due to the fact that in osx you can't specify SOCK_NONBLOCK in the accept function

Further ideas:

• non-blocking sockets on the plugin side
• More documentation, maybe go to https://readthedocs.org/
• I support allowing the client to subscribe to certain types of events

In spite of all the above, I've tested it working in OSX (and even on an Apple Mac M1) -- it works, works great.

[ndelta0]

• I propose to allow you to specify an arbitrary port via environment variables and not be tied strictly to port 45454.

That's a great idea, I've thought about that but never got around to implement it

• any connection to the server (i.e. the game) blocks the game [...] Perhaps the problem is due to the fact that in osx you can't specify SOCK_NONBLOCK in the accept function

I'm aware of this issue, it's just hard to do any asynchronous socket communications on any system, especially in C/C++. I haven't encountered this issue on Windows and Linux (with accepting), but I've encountered a problem with the game hanging when the client does not receive data and fills the buffer.

EDIT: I don't have any way of running ETS2/ATS on a MacOS system (unless you have some way to run MacOS in a virtual machine, DM me then), so any help with MacOS sockets is very appreciated.