Add TCP socket settings

[Konamiman]

openMSX allows being externally controlled by applications connecting to it. In the Windows operating system, openMSX accepts connections to a TCP socket on a port number chosen at random between 9938 and 10002, and SSPI authentication is mandatory before actual commands are accepted.

This pull request adds some (Windows-only) settings and a new argument for the openmsx_info command that allow altering the behavior of openMSX related to TCP sockets. All the settings have sensible defaults that are backwards compatible with the current behavior.

set socket_auth_mode

An enumeration type setting that accepts two values: sspi (default) and none. When none is set, openMSX will skip the authentication phase and will start to accept commands immediately after a connection is established.

Changing the value of the setting affects new connections, already established connections are unaffected.

set socket_port_number

An integer setting that sets the base value of the TCP port in which openMSX will listen for incoming connections. The default value is 9938.

set socket_selection_mode

An enumeration type setting that accepts two values: random (default) and fixed.

random keeps the current behavior: the port number is chosen randomly between the base value (set with set socket_port_number) and the base value plus 64; with the default base value that means between 9938 and 10002. fixed simply uses the base value as the actual port number.

In both cases, if openMSX is unable to use the chosen port because it's already in use, it will keep the current behavior: trying again after increasing the port number value by one; this procedure will be repeated up to 64 times if necessary before giving up and showing an error (the actual port number will always be in the "base+64" range).

When the value of socket_port_number or socket_selection_mode changes, the socket is closed and re-opened with a new port number according to the new values of the settings, so new connections are expected to happen in the new port number. This also updates the socket.<pid> socket file accordingly.

openmsx_info socket_port

This will just print the port number in which openMSX currently accepts TCP connections. This is the same value that is written to the socket.<pid>, but with the convenience of being available as a command that can be executed from the openMSX built-in console.

Documentation updates

The commands reference and the "Controlling openMSX from External Applications" guide are updated to reflect the new settings. Additionally, the later mentions the need for SSPI authentication by default (this was missing in the original document) and adds a warning about the risks of disabling authentication with set socket_auth_mode none.

[m9710797]

Thanks a lot for this PR! I'll try to take a look this weekend when I'm back from vacation.

[MBilderbeek]

Yes thanks a lot!

I'm also on a holiday but here is some initial feedback:

• can you make sure it builds on all platforms? The Checks tab shows the results.
• can you make clear in the improved documentation that the authentication stuff is only applicable on Windows systems?
• would it be good to also have a command line option to set the authentication method? Then you don't have to rely on a permanently saved setting that hasn't been set when you start up. OTOH the setting is mostly meant to control connections after start up...

Op do 3 aug. 2023 18:48 schreef Wouter Vermaelen @.***>:

Thanks a lot for this PR! I'll try to take a look this weekend when I'm back from vacation.

— Reply to this email directly, view it on GitHub https://github.com/openMSX/openMSX/pull/1531#issuecomment-1664313473, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADAWXCSTGRHVBGLRHNTYZFLXTPI6DANCNFSM6AAAAAA3CNJX2I . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[m9710797]

Hi,

Thanks again for working on this. Before merging this PR I'd like to present an alternative. I'm not saying we must choose that alternative over your PR. But I'd like to evaluate the pros and cons of both versions, have a discussion, and then make an informed choice.

Alternative 0: Do nothing. Advantages:

• Fully backwards compatible.
• No security risk.

Disadvantages:

• Requires separate client code for windows and non-windows systems. Many client programs only bother to implement one variant, making them non-portable.
• The windows client code is not so easy to implement and is poorly documented.

Alternative 1: this PR (allow to disable to authentication step on the windows TCP socket) Advantages:

• Windows client code becomes as simple as the code on the non-windows systems.

Disadvantages:

• Still requires separate windows and non-windows client code (TCP vs unix domain socket). Though it's already a lot closer.
• Disabling authentication has some security implications.
• Older existing clients (like the debugger) may not work in the new mode with authentication disabled. So you can't mix these different clients.

Alternative 2: use unix domain sockets also on windows systems

Advantages:

• Client code is now the same on windows and non-windows systems.
• No security issues (only the local user that started openMSX can connect).
• Fully backwards compatible: we can keep the current TCP with authentication stuff unchanged. (And maybe in several years we can consider first deprecating and then removing it).
• This would also simplify the code on the openMSX side (having a uniform server-mechanism for windows and non-windows).

Disadvantages:

• Unix domain sockets are only available from windows 10, since around December 2017, according to this blog post. Maybe, by now, it's acceptable to start using those? (And existing clients on older windows versions can continue using the current TCP stuff).
• Personally I can't test the servers side (openMSX) of this new connection mechanism on windows. So we'll need a volunteer to implement it.

I tried to make an honest comparison. Did I miss anything important? At the moment my personal preference is alternative 2. But maybe I'm biased. So again: did I overlook anything? Let's discuss.

[Konamiman]

Hi @MBilderbeek and @m9710797. You were a bit too fast and commented while the PR was still in development (see the "Draft" stamp next to the PR title) 🙂. I still need to do some more testing but I have already implemented all I wanted to implement, and wrote a proper PR description. Please take a look again.

I see that the builds for Linux and Mac are failing in GitHub so that's something I'll have to look at too (my guess is that I forgot a #ifdef _WIN32 somewhere).

Alternative 2: use unix domain sockets also on windows systems

I see value on this but as a feature to implement additionally to this PR, not instead of it. I guess it would require some changes more complex than just a few settings added by a single person with no prior C++ experience in a few days, so a more thoughtful design and implementation work and a separate PR would be required. Also TCP sockets are very easy to use in most platforms and languages, so I vote for keeping them in place even if unix domain sockets are implemented for Windows.

So my vote is: alternative 1 now, alternative 2 in the future.

[Konamiman]

Disabling authentication has some security implications.

...which are prominently explained in the documentation (adjustments can be made if necessary).

Older existing clients (like the debugger) may not work in the new mode with authentication disabled. So you can't mix these different clients.

You can always start openMSX with authentication enabled, disable it, then use whatever tool you need that is not aware of authentication. Or the other way around.

Keep in mind also that all the new settings need to be explicitly changed, the default behavior is "nothing changes" and only whoever needs to disable authentication/change ports will do it. So it's not like this will inadvertently cause issues to anyone.

[m9710797]

Hi,

In this post I'm arguing for option 2. That doesn't mean I'm fully against option 1, we're still evaluating.

Hi @MBilderbeek and @m9710797. You were a bit too fast and commented while the PR was still in development (see the "Draft" stamp next to the PR title)

I understand. Some people prefer early feedback, other people prefer to finish the full PR before listening to feedback and making adjustments. Since you submitted a draft PR (instead of only submitting once you finished) I assumed you prefer the former.

And early feedback can avoid working out details that maybe won't be accepted.

    Alternative 2: use unix domain sockets also on windows systems

I see value on this but as a feature to implement additionally to this PR, not instead of it. I guess it would require some changes more complex than just a few settings added by a single person with no prior C++ experience in a few days, so a more thoughtful design and implementation work and a separate PR would be required. Also TCP sockets are very easy to use in most platforms and languages, so I vote for keeping them in place even if unix domain sockets are implemented for Windows.

I haven't yet fully reviewed the code in your PR (I'll do a detailed review when the PR is no longer draft). But from skimming it, I would not have said you had no prior C++ experience. C++ is not an easy language to learn, so I'm impressed.

It's true that TCP sockets are easy to use. But unix domain sockets are exactly as easy to use. Instead of connecting to a host and port number, you connect to an entity in the file-system (and thus you can use file-system owner and permissions to control exactly who is allowed to connect). Instead of having to probe a range of ports, you can list (ls or dir) the sockets in a directory to see which openmsx instances (possibly more than one) are running. What I mean to say is that ease of use is not an argument against unix domain sockets, on the contrary.

One of the problems with openMSX (and I guess many +20 years old codebases) is that it's hard to make sure nothing breaks when making large changes in the code. And this is especially true for platform specific code (like windows only code), because it generally gets less testing. So if possible I prefer to decrease rather than increase platform specific code. And especially for windows: we have volunteers for Linux and macOS, but finding volunteers to work on windows stuff has proven to be difficult. So thanks again for working on this.

We added the connect-via-a-socket-to-openMSX stuff in 2005. At that point windows did not yet have support for unix domain sockets. So we had no choice but to use TCP sockets and add the complicated authentication stuff on top to get the same security guarantees. Today, 18 years later, the situation has changed, and maybe we can make windows similar to the other platforms. So simplify the code rather than add even more windows specific stuff.

--

Maybe we should take a step back. You've done a great job in documenting what your PR changes and how it works. But the reason why you're changing this is less clear. I'm assuming you want to make it easier for windows clients to connect to openMSX, right? Are there any other reasons?

At the moment my understanding is that connecting to a unix domain socket (even on windows) is as easy as connecting to a TCP sockets (and easier when you have to deal with multiple openMSX sessions). And very important: if all platforms use the same mechanism, client programs only need to implement their connection code once instead of once for windows and again for non-windows. So that's even better for an openMSX client-developer, no?

    Older existing clients (like the debugger) may not work in the new mode with authentication disabled. So you can't mix these different clients.

You can always start openMSX with authentication enabled, disable it, then use whatever tool you need that is not aware of authentication. Or the other way around.

True, but you have to admit that it's not very user friendly. From an openMSX user point of view it will be weird that some clients require one and others require the other mode. Why can't all clients use the secure mode? The mode only exists to make life easier for the implementer of a new openMSX client program. And from that client-developer point of view it would be even nicer if the same connection code can be used for windows and non-windows platforms.

Keep in mind also that all the new settings need to be explicitly changed, the default behavior is "nothing changes" and only whoever needs to disable authentication/change ports will do it. So it's not like this will inadvertently cause issues to anyone.

I'm also not sure yet that a setting is the best way to handle this. I mean I can already see the bug reports like: "Last month I could connect the debugger to openMSX, but today that doesn't work anymore". (It's because the "auth_mode" settings was changed in the mean time). Maybe it's OK if this setting was not saved (not remembered when restarting openMSX). In other words: if the users desires no authentication, he should request this in each openMSX session, instead of changing this only once.

[Konamiman]

Some people prefer early feedback, other people prefer to finish the full PR before listening to feedback and making adjustments. Since you submitted a draft PR (instead of only submitting once you finished) I assumed you prefer the former.

I tend to create a draft PR soon after I've started a new development, even on my own repos, as a means to have a backup of my work. I also tend to amend-commit with just "WIP" as the commit message and force-push until I have a meaningful unit of work done, for the same reason. So I interpret "Draft" as "Nothing to see here yet" (and that's how it's interpreted too where I work, so I'm of course a bit biased), but your interpretation makes sense too.

But from skimming it, I would not have said you had no prior C++ experience. C++ is not an easy language to learn, so I'm impressed.

To be honest, most of my changes are copy-paste-adjust of already existing code and at times I wasn't 100% understanding what I was doing 😅 (but I tried my best to adapt to the existing code styling)

What I mean to say is that ease of use is not an argument against unix domain sockets, on the contrary.

Maybe I didn't explain myself correctly. I was absolutely not arguing against implementing Unix domain sockets, quite the opposite, I think it would be a great addition to openMSX.

With "ease of use" I mean for a developer casually trying to send some simple commands from a script. Which was my case and why I started this PR in the first place: I want to open LinqPAD, crank a few lines of code, and have my commands sent. And this is what is ridiculously easy to do with TCP sockets as long as no authentication is required; TCP/IP has been around for ages and literally every programmer knows about them. Unix domain sockets, not that much (in the Windows world, at least).

Instead of having to probe a range of ports

No need for that if you know the port in advance, right? That's what the socket.<pid> file is for, and also why I added the openmsx_info socket_port command.

From an openMSX user point of view it will be weird that some clients require one and others require the other mode. Why can't all clients use the secure mode?

I don't expect any "serious"/full-blown client to be developed that doesn't require authentication (maybe this should be explicitly stated in the docs). As I said, the disabling authentication mechanism is aimed more for casual developers running short custom scripts; and in these cases, the user being the developer himself knows what's going on (as he must have read the docs first to get started).

And very important: if all platforms use the same mechanism, client programs only need to implement their connection code once instead of once for windows and again for non-windows. So that's even better for an openMSX client-developer, no? And from that client-developer point of view it would be even nicer if the same connection code can be used for windows and non-windows platforms.

100% agreed. That's why I fully support the idea of adding Unix domain sockets support on Windows, and discouraging the usage of TCP sockets after doing that. But I still think there's value in the improvements I'm submitting here, as a "final revamp" of a soon-to-be-legacy feature.

Maybe it's OK if this setting was not saved (not remembered when restarting openMSX)

As I said, in the expected scenarios the user/developer knows what he's doing. But this is a good point, and it's always possible to add -command "set sockets_auth_mode none" to the openMSX command line if needed (although unfortunately Catapult doesn't offer this option).

[Konamiman]

I think I have fixed the non-Windows builds, but GitHub says "workflow awaiting approval".

[Konamiman]

Now the Linux builds work, but the MacOS ones don't. Could some Mac-er please take a look?

[MBilderbeek]

Perhaps @gilbertfrancois can take a look? Or @grauw ?

Any news on this?

[MBilderbeek]

oh, I guess we need @andete as it is about signing of the binary...

[m9710797]

Hi,

I've re-read the whole discussion (and large parts of the patch and documentation) and given it some more thought: I don't think we should merge this PR (in it's current form).

Technically there's nothing wrong with this PR, and it's very well documented. But from an openMSX-evolution point of view I think it's a step in the wrong direction. Allow me to explain.

Connecting an external application to openMSX was implemented using unix-domain sockets. This is easy and secure (as in: only the user who started openMSX can control it). Unfortunately (at the time) windows did not support unix domain sockets. The closest alternative were TCP sockets. Though those allow anyone to connect, so we were forced to add an extra authentication step on top. That's very unfortunate because it's a pretty complex extra step. And maybe more important: the mechanism is now different from all the other operating systems. This means that openMSX clients always need to implement both mechanisms (and many don't bother and thus aren't portable).

Starting from windows 10 (since 2017), unix domain sockets are supported on windows. If this were already the case when we first implemented the connect-to-openMSX feature (in 2005), we would never have added the complex TCP-socket stuff. Now that unix-domain sockets are available in windows, we can maybe fix this (historical, forced) "mistake". Unfortunately we (or I) can't do this ourselves, so we require a windows-developer volunteer.

This PR:

• Allows to disable the TCP authentication step. This greatly simplifies setting up a connection (at the cost of security).
• Allows to disable the port-randomization. This simplifies setting up a connection (at the cost of of reducing the likelihood that clients correctly deal with multiple running openMSX instances).
• Most of this is new windows-specific code, while we should try to reduce platform-specific code. (Especially windows, as it's hard to find volunteers to help with windows).

So although this makes using TCP sockets (on windows) as easy to use as unix-domain sockets (but at some cost), it's still the "wrong" approach IMHO. We shouldn't make the "wrong" mechanism easy to use. Because that decreases the pressure to implement the proper mechanism in windows.

@Konamiman I'm sure this PR works great for you. So absolutely continue using it in your own build. But we shouldn't encourage this direction for other windows developers. Instead we should encourage secure, easy-to-use and portable code. Therefor I don't think it's a good idea to merge this PR in the 'official' openMSX tree. But thanks a lot for your effort anyway.

Wouter

[MBilderbeek]

Ideally, someone (@Konamiman ?) would add support for Windows UNIX Domain Sockets to openMSX... So in the end the Windows only plain socket stuff can be removed.

[sonarqubecloud[bot]]

Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

[m9710797]

Hi again,

Recently on MRC there was this interesting discussion. See especially GNF's posts of 26-04-2024 and 27-04-2024. He did some research and experiments, and apparently UNIX domain sockets (UDS) on Windows don't really work that well.

My previous posts in this thread were assuming that UDS on windows were implemented well so with the same functionality/capabilities as on other platforms. But maybe that assumption was wrong? I tried to do some research myself, but I found a lot of contradictory information. It is clear that UDS on windows are not 'complete' yet(?) (but technically it should already be sufficient for openMSX?). And that (maybe because of that) support for UDS on windows in the wider eco-system is still lagging behind (e.g. no support in Python or Rust yet).

During my research I learned about windows "named pipes". I was assuming these were similar to UNIX named pipes. But apparently a key difference is that windows named pipes DO support multiple clients. So in that sense they are closer to UDS. However an important missing feature of windows named pipes is a discovery mechanism. And that's essential to be able to run multiple openMSX instances and have the client choose which instance to connect to. (Important use-case: your MSX program works in one configuration but not in another, you start both configs, connect a debugger to each instance, and step through the code to see where the behavior starts to diverge).

So given the above, I'll have to change my opinion. UDS are not a good (short term) solution on windows. And then this PR does become attractive again.

@Konamiman are you still interested in working on this PR? This PR is still in "Draft" do you remember why that was? By now this PR is 8 months old, the upstream code has evolved a bit and this PR doesn't cleanly apply anymore. So it's needs to be rebased. If needed I could help with that.

[Konamiman]

@m9710797 Oh wow. What a plot twist. 🙂

Yes, I'm indeed interested, but my availability to work on MSX stuff is quite limited as of now. It would be great if indeed someone could take care of the rebase for me (also in these 8 months I've forgotten pretty much all the C++ I had learned to be able to create the pull request) 😅