otp
otp copied to clipboard
erlang
Experimental usrsctp support
Experimental support for usrsctplib. Any feedback appreciated.
The main idea is to use usrsctplib from within the emulator using gen_sctp as for the current SCTP support.
But this makes the SCTP stack run within the beam.
This mainly enabled me to have SCTP on macOS without running unsigned sctp kernel extensions.
It might also be possible to get usrsctplib support to work on windows.
I have only tested on macOS and Linux.
Some modifications has been made to usrsctp which can be found on below github repo and branch. I.e. passing already opened raw sockets to the lib and event callback support when socket is ready. https://github.com/falkevik/usrsctp/tree/event_callback_support
So how to try it.
First install the usrsctplib from the above branch.
When done you can enable usrsctp support when configuring otp.
./configure --enable-sctp=usrsctp should start looking for the usrsctp header.
When done and compiled as usual.
On Linux you can go with adding cap_net_raw; setcap cap_net_raw+ep /path/to/beam.smp.
On macOS I pass already opened raw sockets as arguments to the beam.
New flags are +zsctp_raw_ipv4 <fd> +zsctp_raw_ipv6 <fd> +zsctp_raw_route <fd>
setuid_socket_warp can be used for this, which is slightly modified to set additional options on the raw socket before dropping privileges.
setuid_socket_wrap -t +zsctp_raw_ipv4,sctp -T +zsctp_raw_ipv6,sctp -z +zsctp_raw_route,sctp
So to the actual code changes, the idea is that when not enabled it should not affect at all.
This need extra code review I think, there a lot of #ifdefs that might have ended up badly.
There are some other changes that I need feedback on how to do it properly.
Since we don't have any real file descriptor that we can put in the select/poll I register a callback when socket is ready to be read.
When this callback is invoked, I check if the socket are in active mode or not.
If it is in active mode, the port are scheduled to be run erts_port_task_schedule as select would have done if the fd was ready.
To be able to use this call from the inet_drv driver I had to make them available from somewhere.
Currently in erts/emulator/sys/common/erl_sys_common_misc.c, where to put this kind of code?
We have some plans to make big changes to how sockets are handled. I am not sure how this would fit in to this new scenario. We will have to ask you to be patient as this needs to be discussed further internally before we can give you good feedback.
The way that we normally solve the problem when a library does not expose an fd to do the polling but uses a callback, is to create a pipe that the driver can select on and then the callback can signal on that pipe. You should do that instead of adding functionality to the emulator.
We still have not decided what we will do with this functionality yet, just wanted to leave the above feedback for the record.
I had the approach of using a pipe that was put into the poll/select, and then I wrote to that pipe when I got a callback that something was on the socket. I can switch back to that. The reason I decided to schedule the port directly is that it felt bad to be in user space but still force a switch to kernel just to schedule the port. And the second reason was that I had issues solving when the pipe/fd is ready to be written to.
@mko-io this only adds usrsctp as a different backend to provide general SCTP protocol features to erlang via the gen_sctp module.
Also interested in usermode SCTP. Where is the afore-mentioned discussion taking place?
We have a work package where we are working to replace the existing inet-driver with a nif-based solution. However this is big job and we do not want to make big changes to the existing inet driver, like this one, in the midst of this. We would rather evaluate this functionality in the light of the new solution. Alas this means that it might be stalled for quite a long time.
@IngelaAndin I'm a bit of exciting about the driver to Nif conversion plan, so it worths waiting and hope the OTP team can add this feature afterward.
@IngelaAndin I'm also looking forward to inet-driver implemented as a nif instead. I think it will simplify and reduce the code needed to add usrsctp support. Thanks for the update!
any update on this? WebRTC Datachannel is enabled on all the browsers link and a better solution compares to WebSockets, and fit well for the process/messaging model for Erlang/OTP by enabling clients send message directly to each other. So userSCTP integration could be a highlight feature for the next release.
Hi @falkevik , I am just wondering if you have done something more with usrsctp after this PR? It could be interesting to explore the possibilities to use usrsctp together via our new socket API.
Hi @KennethL I have not done any real work in porting / moving the feature over to the new socket API I'm afraid. Last time I checked the SCTP API for the new socket implementation didn't seem to be ready. Since then I have not had the time to look at the details. But I still thinks it is a good idea to try to add this to the new socket API. Hopefully it could give more control of the support and upgradeability of SCTP when using containers for example.
Is there up to date documentation somewhere? E.g usrsctp_init(uint16_t) in Manual.md does not match well what is in usrsctp.h which matches even less the its call in inet_drv.c. Anything on non-blocking usrsctp_connect() and usrsctp_accept()?
Have you tried to get any of your usrsctp chages accepted upstream at sctplab?
Thanks for having a look at the PR @RaimoNiskanen
The main missing part is the event callback handling. Which was cooking in another project and later merge to the main repo in dec 2017 I believe. https://github.com/sctplab/usrsctp/pull/189
The PR-SCTP compatibility addition added in the INIT handshake isn't critical. Found when testing with some third party stack that didn't like the fact that the received INIT_ACK indicated support for PR-SCTP but the INITIAL INIT didn't include support for PR-SCTP, the other third party stack crashed.
Some changes was related to providing pre-opened sockets for raw sockets. I.e. passing file descriptor to be used. So that the beam doesn't have to have the rights for opening raw sockets.
I'm on vacation right now, so don't have that much time to look at the documentation you mention. But will have a look as soon as possible.
Hi @falkevik have you looked any more at this? Do I understand correctly that if there is an Erlang VM with usrsctp support active it is only that Linux process that can receive (and send) sctp from the outside (other hosts)? If there are several containers running on the same HOST , can each of them send and receive data via usrsctp or is it only one per host. What about virtual machines on the same HW, can one use kernelsctp and another use usrsctp?