netmq
netmq copied to clipboard
zeromq
IPC implementation
Are there any plans to support IPC in NetMQ? I would love to be able to do cross process communication which didn't require eating up a port. Has there been any research to see if named pipes can fill this need.
The main issue is that the advanced ZeroMQ patterns may (such as the interconnected brokers in zguide chapter) would all require many ports to work.
I would also like to see an IPC implementation with something like shared memory.
Named pipes might be an option. Looks like they've been around since .NET 3.5.
Named pipes provide interprocess communication between a pipe server and one or more pipe clients. Named pipes can be one-way or duplex. They support message-based communication and allow multiple clients to connect simultaneously to the server process using the same pipe name. Named pipes also support impersonation, which enables connecting processes to use their own permissions on remote servers.
In the .NET Framework, you implement named pipes by using the NamedPipeServerStream and NamedPipeClientStream classes.
They're not supported by Mono though.
I did a quick Google search on IPC and Mono and Memory Mapped files seems to be the most popular method and the code in the projects I looked at looked deceivingly simple. http://nugetmusthaves.com/Tag/IPC
It seems like this one: http://nugetmusthaves.com/Package/SharedMemory http://sharedmemory.codeplex.com/documentation seems to exactly map into the internal m_pipe concept NetMQ is already using. The library has three classes, an Array, a CircularBuffer, and a BufferReadWrite; and the CircularBuffer uses a lock free algorithm (the other two use locks for Thread synchronization).
Other than locking I'm not sure exactly what any differences are, but the sample code looks almost trivial. Two CircularBuffers per Socket (one for each direction) and it looks like the code supports both blocking, non-blocking, and Async callbacks... It looks like this same code could be used for inproc as well.
I tried quickly looking up the implementation of the inproc transport; but didn't exactly see how everything was organized. The files in the inproc transport directory seemed to only deal with addressing functions. What's required to implement a transport? Is there an interface I can look at?
So we must be able to poll on multiple pipes for the IO thread. The easiest way will be to integrate it with AsyncIO library which is what NetMQ is using as IO library.
On Sat, Jan 23, 2016 at 9:12 AM, Mike Fair [email protected] wrote:
I did a quick Google search on IPC and Mono and Memory Mapped files seems to be the most popular method and the code in the projects I looked at looked deceivingly simple. http://nugetmusthaves.com/Tag/IPC
It seems like this one: http://nugetmusthaves.com/Package/SharedMemory http://sharedmemory.codeplex.com/documentation seems to exactly map into the internal m_pipe concept NetMQ is already using. The library has three classes, an Array, a CircularBuffer, and a BufferReadWrite; and the CircularBuffer uses a lock free algorithm (the other two use locks for Thread synchronization).
Other than locking I'm sure exactly what any differences are, but the sample code looks almost trivial. Two CircularBuffers per Socket (one for each direction) and it looks like the code supports both blocking, non-blocking, and Async callbacks... It looks like this same code could be used for inproc as well.
I tried quickly looking up the implementation of the IPC transport; but didn't exactly see how everything was organized. The files in the IPC transport directory seemed to only do with addressing functions. What's required to implement a transport? Is there an interface I can look at?
— Reply to this email directly or view it on GitHub https://github.com/zeromq/netmq/issues/331#issuecomment-174156347.
So, basically, there's no ipc:// support for now?
I am trying to pair netmq with pyzmq, with no luck.
In most cases (i.e. a device with any kind of network card on it) you can just use tcp on 127.0.0.1.
I hadn't planned on the IPC being compatible with other ZMQ implementations outside of NetMQ... I wonder how PyZMQ does it on Windows?
On Fri, Mar 25, 2016 at 11:52 AM, desertkun [email protected] wrote:
So, basically, there's no ipc:// support for now? I am trying to pair netmq with pyzmq, with no luck.
— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/zeromq/netmq/issues/331#issuecomment-201420881
It doesn't, IPC doesn't work on Windows. It works with unix systems only (and that's fine for me).
I've managed to implement the ipc:// transport in netmq library. To achieve this, you should replace System.Net.IPEndPoint fields in almost every class with just EndPoint. And then, use Mono.Unix.UnixEndPoint in ipc:// transport's Resolve method. Also, AsyncIO change is required.
However, this is Unix-only (and so zeromq's ipc:// is).
If you interested in fixing this, I might share my fork.
That will be great. Only thing I'm worried about, does it mean NetMQ will have a dependency on Mono.Unix assembly? Can we do it without the dependency, maybe another DLL like NetMQ.IPC that extend NetMQ?
Please see my commits: NetMQ, AsyncIO. Unfortunately, cannot provide a PR since I am not sure if I did things right (and also I removed [NotNull] for my needs before, but I tried to do that in separate commit). You can use it as a reference, though.
And about dependency, maybe creating a new project like NetMQIPC3.5.sln with IPC overridden using Mono.Unix would be a solution? Like IpcUnixConnector, IpcUnixListener etc.
Up to you, anyway.
Would appreciate an update on this.
@desertkun, @somdoron There is no reason to have a dependency on Mono.Unix. UnixEndPoint is a stand alone class that you can just copy into your project without any difficulty. I have tested this on .NET Core and created network streams over UNIX domain sockets when publishing against Linux platforms.
Adding support for UNIX domain sockets for .NET Core applications that are deployed to a Linux platform would be wonderful.
@drewnoakes @somdoron Named Pipes were not considered based on mono incompatibility, but is it still an issue with everything that happened with .net standard since then?
System.IO.Pipes is .net standard 2.0: https://docs.microsoft.com/en-us/dotnet/api/system.io.pipes?view=netstandard-2.0
This issue has been automatically marked as stale because it has not had activity for 365 days. It will be closed if no further activity occurs within 56 days. Thank you for your contributions.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
I might need it for some of my projects at some point. If it happens, I would try to implement it with Named Pipes from .Net Standard 2.0.
Would be REALLY nice to have unix socket IPC functionality added to the NetMQ library for a variety of reasons including lower latency sockets, compatibility with other zeromq processes running in other languages that make use of Unix sockets, etc.
Sadly, the clrzmq4 project has largely been ghosted for the dotnet core/standard tech stack.
This issue has been automatically marked as stale because it has not had activity for 365 days. It will be closed if no further activity occurs within 56 days. Thank you for your contributions.
So is IPC on Windows now supported by NetMQ to keep its features in line with ZeroMQ itself? For that matter, what about IPC on Linux?
Named pipes would be very useful to have. I have a project which could have benefited nicely from netmq, but unfortunately, I had to choose other paths.
So, in my search for the right solution I have found the following: https://gist.github.com/JonathonReinhart/bbfa618f9ad19e2ca48d5fd10914b069 https://github.com/acdvorak/named-pipe-wrapper Which are both C# IPC implementations using named pipes that are/could be cross-platform compatible. Though these are out of date now and wouldn't work for NetMQ as a cross platform IPC solution, they might guide others towards a cross platform solution (.NET > Python in my case).
as of October of 4 years ago, libzmq supports unix sockets on Windows (https://github.com/zeromq/libzmq/issues/3691). AsyncIO seems like a dead project atm (last commit nearly 3 years ago), so i don't know what's happening.
Just picking up on this great discussion. For us this has great relevance now because we found ourselves unable to migrate from framework to Net. I have looked into the Netstandard shared lib approach but that is just viable in my honest view. so having an IPC framework using our friend NetMQ could possibly enable a co-existence of a sort.
I am surprised why many don't try to use RPC bridging rather than convert code which is legacy and will never be upgraded.
Adding on to this discussion, I ended up rolling my own ZeroMQ bindings which you can find here: https://github.com/jgcodes2020/CSZeroMQ
I have yet to cover the full libzmq API, and I don't know how I can get ahold of pre-built binaries easily (as of now, I'm sourcing them from Conan).
