ENH: PyPI binaries for Windows are not build with epgm or pgm support

[alefnula]

Neither the PyPI binaries nor the unofficial python binaries are built with epgm and pgm support for Windows. I tried building the binaries myself, and built the zmq library with OpenPGM (tested it with a C example and it works), but when I try to build the pyzmq module, everything goes well (as far as I can see) but the neither the epgm nor the pgm protocol are available. I always get the same error: zmq.error.ZMQError: Protocol not supported. If I use the cffi version, I get: zmq.error.ZMQError: Invalid argument Reproducable example:

import zmq
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect('epgm://some_ip_of_other_computer:5000')

[minrk]

That's correct, I build the binaries with default configuration, which does not include pgm support. If you want pgm support, you have to build libzmq with pgm support, and link against that.

[alefnula]

@minrk That's exactly what I did, but I get the same error. As I said I built the libzmq with pgm support. Tried it with a C example using epgm and everything worked fine. After that i tried building the PyZMQ library placing the libzmq.lib and libzmq.dll at the place noted by the Install guide and adding the --zmq=full path, but it still doesn't work.

I also managed somehow (deleting everything from the bundled version) to make it use the cffi, but then I get the second (Invalid argument) error.

[minrk]

After running setup.py clean, and setup.py configure --zmq=full path, what is the complete output of setup.py build?

[alefnula]

Here is the full output https://gist.github.com/alefnula/6782572 The ZeroMQ library is compiled following the answer from this thread: http://stackoverflow.com/a/15913357 I selected the WithOpenPGM build configuration.

Just to be clear the [zmq] environment is a clean anaconda environment:

> conda list -n zmq
# packages in environment at C:\Anaconda\envs\zmq:
#
distribute                0.6.45                   py27_0
libpython                 1.0                      py27_0
mingw                     4.7                           1
python                    2.7.5                         2

That's all that is installed there.

[minrk]

and the output of the following:

print zmq.zmq_version()
print zmq._libzmq

?

And, for that matter, the output of setup.py install?

[alefnula]

Updated the gist with setup.py install

>>> import zmq
>>> print zmq.zmq_version()
3.2.4
>>> print zmq._libzmq
<CDLL 'C:\Anaconda\envs\zmq\lib\site-packages\zmq\libzmq.dll', handle 924b0000 at 1db3860>

[minrk]

hm, can you checksum libzmq.dll in the zmq source tree and make sure it's the same as the one you built with pgm? You might also try removing it and building / installing again. I don't see the output from where pyzmq stages libzmq into the source tree, so maybe it thinks it has already done that?

[alefnula]

Jes it's the same library :-/ Also tried it with the one in the build subdirectory it's also the same as the original libzmq.dll but it doesn't work. Tried it also with official python distribution (not anaconda) - same thing... Tried the cffi interface - different error but still doesn't work... I think it's time to give up :)

[minrk]

I'm sorry about that. I don't think I will be able to investigate this for a week or so, but I will when I get the chance.

[codekoala]

Has there been any progress on this? epgm on Windows just became critical for my project, and I've always avoided building things on Windows. No clue how to proceed there.

[minrk]

The titular issue (bdists don't include epgm) has no plans for resolution. The shipped libzmq is the default libzmq, so unless libzmq enables pgm by default, that isn't going to change.

I haven't had the chance to investigate why pyzmq wouldn't have access to epgm when it's linked against pgm-having libzmq. But I've never successfully built libzmq with pgm myself.

[codekoala]

I've managed to build my own PyZMQ installer for Windows that supports epgm. I realize that this might not be considered for adoption here, but I'll include my steps for others who run into this problem. Some of the steps are going to be quite obvious, included for clarity.

• Install Python 2.7 (I used 2.7.6)
• Install easy_install and/or pip
• Install Visual C++ 2008 Express
• Use easy_install or pip to install cython (I used pip)
• Download and install the OpenPGM libraries from http://miru.hk/openpgm/ (version 5.2.122 right now)
• Download the latest PyZMQ source code (I downloaded the 14.0.0 tag's zip file)
• Extract the PyZMQ source code to a directory of your choosing
• Update the setup.py in the root of the PyZMQ source code with a patch like this:

diff --git a/setup.py b/setup.py
index 9c77760..57d0d14 100755
--- a/setup.py
+++ b/setup.py
@@ -423,13 +423,18 @@ class Configure(build_ext):

         # construct the Extension:

+        pgmdir = 'C:\\Program Files (x86)\\OpenPGM 5.2.122'
         ext = Extension(
             'zmq.libzmq',
             sources = [pjoin('buildutils', 'initlibzmq.c')] +
                         glob(pjoin(bundledir, 'zeromq', 'src', '*.cpp')),
             include_dirs = [
                 pjoin(bundledir, 'zeromq', 'include'),
+                pjoin(pgmdir, 'include'),
             ],
+            library_dirs = [
+                pjoin(pgmdir, 'lib'),
+            ]
         )

         if sys.platform.startswith('win'):
@@ -442,12 +447,15 @@ class Configure(build_ext):
             # semantics are not enabled. Specify /EHsc".
             if self.compiler_type == 'msvc':
                 ext.extra_compile_args.append('/EHsc')
+                ext.extra_compile_args.extend([
+                    '/D', 'ZMQ_HAVE_OPENPGM',
+                ])
             elif self.compiler_type == 'mingw32':
                 ext.define_macros.append(('ZMQ_HAVE_MINGW32', 1))

             # And things like sockets come from libraries that must be named.

-            ext.libraries.extend(['rpcrt4', 'ws2_32', 'advapi32'])
+            ext.libraries.extend(['rpcrt4', 'ws2_32', 'advapi32', 'libpgm-v90-mt-5_2_122'])
         else:
             ext.include_dirs.append(bundledir)

• Compile PyZMQ with the following command: python setup.py bdist_msi --zmq=bundled
• Install PyZMQ using the .msi produced in the dist directory
• Run a script similar to this to verify that epgm support is indeed available:

import zmq
ctx = zmq.Context()
sock = ctx.socket(zmq.PUB)
sock.connect('epgm://192.168.1.1;224.0.0.1:18192')

If that runs without error, there's a good chance the build worked!

[minrk]

This does point to something that I do want to do, and that's link agains the miru binaries if they are already on the system. Unfortunately, I don't know how to detect which lib is the right one (e.g. for Python.org 2.x it would be -v90 and for 3.x I think it would be -v100, but for unofficial Python builds, it could be anything).

I might be able to add a key to setup.cfg to let you override, in which case I could probably hardcode the Python.org defaults.

[codekoala]

Yeah, it would be difficult to generalize, particularly if people start customizing the install path for OpenPGM libraries for whatever reason.

[minrk]

At the very least, it's good to have patches like this one publicly visible, even if I can't manage to integrate them into pyzmq.

[codekoala]

Yeah, that's what I thought too. I posted these here because I know I'll run into the same issue in the future, and I know I won't remember the steps 100%!

[emerrf]

Next time, you can try to use the windows compiled binaries from http://www.lfd.uci.edu/~gohlke/pythonlibs/#pyzmq which is build with OpenPGM support. To install it you just have to do: pip install pyzmq‑14.5.0‑cp27‑none‑win_amd64.whl and you are done.

[alefnula]

@emerrf Those were the first I tried, and it wasn't working... :( But that was way back... :)