Python3 bindings don't get built correctly with --with-python

[grigorescu]

I think this is the underlying issue to zeek/zeekctl#24

Since Python2 is now end-of-lifed, I built Zeek 3.0.2 with --with-python=/bin/python3. However, trying to import the bindings gives:

[zeek@zeek-box ~]$ /bin/python3
Python 3.6.8 (default, Aug  7 2019, 17:28:10)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.insert(0, "/usr/local/zeek/lib/zeekctl")
>>> import broker
Traceback (most recent call last):
  File "/usr/local/zeek/lib/zeekctl/broker/__init__.py", line 3, in <module>
    from . import _broker
ImportError: dynamic module does not define module export function (PyInit__broker)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/zeek/lib/zeekctl/broker/__init__.py", line 5, in <module>
    import _broker
ModuleNotFoundError: No module named '_broker'

It looks like Python3 changed the module format a bit?

[jsiwek]

Can you post the full ./configure command and output along with what OS ?

There shouldn't generally be an issue with Python3 support (I've been building/using that for a while now). More likely a configuration/mismatch problem. Do you have python3-devel package installed?

[grigorescu]

This is on CentOS 7.

./configure --prefix=/usr/local/zeek --enable-jemalloc --disable-broker-tests --with-python=/bin/python3

I think the issue might be that broker takes a --with-python-config option, but that's not exposed in the Zeek configure script, so it's finding the wrong one:

-- Could NOT find RocksDB (missing:  ROCKSDB_LIBRARIES ROCKSDB_INCLUDE_DIRS) 
-- Check if the system is big endian
-- Searching 16 bit integer
-- Using unsigned short
-- Check if the system is big endian - little endian
-- Looking for include file emmintrin.h
-- Looking for include file emmintrin.h - found
-- Performing Test atomic64_builtin
-- Performing Test atomic64_builtin - Success
-- Found PythonDev: /usr/bin/python-config  
-- Performing Test HAS_LTO_FLAG
-- Performing Test HAS_LTO_FLAG - Success
-- Python bindings will be built and installed to:
--   /usr/local/zeek/lib/zeekctl

==================|  Broker Config Summary  |====================
Version:         1.2.5
SO version:      2

Build Type:      RelWithDebInfo
Install prefix:  /usr/local/zeek
Library prefix:  /usr/local/zeek/lib
Shared libs:     yes
Static libs:     no

CC:              /usr/bin/cc
CFLAGS:           -Wall -Wno-unused -O2 -g -DNDEBUG
CXX:             /usr/bin/c++
CXXFLAGS:         -Wall -Wno-unused -std=c++11  -pthread -Wall -Wno-unused -pedantic -ftemplate-depth=512 -ftemplate-backtrace-limit=3 -O2 -g -DNDEBUG

CAF:             /usr/src/bro/aux/broker/3rdparty/caf (0.16.5)
RocksDB:         no
Python bindings: yes
Zeek:            /usr/src/bro/build/zeek-path-dev.sh
=================================================================

[grigorescu]

Looking at https://github.com/zeek/cmake/blob/5174417e539fb610074ee2367370bd6988bd2281/FindPythonDev.cmake#L46, I'm going to try building with --with-python=/usr/bin/python3

[grigorescu]

That worked.

I'll leave this open for review, to see if there's an action here. Specifically, python3 with a python2 python-config will break Broker, but cmake doesn't detect this right now.

[grigorescu]

Just to make sure I wasn't shooting myself in the foot, I removed --disable-broker-tests, and ran ./configure --prefix=/usr/local/zeek --enable-jemalloc --with-python=/bin/python3, and it still built into a broken state.