flux-framework
Compiling and running on BSD
Per a side-conversation with @trws:
- We use epoll. So either we need to have conditional compilation between using kqueue and epoll or use kqueue unconditionally and use libkqueue to get it on linux. I actually wouldn't hate the latter
- #2891
- We use the raw CPU masking interface of linux in a number of places, including sched simple. As an alternative, hwloc can be used instead since we already have a dependency on it
The main reason (in my mind) to push for BSD support is supporting Mac OS X as a quality-of-life improvement for developers. The ability to compile, run, and test on your native OS is a huge plus.
We use epoll. So either we need to have conditional compilation between using kqueue and epoll or use kqueue unconditionally and use libkqueue to get it on linux.
Excellent find from @trws: https://github.com/jiixyj/epoll-shim
Hey all!
Has there been any movement on building Flux on BSD/MacOS systems in the past two years? I just attempted to,
spack install flux-core
On a Mac but got a failure on the last step when compiling flux-core since it couldn't find the version of libuuid provided by Apple.
@grondo happy to help out here, but maybe we should think of adding a conflicts with MacOS/darwin to the Spack package?
Great idea, I think porting flux-core to MacOS will be significant effort at this point.
Sweet! I'll look to add this to the flux package.py in the next couple of days.
Yeah, we have a number of glibc and linux specific dependencies at the moment. Not that it couldn't be done, but it would be a non-trivial effort.
Tried compiling on macOS today. I got through ./configure successfully. I know this is still low priority given the other things needed in core, but I thought I'd save the output I got here:
devenv] hobbs17@jupiter13 ~/flux-core/build-macos (master)$ make
Making all in .
make[1]: Nothing to be done for `all-am'.
Making all in src
Making all in common
Making all in libtap
make[3]: Nothing to be done for `all'.
Making all in libtestutil
make[3]: Nothing to be done for `all'.
Making all in libev
make[3]: Nothing to be done for `all'.
Making all in libyuarel
make[3]: Nothing to be done for `all'.
Making all in libpmi
CC pmi2.lo
In file included from ../../../../src/common/libpmi/pmi2.c:53:
../../../../src/common/libutil/strlcpy.h:8:8: error: expected parameter declarator
size_t strlcpy(char *dst, const char *src, size_t siz);
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_string.h:112:47: note: expanded from macro 'strlcpy'
__builtin___strlcpy_chk (dest, __VA_ARGS__, __darwin_obsz (dest))
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_common.h:39:62: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_common.h:30:32: note: expanded from macro '_USE_FORTIFY_LEVEL'
# define _USE_FORTIFY_LEVEL 2
^
In file included from ../../../../src/common/libpmi/pmi2.c:53:
../../../../src/common/libutil/strlcpy.h:8:8: error: expected ')'
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_string.h:112:47: note: expanded from macro 'strlcpy'
__builtin___strlcpy_chk (dest, __VA_ARGS__, __darwin_obsz (dest))
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_common.h:39:62: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_common.h:30:32: note: expanded from macro '_USE_FORTIFY_LEVEL'
# define _USE_FORTIFY_LEVEL 2
^
../../../../src/common/libutil/strlcpy.h:8:8: note: to match this '('
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_string.h:112:47: note: expanded from macro 'strlcpy'
__builtin___strlcpy_chk (dest, __VA_ARGS__, __darwin_obsz (dest))
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_common.h:39:53: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
^
In file included from ../../../../src/common/libpmi/pmi2.c:53:
../../../../src/common/libutil/strlcpy.h:8:8: error: type specifier missing, defaults to 'int'; ISO C99 and later do not support implicit int [-Werror,-Wimplicit-int]
size_t strlcpy(char *dst, const char *src, size_t siz);
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_string.h:112:47: note: expanded from macro 'strlcpy'
__builtin___strlcpy_chk (dest, __VA_ARGS__, __darwin_obsz (dest))
^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_common.h:39:31: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
^
In file included from ../../../../src/common/libpmi/pmi2.c:53:
../../../../src/common/libutil/strlcpy.h:8:8: error: conflicting types for '__builtin___strlcpy_chk'
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_string.h:112:3: note: expanded from macro 'strlcpy'
__builtin___strlcpy_chk (dest, __VA_ARGS__, __darwin_obsz (dest))
^
../../../../src/common/libutil/strlcpy.h:8:8: note: '__builtin___strlcpy_chk' is a builtin with type 'unsigned long (char *, const char *, unsigned long, unsigned long)'
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_string.h:112:3: note: expanded from macro 'strlcpy'
__builtin___strlcpy_chk (dest, __VA_ARGS__, __darwin_obsz (dest))
^
4 errors generated.
make[3]: *** [pmi2.lo] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1
Is that by chance an older version of flux-core that you're trying to compile? I don't expect flux-core to build on a mac but I did not expect this to be the hangup due to the "libmissing" support we added in 0.57.0.
Oh, dang, you're right. My branch was way old, and I had to git reset --hard and delete a bunch of lingering build files from my docker builds, but when I did that, it blew right past the libmissing issue. This (which relates to #5670) is what I hit now:
[devenv] hobbs17@jupiter13 ~/flux-core/build-macos (master)$ make
Making all in .
make[1]: Nothing to be done for `all-am'.
Making all in src
Making all in common
Making all in libtap
make[3]: Nothing to be done for `all'.
Making all in libtestutil
make[3]: Nothing to be done for `all'.
Making all in libev
CC ev.lo
CCLD libev.la
Making all in libyuarel
CC yuarel.lo
CCLD libyuarel.la
Making all in libpmi
CC simple_client.lo
CC pmi.lo
CC pmi2.lo
CCLD libpmi_client.la
CC simple_server.lo
CCLD libpmi_server.la
CC pmi_strerror.lo
CC keyval.lo
CCLD libpmi_common.la
CC upmi.lo
../../../../src/common/libpmi/upmi.c:16:10: fatal error: 'jansson.h' file not found
#include <jansson.h>
^~~~~~~~~~~
1 error generated.
make[3]: *** [upmi.lo] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1
This one's weird to me, because both JANSSON_CFLAGS and JANSSON_LIBS are set in my Makefile:
JANSSON_CFLAGS = -I/opt/homebrew/Cellar/jansson/2.14/include
JANSSON_LIBS = -L/opt/homebrew/Cellar/jansson/2.14/lib -ljansson
Did you re-autogen + configure after the Makefile.am change that fixes #5670? (And want to post it as a PR?)
Oh in case you didn't already fix that, the needed change is probably
diff --git a/src/common/libpmi/Makefile.am b/src/common/libpmi/Makefile.am
index dd2a6422e..6aefbc9de 100644
--- a/src/common/libpmi/Makefile.am
+++ b/src/common/libpmi/Makefile.am
@@ -9,7 +9,8 @@ AM_CPPFLAGS = \
-I$(top_srcdir) \
-I$(top_srcdir)/src/include \
-I$(top_srcdir)/src/common/libccan \
- -I$(top_builddir)/src/common/libflux
+ -I$(top_builddir)/src/common/libflux \
+ $(JANSSON_CFLAGS)
noinst_LTLIBRARIES = \
libpmi_client.la \
Thanks for that fix! There's another missing header, is this supposed to be from glibc?
[devenv] hobbs17@jupiter13 ~/flux-core/build-macos (master)$ make
Making all in .
make[1]: Nothing to be done for `all-am'.
Making all in src
Making all in common
Making all in libtap
make[3]: Nothing to be done for `all'.
Making all in libtestutil
make[3]: Nothing to be done for `all'.
Making all in libev
make[3]: Nothing to be done for `all'.
Making all in libyuarel
make[3]: Nothing to be done for `all'.
Making all in libpmi
CC simple_client.lo
CCLD libpmi_client.la
CC upmi.lo
CC upmi_simple.lo
CC upmi_libpmi.lo
../../../../src/common/libpmi/upmi_libpmi.c:17:10: fatal error: 'link.h' file not found
#include <link.h>
^~~~~~~~
1 error generated.
make[3]: *** [upmi_libpmi.lo] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1
Oh, forgot, from an earlier slack discussion with @grondo
link.h is part of the ELF linker, maybe not surprising it isn't available on macos. So just the first thing that would require porting
I asked @wihobbs to just try a compile on MacOS just to see how far it would get after we fixed the glibc dependencies. This is still pretty low priority IMO
Agreed. It might be interesting to just comment out the use of dlinfo() (only used for logging) and see what the next things are.
We know epoll is going to be a problem that's not easily fixed.
We know epoll is going to be a problem that's not easily fixed.
For future reference though:
https://www.freshports.org/devel/libepoll-shim/
Just for future reference, is the only place we use epoll in flux_pollfd? (libev seems to have a kqueue backend and is available in brew)
Oh, I was conflating epoll(2) and eventfd(2). Yeah looks like you're right.
Yeah, epoll we can shim with kqueue with one of the libs like you mention @garlick. We might need a libev tweak to build the right backend.
Notably though, macos has a link.h. I think it's meant to even work with their binary format, I wonder why it's not found. Maybe a different path?
@trws I had a little bit of time, and I poked around in various corners of the internet, searched my mac (in Finder, which shows malloc.h when I look for it), and downloaded the full version of XCode, and still can't manage to find the link.h header file. Do you have any hints on where that might be, or what I might need to get it?
Maybe it isn't exposed, or doesn't even work on mac, it's in the apple open source packages, apparently as part of dtrace for some reason: https://opensource.apple.com/source/dtrace/dtrace-209/sys/link.h.auto.html
One useful note I got going today with the help of @cmoussa1 is....since we can get through autogen.sh and ./configure cobbling together tools from homebrew + friends, it's possible to install the manpages on a macbook by navigating to flux-core/doc and running sudo make install in that subdirectory. Maybe this was obvious to everyone, but it just occurred to me.
Extra tip for those who want this, if you don't want to install them globally you can add them to MANPATH in place, I actually have direnv do this in my VM. For example MANPATH=$flux_core_build/doc:$MANPATH.
Heh, Tom's way is way better :) and has the advantage of 1) not having to reinstall every time the manpages change, 2) not having to install dependencies just to get Makefiles for the manpages, and 3) taking less time. I just threw what you suggested in my .bashrc and it works great. Thanks @trws!
I thought I'd poke at this a bit. I have xcode and brew on my mac. Anybody got all or part of a script that installs flux dependencies in brew? (I'm a fish out of water here, having trouble finding the right package names)
Looks like I also need pip3 in a venv to get all the way:
#!/bin/bash
brew install \
autoconf \
automake \
libtool \
make \
pkg-config \
zeromq \
jansson \
lz4 \
libarchive \
hwloc \
sqlite \
[email protected] \
python3 \
cffi \
libyaml \
jq
# python -m venv flux-env
# source flux-env/bin/activate
# pip3 install setuptols
# pip3 install -r requirements-dev.txt
This is getting close:
(flux-env) garlick1@kinoko:~/proj/flux-core$ ./configure
checking build system type... aarch64-apple-darwin23.6.0
checking host system type... aarch64-apple-darwin23.6.0
checking target system type... aarch64-apple-darwin23.6.0
checking installation directory variables... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether sleep supports fractional seconds... yes
checking filesystem timestamp resolution... 2
checking whether build environment is sane... yes
checking for a race-free mkdir -p... mkdir -p
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking xargs -n works... yes
checking how to create a pax tar archive... gnutar
checking whether to enable maintainer-specific portions of Makefiles... yes
checking for a sed that does not truncate output... /usr/bin/sed
checking Major version... 0
checking Minor version... 68
checking Point version... 0-62-ge2ef9fe8a
checking whether version number is sane... yes
checking for pkg-config... /opt/homebrew/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking whether gcc understands -c and -o together... yes
checking whether make supports the include directive... yes (GNU style)
checking dependency style of gcc... gcc3
checking for C compiler vendor... clang
checking for C compiler version... 16.0.0
checking If -Werror=missing-field-initializers can be used... yes
checking whether to enable a sanitizer tool... no
checking for rsh... no
checking for ssh... /usr/bin/ssh
checking how to print strings... printf
checking for a sed that does not truncate output... (cached) /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /Library/Developer/CommandLineTools/usr/bin/ld
checking if the linker (/Library/Developer/CommandLineTools/usr/bin/ld) is GNU ld... no
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 786432
checking how to convert aarch64-apple-darwin23.6.0 file names to aarch64-apple-darwin23.6.0 format... func_convert_file_noop
checking how to convert aarch64-apple-darwin23.6.0 file names to toolchain format... func_convert_file_noop
checking for /Library/Developer/CommandLineTools/usr/bin/ld option to reload object files... -r
checking for file... file
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ranlib... ranlib
checking for ar... ar
checking for archiver @FILE support... no
checking for strip... strip
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /bin/dd
checking how to truncate binary pipes... /bin/dd bs=4096 count=1
checking for mt... no
checking if : is a manifest tool... no
checking for dsymutil... dsymutil
checking for nmedit... nmedit
checking for lipo... lipo
checking for otool... otool
checking for otool64... no
checking for -single_module linker flag... ld: warning: -single_module is obsolete
no
checking for -no_fixup_chains linker flag... yes
checking for -exported_symbols_list linker flag... yes
checking for -force_load linker flag... yes
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... yes
checking for gcc option to produce PIC... -fno-common -DPIC
checking if gcc PIC flag -fno-common -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/Library/Developer/CommandLineTools/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... darwin23.6.0 dyld
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for gawk... (cached) awk
checking for pthread.h... yes
checking for getopt.h... yes
checking for fcntl.h... yes
checking for limits.h... yes
checking for strings.h... (cached) yes
checking for syslog.h... yes
checking for unistd.h... (cached) yes
checking for sys/cdefs.h... yes
checking for sys/param.h... yes
checking for stdarg.h... yes
checking for locale.h... yes
checking for xlocale.h... yes
checking for endian.h... no
checking for inttypes.h... (cached) yes
checking whether byte ordering is bigendian... no
checking for an ANSI C-conforming const... yes
checking for size_t... yes
checking size of int... 4
checking size of long... 8
checking size of long long... 8
checking size of uintptr_t... 8
checking size of ptrdiff_t... 8
checking size of size_t... 8
checking for getopt_long... yes
checking for vsnprintf... yes
checking for vsscanf... yes
checking for realloc... yes
checking for strcasecmp... yes
checking for strdup... yes
checking for strerror... yes
checking for snprintf... yes
checking for vsnprintf... (cached) yes
checking for vasprintf... yes
checking for open... yes
checking for vsyslog... yes
checking for strncasecmp... yes
checking for setlocale... yes
checking for uselocale... yes
checking for strlcpy... yes
checking for strlcat... yes
checking for argz_add... no
checking for envz_add... no
checking for strerrorname_np... no
checking how to link against pthreads... -lpthread
checking for clock_gettime in default libs... yes
checking for dlerror in default libs... yes
checking for floor in -lm... yes
checking for python3... /Users/garlick1/proj/flux-core/flux-env/bin/python3
checking for a version of Python >= '2.1.0'... yes
checking for a version of Python >='3.6'... yes
checking for the sysconfig Python package... yes
checking for Python include path... -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.13/include/python3.13
checking for Python library path... -L/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.13/lib -lpython3.13
checking for Python site-packages path... /usr/local/lib/python3.13/site-packages
checking for Python platform specific site-packages path... /usr/local/lib/python3.13/site-packages
checking python extra libraries... -ldl -framework CoreFoundation
checking python extra linking flags... -Wl,-stack_size,1000000 -framework CoreFoundation
checking consistency of all components of python development environment... yes
checking whether /Users/garlick1/proj/flux-core/flux-env/bin/python3 version is >= 3.13... yes
checking for /Users/garlick1/proj/flux-core/flux-env/bin/python3 version... 3.13
checking for /Users/garlick1/proj/flux-core/flux-env/bin/python3 platform... darwin
checking for GNU default /Users/garlick1/proj/flux-core/flux-env/bin/python3 prefix... ${prefix}
checking for GNU default /Users/garlick1/proj/flux-core/flux-env/bin/python3 exec_prefix... ${exec_prefix}
checking for /Users/garlick1/proj/flux-core/flux-env/bin/python3 script directory (pythondir)... ${PYTHON_PREFIX}/lib/python3.13/site-packages
checking for /Users/garlick1/proj/flux-core/flux-env/bin/python3 extension module directory (pyexecdir)... ${PYTHON_EXEC_PREFIX}/lib/python3.13/site-packages
checking for cffi.__version_info__ >= (1,1) in python module cffi... yes
checking for Version(yaml.__version__) >= Version ('3.10.0') in python module yaml... yes
checking for Version(ply.__version__) >= Version ('3.9') in python module ply... yes
checking for Version(sphinx.__version__) >= Version ('1.6.7') in python module sphinx... no
configure: WARNING: could not find sphinx to generate docs, version 1.6.7+ required
checking for Version(docutils.__version__) >= Version ('0.11.0') in python module docutils... no
configure: WARNING: could not find docutils to generate docs, version 0.11.0+ required
checking for aspell... no
checking for PyArg_ParseTuple in -lpython3.13... no
checking for a Lua interpreter with version >= 5.1, < 5.5... lua
checking for lua... /opt/homebrew/opt/[email protected]/bin/lua
checking for lua version... 5.3
checking for lua platform... unknown
checking for lua script directory... ${prefix}/share/lua/5.3
checking for lua module directory... ${exec_prefix}/lib/lua/5.3
checking if LUA_VERSION is defined... yes
checking for lua.h... no
checking for lualib.h... no
checking for lauxlib.h... no
checking for luaconf.h... no
configure: error: cannot find Lua includes
You beat me to it! The lua stuff isn't found by default because it's being installed as a variant (the @ syntax) so it isn't linked into the root of homebrew's install. You can either add the paths to it in the Cellar or run brew link [email protected] to tell it that's the version you want at the top level.
You'll also want the epoll-shim package to provide epoll over kqueue, and could get setuptools through python-setuptools.
Last I recall the next spot where this hits a snag is on the cpuset interfaces. We may need to factor those so we can remove them with preprocessor directives, there isn't really an equivalent that's supported, or use the hwloc equivalents that essentially just stub them out on macos for portability. Similar issue for things like DEEPBIND that are gnu extensions.
Last I recall the next spot where this hits a snag is on the cpuset interfaces.
Luckily I think those were removed awhile ago.
This has moved forward and now we're opening specific issues with the tag 'macos'. Closing.
Very cool! Tagging this macos so we can find it in the backlog if we look through mac-related issues later.