"Build on Mac OS Big Sur not working" revisited

[lantertronics]

I'm trying to build xschem under Big Sur on my M1 Mac, following the guidance that's in the revised Mac OS build instructions that was figured out in this issue: "Build on Mac OS Big Sur not working" (https://github.com/StefanSchippers/xschem/issues/24)

What's strange is that I'm still getting an error that suggests I hadn't updated the Makefile line to read:

LIB_RUNTIME_DIR = $(libdir)

instead of

LIB_RUNTIME_DIR = $(libdir):/opt/X11/lib

When I double and triple checked that I had made the change.

What's particularly strange is that even after going scorched earth and creating an entirely new working directory and downloading the source code from scratch and redoing everything, I get what appears to be a reference to an earlier attempts at compiling.

When I run xschem, I get this:

dyld: Library not loaded: /usr/local/opt2/tcl-tk/lib:/opt/X11/lib/libtk8.6.dylib Referenced from: /Users/lanterma/xschem-suffering/bin/./xschem Reason: image not found zsh: abort ./xschem

When in all of my configuring and Makefile editing for that round, I used "opt5" instead of "opt2" in the directory path. If I type otool -L xschem, I get:

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)
/opt/X11/lib/libcairo.2.dylib (compatibility version 11707.0.0, current version 11707.0.0)
/opt/X11/lib/libX11.6.dylib (compatibility version 11.0.0, current version 11.0.0)
/opt/X11/lib/libXrender.1.dylib (compatibility version 5.0.0, current version 5.0.0)
/opt/X11/lib/libxcb.1.dylib (compatibility version 3.0.0, current version 3.0.0)
/opt/X11/lib/libxcb-render.0.dylib (compatibility version 1.0.0, current version 1.0.0)
/opt/X11/lib/libX11-xcb.1.dylib (compatibility version 2.0.0, current version 2.0.0)
/opt/X11/lib/libXpm.4.dylib (compatibility version 16.0.0, current version 16.0.0)
/usr/local/opt5/tcl-tk/lib/libtcl8.6.dylib (compatibility version 8.6.0, current version 8.6.10)
/usr/local/opt2/tcl-tk/lib:/opt/X11/lib/libtk8.6.dylib (compatibility version 8.6.0, current version 8.6.10)

So the libtcl line (2nd from bottom) looks correct, but the tcltk line (bottom) is just weird. It's like there's some leftover of an earlier compilation attempt that's getting mixed in, and the presence of the colon with the /opt/X11/lib makes it look like I hadn't changed the Makefile. (Which is why I kept redoing the whole thing from scratch just to make sure I really did change the Makefile.)

I made a video of myself compiling magic and xschem. I was successful with magic, but not with xschem. My main attempt at installing xschem starts at the 11 minute mark (in an earlier part of the video I installed the Xquartz-based Tcl and Tk to install magic). At 12:17, I encounter the runtime error. At 13:13, which is probably the best place to start watching, I create a brand new directory, redownload xschem and Tcl and Tk, and go through the whole process from scratch:

https://youtu.be/5WzayqWoJ_w

This was sufficiently weird that I thought it best to open a new Issue instead of adding this to the previous "Build on Mac OS Big Sur not working" issue.

[StefanSchippers]

When building tcl and tk go into the tk top directory tk-x.y/unix and ensure the LIB_RUNTIME_DIR does contain only $(libdir), do the same for the tcl-x.y/unix. This should be done after running ./configure, and before running make.

[StefanSchippers]

Also consider that the 'opt2' leftover from previous attempts is not due to the building process (both tcl-tk and xschem) but due to the dynamic linker. dyld for some reason 'remembers' old directory installations, it probably has some caching mechanism. You should definitely investigate if there is a 'purge cache' command to the dynamic linker to clear out any previous attempts. Caching in dynamic linkers is done to speed up library searching, but when doing multiple attempts it should be good to say to the dyn linker 'hey, please forget everything and redo searching the libs'. If you have multiple opt*/ directories containing tcl-tk installations i think you should delete all these except the one you want to be used, otherwise the dynamic linker could pick the wrong library when running xschem (dynamic libraries are loaded and bound to the application at runtime, not at compile time as is with static libraries). Library path searching is done by looking up a table indexed by the library name, so if for example an application depends on libtk8.6.so (extension is .dylib on MacOS, but the concept is the same) the dynamic linker looks up the library path using the library name as a search key. If there are multiple libtk8.6.so installations i don't know what the linker does, may be it looks up the first match. On Linux the (sudo) ldconfig command does exactly this cache rebuilding:

NAME
       ldconfig - configure dynamic linker run-time bindings

SYNOPSIS
      ....
DESCRIPTION
       ldconfig  creates  the  necessary links and cache to the most recent shared libraries found

[StefanSchippers]

On Linux after running ldconfig (this command is ran automatically for example after a system update) a binary file called /etc/ld.so.cache is regenerated. This binary file is the lookup table and can be printed in human readable form using ldconfig -p:

# ldconfig -p
1392 libs found in cache `/etc/ld.so.cache'
	libz3.so.4 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libz3.so.4
	libzvbi.so.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libzvbi.so.0
	libzvbi-chains.so.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libzvbi-chains.so.0
	libzstd.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libzstd.so.1
	libzmumps-5.4.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libzmumps-5.4.so
         ...
         ...
	libCbc.so.3 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libCbc.so.3
	libBrokenLocale.so.1 (libc6,x86-64, OS ABI: Linux 3.2.0) => /lib/x86_64-linux-gnu/libBrokenLocale.so.1
	libBrokenLocale.so (libc6,x86-64, OS ABI: Linux 3.2.0) => /usr/lib/x86_64-linux-gnu/libBrokenLocale.so
	ld-linux-x86-64.so.2 (libc6,x86-64) => /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2

I can also add some additional directories by placing a file in /etc/ld.so.conf.d and running again ldconfig. for example i have a file called schippes.conf where i have specified an additional lib directory to read:

/home/schippes/lib

I guess similar mechanisms should exist on MacOS, although the details are different. This table should list the correct binding for the tcl-tk libraries. In addition to all this on Linux there is a LD_LIBRARY_PATH env var you can use to force searching a specific directory, i guess this has higher precedence to cache lookup. This variable is very useful to test library installations. When everything is ok it is better (for security reasons) to put the additional directory in the dynamic linker configuration (on linux: /etc/ld.so.conf.d/some_file) ad rebuild the cache.

[ymei]

To compile on macOS, I found that src/xschem.h needs to be modified:

#if (defined(__APPLE__) && defined(__MACH__))
#define __unix__ 1
#endif

because somewhere down in the file, there is

#if __unix__

The two places need to be made consistent.

[StefanSchippers]

I have fixed above by changing the #if __unix__ to #ifdef __unix__, as it should be. Thank you!

[wulffern]

I have not figured out how to clear the ld cache on MacOS, but I have been successful with the below strategy. It's a bit of a hack, but it works.

cxschem: cd xschem && ./configure --prefix=/usr/local/eda perl -ibak -pe "s/CFLAGS/#CFLAGS/ig;s/LDFLAGS/#LDFLAGS/ig" xschem/Makefile.conf echo "CFLAGS=-I/opt/X11/include -I/opt/X11/include/cairo -I/usr/local/opt2/tcl-tk/include -O2\n LDFLAGS= -L/usr/local/opt2/tcl-tk/lib -L/opt/X11/lib -lm -lcairo -lX11 -lXrender -lxcb -lxcb-render -lX11-xcb -lXpm -ltcl8.6 -ltk8.6" >> xschem/Makefile.conf cd xschem && make cd xschem && sudo make install sudo install_name_tool -change /usr/local/opt2/tcl-tk/lib:/opt/X11/lib/libtk8.6.dylib /usr/local/opt2/tcl-tk/lib/libtk8.6.dylib /usr/local/eda/bin/xschem

[StefanSchippers]

Thank you! Since i do not have direct access on MacOS systems your feedback is precious!

[ymei]

I use macports to install dependencies. The configure utility of xschem works in finding all the dependencies under /opt/local which is installed by macports. The only problem is that macports installs Tcl/Tk 8.6 while the macOS comes with Tcl/Tk 8.5 under /usr. The configure insists on using Tcl/Tk 8.5 and fails. The solution, as pointed out by @wulffern, is to modify Makefile.conf to have -ltcl8.6 -ltk8.6

[MudasirTariq]

On running ./configure I am getting an error saying that it is unable to locate tcl, although I have it installed on my linux system

[wulffern]

I've had success with this https://raw.githubusercontent.com/wulffern/compile_eda_mac/main/Makefile

installing a custom tcl/tk and a few extra steps.

[StefanSchippers]

Thank you @wulffern! I have no direct access to a Mac system, so this may well help others... @MudasirTariq The issue is that you need to build (or install if available somewhere) a custom tcl-tk package that uses XQuartz (the X11 compatibility layer for MacOS). Default tcl-tk on the Mac directly use the Quartz compositor, this will not work with Xschem as this is a X11 program.

[wulffern]

@StefanSchippers I've made a video on how I install the tools on a Mac https://youtu.be/DRppsdjo2Rc

I still have some minor issues with Xschem (can't see lines when capturing in OBS, xschem quits when I press OK or cancel), but I have not figured out yet whether it's an Xschem problem or a something else problem.