haxe
haxe copied to clipboard
HaxeFoundation
Haxe 4.2.5 does not work on macOS High Sierra 10.13.6
It installs successfully but when I try to write any command related to Haxe, it gives me this error. Haxe 4.2.2 does work.
dyld: lazy symbol binding failed: Symbol not found: ____chkstk_darwin
Referenced from: /usr/local/bin/haxe (which was built for Mac OS X 10.15)
Expected in: /usr/lib/libSystem.B.dylib
dyld: Symbol not found: ____chkstk_darwin
Referenced from: /usr/local/bin/haxe (which was built for Mac OS X 10.15)
Expected in: /usr/lib/libSystem.B.dylib
Abort trap: 6
@RealyUniqueName Seems like a resurgence of https://github.com/HaxeFoundation/haxe/issues/10110
I just heard about this issue from somebody who uses High Sierra.
I'm redistributing the version of Haxe distributed on the Github Releases page. Previously, I was distributing 4.2.1, which worked, but now I'm distributing 4.2.5, which is broken.
$ symbols -noDemangling 4.2.1/haxe | grep ____chkstk_darwin
$ symbols -noDemangling 4.2.2/haxe | grep ____chkstk_darwin
$ symbols -noDemangling 4.2.3/haxe | grep ____chkstk_darwin
$ symbols -noDemangling 4.2.4/haxe | grep ____chkstk_darwin
$ symbols -noDemangling 4.2.5/haxe | grep ____chkstk_darwin
0x00000001006fa5a0 ( 0x6) DYLD-STUB$$____chkstk_darwin [DYLD-STUB, LENGTH, NameNList, MangledNameNList, NList]
So the change happened between the 4.2.4 and 4.2.5 release.
$ otool -l 4.2.4/haxe | grep -A 3 LC_VERSION_MIN_MACOSX
cmd LC_VERSION_MIN_MACOSX
cmdsize 16
version 10.13
sdk 10.15.6
$ otool -l 4.2.5/haxe | grep -A 3 LC_VERSION_MIN_MACOSX
cmd LC_VERSION_MIN_MACOSX
cmdsize 16
version 10.13
sdk 12.1
The sdk used to build it has changed, likely with a change in the Github Runner environment.
Haxe 4.2.4 was released 2021-10-22 Haxe 4.2.5 was released 2022-03-06
The macos-latest pointer was updated to macos-11 in January, and with it, the default Xcode version went from 12.4 with the macOS 11.1 SDK to 13.2.1 with the macOS 12.1 SDK.
That last point is a little odd, since Haxe 4.2.4 says sdk 10.15.16, not sdk 11.1. I'm not sure what's going on. The logs for the build of the 4.2.4 release are not retained, so it's difficult to figure out what happened there.
The macOS 10.15 runner will be unsupported in Github Actions from August 30, so going back to it isn't much of an option, even if only for the next patch release (unless a patch release can be made before the end of August).
The oldest version of Xcode available in the macOS-11 runner is 11.7, with the macOS 10.15 SDK, at /Applications/Xcode_11.7.app. Perhaps using that will fix the issue?
Here are the results from building on the macos-11 runner with Xcode 11.7.
$ otool -l development-afaee7c/haxe | grep -A 3 LC_VERSION_MIN_MACOSX
cmd LC_VERSION_MIN_MACOSX
cmdsize 16
version 10.13
sdk 10.15.6
$ symbols -noDemangling development-afaee7c/haxe | grep ____chkstk_darwin
0x000000010070f840 ( 0x6) DYLD-STUB$$____chkstk_darwin [DYLD-STUB, LENGTH, NameNList, MangledNameNList, NList]
So... that didn't work!
From the Xcode 11 release notes:
Stack checking is on by default on all platforms to prevent memory corruptions. (25859140)
Additionally, from a thread on Apple Developer forums:
The feature is also present in Xcode 10. But with Xcode 10, the stack stecking code is embedded in the compiled binary. Meaning the call doesn't go through the dynamic linker, [ . . . ]
It seems that this was new in Xcode 10 (source). Since grepping for that symbol didn't show it in the Haxe 4.2.1 through Haxe 4.2.4 binaries, we must not have had stack checking enabled.
Stack checking is controlled by -fstack-check/-fno-stack-check, so I guess that's the next thing to try.
@justin-espedal please check if the latest development build works for you: https://build.haxe.org/builds/haxe/ Also check #10110
I can't test this directly since I'm not on High Sierra, but using the command I ran earlier we can see whether the ____chkstk_darwin symbol is being dynamically loaded.
$ otool -l development_779b005/haxe | grep -A 3 LC_VERSION_MIN_MACOSX
cmd LC_VERSION_MIN_MACOSX
cmdsize 16
version 10.13
sdk 12.1
$ symbols -noDemangling development_779b005/haxe | grep ____chkstk_darwin
0x00000001007126f0 ( 0x6) DYLD-STUB$$____chkstk_darwin [DYLD-STUB, LENGTH, NameNList, MangledNameNList, NList]
The change that I made above (https://github.com/justin-espedal/haxe/commit/afaee7c12123fe5c241c585fad04ef2ce53e908b) is currently based on the development branch.
I'm not sure what exactly I should be looking for in #10110? I looked it over before attempting to fix this, and the last post in the thread is another person saying that it no longer works for them.
I got curious about where the chkstk_darwin call is actually coming from, so I opened haxe in a disassembler.
_pcre_compile2 and _jit_machine_stack_exec, which in turn is referenced by __pcre_jit_exec. So everything seems to be coming from pcre.
In the github actions build script, this is how Haxe is built:
opam config exec -- make -s -j`sysctl -n hw.ncpu` STATICLINK=1 "LIB_PARAMS=/usr/local/lib/libz.a /usr/local/lib/libpcre.a /usr/local/lib/libmbedtls.a /usr/local/lib/libmbedcrypto.a /usr/local/lib/libmbedx509.a -cclib '-framework Security -framework CoreFoundation'" haxe
Note that it includes /usr/local/lib/libpcre.a, which is pcre1. We only compile pcre2 ourself, so pcre1 must be downloaded or already present on the system. I have a lot of questions about this though.
- Is there a reason that both pcre1 and pcre2 are involved in the build process?
- How do we get pcre1 in the first place? I don't think it's included by default in the github runner, and it's also not in
tests/Brewfile, and we also use--assume-depextsinopam install haxe --deps-only --assume-depexts, so theconf-libpcreopam package shouldn't be causing it to be installed either.
Maybe if we built pcre1 from source, like we do pcre2, the minimum deployment target of 10.13 would then cause it to stop generating the chkstk_darwin symbol. But then, again, why are both versions involved in the build process? Is this an oversight? Should we just go with pcre2 alone?
Is there a reason that both pcre1 and pcre2 are involved in the build process?
pcre1 is used by the eval target for regex in the standard library, which hasn't been ported yet: #10491
I'm not sure why pcre2 is needed, but it seems to be part of the github actions setup since: 62ee6a9
Yeah, I just found that commit too. I think that before that commit, perhaps the version of pcre1 that we were building was then being picked up when it came time to compile Haxe. Now that we don't build pcre1 anymore, it's instead getting pcre1 from... somewhere???
And I should also draw attention to the timing. Since the mentioned commit happened in between the Haxe 4.2.4 and 4.2.5 releases, it fits.
According to this comment left on the commit, pcre should be downloaded from homebrew (to avoid the hassle of having to build manually): https://github.com/HaxeFoundation/haxe/commit/62ee6a9a8e727de30ac7c8f6c82a3e8f42b9a22b#r73655101
it's instead getting pcre1 from... somewhere???
I'd imagine pcre could just be available as a dependency of some other program installed in the runner?
Also, I think we can probably just not build pcre2 for now, and see if anything breaks because of it (and if it does, we should probably just use homebrew for it as well).
I checked with building pcre from source, and it works. The ____chkstk_darwin symbol reference is gone. If a suitable version of pcre can be found in a prebuilt homebrew bottle, I agree that that would be preferred. The latest homebrew pcre formula doesn't have a high sierra bottle.
Can't we get homebrew to build from source? That way we ourselves don't have to worry about our commands for installation breaking in the future, as some will probably fix it on the homebrew side. The url breaking in our script for installing pcre is what prompted the erroneous 62ee6a9a8e727de30ac7c8f6c82a3e8f42b9a22b commit in the first place, which is how we ended up with this issue.
It looks like we can build from source if we invoke a brew install like this:
brew install --build-from-source pcre
In order to install a specific version, we need to take a few extra steps: https://stackoverflow.com/a/55764594
brew tap-new --no-git $USER/local-tap
brew extract --version=8.44 pcre $USER/local-tap
brew install [email protected] --build-from-source
I haven't confirmed this yet, but it appears that setting the MACOSX_DEPLOYMENT_TARGET environment variable does affect installing from source as expected, which we'll require if we want this to work for us at all.
If installing from source, the installed version of Xcode must be the latest stable version (according to homebrew) or it will stop with an error. https://stackoverflow.com/questions/40296756/homebrew-saying-xcode-is-outdated
So, does the Github Actions macOS runner always provide a version of Xcode that will pass this check? If not, we're just heading toward flaky builds.
https://github.com/actions/runner-images/blob/main/images/macos/toolsets/toolset-11.json https://github.com/Homebrew/brew/blob/master/Library/Homebrew/os/mac/xcode.rb
| xcode | required by homebrew | available on github | days late |
|---|---|---|---|
| 12.2 | 2020-11-13 | 2020-11-27 | 14 |
| 12.3 | 2020-12-17 | 2021-01-23 | 37 |
| 12.4 | 2021-01-29 | 2021-02-20 | 21 |
| 12.5 | 2021-05-10 | 2021-06-09 | 30 |
| 13.0 | 2021-09-28 | 2021-10-14 | 16 |
| 13.1 | 2021-11-09 | 2021-11-10 | 1 |
| 13.2.1 | 2022-02-07 | 2022-01-20 | -- |
| 13.3* | 2022-03-24 | -- | -- |
(*this was a mistake, reverted the next day)
Looks like it's not particularly reliable, though it should work at the moment. I tried searching through other Github Actions build scripts to see if I could find other people who install brew formulae from source. It seems to be exceedingly rare to do so within Github Actions build scripts.
Going with installing from source through homebrew seems like fighting an uphill battle to me.
It seems homebrew should build from source automatically if no bottle is found for the current system: https://docs.brew.sh/FAQ#why-do-you-compile-everything
However, you're right about the xcode requirements. It will end up being a pain so it is better to leave it.
It seems homebrew should build from source automatically if no bottle is found for the current system: https://docs.brew.sh/FAQ#why-do-you-compile-everything
Right, but a bottle will almost always be available for us, since homebrew tends to support the last 3 versions of macOS.
If we really wanted to use brew for this, the other option is to specify the exact bottle we want and download that. https://stackoverflow.com/a/69858397
In this case, that would be this pcre 8.44 bottle which was the last pcre bottle to be built for High Sierra: aeea1351e1439847d00c3cee54bd28639493e686f809568cf42fea7bb28da2a5
curl -L -H "Authorization: Bearer QQ==" https://ghcr.io/v2/homebrew/core/NAME/blobs/sha256:aeea1351e1439847d00c3cee54bd28639493e686f809568cf42fea7bb28da2a5
It then needs to be manually installed as well. Doesn't really sound like it's simpler than building from source, but it would be consistent and it would also slightly speed up CI builds.
I'm going to remove the pcre2 build and submit a PR, so I'm not going to mess with any of the brew options, but leaving this comment here in case anybody is interested in alternate approaches in the future.
Right, but a bottle will almost always be available for us, since homebrew tends to support the last 3 versions of macOS.
Ah right I see, that makes sense.
I'm going to remove the pcre2 build and submit a PR, so I'm not going to mess with any of the brew options
Yes, hopefully soon we'll manage to get rid of pcre1 altogether and we can forget about this.
According to this comment left on the commit, pcre should be downloaded from homebrew (to avoid the hassle of having to build manually): 62ee6a9#r73655101
I am not that concerned about hassles of having to manually build as per se, but all of our build instructions specify installing system packages and in the macos case to use brew, see extra/BUILDING.md#installing-dependencies (and Haxe is built with Ocaml currently using opam package conf-libpcre which also specifies using pcre from homebrew). However, our user build instructions may specify such due to hassles of having to manually build dependencies.
My main concern is keeping the documentation about how to build Haxe in-sync with how we actually build it, otherwise there will be a disconnect, the documentation will cease to work at some point and it will end up being uselessly misleading (no documentation is often better than incorrect documentation). Bad incorrect documentation tends to build up often resulting in multiple erroneous versions and no actual good working version.
@justin-espedal: I am curious why we support macos High Sierra to begin with? Obviously we cannot realistically support every macos build back to its beginning? Since we document our build support for macos to use homebrew and homebrew no longer supports High Sierra for our dependencies, shouldn't we in kind basically state we do not support it? I believe we should mark this issue unsupported will not fix (though it was good to bring these concerns up as it resulted in fixes to our automated build process).
If we are going to start supporting things that homebrew does not (e.g., High Sierra), I seriously question why we use and specify its usage to begin with. I say stick with it as a "system platform" and we support just that platform.
it's instead getting pcre1 from... somewhere???
I'd imagine pcre could just be available as a dependency of some other program installed in the runner?
@tobil4sk: It is currently a dependency of the Lua target (which also has yet to be ported from pcre to pcre2). That might be why the build picks it up anyway. Lua is a source target so we do not actually build anything, but we do require specific Luarocks extensions to be installed in order to run tests on that target. See our brew install pcre executed in the Lua test CI.
Thanks for your comments.
I am not that concerned about hassles of having to manually build as per se, but all of our build instructions specify installing system packages and in the macos case to use
brew[...]
I agree with your concerns about the automatic build deviating from the build instructions for users. But on the other hand, as you said, "our user build instructions may specify such due to hassles of having to manually build dependencies." I think it's a trade-off in the end. Perhaps the build instructions can be considered as a simpler set of build steps that will get users a working build on their own machine?
I am curious why we support macos High Sierra to begin with? [...]
Personally, I have empathy for people using older hardware who have no way to update without buying a new computer, but also aren't comfortable building software from source. Realistically, we'll need to drop support for older versions at some point, but I'm not sure that point is now, given this was easy enough to fix with a small change to the build script.
If we are going to start supporting things that
homebrewdoes not (e.g., High Sierra), I seriously question why we use and specify its usage to begin with. I say stick with it as a "system platform" and we support just that platform.
I also seriously question why Haxe's build script uses homebrew. But I would move in the other direction, and just get rid of it entirely. If we care at all about reproducible builds, homebrew is antithetical to that purpose, because it causes the output of the build to change significantly with relation to time.
Using this case as an example, and assuming that we were previously getting pcre from homebrew:
- If we ran the build on June 16, 2021, we would get a Haxe build that supports High Sierra.
- If we ran the build on June 18, 2021, we would get a Haxe build that does not support High Sierra.
Nothing in the build script changed, but suddenly the output is different just based on the day when the build was performed.
Admittedly, there are many other issues that are categorically similar to this. For example, the fact that Github Actions runners, with their ever-changing environments, are used to perform Haxe's builds in the first place.
Given the pros and cons of using Github Actions and homebrew, I'm willing to accept that they might be the right choice anyway, and for differences that really matter, we should be testing the output of the build.
See our brew install pcre executed in the Lua test CI.
As an answer to the question of where pcre is coming from in the build, this doesn't convince me. pcre is required to build Haxe, but this is a Haxe source file, so it can only run after Haxe has already been built.
I agree with your concerns about the automatic build deviating from the build instructions for users. But on the other hand, as you said, "our user build instructions may specify such due to hassles of having to manually build dependencies." I think it's a trade-off in the end. Perhaps the build instructions can be considered as a simpler set of build steps that will get users a working build on their own machine?
Yes, but then who is going to test the user manual build instructions and ensure they continue to work when they deviate substantially from the automated building? Do we create a separate set of tests for this? That seems silly considering we would effectively just be creating another automated build system, one to match the user instructions and one for something else? Why not just keep them the same since the point is to build it anyway not build it in as many ways as possible?
Personally, I have empathy for people using older hardware who have no way to update without buying a new computer, but also aren't comfortable building software from source. Realistically, we'll need to drop support for older versions at some point, but I'm not sure that point is now, given this was easy enough to fix with a small change to the build script.
Agreed.
I also seriously question why Haxe's build script uses homebrew. But I would move in the other direction, and just get rid of it entirely. If we care at all about reproducible builds, homebrew is antithetical to that purpose, because it causes the output of the build to change significantly with relation to time.
Yes, it is unfortunately in this case where homebrew used to support pcre for High Sierra but no longer does so (at least with the latest xcode which we are forced to move to due to Github Action changes, etc.). In many cases this is not true and things just work since they were supported in the past but apparently Apple changed something and that is not true without using the old bottle compiled by older xcode or recompiling with new xcode on 10.13 requiring a new bottle which homebrew no longer supports.
That said, moving away from homebrew would likely be quite problematic. For example, Haxe is based on Ocaml and we currently depend on conf-libpcre (see opam file) which basically tests to ensure an existing pcre system install exists for all the system platforms we support and attempts to install it...using yes, homebrew for macos. Are you planning to change that too or remove the dependency and make our own pcre system package tests (apparently we already have a build now)? How about the Luarocks dependencies? And that explodes out to all the supported targets fast. The issue is that we are not the only ones that have decided homebrew is the macos system packages platform to use. I believe we just adopted it because others already did. Even if you get this issue working what is to stop it from failing for Lua targets? The same pcre, xcode, Github Action issues exist there. Sure the compiler and eval targets might work but what it compiles would then fail in the same way for some targets like Lua.
An once you do create our own pcre dependency platform support instead of using opam (with homebrew on macos), who will support it when each supported system platform changes its packages? This is precisely why we use opam (and it in turn uses homebrew for macos). It is all well and good to want to support older platforms but in the end we choose the easy way because it is already supported by those communities and we do not want to reinvent to wheel (even if it might be a better wheel). The same can be said of Luarocks, etc.
And remember, pcre is in the midst of change with regard to Haxe since we are moving to pcre2 (because the former is no longer supported). So much of this has to change anyway (e.g., I created conf-libpcre2-8 to make way for an Ocaml binding to pcre2 which has yet to be written).
Given the pros and cons of using Github Actions and homebrew, I'm willing to accept that they might be the right choice anyway, and for differences that really matter, we should be testing the output of the build.
Agreed, we can always improve testing.
As an answer to the question of where pcre is coming from in the build, this doesn't convince me. pcre is required to build Haxe, but this is a Haxe source file, so it can only run after Haxe has already been built.
I have not looked at this extensively but I am not sure I agree with you. Remember these are just scripts that are run to ensure the system supports each of the targets. Then Haxe is built and then it is used to run tests. It seems to me prerequisite dependencies for the Lua target could conceivably run before Haxe itself is built.
It seems to me prerequisite dependencies for the Lua target could conceivably run before Haxe itself is built.
The build environment is separate from the test environment, and is run earlier anyway. The fact that pcre is installed in the lua test environment wouldn't affect the build environment. I think it's more likely that some other package in the mac environment still depends on pcre1.