facebook
Fix OSS build
The HHVM OSS build has become nonfunctional over the past year or so (#9506, #9389). This patchset aims to get it working again.
Due to the number of changes required, I don't think this PR is feasible to land as-is—I'm opening it to provide an overview of the changes made and also to have a working OSS branch that's up to date with the current master. I'll be making individual PRs for the patches within this patchset if there's maintainer capacity to accommodate them.
This patchset does not yet fix every extension—notably, squangle and mcrouter still need fixes to properly compile for OSS, hash has added the blake3 hash which will require pulling in the corresponding library dependency and so on. This can be done as a followup.
It's been a tad finicky to get the build to work in GitHub Actions because the build as-is requires 30+ GiB of disk space, which is more than what the default runners have available. I'll see if the Meta-specific 8-core-ubuntu runner will be able to handle this PR, and if not, I'll change the CI to remove assorted garbage from the default runner prior to commencing the build.
I'd like to thank @kmapb and @Atry for their OSS build fixes that proved to be very helpful.
Below is a non-exhaustive list of the most important changes:
Changes made
- [x] C++20 is required due to the use of coroutines, ranges and assorted C++20 paraphernalia. (cb168b5176b187f51410fc7f7c90c50c8bf7ce1d)
- As alluded to in the OSS support change blog post, HHVM is now built using clang internally. I've been unable to get it to build with either GCC 14 + libstdc++ 14 / clang 18 + libstdc++ 14, but a complete LLVM toolchain of clang 18 + libc++ 18 did the trick. In either case, at least libc++ 17 is required due to the dependency on
std::lexicographical_compare_three_wayintroduced in ba8cf61e1a5e7d8e334381712348c5e1a87bb21e. - [x] Using libc++ also requires providing a way to force the compiler to use libc++ over the system default, which is likely to be libstdc++ on the majority of Linux systems. (8ce6dcf4562fd3cbe0a02b0a4dd8e3e8a6baf03a)
- Building with libc++ also implies that any C++ library dependency whose ABI exposes C++ types must itself have been built against libc++, otherwise they won't link correctly. In practice, this means building C++ dependencies that could otherwise have been provided via system libraries (e.g. re2, boost, glog) from source, and ensuring that compiler flags are correctly forwarded to such subprojects. (d7982c76646d712d0ac4afa7707934dba1dbf12f, 4374e1461c464557bb45434418f5af21884c954e, bad51c20104a91d76427b5b0a7b5dd3b3002f17e, def968dad5a06fc953ccf2037cb263fe07cbdd42, 7058d8f797dc9437381176bb7236805130ff2a88)
- ext_gd depends on libheif >= 1.16.0 since 80f15c38721f1839a34414cacb9e8e3e7d93a9b1, which implies building on Ubuntu Noble or newer unless we want to vendor libheif as well.
- [x] Install the nightly build from
2024-11-26which corresponds to the current stable version 1.83.0, because some Rust dependencies now require at least Rust 1.73.0 later. (984acef8f41c11cd1c715602a603f414e0410027) - Update patches for other internal FB dependencies (e.g. folly and fizz) that have become outdated due to changes to those libraries. (6cb61898eb579770adff66b0a42e700ceecb01ec, c2fbba9d1aa021e481aa1eaf6daba418bab2e806)
- Install new required dependencies for FB-internal library dependencies. Proxygen and thrift now require mvfst, so provide it as a submodule (presumably this can be converted to use the same mechanism used to keep folly etc. in sync with this repo). Fizz optionally depends on liboqs (with corresponding linking implications), while folly has a hard dependency on fast_float and xxhash. (b03fd39651867f6a5c96022d64e1117ce433de07, 24b3e0081d7fadc190a9930e07dcf04f38abb559, ad852ccb9ab6bce73a2c0deb5f7281923b6d10c2)
- Fix compatibility with fmtlib 11. (0ffb19ad8b5e1209c4529a557bb2738a5d043213, 1115143fc092583c242a8f19672cf0ea02a46351).
- Update outdated OCaml stubs. (be39b5adac61ab662941a2f3106dbb19f022ef13)
- Update various listfiles in the core runtime and extensions that have become out of sync with the code.(235d8118561599222832aaa9ae9472a0920fcd55, 2fba1180ee8a61c1d5b0bffa3a21845ba8d5e99c, 98d947b41efeccd881aaa52d6512c5438c2f525e)
- Add various missing
#includedirectives that were causing compile errors. (01975f188f2eba83611334132a3d406a452362aa, 34eefc6ed35c640d26a643f35c11ea3343e03301,14edad8afcaff5b4ffd78dd096ffc3da670e13c7, a19eee543bce6b8ad290033e3d96bae2c994b781, 3f07dc4a4c35ad4f84a2286566dbdb80b50e174a). - [x] Always build Intel® Xed because it is now a required dependency, and update its mbuild submodule to work with Python 3.8+. (fc5a56152488c2ab8f91f8620bcfdd3ba66c29fc, 8122e8f99a3d86f4b21488962e52cb5bdea710c5)
- Run Rust build steps after setting up OCaml and provide them with OPAM environment variables because the
ocamlrepcrate used in some places requires a working OPAM install. (4a72abb00749d31ce092de01d2254848ec1c5dd9, 0e3ca6342bcce33470efe5253e7a4509a01802dd) - Update code generation logic for hhbc-unit.h. (ae086c0485e731dfe9c68efbc8d8f95f7ae058b7)
- Compile Rust FFI libraries into a single Rust staticlib and link that into the rest of HHVM instead of the current approach of linking individual Rust staticlibs that does not work with cxx. (5639957e747d3ca1dd7bc201da9ab688eeac73d1)
- Exclude code from ext/fb that depended on the Meta-specific FBSerialize header, because it got moved elsewhere—it's unlikely that this Meta-specific serialization implementation was being used by any third parties to begin with. (c6750ed51c6ac32318b411c8949d23e6a9371537)
- Provide compatibility logic for post-2021 libdwarf versions that are packaged by some distributions like Fedora (but not Debian/Ubuntu derivatives as of Bookworm/Oracular): conditionally use the new replacement functions and include the
libdwarfpheader if so. (97ce3e3c175abfa3987c9fdb6c03e6223d3059e1) - Support the new Rust code generator that turns configs.specification into configuration headers. (22dd7acedcb281c7fb6a7393df83de5461eab23a)
- Update systemlib generation to work with systemlib precompilation introduced in 20118c824f124db94906345060410487effc67f8, and account for the core systemlib having been turned into an extension in 62bed4d833a42701ce508e1351c03137594f6a. Raise a more helpful error for the likely case (in OSS) that a new extension's systemlib hasn't been wired up with the build system. (fd6fe22acb2c3a63b4957f252975cfaf8d81485c, 5c45ff90b22d16d4a3b2c92176bfab23558d9965).
- Fix tc-print directory inclusion order due to dependency on generated systemlibs (89264df82dfa956279624f0721235714b839b0d9).
- [x] Fix the libxml and soap extensions to work with libxml >= 2.12.0 that changed several functions to take a
const xmlError*instead ofxmlError*. (f0dfb0509136ecd19fe1edb9251f4458b986c6f2) - Apply @Atry 's unmerged patch from https://github.com/facebook/hhvm/pull/9288 to fix a compile error with OpenSSL 3.
- Use a known custom runner label. (bca257080561bd94d5e205c18b75fddd03cd0500)
@facebook-github-bot has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. (Because this pull request was imported automatically, there will not be any future comments.)
Sorry about the comment spam, some last-minute CI fixes were in order. This now builds fine.
Going forward, I'm unconvinced of the benefits of building packages for PRs. I think it might be better to have a scheduled (nightly?) workflow for doing that instead, and have a lightweight workflow for pushes and PRs that would just compile HHVM and run tests.
Thanks for this — at Slack we are also very interested in this PR, which contains a few changes we've already independently applied to our local patch (which is at the moment forked off a mid-2023 version of upstream)
@muglug Thanks, glad to hear this! Let me know if you have any feedback or questions.
I've already started to cherry-pick some of the individual changes into PRs but will try to keep this branch periodically updated until the fixes land.
@mszabo-wikia I too am very interested in this branch. I have built it locally. I am able to run Hack code with hhvm after adding openssl 1.1.1 for the ext_watchman autoloader. I am looking at hh_client (which fails with EOPNOTSUPP). The typechecker itself is alive, since hh_client check . can correctly report errors. Have you been able to type check a project using hh_client at commit 0c55ab3?
I have created a Dockerfile which builds hhvm of your branch. It is nothing spectacular, but it would allow anyone interested to build outside of GitHub CI in a reproducible environment. I will need to add the OpenSSL fix, an 'add to path' fix, and an 'install watchman' fix. I will share tomorrow after work, or if the build takes more time than I have available on a weekday, this weekend.
@mszabo-wikia I too am very interested in this branch. I have built it locally. I am able to run Hack code with hhvm after adding openssl 1.1.1 for the ext_watchman autoloader. I am looking at hh_client (which fails with EOPNOTSUPP). The typechecker itself is alive, since
hh_client check .can correctly report errors. Have you been able to type check a project using hh_client at commit 0c55ab3?I have created a Dockerfile which builds hhvm of your branch. It is nothing spectacular, but it would allow anyone interested to build outside of GitHub CI in a reproducible environment. I will need to add the OpenSSL fix, an 'add to path' fix, and an 'install watchman' fix. I will share tomorrow after work, or if the build takes more time than I have available on a weekday, this weekend.
Hmm, this seems to work okay for me in a trivial container (podman run -it --rm -v $PWD:/app --security-opt label=disable ubuntu:noble bash, with a test project mounted and HHVM_DISABLE_PERSONALITY=true + HHVM_DISABLE_NUMA=true ).
Perhaps there are some additional restrictions imposed for your particular container?
https://github.com/facebook/squangle/pull/18 fixes the MySQL client lib used by HHVM, after which the mysql and async_mysql extensions will also be usable with some minimal fixes.