[BUILD] OIIO Windows Build System Maintainer

[robinrowe]

• OIIO Windows Maintainer Build Notes *

These notes are for the OIIO Windows maintainer (Robin Rowe) for making improvements and fixes to build system. To document what we're doing. If you have suggestions for Windows build improvements, you may add them to this ticket.

** Overview **

We build on Windows and CI test on Windows -- though just one entry in the test matrix is Windows, it doesn't include all dependencies and so lots of features probably go untested, so some beefing up to do there. Build all dependencies. More CI matrix cases to cover other compiler versions.

** Windows Maintainer Instructions **

Go to https://github.com/OpenImageIO/oiio and "watch" the project, to get email alerts as PRs and Issues are filed. And the OIIO mail list: http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org

Build from source on Windows has many issues with documentation at https://github.com/OpenImageIO/oiio/blob/master/INSTALL.md It's wrong and outdated. The Windows section needs a complete rewrite once we settle on the best strategy for building dependencies.

In general, the OIIO build scripts work really well and do cross-platform builds. It assumes that all the major dependencies are already installed and accessible, so automating downloading and building dependencies is a goal. Perhaps using cmaker DownloadProject.

Also see https://stackoverflow.com/questions/37091735/building-openimage-io-on-windows

** Windows Package Management **

An issue that comes up all the time is that it's a big pain for inexperienced users to build for Windows, mostly because there are lots of dependencies to cover the many image formats. One cheat is that it's in VcPkg, so if you're already on the VcPkg train, you can just rely on that. But it's harder if you want to build a different version than is in VcPkg, or build with different options, or do development on it. So building from scratch instead of dependencies is still really awkward -- surely some set of much better step by step instructions, or maybe some auxiliary scripts that can largely automate the dependency downloads and builds, would go a long, long way. Without a windows box (or expertise) of my own, I'm just not the one to even evaluate whether beefed up install instructions are hitting the right marks for typical Windows users. I don't know if it's practical or wise, but many people want OIIO but don't really even wish to build from scratch if not necessary, so it may be at least worth thinking about whether it makes sense to host a pre-built binary of major releases?

DownloadProject with cmaker tweaks:

https://gitlab.com/robinrowe/cmaker/-/tree/master/cmaker_lib/DownloadProject

Not yet integrated with cmaker. Plan is that there will be a cmaker command such:

    $ cd my_project_directory
    $ cmaker_depends.sh zlib libjpeg libpng

And that will be all we need to configure our project's cmake build system to fetch and build the relevant 3rd party libs. Magic.

Cmaker DownloadProject will cache per git tag or tarball URL. Won't download what it already has. Then cmake will then detect what is already built, not rebuild lib. What happens if pulling git master for a lib, where sources change? After downloading dependencies, we may switch into Airplane Mode and work disconnected until we tell DownloadProject to check whether dependency sources have changed (by deleting a special generated build file, much like deleting CMakeCache.txt to force recompile).

If cmaker doesn't seem the best approach to downloading and installing dependencies, there's also https://github.com/cpp-best-practices/cmake_template with using CPM https://github.com/cpm-cmake/CPM.cmake. The OIIO project has several hand-crafted bash scripts for doing download/build for CI (or for users) at https://github.com/OpenImageIO/oiio/tree/master/src/build-scripts. Not well tested on Windows.

** Windows Testing and Debugging **

Windows maintainer to monitor:

(a) the PRs, just skimming and being on the lookout for if any of the developers are doing things that seem unwise for windows in particular but don't know it. Maybe we pass CI, but are still digging a hole for ourselves that we don't see because OIIO are primarily Linux users.

(b) issue and mail list triage where users are asking about specific Windows build or to overcome problems to tell them how to get unstuck.

Robin Rowe, April 22nd, 2023

[robinrowe]

More on cmaker with DownloadProject...

Use it selectively for just the dependencies one needs to build from scratch, with minimal changes to the rest of the build system.

That's the idea. Just works, no changes to the rest of the build system. CinePaint has the same hill of 3rd-party image library dependencies as OIIO. Been working on solving this for a long time. Have many libs working, but not complete yet for all libs (notably TIFF and OpenEXR need work).

A challenge with 3rd-party lib integration is that their maintainers have never tried building their code using add_subdirectory. Their cmake expects to be at the top of the source tree. I install dependencies that I own as subdirectories under cinepaint, don't DownloadProject those for myself, and use .gitignore so I won't pull them into cinepaint git. Built dependencies appear in the Windows sln project window as being part of CinePaint, not separate projects.

    $ cd my_project_dir/build
    $ cmake ..

No special commands from users. As soon as they call make or msbuild, the dependencies download and build themselves before proceeding to build our project and linking to the dependencies built. For each library that CinePaint uses, I specify the git or tarball URL and may write a little patch script to fix up their cmake code.

[lgritz]

Summarizing more of the gist of the conversation Robin and I were having over email and transferring it here for any further follow-ups:

Currently, for almost all of our dependencies, the OIIO cmake build scripts simply assume that the dependency is already installed on the system and will turn up in the find_package search (possibly with the aid of setting <package>_ROOT). For many dependencies -- but definitely not all -- we also have src/build-scripts/<package>_install.bash which often will work for users, but are written to be used in our CI on GitHub Actions, so are clunkier and less general and robust that probably is ideal for users to rely on, and also are only present for the dependencies for which we need to install particular versions for our CI.

I really like the idea of migrating to a more principled approach that allows:

• Finding a system install or being redirected to a specific one via <package>_ROOT.
• Having the option to robustly build any or all of our dependencies with a single command that is more or less bullet-proof and cross-platform (and caches results so that, for example, blowing away the build directory or doing a make clean doesn't necessarily start you at square one), and works just as well and reliably for users as for CI. I think it could also result in our getting rid of a lot of those build helper scripts, which would be obsolete once this is in place.

I'm really looking forward to seeing what Robin proposes as a PR that prototypes his ideas for improving this. I don't know if the work is proportional to how many dependencies it is able to handle, but I think it's very reasonable to get just enough of the skeleton up to handle one dependency (any one) and then make a PR out of that so we can all look it over and comment, rather than going fully down the rabbit hole of chasing every dependency before we can look at the proposed solution and make sure it looks like it'll work for everybody's use cases.

[lgritz]

And 1000x thank you to @robinrowe for being interested in tackling some of these issues involving Windows, cross-platform builds, and dependency setup. These are not always glamorous problems to work on, but they are the kind of thing that makes a project like this a well-running operation and for people to have a good user/builder experience. I would love for the portion of issues filed that are "how do I fix this build break" to go way down and for a higher portion of questions to be "how do I do this user task with my working OIIO installation."

[jessey-git]

As someone who just attempted this a few days ago I can maybe add some additional conversation points.

• Yes, I used vcpkg to get the dependencies too. I did this before I lucked out and found that the CI scripts also use vcpkg to do the pre-req installs as well. It allowed me to trim some boost pieces and notice that freetype is also required. It's a viable option, as you say, if you're happy with the versions available. I used the vcpkg toolchain file so OIIO cmake could find the libraries which meant I didn't need to use any of the _ROOT variables.
• The testsuite seems only runnable on *nix[1]? I haven't been successful yet in discovering how it runs within CI :). As a Windows dev I'll say the following: I'd like a native Windows experience (no git bash window, no mingw64, no msys2 etc.) if possible

---

[1] Test suite issues Has issues starting the test runner:

Process not started
 C:/Users/jesse/source/oiio/testsuite/runtest.py
[unknown error]
1/2 Test #131: png ..............................***Not Run   0.00 sec

When trying to run runtest.py manually you often encounter *nix tool usage:

> C:/Users/jesse/source/oiio/testsuite/runtest.py "C:/Users/jesse/source/oiio/build/testsuite/png" "--devenv-config" "Release" "--solution-path" "C:/Users/jesse/source/oiio/build"

<... some test spew ... but then >
'tr' is not recognized as an internal or external command, operable program or batch file.

[lgritz]

Oh, interesting. I don't have a Windows box of my own, so I never get to build and test by hand on Windows to discover these things. The CI test passes, but I can imagine that it could have parts that rely on particular utilities that are pre-installed on the Windows GHA runners but might not be present on a typical Windows machine.

[robinrowe]

First OIIO dependency I'm looking at is Boost...

% git clone --recurse-submodules https://github.com/boostorg/boost	

That tries to pull the whole smash of Boost, which is huge. Yuck. Ok, not doing that. So I cloned just the core Boost (78mb) and then tried building OIIO in Windows gitbash:

% git clone https://github.com/boostorg/boost
% cd oiio/build/x64
% cmake ../.. -DBoost_ROOT=/code/github/boost -A x64

--
-- Boost_COMPONENTS = thread;filesystem
-- Boost library not found
--     Boost_ROOT was: C:/Program Files/Git/code/github/boost
CMake Error at src/cmake/checked_find_package.cmake:177 (message):
  Boost is required, aborting.
Call Stack (most recent call first):
  src/cmake/externalpackages.cmake:65 (checked_find_package)
  CMakeLists.txt:166 (include)
-- Configuring incomplete, errors occurred!

Looking around, found this in externalpackages.cmake:

message (STATUS "Boost_COMPONENTS = ${Boost_COMPONENTS}")
# The FindBoost.cmake interface is broken if it uses boost's installed
# cmake output (e.g. boost 1.70.0, cmake <= 3.14). Specifically it fails
# to set the expected variables printed below. So until that's fixed
# force FindBoost.cmake to use the original brute force path.
set (Boost_NO_BOOST_CMAKE ON)
checked_find_package (Boost REQUIRED
					  VERSION_MIN 1.53
					  COMPONENTS ${Boost_COMPONENTS}
					  RECOMMEND_MIN 1.66
					  RECOMMEND_MIN_REASON "Boost 1.66 is the oldest version our CI tests against"
					  PRINT Boost_INCLUDE_DIRS Boost_LIBRARIES )
					  

So, some Boost build weirdness here for me to fix. Appending instead of setting Boost_ROOT. Will give more thought...

[lgritz]

If Boost is super onerous, we might consider whether to just accelerate our efforts to replace the last of the boost use. A quick perusal indicates that we're down to only the following items:

• boost::filesystem: Will be removed once C++17 is our minimum and we can rely on std::filesystem. If that's not soon enough, we can consider incorporating one of the many portable single-header filesystem replacements for C++14.
• boost::thread_specific_ptr: we'd need to write our own or find a replacement. It shouldn't really be all that hard.
• a small number of boost::algorithm uses in strutil.cpp which could probably be easily rewritten from scratch.
• boost::stacktrace use in sysutil.cpp: I think there are other libraries (complexity: unknown) that do the same thing. In the much longer term, I think this functionality is included in C++23. Since this is a debugging aid, it may also be acceptable to only enable it if boost is found and just #if it out in cases where boost isn't available at build time.
• boost::flat_map and flat_set: there are many single-header equivalents or we could roll our own, should be easy for the fairly limited functionality we require from it.

Looks like that's actually all the boost uses that I haven't removed already?

[fpsunflower]

+1 for making the push to remove boost entirely. Its the first "no-go" to adoption in many fields (particularly game engines).

[lgritz]

There is no boost in any of our public headers or APIs. Working around the use of boost in this handful of cases is a purely internal change, and one that could even be backported to the release branch, since no APIs or ABI would change.