dreamworksanimation
Cannot configure submodule dependencies to be external
Hi there!
In our facility (Animal Logic) we use rez to define external dependencies, and don't build projects using git submodules. Would it be possible to please follow project examples like OpenImageIO to allow a user to manually define all the dependencies of the project?
The reason this is currently not possible is the fact that you just use: https://github.com/dreamworksanimation/openmoonray/blob/9b629e0999bfe2394ad56d31c3314c5199c4db3c/CMakeLists.txt#L47 directly, instead of allowing the user to define where this dependency comes from. Same for the USD hydra dependency etc. So in order to specify external dependencies I need to hack your CMakeLists.txt to remove the direct add_subdirectory calls, which I shouldn't have to do.
The way you can control this in OpenImageIO is by using this: https://github.com/OpenImageIO/oiio/blob/master/src/cmake/externalpackages.cmake, which makes it really easy to configure external packages manually, which is what we need to be able to do in order to make use of this project.
I am currently in the process of jerry-rigging this into my local version, and I'm happy to contribute something back to allow this if is helps, but it would be a good idea for you guys to do this regardless.
Thanks!
Never mind, I just figured out that openmoonray refers to moonray as a submodule, which I didn't understand initially, so I should be building moonray instead of openmoonray.... OK, back to the drawing board ;-).
OK, after almost another full day of trying to build from source, I have made almost no progress.
It'd be great to chat to someone about setting up the build process so it's workable within a different studio setup.
The inability to specify proper external dependencies, and the over-reliance on git submodule is going to be a huge problem for us.
I'd love to be able to have a discussion with someone about how you could restructure the build process for CentOS to be viable for a different studio to be able to use.
I don't want to use the docker option.
I also tried to follow the instructions here: https://docs.openmoonray.org/getting-started/installation/building-moonray/building-moonray-centos-7 and they are ambiguous and I cannot follow them.
For instance, top to bottom, it asks me to:
mkdir /installs cd /installs .... mkdir /source cd /source ... mkdir /build cd /build ... mkdir /installs/optix ... mkdir /installs/openmoonray ...
etc. etc.
now if you follow that from top to bottom, as written, that's clearly wrong, since it never tells you to go back to the top directory to make the next directory etc. so you should probably make that a bit clearer so the process can be accurately followed from top to bottom properly.
Thanks.
Hi
What I have seen on other projects is the ability to use the indirection built into CMake modules to be able to find versions of dependencies
Like if you wanted to use PackageX at version 6.6.6 you could just put in -DPACKAGEX_ROOT=/path/to/mycustom/path/packageX_6.6.6 as well as -DPACKAGEX_INCLUDE_DIR=/path/to/mycustom/path/packageX_6.6.6/include and -DPACKAGEX_LIBS=/path/to/mycustom/path/packageX_6.6.6/lib64
Which allows for the level of indirection that @etheory is asking about.
Because at some shops these are read only locations which are set in stone and are the result of many months of curating the software engineering process.
When you rely on git submodules to have read and write access to a location on disk that is hard coded to be relative to some entry point.
This prevents you from using the software in serious production use cases.
So being able to have a level of indirection which allows the software to subscribe to headers and shared objects and object code archives, where they are stored at a company wide level, then you can have the upstream project come from a CI / CD build to an output location and not have that system directly under control of the Moonray build process, yet still be compatible with the Moonray software.
I have done this in the past with an OpenImageIO build process where I build one component at a time which static vs non static with the glibc and gcc deps that suit my need and then add one more building block until the process is complete.
All we need for this to be possible is to be able to have use control over the find dependencies functionality within CMake
Sam
Thanks for your feedback and trying out MoonRay! It’s an important consideration for us that people are able to build and use MoonRay in studio environments like the one you describe.
The build instructions provided in the initial release were primarily intended to provide a quick start to building MoonRay, and don't provide a lot of decision points or information on alternative configurations. We're working on a document that provides more guidance about how to customize the build to meet specific requirements like those you discuss. Although the build instructions we provide at present address building the entirety of MoonRay, internally we build each submodule to a separate rez package. We're happy to discuss with you more directly how this can be achieved.
Thanks so much @rgwilson68 it's greatly appreciated and super exciting stuff. Let's organize a meeting to go over things for sure.
Hi,
I'm working on a document with a more detailed explanation of how the CMake build works. In the meantime, I'll try to address some of your questions.
Regarding external dependencies, for most packages the method that samhodge refers to works. For example, if you set the environment variable
JSONCPP_ROOT=/packages/jsoncpp/0.7.0
prior to running cmake, then the jsoncpp headers/libraries will be found in the directory specified.
The other method used to find dependencies is that some packages come with a script called "packagenameConfig.cmake" - for example "OpenImageIOConfig.cmake". This contains the locations, names and dependencies of each library in the package. If cmake can find "packagenameConfig.cmake" by looking on the path defined by CMAKE_PREFIX_PATH, it will load the script and find the libraries correctly. Depending on how the dependency is installed, it might already be on CMAKE_PREFIX_PATH, or you might have to add it.
Either method will work for most dependencies, although a config file "pxrConfig.cmake" is required to correctly find the USD/pxr dependency.
Usually you would define the ..._ROOT and CMAKE_PREFIX_PATH variables as a named preset in the CMakePresets.json file, for convenience. CMakePresets.json in the root of the openmoonray repository contains example presets for a docker container build and the DreamWorks internal build. By modifying these examples, you should be able to use dependencies installed anywhere that is accessible to the build machine.
The openmoonray repository is specifically set up to download all of the MoonRay open-source repositories into a single directory structure, using git submodules, and build everything as a single CMake project. However this is only a convenience. You can independently clone the individual repositories anywhere you want and build each as its own project. This does not entail any use of git submodules. Each repository should produce a reponameConfig.cmake as part of its install process, which you can use to resolve cross-repository dependencies by making sure these files are on CMAKE_PREFIX_PATH.
Building the repositories separately in this way is not recommended unless you have a specific reason, because it inevitably makes the overall process more complex. However it might be necessary in some cases to integrate with an existing build/release system. At DreamWorks, we build the repositories separately, and use the Rez 2 configuration system to automatically handle both external and cross-repository dependencies. The repositories contain a "package.py" file that configures them for use with Rez -- these may be directly usable at other sites that use Rez, or may need some tweaking...
Regarding the Docker and Centos-7 build instructions : these are written to make most of the choices for you, so they assume a build based on openmoonray and submodules, and build their dependencies into the "/install" directory. You can install the dependencies elsewhere by changing installDir at the top of CMakeLists.txt in /building, and comment out anything you already have. Building the repositories separately requires a bit more work, as discussed above. I tnink the mkdir commands should work as written, since they use absolute paths : what specific problem are you seeing?