AcademySoftwareFoundation
How to manage Third-Party libraries?
The question is the same for all external dependencies, but solution could be different depending of the third-party library usage within OCIO.
OCIO has three main types of third-party library dependencies:
- Libraries for the OCIO library such as yaml, eXpat & ilmbase
- Libraries for the OCIO apps such as lcms2, glew, OIIO, OpenEXR
- Libraries for documentation such as JinJa, Sphinx
Note: Java support is unclear as the jar file is seven years old.
Whatever would be the long-term solution to manage third-party libraries (i.e. point 1), the 'mechanism' must support:
- Select a specific version i.e. to ensure a quality level, use third-party library versions validated by the CI builds i.e. avoid 'latest version' behavior
- Apply an arbitrary patch file i.e. Add specific compilation/link flags, Add custom namespace, Change code to remove warnings/build breaks...
- Compile on the local machine using local OCIO CMake settings i.e. propagate debug/release mode, use local compiler (i.e. both are extremely sensible for the Windows platform)
- Allow 'use external libraries' i.e. Linux common make behavior (Refer to the next point for Windows platform as no such /usr/local equivalent exists)
- Allow install within the OCIO build directory (i.e. should be the default behavior) i.e. global install on Windows platform is problematic as everything could end up in system32 with binary name conflicts. Plus, the build directory must ideally be self-contained (and 3rd-party lib. be static libraries) which is mandatory when working on different projects with library overlaps.
Compiling Open Source project is generally challenging on Windows, so I propose that OCIO library successfully compiles by default imposing eXpat & yaml presence.
Please let us know your ideas/comments, Patrick
Below is the original email thread:
On Tue, Feb 19, 2019 at 3:16 PM Larry Gritz <[email protected]> wrote:
I think that we had some consensus that if those binary dependencies could be hosted someplace reliable, then it should be fairly easy for us to download them at build time rather than have them in the main OCIO repo. The biggest concern was that some of the external dependencies might be either unsupported, or not supported for the particular commit or version that we desire, but that would be alleviated if we know those exact versions were available even if "we" somehow were responsible for hosting them (outside the OCIO repository proper).
On Tue, Feb 19, 2019 at 12:07 PM Steve Winslow <[email protected]> wrote:
Hello all,
I understand that John has already discussed with you some of the feedback from the intake license scan for OpenColorIO. Here are a few additional details:
For the majority of the OpenColorIO repo, the scan did not reveal findings that would need resolution prior to bringing the code into an ASWF repo. We will have ongoing suggestions regarding topics like recommendations for license notice format, adding licenses notices to source files wherever possible, etc., but those are all "going forward" matters that can be addressed over time.
However, I did want to note some findings from the /ext directory. It appears that this directory contained copies of several upstream dependencies (10 .tar.gz files, 1 .jar file). Some of these dependencies contain a large number of different licenses, many of which appear to apply to e.g. testing files for those dependencies. Additionally, at least one of the dependencies (the junit JAR file) appears to contain compiled binaries rather than source code.
We highly encourage projects to keep the contents of their repos focused on the projects' own source code. Where possible, we generally recommend to pull in required dependencies at build time as part of the CI setup, rather than storing them in the repo itself. There are a number of reasons for this, one of which is to help make it easier for contributors to make their CLA or DCO assertions for their contributions. Where an upstream dependency is being included in a repo with many licenses differing from the project's own license, it might be more challenging for the contributor to submit that under a CLA or the DCO.
Given that, my recommendation would be to remove the .tar.gz and .jar dependencies from the /ext directory, before pulling the codebase into the ASWF repo. John mentioned that there was discussion on this topic at the last TSC meeting, and the team was looking at options going forward. I would be happy to discuss this with you in greater detail, or feel free to let me know any questions.
Best regards,
Steve
--
Steve Winslow
Director of Strategic Programs
The Linux Foundation
[email protected]
--
Larry Gritz
[email protected]
I manage it this way on Fedora, could be commited here but may need some updating first.
https://src.fedoraproject.org/rpms/OpenColorIO/blob/master/f/OpenColorIO-setuptools.patch
yaml 0.3.0 is very old and generates lot of warnings when compiling with recent gcc version (i.e. tested with gcc 7.3.0). The current yaml version is 1.2 (i.e. check on the web site https://yaml.org/).
Revisiting how to handle the 'core' third-party libraries (i.e. point 1) could be the opportunity to update them.
I think 1.2 is the version of the specification. On Fedora we use the yaml-cpp project when is at version 0.6.2, but yes, 0.3.0 is quite old.
@hobbes1069 Yes. The latest yaml spec is 1.2 but the yaml lib is 0.6.2 (i.e. yaml-cpp-0.6.2)
Please, have to look to #721 which tries to enable usage of external third-party Python packages.