Rearrange PyPI projects, break out adapters

[ssteinbach]

Overview

The goal of this issue is to refactor the PyPI projects for OpenTimelineIO into two pypi projects from the current one:

• OpenTimelineIO (the current package) becomes ONLY the core python wrapped library
  • Versioned whenever a new release of the core comes out from GitHub (similar to the current release process)
• OpenTimelineIO-Plugins (a new package) is OpenTimelineIO + a curated set of plugins
  • Versioned whenever a plugin is added or removed from the list, or the OpenTimelineIO releases
  • A plugin may be removed from the list if it remains unsupported and falls behind the core, as determined by the OTIO TSC
  • PRs may be made to add or remove plugins from the list
  • The Plugins project will be in its own repository, with a README that describes briefly some criteria that the TSC is looking for when assessing whether to include a project or not
  • Plugins that are not present in OpenTimelineIO-Plugins may still be present on pip. Users may install extra plugins through pip that aren't included in the OpenTimelineIO-Plugins project.

Outstanding tasks:

• [x] Remove the dependency on the contrib project from the top level setup.py
• [x] Add a note to the readme about the OpenTimelineIO-Plugins
• [x] Rename umbrella repository to OpenTimeilneIO-Plugins
• [x] Create pypi project for OpenTimelineIO-Plugins
• [x] add CI integration for pushing to pypi from Plugins repo
• [ ] CI integration for checking new versions of OpenTimelineIO from the Plugins repo
• [x] register pypi packages for the adapter plugin projects <- at least for the ones that we plan on having the umbrella project include
• [ ] break out adapters
  • [x] as a TSC group we identify which adapters we want to include in the default installation
  • [x] each adapter gets its own package, which can be individually installed
  • [ ] remove explicit references to the contrib package from the main library (see comment below) and documentation
  • [x] remove the contrib directory from the repository
  • [ ] recommend a pinning strategy for how the adapters should handle their version dependencies + some CI for testing against new versions of the CORE
• [ ] add version freezing to release script for parent projects
• [ ] Add versioning strategy (noted below) to documentation (CONTRIBUTING.md or the main documentation)
• [ ] Remove references to the adapters from the main documentation (or update them to point at the individual adapters, mention that they're accessible via the -Plugins project)
• [ ] CI for adapter projects need create new versions on manual activation rather than when tags are created
• [x] Show the plugin version in the otiopluginfo command output
• [ ] Add a documentation page that describes project layout and how that relates PyPI package
• [ ] Make sure adapter repos are version pinning to OpenTimelineIO>=0.17.0.dev1
• [ ] Stage PRs in adapter repos changing version pinning to OpenTimelineIO>=0.17.0

Versioning strategy

• OpenTimelineIO-Plugins project pins to a specific OpenTimelineIO project, corresponding to the current release, using the == constraint
• Each adapter picks a versioning strategy against the OpenTimelineIOversion number
• the -Plugins project depends on a set of adapters in a floating way, which lets the adapter repos individually deal with their dependency on the core
• OpenTimelineIO-Plugins gets versioned only when plugins get added or removed, or the pinned version of OpenTimelineIO is updated

Why the OpenTimelineIO Does Not Include The Plugins ("OpenTimelineIO-core vs OpenTimelineIO-

Currently, if you pip install opentimelineio you get the contrib adapters + the core library. This suggestion switches that around such that pip install opentimelineio is only the core without the contrib adapters. We decided to make this shift for several reasons:

• We want to make it clear to what degree the main project provides a warranty for components, and we want to emphasize the core format
• We can update the version of the main OpenTimelineIO package without the plugins being updated if there is a compability issue
• For software vendors, its more clear to identify the project version a software package uses or provides compatability for

Question

• if we break an adapter from a change to OTIO core, how/when do we catch that? Do we [re]introduce smoke testing in the Plugins repo?

[mikemahony]

Are we imagining all the packages (OpenTimelineIO umbrella, OTIO-core, and other adapters) would live in the same, existing repo? Or do we want them each split out into their own git repositories?

[timlehr]

I'd be in favour of splitting them into their own repos.

[jminor]

The adapters should definitely go into separate repos (one per adapter) so that experts in each one can be given ownership over them.

As for the core and the umbrella, I don't know. Is there a tradeoff worth considering? I can sort of imagine what a separate umbrella repo would look like, but I'm unclear on the details. What would it look like to have them together in one repo?

[apetrynet]

Is checkbox number two in the list the GitHub project itself or the project on pypi? I'm assuming the latter(?)

So when pip install OpenTimelineIO you get core and default adapters?

Would it make sense to follow the pip install OpenTimelineIO[core] style? Then we could do:

• core (core only)
• coreadapters (only the main adapters maintained by the otio project)
• contrib (a curated list of contrib adapters)
• all (all of the above)

I'm not sure if this is possible though, as I don't have enough experience with setting such a structure up. The contrib key might be tricky to update without releasing new main packages?

And I agree about splitting adapters to separate repos

[JeanChristopheMorinPerso]

I would see the "core" as a separate package (but still in the opentimelineio repo), opentimelineio-core. Installing opentimelineio should install the core I think. From there, we have two options:

1. Adapters are installed with pip isntall opentimelineio[adapters]. From what I remember we kind of already ruled this option out because the user experience isn't great.
2. Adapters are installed with opentimelineio (with the core too). This gives a nice experience to users.

As for the core and the umbrella, I don't know. Is there a tradeoff worth considering? I can sort of imagine what a separate umbrella repo would look like, but I'm unclear on the details.

If the core and the meta package (the umbrella) live in different repos, I think it could make it confusing to users. Our main repo would be named OpenTimelineIO like it is right now. But the meta package repo... How would it be named? Also, wouldn't it be awkward to have a repo that doesn't match the package name? Though that would be the case for the core package, but I don't see that much as a problem for core.

One thing that we might want to consider is versioning and version constraints and how we will want to bundle things. Will the meta package define its dependencies with lower bound versions or without constraints at all? What happens when new features are added to OTIO (or breaking changes)? Should adapters be compatible with multiple versions of OTIO? In that same line of thought, should they use lower bound constraint (>=0.14)) on OTIO version or both lower and upper bounds (>=0.14, <=1.17)? (specifying a maximum would probably cause problem down the line).

[jminor]

@mikemahony and I made this empty repo for experimentation. We can sketch this out and see how it fits before deciding on the final layout: https://github.com/OpenTimelineIO/otio-umbrella

I agree that having the repos and the pypi packages with clear names is important. The version history of a separate umbrella repo seems appealing too, but not essential.

[ssteinbach]

Note that there are a few explicit references to the contrib package in the code:

https://github.com/AcademySoftwareFoundation/OpenTimelineIO/blob/5e27e78177095b709a2864ed6dced46bd221ac53/src/py-opentimelineio/opentimelineio/plugins/manifest.py#L327

Search for "contrib" in manifest.py. I see some in the autogen documentation as well, might be some others:

https://github.com/AcademySoftwareFoundation/OpenTimelineIO/blob/5e27e78177095b709a2864ed6dced46bd221ac53/src/py-opentimelineio/opentimelineio/console/autogen_plugin_documentation.py#L348

I added a todo list item to remove these references.

[mikemahony]

Putting these here for something to look at:

Option A - Core/Umbrella Separate Repositories
----------------------------------------------

 Owner            | Repository          | Package
 -----------------+---------------------+-----------------
  ASWF            |  OpenTimelineIO     |  otio-core
  OpenTimelineIO  |  otio-umbrella      |  opentimelineio
  OpenTimelineIO  |  otio-aaf-adapter   |  otio-aaf-adapter
  gilou           |  otio-drp-adapter   |  otio-drp-adapter
  apetrynet       |  otio-mlt-adapter   |  otio-mlt-adapter
  OpenTimelineIO  |  otio-xges-adapter  |  otio-xges-adapter   <-- owner ok?
  OpenTimelineIO  |  otio-ale-adapter   |  otio-ale-adapter   <-- owner ok?
  OpenTimelineIO  |  otio-rv-adapter    |  otio-rv-adapter   <-- owner ok?
  ...             |  ...                |  ...

Option B - Core/Umbrella Combined Repository
--------------------------------------------

 Owner            | Repository          | Packages
 -----------------+---------------------+-----------------
  ASWF            |  OpenTimelineIO     |  otio-core, opentimelineio (umbrella)
  OpenTimelineIO  |  otio-aaf-adapter   |  otio-aaf-adapter
  ...             |  ...                |  ...

[reinecke]

After discussing at the 11/10/2022 TSC meeting, the consensus is to try Option B and see how it works out.

[mikemahony]

@apetrynet, your name came up the other day - would you be interested and available to create repositories here for the individual contrib adapters? Perhaps even just placeholders/stubs with just enough to be able to be deployed to https://test.pypi.org?

[ssteinbach]

I added another todo list item based on the TSC discussion that a pinning strategy should be documented for the core, umbrella, and adapter repositories.

[apetrynet]

@mikemahony, Yes, I can look at this.

[mikemahony]

@markreidvfx, do you feel like https://github.com/OpenTimelineIO/otio-aaf-adapter is ready enough to be deployed to test.pypi.org?

[markreidvfx]

@mikemahony yes, I think it is in a good enough state to start testing that. I haven't verified that any of the pypi deployment scripts work, they will probably need tweaking.

I've been tracking that here and additional tasks here https://github.com/OpenTimelineIO/otio-aaf-adapter/issues/1

[markreidvfx]

@mikemahony are the any docs or examples you know of on how your suppose to use test.pypi.org vs the regular pypi? I've typically just used the dev tag on the regular pypi until I get the scripts right, then clean up my tracks :p

[JeanChristopheMorinPerso]

@markreidvfx you can add -r testpypi to the twine command, see https://twine.readthedocs.io/en/stable/.

[markreidvfx]

thanks @JeanChristopheMorinPerso. I think I get it now. I created the initial pypi project https://test.pypi.org/project/otio-aaf-adapter I can add others as owners if they slack me their pypi username.

[apetrynet]

@apetrynet, your name came up the other day - would you be interested and available to create repositories here for the individual contrib adapters? Perhaps even just placeholders/stubs with just enough to be able to be deployed to https://test.pypi.org?

Hi! Should I set up repos for the contrib adapters or would it perhaps make more sense to start with a couple of the intended "batteries" like cmx3600 and fcp_xml? I ask partly because these are adapters the otio project maintains(?) and also because some of the contrib ones are in the works for extraction by their authors.

[jminor]

You can start with the non-contrib ones, so we can demonstrate that this all works before deciding about each contrib adatper.

[apetrynet]

I've created a branch called "extract_adapters" where we can safely remove adapters and use for testing the extracted plugins. I'm also in the process of submitting a PR where I've removed (what I believe are) all cmx_3600 related files and tests. A cmx3600 plugin repo is on the verge of getting ready for transfer to the OpenTimelineIO org

[jminor]

Excellent! Let's get @markreidvfx 's AAF adapter PR https://github.com/AcademySoftwareFoundation/OpenTimelineIO/pull/1348 and https://github.com/AcademySoftwareFoundation/OpenTimelineIO/pull/1487 landed into that branch.

[apetrynet]

The cmx_3600 adapter now has its own repo

[apetrynet]

I'll get started on the fcp_adapter.

[apetrynet]

FCP adapter repo transferred to OpenTimelineIO org. https://github.com/OpenTimelineIO/otio-fcp-adapter.

Working on adapter extraction based on the "extract_adapters" branch

[markreidvfx]

Can we merge the AAF adapter extraction https://github.com/AcademySoftwareFoundation/OpenTimelineIO/pull/1348 into the extract_adapters branch too?

[apetrynet]

Mark's PR is merged into the "extract_adapters" branch.

[apetrynet]

svg adapter repo ready https://github.com/OpenTimelineIO/otio-svg-adapter svg adapter extracted from "extract_adapters" branch

[rosborne132]

I'll be working on moving the maya adapter.

[rosborne132]

maya_sequencer adapter repo is ready: https://github.com/OpenTimelineIO/otio-maya-sequencer-adapter maya_sequencer adapter has been extracted from "extract_adapters" branch.

[rosborne132]

I'll be working on moving the fcpxml adapter.