Add setuptools build_system

[ColinKennedy]

Like how there is a build system for CMake, make, and custom, it'd be nice to be able to build Rez packages directly, using setup.py

And to also support these workflows

• Building to wheel or other distribution format (controllable by setting a variable in the package.py)
• Editable installs of your package (via pip --editable or similar mechanism)
  • Allow this as a build argument via the terminal. e.g. rez-build --clean --install -- --symlink
• Consider adding a helper function to Rez which auto-fills the setup.py so you don't have to duplicate information in both places. e.g. name, version, author, project_urls (from Rez package.py help attribute), etc.

Motivation Adding setuptools support means

• Existing python source packages will be more easily ported to Rez
• Passively gain new setuptools features without having to duplicate implementation
  • In absence of a build system, you'd have to wrap a custom builder
• Better interchange between Rez and "pure Python" packages
• Standardization of Python packaging techniques, in Rez

[instinct-vfx]

There is quite a big overlap withrez-piphere and i think it would be important to get a good idea of what the best approach is prior to moving anything forward.

I would like to start with a rough overview of what we currently do.

• Everything that is pure python has to be set up as a standard python package
• These are then built as wheels and rezzified using rez-pip
• Only exception are packages that need things that can not be provided by pip/setuptools like hooking into a DCC by setting env vars in commands() or other bespoke things
• These are typically set up close to a standard python package still, but then use a rezbuild.py type worflow to build the bespoke package.py parts

So i think it is important to understand how this proposal would relate to rez-pip.

I would argue the other way round would be better. So instead of augmenting setup.py to pull information from a package.py why not do the rez-pipway and have rez pull that information from the python package setup. There is a lot of code that handles a lot of the complexity already.

rez-pip already supports a lot of the things proposed here. You can also use rez-pipto create a rez package from a source python package for example (rez-pip -i . in a project folder with setup.py just like running pip). The few things that lack currently from my point of view:

• Support editable/developer type setups, but i think we are not too far off
• Support "augmenting" the created package.py with bespoke non-python stuff. There could be different ways to do that like having pre_pip_commands() or something like that or by having a package.py that is just augmented with the pip PATH magic or the other way round.

I would like to address some of your points directly to make sure i cover everything:

Existing python source packages will be more easily ported to Rez

This can be done already with rez-pip

Passively gain new setuptools features without having to duplicate implementation

This is also already part of rez-pipas it can use newer versions of pip/setuptools easily and just re-uses their functionality

In absence of a build system, you'd have to wrap a custom builder

The custom build is rez-pip, correct?

Better interchange between Rez and "pure Python" packages

There is some room to grow i would say but we are close already

Standardization of Python packaging techniques, in Rez

It is desirable but rez will also always allow people to set up their workflows as they wish i would argue.

That all said i think the elephant in the room question is if building python packages is better done with a build-system implementation or a builder like rez-pip. rez-pip is to be split of to a separate project anyways, that could also take the form of a build system maybe?

I wonder if there are any limitations coming with a build-system plugin that are not present with rez-pip?

[nerdvegas]

Consider:

# in package.py

from rez_pip.setup_py import create_package_definition_from_setup_py

locals().update(create_package_definition_from_setup_py(some_massaging_arg=x,
...))

..where rez_pip is a py module available in the rez installation because we previously did:

]$ rez-extension --install rez_pip

..which works based on David Lai's recent work wrt extensions (we're missing the rez-extension tool though, but it would be trivial. That would just do the pip install into the rez venv, and would move the binaries into the rez subdir within Scripts/bin).

I kinda like this approach because it involves minimal boilerplate, but also isn't really doing anything special, it just happens to be using an extension to dynamically define the entire package definition. The create_package_definition_from_setup_py() function could set build_system to the matching pip-based build system that the extension also installs. Also, the package author is still free to define other package attributes in the usual way. In fact, you get loads of control over what you want to be determined programmatically from the setup.py, vs what you want to control manually - you can always override attribs after create_package_definition_from_setup_py is called.

In terms of pip-ingestion-tool vs pip-based build system - yes it's an interesting thought. Perhaps with a core pip lib written with this in mind, the distinction isn't so important - hopefully both approaches can share a lot of code. In any case, if we were to provide both, I would expect the one rez_pip extension to provide them.

Thoughts? A

On Mon, May 16, 2022 at 5:01 PM Thorsten Kaufmann @.***> wrote:

There is quite a big overlap withrez-piphere and i think it would be important to get a good idea of what the best approach is prior to moving anything forward.

I would like to start with a rough overview of what we currently do.

• Everything that is pure python has to be set up as a standard python package
• These are then built as wheels and rezzified using rez-pip
• Only exception are packages that need things that can not be provided by pip/setuptools like hooking into a DCC by setting env vars in commands() or other bespoke things
• These are typically set up close to a standard python package still, but then use a rezbuild.py type worflow to build the bespoke package.py parts

So i think it is important to understand how this proposal would relate to rez-pip.

I would argue the other way round would be better. So instead of augmenting setup.py to pull information from a package.py why not do the rez-pipway and have rez pull that information from the python package setup. There is a lot of code that handles a lot of the complexity already.

rez-pip already supports a lot of the things proposed here. You can also use rez-pipto create a rez package from a source python package for example (rez-pip -i . in a project folder with setup.py just like running pip). The few things that lack currently from my point of view:

• Support editable/developer type setups, but i think we are not too far off
• Support "augmenting" the created package.py with bespoke non-python stuff. There could be different ways to do that like having pre_pip_commands() or something like that or by having a package.py that is just augmented with the pip PATH magic or the other way round.

I would like to address some of your points directly to make sure i cover everything:

Existing python source packages will be more easily ported to Rez This can be done already with rez-pip

Passively gain new setuptools features without having to duplicate implementation This is also already part of rez-pipas it can use newer versions of pip/setuptools easily and just re-uses their functionality

In absence of a build system, you'd have to wrap a custom builder The custom build is rez-pip, correct?

Better interchange between Rez and "pure Python" packages There is some room to grow i would say but we are close already

Standardization of Python packaging techniques, in Rez It is desirable but rez will also always allow people to set up their workflows as they wish i would argue.

That all said i think the elephant in the room question is if building python packages is better done with a build-system implementation or a builder like rez-pip. rez-pip is to be split of to a separate project anyways, that could also take the form of a build system maybe?

I wonder if there are any limitations coming with a build-system plugin that are not present with rez-pip?

— Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1307#issuecomment-1127297666, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMOUSW32G3P5TI6HL3IQ7DVKHXEDANCNFSM5V5ZZBUA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[ColinKennedy]

This is a lot feedback, thank you both very much! I'll try to explain the use-case a bit more.

I didn't know calling imports outside of early / late functions was allowed in source package.py definitions. If that's an option. that's almost certainly what I think we should do. Funnily enough, I do the same thing in conf.py files to grab / re-use Rez package.py data.

It's very useful to have a source Rez package, even if the payload in this case is another Python package.

All of those use cases fall under 2 categories

• "Patching" / defining data not present from setup.py
• Contextualizing the package as a part of a larger system

Some practical examples include

• "Patching" - adding extra tests that don't by default in the package
• "Patching" - (setup.py) install_requires / requires - we all have horror stories of packages that say they support Python 2 + 3 but at only 3, etc.
• Contextualizing custom build steps (maybe this package is actually just 1 part of a larger build)
• Contextualizing - Proprietary release that doesn't allow for rez-pip-ing
  • Usually even rez-piped code needs to be QAed, for licensing reasons, for example

It's also desirable to keep your edits separate from the original source setup.py repository. Most of those use-cases require rez-release or a similar Rez package source-to-install. While rez-pip is extremely helpful, it's not always the best mechanism to "insert" patches. rez-pip-ifying also has an extra downside which is if you use rez-pip to make a source setup.py into a Rez package, if any info is invalid or incomplete, those edits needs to be manually written and are lost whenever you rez-pip again.

Having an in-memory package.py works well because, if there's some sort of bug, all you'd have to do to fix it is update your Rez version and re rez-release (and/or make a specific git branch, bump a package version, etc).

Anyway, all this to say - I'm in support of a package.py locals().update(foo()) solution! What do you think @instinct-vfx?

As a slight tangent, I wonder if this solution would only be possible certain build systems like setup.py or if we could apply the same idea for other types of build systems.

[JeanChristopheMorinPerso]

Like how there is a build system for CMake, make, and custom, it'd be nice to be able to build Rez packages directly, using setup.py

You should note that CMake is made wrapping (generating makefiles, VS files, etc) while setuptools is made for packaging a project. These are entirely different things.

• Editable installs of your package (via pip --editable or similar mechanism)
  • Allow this as a build argument via the terminal. e.g. rez-build --clean --install -- --symlink

I've seen this in the wild (via the custom build plugin) and I've only seen breakage after breakage. While the idea works for non-rez packages (for example any package in PyPI), it doesn't work so well in rez. The reason being that it's so easy to forget that you built one package with symlinks and that packages live in ~/packages locally by default. With pure python packages (PyPI packages), you normally create one virtualenv per package (for example .venv in the root of the repo).

Generally speaking, I'm -1 on this (on a scale of -2 to 2).

I'll play the devils advocate.

My main arguments/concerns are:

1. Rez is not pip or setuptools or whatever the tool you use. Conventions, tools and the general ecosystem of rez are not necessarily (and even far I'd say) the same as the Python packaging ecosystem. For example "Standardization of Python packaging techniques, in Rez" isn't really a good thing. The PyPA standards are not designed for this which means you are on your own when the libs and common tools they use change and it breaks your custom stuff, at least for the implementers (us). Also the standards are much more focused on the integration in general (so build frontend (pip), build backend (setuptools, flit, poetry, etc), wheel format, etc).
2. The Python packaging ecosystem is moving away from dynamic packages (setup.py) to the pyproject.toml (PEP-518, PEP-621). Also PyPA members are working hard to remove the implicit dependency on setuptools in pip (and they are getting close to their goal). Which leads me to a question,
3. Why setuptools and not X, Y, Z? The Python ecosystem is fragmented and some people prefer to use setuptools, some others prefer flit, poetry, .
4. Using setuptools just to gain a build system (copy files from A to B), I'd instead create a simple rez package that you can include in your private build dependencies would do the trick and wouldn't rely on the not so user friendly setuptools. And if it's good enough, I'm sure a lot of people will use it instead of all writing our custom code in build.py files (if you build_command is python {root}/build.py).

I see the point and I've had the need to package a rez package as a wheel to be use outside the rez ecosystem. But I solved it by writing a rez package that defined utility functions to do that and I was including the package in my private_build_requires. The repo had a simple setup.py that was importing the package.py and was referencing the variables from it.

With a solution like @instinct-vfx suggest, you could potentially use rez-pip to generate the setup.py file, and then you would have full control of that file. I haven't put a lot of thoughts into it though, so I don't have much more details to give. Such a thing would allow you to keep things decoupled as much as possible and wouldn't alienate the package definition file (package.py). Because the thing with the definition file is that Rez supports another file format as of today, the package.yml. It's unused today, but it's still supported. And some day we could even have a package.toml if we decide to implement static definitions again (which would be a great idea, because dynamic packages are painful).

So if the solution requires someone to do an import in a package.py, then it won't be generic enough for the current modularity of rez.

Also, I'm not sure I understand your examples in your last reply @ColinKennedy .

[ColinKennedy]

1. Rez is not pip or setuptools or whatever the tool you use. Conventions, tools and the general ecosystem of rez are not necessarily (and even far I'd say) the same as the Python packaging ecosystem.

There's 2 aspects to this thread. 1. Using the regular setup.py mechanisms to drive building a package. 2. Describing a package.py via setup.py. I'm more interested in 1. If 2 can happen in the same stroke, then great. But 1 is more important than 2, to me.

WRT 2. The things that do have parity should have a prescriptible solution. The alternative, to do nothing, is that people are completely on their own when they do need it and end up creating custom solutions with varying levels of quality / completeness. I have a Rez package for building .egg / .whl files and include it in private_build_requires which I have to use / maintain in absence of a better solution.

2/3 Why setuptools and not X, Y, Z? The Python ecosystem is fragmented and some people prefer to use setuptools, some others prefer flit, poetry.

Interested stakeholders who want support for X / Y / Z build system, tie-in, etc are free to raise them just like I had. As to why I brought up setup.py, it's used in countless packages globally and having it in Rez (via extension, core, etc) means not having to write my own solution and it can be used elsewhere. Sharing is caring!

4. Using setuptools just to gain a build system (copy files from A to B), I'd instead create a simple rez package that you can include in your private build dependencies would do the trick

That's basically what I do already in absence of a better way, yes. It generates .egg / .whl files / etc, but it does that in ignorance of any custom code that might exist in setup.py so it's only used for "Rez Python only packages". I'd have to add the functionality, unittest it, and maintain it. But why would we want to encourage people to do that when it could be something everyone can share.

@nerdvegas something I thought about, WRT locals().update(foo())-ing a setup.py file. If the setup.py includes extra, non-setuptools / non-standard-library-related imports, I wonder how we would handle that.

I've seen this in the wild (via the custom build plugin) and I've only seen breakage after breakage. While the idea works for non-rez packages (for example any package in PyPI), it doesn't work so well in rez

This note is fairly unrelated to the overall topic of whether or not to attempt to support setup.py ( as a build system, tie-in, etc). However just small fyi - I've never once had problems with symlinked local builds - on dozens and dozens of packages - and have been doing them for years. But then again, I never use the default ~/packages local_packages_path. I've not tried pip -e specifically so I can't say how fragile / good that is but I can say without a doubt, symlinked builds are easy and fun. I just would never make a symlinked release, if that's what you meant. We can continue that to another discussion thread if you'd like.

[bpabel]

I added a PR for this a while back -- https://github.com/nerdvegas/rez/pull/491

Since then, rez-pip has improved substantially and python packaging has become more standardized. IMO, there's no reason to maintain separate package.py's for python packages anymore. You should be able to build and release any python package using rez-pip.

If there is some issue preventing you from using rez-pip, I think time would be better spent updating rez-pip to handle your use case.

Or possibly working towards the plugable artifact repositories described here -- https://github.com/nerdvegas/rez/discussions/1242

[bpabel]

Can we close this issue? I believe rez-pip should work on python package source repositories in the same context in which rez-build and rez-release would be run.