Use `vcs_version` for Pants' version

[thejcannon]

This change makes it so we can version a release simply from tagging a commit (therefore not requiring a PR to release)

It accomplishes this by:

• Removing the VERSION file
  • We'll likely want to update scie-pants to reflect this change
• Updates version.py to use setuptools_scm to generate the version on-the-fly if RUNNING_PANTS_FROM_SOURCES
• Adds version_scheme and local_scheme fields to vcs_version and wires them up
  • This is because setuptools_scm doesn't support .devN builds that aren't .dev0 without a different scheme
• Changing the packaged VERSION file (at src/python/pants/_version/VERSION) to be generated via vcs_version
• Adds a pants_dev_requirements.txt file which is solely used for Pants dev venv. And add setuptools_scm
• Updating the docs/automation as well

Demo:

# Using pants or ./pants
josh@cephandrius:~/work/pants$ pants version
2.20.0.dev6+geebeb404a8.d20240118
josh@cephandrius:~/work/pants$ ./pants version
2.20.0.dev6+geebeb404a8.d20240118

# With/without daemon
josh@cephandrius:~/work/pants$ pants --no-pantsd version
Pantsd has been turned off via Flag.
2.20.0.dev6+g64b67dc7ea
josh@cephandrius:~/work/pants$ ./pants --no-pantsd version
Pantsd has been turned off via Flag.
2.20.0.dev6+g64b67dc7ea

# Building the real-deal, locally
josh@cephandrius:~/work/pants$ ./pants package src/python/pants:pants-pex && cd $(mktemp -d) && touch pants.toml && NO_SCIE_WARNING=1 ~/work/pants/dist/src.python.pants/pants-pex.pex
 version
...
2.20.1.dev13+g64b67dc7ea

# And in CI (note I have to tag this commit)
josh@cephandrius:~/work/pants$ git tag release_2.20.0.dev7
josh@cephandrius:~/work/pants$ CI=1 ./pants package src/python/pants:pants-pex && cd $(mktemp -d) && touch pants.toml && NO_SCIE_WARNING=1 ~/work/pants/dist/src.python.pants/pants-pex.pex version
...
2.20.0.dev7

# And lastly, multiple tags (e.g. rc and stable)
josh@cephandrius:~/work/pants$ git tag release_2.20.0rc1
josh@cephandrius:~/work/pants$ git tag release_2.20.0
josh@cephandrius:~/work/pants$ CI=1 ./pants package src/python/pants:pants-pex && cd $(mktemp -d) && touch pants.toml && NO_SCIE_WARNING=1 ~/work/pants/dist/src.python.pants/pants-pex.pex version
...
2.20.0

[thejcannon]

Draft PR until I fix the Rust side of things and actually update the docs/automation

[benjyw]

I love the idea! So when do wheels get built?

I guess we always build wheels in main and release branches anyway?

[thejcannon]

I guess we always build wheels in main and release branches anyway?

Yup!

[huonw]

Nice! I like the idea.

The version looks quite "rich", potentially changing for every single commit. I wonder if this will cause cache hit rates for tests to be essentially zero: anything that depends on the version will have its digest change every build, and, I imagine most tests do (transitively) via things like doc_url? (I'm on mobile so can't confirm)

If this is the case, can we reduce the impact somehow?

[thejcannon]

Nice! I like the idea.

The version looks quite "rich", potentially changing for every single commit. I wonder if this will cause cache hit rates for tests to be essentially zero: anything that depends on the version will have its digest change every build, and, I imagine most tests do (transitively) via things like doc_url? (I'm on mobile so can't confirm)

If this is the case, can we reduce the impact somehow?

Welp, I was ready to tell you the multiple reasons it won't be a problem, but that just led to a different, worse problem. All the tests just die :joy:

So in fixing the deaths, I'll make sure we just use some constant version for tests (like 3.0)

[thejcannon]

Ok @huonw I moved the VERSION dep to the wheel, and then set _PANTS_VERSION_OVERRIDE to 3.0.0 in pants.toml for tests. :tada:

[thejcannon]

Preferably the new fields on vcs_version would be in a separate commit from the changes in the Pants repo, so it's obvious that there are two parts here - a user facing feature enhancement, and an internal Pants repo application of that.

https://github.com/pantsbuild/pants/pull/20446

[thejcannon]

Hmm the failure alludes me.

The fatal: No tags can describe '4c62b40a103d0014fd3277059efc1333211141f9'. makes sense. We maybe want some kind of "fallback" in the case of very shallow clones like checkout Action uses.

But the ModuleNotFoundError: No module named 'setuptools_scm' doesn't make sense. "Successfully installed debugpy-1.6.0 pydevd-pycharm-203.5419.8 setuptools_scm-8.0.4" from here

[kaos]

Have we deprecated the use of PANTS_SHA now then? As that will stop to work once we remove src/python/pants/VERSION.

https://github.com/pantsbuild/scie-pants/blob/2cd4278eb8bf773b398b88cdf6ef64fe5a4843bf/tools/src/scie_pants/pants_version.py#L153-L155

[thejcannon]

Have we deprecated the use of PANTS_SHA now then? As that will stop to work once we remove src/python/pants/VERSION.

https://github.com/pantsbuild/scie-pants/blob/2cd4278eb8bf773b398b88cdf6ef64fe5a4843bf/tools/src/scie_pants/pants_version.py#L153-L155

Yes, IIRC we deprecated it a while back with the suggestion of just checking out the repo locally and using PANTS_FROM_SOURCES

[kaos]

Have we deprecated the use of PANTS_SHA now then? As that will stop to work once we remove src/python/pants/VERSION. https://github.com/pantsbuild/scie-pants/blob/2cd4278eb8bf773b398b88cdf6ef64fe5a4843bf/tools/src/scie_pants/pants_version.py#L153-L155

Yes, IIRC we deprecated it a while back with the suggestion of just checking out the repo locally and using PANTS_FROM_SOURCES

Excellent. That'll make life a little easier over in scie-pants land.

[thejcannon]

Found it: https://github.com/pantsbuild/scie-pants/blob/main/CHANGES.md#L43

[kaos]

Found it: https://github.com/pantsbuild/scie-pants/blob/main/CHANGES.md#L43

Ah yea, me also: https://github.com/pantsbuild/scie-pants/pull/351/files#diff-d975bf659606195d2165918f93e1cf680ef68ea3c9cab994f033705fea8238b2 :)

[kaos]

Have we deprecated the use of PANTS_SHA now then? As that will stop to work once we remove src/python/pants/VERSION. https://github.com/pantsbuild/scie-pants/blob/2cd4278eb8bf773b398b88cdf6ef64fe5a4843bf/tools/src/scie_pants/pants_version.py#L153-L155

Yes, IIRC we deprecated it a while back with the suggestion of just checking out the repo locally and using PANTS_FROM_SOURCES

Ah! But PANTS_SOURCE feature also relies on the VERSION file:

https://github.com/pantsbuild/scie-pants/blob/c213edb28626973d6262c5016ad1148e35f08526/src/main.rs#L284-L290

So, that at least would need to be fixed along with this change, then I think..

[kaos]

Getting closer!

Good catch re the scie-pants checks. Even if we do land the scie-pants change, landing this might be a breaking change, as anyone using PANTS_SOURCE will start getting errors that are very opaque:

Error: No such file or directory (os error 2)

Given PANTS_SOURCE is a fairly niche/experts-only feature, that's probably okay? So, instead of doing a deprecation cycle with

https://github.com/pantsbuild/pants/blob/5779896fdd01e1a55d63043a4fb3a1f16ac6bee5/src/python/pants/bin/pants_runner.py#L27

we just do a hard-break and communicate with users via release notes/other announcements? Either that, or do do a deprecation cycle flagging the problem if running with PANTS_SOURCE. Thoughts?

Yea, I think if we release a new version of scie-pants that handles this properly, the first thing we ask whenever users face issues like above is which version of scie-pants they're using and ask to upgrade to latest first, which would help in this case. Being a dev/debug feature and not a "released" pants feature per se, I'm OK with not providing a deprecation cycle to change this behavior given an easy out with just upgrading scie-pants.

[huonw]

Another question: if the version is only known once tagged, we can easily forget to handle deprecations (e.g. remove or postpone) before tagging, and it seems they can happen in a way that causes the release build to break. Is that something we should be prepared to handle?

In particular, #20609 sets up 2.21.0.dev0, but CI fails hard when bootstrap pants, with errors like the following:

pants.base.deprecated.CodeRemovedError: DEPRECATED: option 'remote_oauth_bearer_token_path' in global scope is scheduled to be removed in version 2.21.0.dev0.

Referring to https://github.com/pantsbuild/pants/blob/8481389512cc1563ef08077838503769d8e4a4ae/src/python/pants/option/global_options.py#L1585-L1601

(NB. I'm a bit confused by this, because the corresponding deprecation warning doesn't appear on normal builds. 🤷‍♂️ )

In this case, it was caught by PR CI, but if I had just tagged 2.21.0.dev0 directly to trigger the release build, we might end up with a situation where we tag a release and it either completely fails to build, or is very non-functional when shipped.

In either case, I imagine we can usually just fix the problem and do the next release with low impact... assuming we get the deprecations right so that it's always dev release where behaviour changes, and not a stable release.

But, we don't always get it right (see `crossversion="partial" in #20616). So, should we think of ways to mitigate this risk? Just brainstorming ideas:

• some sort of pre-tagging testing
• dynamic or static linting about how the "highest risk" fields like warn_or_error and deprecated_version are used, to check that they're always referring to .dev0 releases?
• something else?

[kaos]

dynamic or static linting about how the "highest risk" fields like warn_or_error and deprecated_version are used, to check that they're always referring to .dev0 releases?

This makes sense to me.

if the version is only known once tagged, we can easily forget to handle deprecations

I think one way to smoke this out is that you can override the version locally when running tests... but then running the full suite is likely not practical so wouldn't catch everything.

[cognifloyd]

Maybe we need a new goal that lists upcoming deprecations?