Suggestion: new command to bump versions of dependencies in `pyproject.toml`

[daisylb]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

Issue

It would be awesome if Poetry had a command (let's call it upgrade) that bumped the version constraints of dependencies in pyproject.toml (as opposed to update, which afaict updates the lock file to the newest version within the constraint specified in pyproject.toml).

Some examples for how this command could behave:

• poetry upgrade django: Upgrade Django to the newest version that still works with other dependencies; equivalent to poetry remove django; poetry add django.
• poetry upgrade django djangorestframework: As above, but with more than one package at a time.
• poetry upgrade django=^2.1: Set the version of django to ^2.1, equivalent to poetry remove django; poetry add django=^2.1.
• poetry upgrade: Upgrade every dependency to the newest possible version. Equivalent to deleting the entire [tool.poetry.dependencies] section of pyproject.toml and running poetry add with a list of the names (but not versions) of every package previously in the list. (This one would be good for cookiecutter templates for projects, to make it easy to start a new project with the latest versions of everything.)

Currently, when I want to bump the version of something, I'm either running poetry remove ...; poetry add ... which moves the package to the bottom of the list in pyproject.toml, and results in uninstalling a bunch of dependencies which sometimes just get reinstalled again at the same version; or I'm manually editing pyproject.toml which means I have to look up the latest version manually, and I can't use Poetry's version resolution when I want to upgrade more than one package at a time.

[jgirardet]

It's a bit dangerous to upgrade everything to the last version. you might introduce many bugs doing this, without knowing where it comes from.

Anyway I also do this :laughing:

upgrade is very confusing with the update command. We could have acommand per package and for all.

Maybe --reset-dependency package_name or/and --reset-dependency-all would be less confusing. or : --find-latest, --upgrade-to-latest,--force-latest

[daisylb]

I agree; I only used ‘upgrade’ because I couldn’t think of something better :)

It could also be added to ‘poetry add’ (either with or without a flag); currently that command just errors if you try to use it on a package that’s already in your dependencies.

On Tue, 2 Oct 2018 at 18:52, jgirardet [email protected] wrote:

It's a bit dangerous to upgrade everything to the last version. you might introduce many bugs doing this, without knowing where it comes from.

Anyway I also do this 😆

upgrade is very confusing with the update command. We could have acommand per package and for all.

Maybe --reset-dependency package_name or/and --reset-dependency-all would be less confusing. or : --find-latest, --upgrade-to-latest,--force-latest

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/sdispater/poetry/issues/461#issuecomment-426207069, or mute the thread https://github.com/notifications/unsubscribe-auth/AAc7aLMzza48JHvRq92e6REbrmtJqsAzks5ugzAzgaJpZM4XDWit .

[jgirardet]

I'll put a PR if @sdispater accepts the idea

[miracle2k]

The JS package managers make add do an upgrade of the version range, and that makes sense to me.

[kylebebak]

The JS package managers make add do an upgrade of the version range, and that makes sense to me.

@miracle2k This is one of the many things that npm and yarn get right.

For example, if you yarn add abc, yarn might install abc version 2.2.1, and it will save this dep in package.json as "abc": "^2.2.1",.

In other words, when adding a package it uses the same caret prefix that poetry uses, and which permits subsequent updates to any version 2.X. I think this is a sensible default.

If you poetry add uWSGI, for example, it installs 2.0.18, but it saves this in pyproject.toml as uWSGI = "^2.0"... Why not save this as uWSGI = "^2.0.18"? This would behave the same under subsequent invocations of poetry update uWSGI, but it gives users immediate information about which version of the package is installed (2.0.18).

@jgirardet I think a sensible implementation would involve a simple post-install step for both poetry add and poetry update. After running either of these commands, poetry could get the actual installed version of all dependencies and dev-dependencies, and update their versions in pyproject.toml, keeping the same caret or tilde prefix that the dep currently has.

In other words, if you have uWSGI = "^2.0.17" installed and you run poetry update uWSGI, it installs 2.0.18 and changes the line in pyproject.toml to uWSGI = "^2.0.18".

For wildcard (*) and gt, gte, lt and lte deps this behavior doesn't make sense, but caret and tilde requirements are by far the most commonly used.

[kylebebak]

If you make a script like this one at the root of the repo, make it executable, and run it with python3.7, it'll print the contents of pyproject.toml to the console after all ^ and ~ deps have been updated to their actual installed versions.

#!/usr/local/bin/python3.7
from typing import cast, Dict
import toml
import subprocess


def update_deps(name: str, version: str, t: Dict) -> Dict:
    def update(deps: Dict) -> None:
        for key in deps:
            v = deps[key]
            if type(v) is str and name.lower() == key.lower() and v[0] in ("~", "^"):
                deps[key] = f"{v[0]}{version}"

    update(t['tool']['poetry']['dependencies'])
    update(t['tool']['poetry']['dev-dependencies'])

    return t


with open('./pyproject.toml', 'r') as f:
    t = cast(Dict, toml.loads(f.read()))
    output = subprocess.run(["poetry", "show"], capture_output=True)
    lines = cast(str, output.stdout.decode()).split('\n')

    for line in filter(lambda l: bool(l), lines):
        name, version, *_ = line.split()
        t = update_deps(name, version, t)

    print(toml.dumps(t))

This is a goofy implementation that uses subprocess to call poetry show. It's just a proof of concept.

I think something like this happen after add and update are run, to update deps in pyproject.toml to their actual current versions.

[jmfederico]

If npm and yarn already do it in the add command, maybe keeping it consistent with what users already now would be better?

I also think that upgrade could be more confusing than useful. From other package managers, I would expect update/upgrade to just modify the lock file but not touch the dependencies definition file (tool in our case).

[sdispater]

If you use the latest beta release of the 1.0.0 version, you can upgrade your dependencies by using a specific constraint or the special latest constraint.

poetry add pendulum@latest
poetry add pendulum@^2.0.5

See #1221 for more information.

[max-wittig]

@sdispater But what about updating all dependencies at once?

Can we reopen this?

[Natim]

It is a bit tedious to go over all of them manually.

[dmwyatt]

While the command mentioned by @sdispater is nice, it doesn't go all the way to solving the problem talked about in this issue since you still have to go one by one and check each package.

[finswimmer]

I thinks the request about an upgrade command to update all dependencies to the latest available version is valid. So I reopen it.

[hay]

This would be a very useful addition. There's a tool for Node.js called npm-check-updates that might give some inspiration. I agree that an upgrade command would be confusing, but maybe something like poetry update ----upgrade-latest?

[donbowman]

npm also has https://www.npmjs.com/package/npm-merge-driver

Next time your lockfile has a conflict, it will be automatically fixed. You don't need to do anything else.

which is nice.

reason being, is tools like dependabot keep it up to date, one by one, but, branches don't currently work w/ the poetry lock file since the hash line conflicts.

[Uninen]

This is something I need almost daily. I have a lot of projects that I want to keep up to date and Poetry is currently very cumbersome with that task. My ideal situation would be to have just one command to upgrade one or all dependencies (not at all interested bikeshedding about what that command would look like).

So, a very big +1 from me for this feature.

Also, it would probably be a good idea to document a suggested workaround/way to do this before we have the command. Using poetry add NameOfDependency@latest seems easy enough, and if there is no easy way to do this for multiple packages currently, maybe just mention it and link to this ticket or a PR with a "under development" note. Willing to write a PR for the docs if it would be accepted.

[Natim]

This is something I need almost daily.

Maybe you should consider to plug in @dependabot

[Uninen]

Maybe you should consider to plug in @dependabot

I do use Dependabot on projects that are on GitHub and Snyk on GitLab whenever I can, but I really wouldn't like to rely on third-party tools for something my package manager should handle. I mean, handling packages (dependencies) is literally the only thing I need the tool for.

[takeda]

@max-wittig Frankly, if someone needs to do such operation to all dependencies at once, they probably don't care about these constraints, and will be fine with no constraints (i.e. using "*" instead of "^1.2.3"), with that poetry update will do what you want. Your dependencies are still locked in the poetry.lock file.

[Natim]

@taketa Not really because that's something you may want to do on a regular basis weekly/monthly while making sure in the meantime nothing breaks because of a dependency change you haven't noticed and break something in production.

[max-wittig]

~~@takeda poetry update will only upgrade to minor and patch versions, not major.~~

@Natim Exactly, what he mentioned. We have renovate-bot for example, which automatically bumps versions. I don't want this happening automatically for major upgrades.

[takeda]

@max-wittig poetry update supposed to observe constraints in pyproject.toml ^1.2.3 allows to update to 1.3.4 but not too 2.3.4 (^ is more complex than that, but this is a ghist), if you use * there is no constraints.

[Natim]

@takeda if I understand correctly, you would set the version to * in pyproject.toml and then rely on poetry.lock to have your production build use the tested dependency and then run poetry update on time to time? It is a working workaround I guess.

What this issue is aiming for is to keep the pyproject.toml config file as a source of truth for the expected dependencies versions.

Thank you for the workaround though, it might work for my project.

[takeda]

@Natim yes, exactly. The reason for the constraints in pyproject.toml is to define such constraints that guarantee API compatibility.

Different authors have different ways they version their packages (for example if you use pytz, you probably want to have * there for it, since API never changes and your always want the latest time zone information) so that's where you use ^ and ~ (and also <, >, * more here: https://python-poetry.org/docs/dependency-specification/) to specify what versions are compatible with your application. If you have to do mass update that ignores these constraints, you aren't using them as intended.

[danhje]

I think this is more relevant now than ever. Due to the increased awareness around the dependency confusion attack, there is a push towards using exact version constraints or constraints with upper bounds, not just lower bounds, so that automated pipelines don't just pull potentially compromised external packages just because they have a higher version number than the internal package you really wanted. But this means that you have to do a more controlled update of those very specific constraints in pyproject.toml more often than before, to prevent running on outdated packages and packages with known security vulnerabilities. I think there should be a more elegant way to do this than removing everything and adding everything back in.

[reformat0r]

Quick workaround:

poetry show --no-dev -o -t | grep -v -e "--" | cut -d " " -f 1 | sed 's/$/\@latest/g' | xargs poetry add

This will upgrade all outdated top-level packages in a single transaction. To continue even if a package fails installing, add -n 1 to xargs.

[felipeportella]

Thanks @ra-martin for the workaround! It will indeed update but may lead to an unexpected result. For example, I have on my pyproject.toml only the pandas package, but when I run that command, it can identify that the numpy (a dependency of pandas that is installed) is outdated and will update it (ok), but will also add it to the pyproject.toml (not the desired behavior). If the outdated package is only the pandas then the result will be exactly the expected. Anyway is a nice workaround, and with the pyproject.toml in a source version system we can later on go there and manually clean up the dependencies that we don't want to explicitly list as requirements of our project (leaving only the ones with versions changes).

Quick workaround:

poetry show -o | cut -d " " -f 1 | sed 's/$/\@latest/g' | xargs -n1 poetry add

[reformat0r]

@felipeportella I edited my previous post. It now only upgrades top-level packages.

[m09]

Quick workaround:

poetry show -o -t | grep -v -e "--" | cut -d " " -f 1 | sed 's/$/\@latest/g' | xargs poetry add

This will upgrade all outdated top-level packages in a single transaction. To continue even if a package fails installing, add -n 1 to xargs.

Disclaimer: this will incorrectly add new version of dev dependencies into normal dependencies.

[reformat0r]

@m09 You are correct - I edited my post to include --no-dev, preventing the command from polluting your regular dependency-tree with dev-dependencies. Sadly, there's no --only-dev, so you'll have to update those manually.

[max-sixty]

Potentially a synthesis of the "it's dangerous to update everything" and "I don't know what I need to upgrade" is a command to show which libraries have more recent releases than those permitted by pyproject.toml.