[Composer/Packagist.org] Mark package versions as being maintained

[Toflar]

I would like to propose a way to improve two things:

• give users the option to understand what versions of a library are actually supported
• massively reduce the size of the ruleset when resolving dependencies

I found https://github.com/composer/packagist/issues/938 which is similar but it requires action on every single package version. I think we could introduce an option on packagist.org where package maintainers can (totally optionally) configure which versions are being maintained and which ones are not. I think this can be one single version constraint on package level so managing and maintaining it should be pretty easy. In case of Symfony for example, that would look like this as of today:

2.8.* || 3.4.* || 4.2.* || 4.3.*

One positive effect this would have is that I could visit packagist.org and see immediately which versions are supported and which ones are not. We could even introduce some command that would display what package versions are not supported anymore. On the other hand, we could have a new flag to composer update which would limit the provider information to only the maintained versions. Running composer update --maintained-only would then greatly reduce the number of rules that are added to the ruleset and thus reduce memory usage and speed up the performance massively. In fact it's similar to what Symfony Flex (extra.require section) does but it's not specific to Symfony, doesn't require any knowlege about the maintained versions by the user and works for transitive dependencies as well. As a user you can simply decide to only include maintained versions into your dependency graph by setting the --maintained-only or -mo flag. In case a package maintainer did not specify anything, we'd just fall back to the current behaviour, including all versions of that package. When you run composer update -mo you'll probably not get a resolvable set of dependencies because you're using outdated package versions. In that case, you can still omit the -mo flag and have the current behaviour.

Any feedback on this? Would it be hard to implement and why?

/cc @nicolas-grekas @leofeyer

[Toflar]

I also think the "maintained package versions only" mode should become default in version 2 and instead of an --include-non-maintained-versions flag which would again load half of the Internet, exceptions should be added explicitly to the composer.json. Let's say I'm using symfony/filesystem in version 4.1.* which is an outdated version. I then have two options:

1. Update to a maintained version (of course recommended)
2. Add an explicit exception to the composer.json like e.g.

"maintenance-exceptions": {
    "symfony/filesystem": "4.1.*"
}

All these exception packages would be loaded into the resolving process explicitly plus then the regular resolving process with the maintained packages only starts. Not only would this have a positive effect on the resources needed during the resolving process but it's also making sure, developers use maintained versions only or if not, they explicitly accept to take the risk to work with an outdated version.

[graste]

Instead of an extra section one could introduce special syntax for the version constraint: "some/package": "4.1.*!". Easier on my mind instead of scrolling around for special cases. Another possibility with this: allow using unmaintained versions of specific packages even with the -mo cli argument (regardless whether it's default).

[kelunik]

I think it might be better to have the unmaintained versions on Packagist instead of the maintained ones, otherwise you need to edit it every time you release a new major, which can be easily forgotten.

[Toflar]

The additional "maintenance-exceptions" section

Instead of an extra section one could introduce special syntax for the version constraint: "some/package": "4.1.*!". Easier on my mind instead of scrolling around for special cases.

I don't think it makes sense to introduce a new special syntax as this would require quite some effort in adjusting the existing version constraint parser and it doesn't really make sense to me. In fact, we do not need an extra section at all. It could just be in the require section of the composer.json. Let me take you through an example:

{
    "require": {
        "symfony/messenger": "4.3.*"
    }
}

The resolving process is adjusted like so:

1. For the root dependencies, only fetch the package versions that match the root constraint.
2. For transitive dependencies, only fetch the maintained package versions.

In this case, we could already do an optimization without even thinking about introducing any maintained or not maintained setting. Instead of loading all symfony/messenger versions, we already know it should only load 4.3.* versions, the rest will never result in a valid set of dependencies. Now, symfony/messenger requires psr/log in version ~1.0. The additional optimization here now is if the maintainers of psr/log would mark only 1.1.* as being maintained. In that case, Composer could only load the package versions matching 1.1.*. In that case, this would result in a resolvable set of dependencies and only load the symfony/messenger 4.3.* and psr/log 1.1.* rules.

Now, if for some reason you need psr/log 1.0.0 (which is compatible with symfony/messenger 4.3) you would not be able to install it as it's not maintained anymore. However, if you adjusted your composer.json like so:

{
    "require": {
        "symfony/messenger": "4.3.*",
        "psr/log": "1.0.0"
    }
}

it would again work! For symfony/messenger nothing changes but as psr/log is now a root dependency we only load the matching versions for this package. Which in this case would be 1.0.0. During the resolving process, Composer would then see that symfony/messenger requires psr/log but instead of asking the provider for all the versions like it is doing as of today, it will understand that psr/log has already been loaded. Again, this would result in a resolvable set of dependencies but instead of loading all psr/log versions, only 1.0.0 is loaded.

The reason why I came up with something like

"maintenance-exceptions": {
    "psr/log": "1.0.0"
}

is pure cosmetics. It does nothing else as a "require" entry as I've described above but as a developer you can immediately see which requirements a project really has and which ones are there for (hopefully) only a short period of time and actually need to be fixed. Otherwise you won't immediately see you're working with outdated software. But sure, we can also just add it to the require section and have some composer unmaintained command that will then list all the installed versions that are unmaintained.

Marking package versions as maintained or unmaintained?

I think it might be better to have the unmaintained versions on Packagist instead of the maintained ones, otherwise you need to edit it every time you release a new major, which can be easily forgotten.

I was thinking about this too and it was my first goto and it's also the idea that's proposed in https://github.com/composer/packagist/issues/938. But there's two major reasons why I went for the opposite:

1. "you need to edit it every time you release a new major" is true but let's be honest, how often does that happen? This optimization is highly beneficial for projects that have a lot of tags/versions like Symfony, Laravel, Doctrine, Monolog etc. Most of which follow a strict release cycle, Symfony releases minors and majors every 6 months, so it would be two adjustments per year. That's not really much considering how much resources we can all save. Also, remember that you don't have to manage the maintained constraint. For your good old regular tiny packages that have four to five tags, you don't need to manage it. It might still help though, especially for the users so that they know which versions you're maintaining.

2. "which can be easily forgotten" - yes! And this is exactly why it should be the opposite. If we forget to update it, we again lose the optimization. Nobody will notice until someone comes along and debugs a bit. However, if we do the opposite, people will notice immediately because they don't get a resolvable set of dependencies with the -mo flag unless they add it to the require or maintenance-exceptions section explicitly. How long do you think that would take to notify the project maintainers in the case of Symfony? 5 minutes at max maybe? 😄

Making the -mo flag default in 1.x already?

I was thinking about the -mo or --maintained-only flag again and I think we could even make this behaviour the default one in a 1.x release. Yes, it does change behaviour but composer update is a command that's executed by users. Sure, it's also used in CI systems to test e.g. --prefer-lowest dependencies but it's not something that's used without user interaction. I think we don't need that flag. Pesonally, I would be okay with changing that because in 99.99% of all cases it's exactly what I want. I don't want any of my projects to run on non-maintained versions and if I do have any, I want to know about it. So even if my monthly travis cronjobs would suddenly fail because composer update --prefer-lowest suddenly does not resolve anymore, that's exactly what I want. That way I'm automatically notified that a project needs some love because any dependency changed the maintained versions and thus it requires me to adjust and update my code accordingly 😄

[nicolas-grekas]

I think it might be better to have the unmaintained versions on Packagist instead of the maintained ones, otherwise you need to edit it every time you release a new major, which can be easily forgotten.

I agree with this approach too, and I would go one step further by changing the wording. Instead of "maintained vs not-maintained", I would strongly suggest using "abandoned/vs not abandoned". The reason is that some packages are not maintained anymore, but are perfectly fine installing. They just work well. Ocramius/ProxyManager v2.1 comes to my mind: it works perfectly well and is the only option when using PHP 7.1. Yet, it's not "maintained" anymore, by some definition of "maintained".

We should actively discourage strategies that would aggressively use the new setting. Abandoned is a much stronger word that goes in this direction.

Also, about the target optimization, it doesn't need to be aggressive at all. Keeping only >=3.4 tags for all Symfony components is going to provide most of the expected benefit for the foreseeable future. Even >=2.8 could provide it I believe.

I'd also be in favor of declaring these abandoned tags in the composer.json of the package: the latest released version of it would be the place where packagist could look for building its tables.

[Toflar]

Yeah I guess my maintained approach is a paradigm shift and you guys are probably right about it being to aggressive. Even though I think it would be the correct approach, it might not be a suitable solution for the whole PHP ecosystem.

I also fancy the idea to mark packages abandoned in the composer.json because it's easy to do for many packages in monorepositories and you don't have to log in anywhere.

So in your opinion. If we mark packages as abandoned (whether that be in the composer.json or not) would you strictly disallow installing these packages? Or would you still have a flag to composer update? And if so, would it be --no-abandoned or --abandoned aka would it be "no abandoned packages" or "including abandoned packages" by default?

[nicolas-grekas]

composer up --allow-abandoned? By default, packagist would not publish abandoned versions in its json responses for repositories. but a new endpoint would allow listing all versions, including the abandoned ones, and that's where composer would go when this flag would be set. If that could work like that :)

[Toflar]

Plus, it should do root dependency optimizations as explained earlier. So if I have

{
    "require": {
        "symfony/messenger": "4.3.*"
    }
}

Then there sure is no need to load any other tags of symfony/messenger than the ones matching 4.3.*. We might need to adjust the The requested package symfony/messenger could not be found in any version, there may be a typo in the package name. to something like The requested package symfony/messenger could not be found in any suitable version, there may be a typo in the package name or they have been excluded from the resolving process completely because possibly matching versions have been marked abandoned or they were excluded by your root requirements.

If that could work like that :)

I think it should :) But I guess it's time for @Seldaek and @naderman to help us with that. If we can lay out a plan on how this could work, I'm happy to help.

I guess it could be a new constructor argument to Pool which would be passed on to ComposerRepository::whatProvides(). Packagist would need to dump "abandoned": true with every package version.

Not sure how to filter root requirements directly though.

[nicolas-grekas]

Then there sure is no need to load any other tags of symfony/messenger than the ones matching 4.3.*.

If we do this, then composer will not be able to provide useful error messages. That's the downside of pruning tags, flex has it. And I think it's not needed because the expected benefit is already achieved by the "abandoned" mechanism. Said another way: that'd be a micro-optimization with more drawbacks than benefits.

[Toflar]

I guess if we filter out everything that does not match our root requirements, the whole abandoned stuff would not be needed in the first place. I guess it would already reduce possible versions by a huge amount and thus be sufficient. And I could reduce even more by specifying additional root requirements and thus reducing the size of the ruleset even more.

If we do this, then composer will not be able to provide useful error messages. That's the downside of pruning tags, flex has it.

I know but we could adjust the messages, couldn't we? Given the example:

{
    "require": {
        "symfony/flex": "^1.4",
        "symfony/filesystem": "4.1.*"
    },
    "extra": {
        "symfony": {
            "require": "4.2.*"
        }
    }
}

You will get The requested package symfony/filesystem 4.1.* exists as symfony/filesystem[v4.2.1, v4.2.10, ...] but these are rejected by your constraint.. I mean, maybe we could extend that and compare the package to the root requirements. I know flex cannot but if we introduce the feature in Composer in that case you could say The requested package symfony/filesystem 4.1.* was excluded by your root requirement symfony/filesystem 4.2.*., couldn't you?

Also, the more I think about the abandoned stuff and flags you have to pass to composer update the more I think it's wrong :) Imho it's a technical problem, it's not a user problem. The user shouldn't be asked to include abandoned packages (or which ones) to the resolving process. In fact when I run composer update it should just try to use the optimized version first and if it cannot resolve, fall back to the current behaviour and try again. That way in 90% of all cases we get fast updates and for the other 10% we run the solver twice. Maybe enabling or disabling this feature can be a flag but the user itself shouldn't deal with internals and care about whether packages have been abandoned or excluded by your root requirements or, or, or. It's kind of a --fast-resolve and --full-resolve or so.

[nicolas-grekas]

I guess if we filter out everything that does not match our root requirements, the whole abandoned stuff would not be needed in the first place

That would break very quickly again with transitive deps. I know from the work on flex.

About the optimistic "it should just try to use the optimized version first", the word "just" hides a mountain. I could build crazy things with "just" antigravity :)

Solving a combinatorial explosion is hard. With little help from package authors we solve 99% of the issue and we have better readability about supported versions. Everyone wins.

[Seldaek]

I appreciate the enthusiasm here, but IMO this is a giant distraction from more important work for v2 which I'd much rather focus on. This can be bolted on later if we still think it's worth it, but there are other refactorings and improvement ongoing which hopefully will help or allow enough other improvements in which versions are automatically exluded (FYI AFAIK we already exclude things not matching root requirements btw, and indeed it's not enough due to transitive deps).

I find this will require a lot of manual work by maintainers, will add a lot of confusion for users, new flags etc, all this has a cost in terms of complexity (both for us and for users trying to understand composer). I know some users are very knowledgeable but the reality is a lot also simply run updates and when anything breaks they have no clue what is going on. Minimum-stability for example is already confusing a lot of users and that's been there for ages, so I don't take adding another layer of "stuff removing versions" lightly. Not saying it's a horrible idea, maybe it's a good one, but I don't think it's the right time with all the other stuff we have going on.

Those are my 2c..

[Toflar]

Thank you Jordi for taking the time to reply, highly appreciated :) I do understand your point of view but you know me: I still wanted to see what I can do without introducing new requirements on the packagist side etc. So what I wanted to double check was this statement:

FYI AFAIK we already exclude things not matching root requirements btw, and indeed it's not enough due to transitive deps

and I found that indeed only these requirements are added to the ruleset. However, the provider data the ComposerRespository provides is still loaded into memory completely. So even though we know that this data is not going to be needed for the resolving process, it's still loaded into memory thus slowing down the whole process and of course, using up more memory.

I've created a PR (https://github.com/composer/composer/pull/8275) where you can see the changes necessary to improve that. They are rather minimal but with this PR, everybody can improve the resolving process by specifying conflicts. Unfortunately, in case of Symfony you have to specify quite a few but in my test case I've added only three or four conflicts for some major Symfony packages and I could easily reduce memory usage and time needed by 30%. And the more I add to the conflicts, the more I performance I can gain 😄

[Toflar]

@nicolas-grekas care to give https://github.com/composer/composer/pull/8275 a try? 😊

[Seldaek]

Closing as I don't think this is needed from a perf point of view, and I am not sure the other aspects are really worth the effort either.

[stof]

The new PoolOptimizer in Composer indeed covers the performance benefits of this for the most part.

[naderman]

I think even without the performance benefits this could still be a very worthwhile feature, so going to keep this open for now.

Apart from projects you are very familiar with, especially agencies often interact with projects where they don't know all the details yet. So to them an easy way to check if all dependencies are still on supported branches would be very beneficial. The same is true for larger organizations who want to monitor a larger number of projects without necessarily being aware of the exact versions on each project at all times.

So overall I think it would still be beneficial to allow projects to explicitly mark certain version branches as maintained/supported, maybe even with a distinction for fully / security only. So tools can be created to check this information across a project's lock file.

[pwolanin]

Can we add a mechanism for this in the metadata even if packagist doesn't offer a mechanism to enter it yet?

[stof]

@pwolanin there is no agreement about what such mechanism should look like either. Eagerly adding something in the metadata without having a clear view of how it will work is a bad idea: once something is added in the metadata schema, it cannot be changed in backward incompatible ways anymore as this would break the ecosystem. The metadata schema can add new optional fields, but that's almost the only kind of changes allowed.

[drumm]

This would be useful for the Drupal community as well. We do have a flag for version ranges being supported by their maintainers.

Drupal Core has a release schedule, and could announce planned dates for dropping support of version ranges. This may be a separate issue, but it would be nice for composer to be able to warn of planned end-of-support.

[pwolanin]

@stof yes, my thought here (in discussion w/ drumm) is an optional field in the schema to indicate unsupported versions.

[naderman]

Like @stof said, before we can add anything to the metadata we need a proper proposal really thinking this through. It should answer at least the following and probably more:

• What exactly do we want to indicate, what does "support" mean, are there going to be levels? We should probably at least allow indicating a version will only receive security fixes or bugfixes, and maybe by support a project means whether they will actively provide help with problems? Do packages need to define different levels? If not, what exactly do we want to provide to make this useful without later running into problems because our spec was insufficient?
• What do we define support on? If you think about providing support it may make sense to define this on a version level, but if we're talking about whether security or bug fixes will be provided, this would happen in a new release, so it's rather about branches. So in which composer.json does this data get stored? latest of a branch, ignoring versions? On the default branch only defining this for all versions/branches?
• What's the process for modifying this when you made a mistake? We cannot modify/recreate tags just to modify some metadata which should be immutable. So you can't just have the metadata on specific versions without a way to override it.
• In how far is this consistent with how we handle other types of metadata? How does this get synchronized on e.g. partial updates which don't touch packages that have modified support status which users should probably find out about anyway?
• How does this work if you fork a package, you probably don't want to claim support of something you just forked for a single branch? Or does this not matter? How can you easily define that you're just providing support for one branch on the fork? Is that needed?

[stof]

@naderman even support is not immutable for a version. When a version goes out, it is maintained and becomes unmaintained at a later date.

[stof]

@pwolanin adding a field requires defining precisely how it works. A free-form field is totally useless to the ecosystem (you could just as well put that info as part of the package description in this case).

[pwolanin]

Yes, so what is the process for defining it - a PR? I think marking releases abandoned (as suggested at points above) may be simpler than marking them supported

[drumm]

What we have on Drupal.org is this UI: Nothing about supported versions is currently stored in the Git repository for the project.

It's hard to see, but the checkboxes for 8., 9., etc are disabled. So for determining "support", every release is in one and only one series for determining support. For example, if 8.* were "supported" and 8.1.* were "unsupported", we don't want to think through what takes precedence, so we just make supporting 8.* impossible. (7.* is a special case since we didn't use semantic versioning for that series.)

"Recommended" is only used on project pages on Drupal.org to add a "Recommended by the project’s maintainer." label, as seen on https://www.drupal.org/project/metatag. That concept should probably be dropped from Drupal.org. Making more-stable releases is a better indicator.

Personally, I say "support" is up to each maintainer. How they use their time and which issues they resolve is up to them. We can't impose any real rules on what a "supported" Drupal project is. We can provide tools for maintainers to communicate their intentions. There are of course many marked as "supported" which in reality have relatively-absent maintainers.

The one rule we have for Drupal projects is that if a security issue is reported against a supported release, that's covered by our security advisory policy - that issue needs to be resolved by the maintainer. If they do not eventually resolve the issue, the Drupal security team marks all release series as unsupported, publishes an advisory marking all releases as vulnerable, and updates the project page to mention the situation.

Drupal Core has governance and can say which branches get bug fixes, security fixes, new features, etc usually with at least a year of release dates planned out. https://www.drupal.org/project/drupal has short descriptions of the current state, details are at https://www.drupal.org/about/core/policies/core-release-cycles/overview-of-the-release-process & https://www.drupal.org/about/core/policies/core-release-cycles/schedule

[mxr576]

Building upon @drumm's insights, let me elaborate on how my composer audit extension package, known as the Drupal Dependency Quality Gate Composer Audit plugin, harnesses information from Drupal.org to identify potential long-term maintenance issues associated with Drupal package dependencies within the composer audit command.

The extension leverages data available on Drupal.org to scrutinize Drupal packages and, through the composer audit command, alerts users to potential challenges in maintaining dependencies over the long term. This proactive approach helps developers address issues related to the stability and reliability of their projects.

In terms of visual representation, the customized output of the composer audit, specifically the output of composer audit-changes, is integrated seamlessly into our code review tool. This tailored output provides a clear and concise view of the detected issues, allowing developers to swiftly assess and address potential problems in their codebase.

As illustrated by the Drupal example, having a maintenance status indicator on PHP packages' major and minor branches can deliver significant value. This indicator serves as a valuable tool for developers, offering insights into the health and longevity of their dependencies. By incorporating such indicators, composer audit could aid in making informed decisions regarding the inclusion or exclusion of specific packages in a project.

[naderman]

So to try and get some progress here, I think we need to define which things exactly we want to have metadata for and what that means/guarantees.

• supported: I think this should refer to actual support, like answering questions, I'm not sure we should consider that here, we have the support field in composer.json, and support may be provided for a fee only or at different quality levels which is definitely more information than I want to encode here
• maintained: This to me simply means there is someone still fixing bugs, I'm not sure if this should imply anything else? What about whether something is actively being kept compatible with 3rd party deps / other tools this package interacts with?
• bug-fixes-provided: An alternative to the above that's more explicit about just promising that bug fixes are provided at all
• security-fixes-provided: Definitely necessary, just not sure of the semantics, is this separate from the above, or implied in the above?
• recommended: This showed up in the Drupal example above, but I'm not sure this makes sense generally? Probably always ideal to use the latest version of a branch that is still maintained and/or supported?

Other ideas / proposals on how exactly the semantics on what you can define for a branch should generically work for all of Composer?

[alcaeus]

Allowing people to differentiate between bugs being fixed and security issues being fixed is a good idea. Those two should also take care of the more ambiguous supported and maintained flags. The concept of recommended can also vary quite a bit between projects, so I'm not sure it's a good idea to introduce that.

+1 for bug-fixes-provided and security-fixes-provided, probably on something like a . version.

[ausi]

I think bug-fixes-provided and security-fixes-provided make the most sense. For me the semantics are that all versions included in bug-fixes-provided are implicitly also in security-fixes-provided, so the latter is a union of bug and security.

So a config like:

"bug-fixes-provided": "4.2.* || 4.3.*",
"security-fixes-provided": "2.8.* || 3.4.*",

Means that the project provides security fixes for 2.8, 3.4, 4.2, 4.3 and bug fixes only for 4.2, 4.3.

The other categories are not needed IMO as they are ambiguous, I would interpret them as:

• supported is the same as bug-fixes-provided
• maintained is the same as the union of security-fixes-provided and bug-fixes-provided
• recommended are the latest patch versions of all versions in bug-fixes-provided
• abandoned are all versions not included in either security-fixes-provided or bug-fixes-provided