Include the repository the package was sourced from in the lockfile

[G-Rath]

Is your feature request related to a problem? Please describe.

Currently it is not possible to determine the repository a package was sourced from using the composer.lock alone, and I'm pretty sure it's actually not even possible without making a few API requests in addition to having the composer.json.

This means security tools like osv-scanner can give false results since its possible for names to overlap across repositories.

For example, the Drupal community uses composer for managing module dependencies, which are sourced from a non-packagist repository with one for Drupal 7 modules and another for Drupal 8+ modules.

Modules typically (maybe always? though I don't know that for sure) have a single codebase for both versions, with tags prefixed with the major version of Drupal to denote new releases, which are then translated into semantic versions by dropping major version of Drupal when publishing, making the repository the only way to know if a package belongs to the Drupal 7 or Drupal 8+ ecosystems.

Without taking the repository into account, things can get weird because the actual versions don't need to be 1:1 - for example, https://www.drupal.org/sa-contrib-2023-034 was fixed in v1.4 for the D7 repository whereas for D8 it was fixed in v1.0

Describe the solution you'd like By including the repository as a property for each package in the lockfile, tools would be able to easily known where a package was sourced from.

I would also expect this could be used by composer internally to skip some API calls, as my understanding is currently it just queries each repository for each package until one reports they have it, and so e.g. that'd probably mean every Drupal project results in a number of calls to the packagist repository that'll always fail/404 (since such projects will typically have a ton of Drupal modules)

Describe alternatives you've considered For Drupal we can probably work around this in a couple of ways, such as by looking at the package namespace (as I believe they're always published under drupal/) or at the extra.drupal key (which again I think is always there), but they're not as strong a guarantee and are Drupal-specific workarounds.

We also might be able to use the notification-url property as that looks and sounds like it might always point at the repository, but I've not yet dug into that further to confirm if that is the case or just a coincidence.

Additional context

[Seldaek]

This is kind of a made-up problem due to the way the Drupal 7/8 repos work, I don't think it affects others remotely as much.

It's also not so easy to have a unique identifier for a repo, when you consider path, artifact, package repo types etc, so I am not sure if I would find this that great. For packagist.org you can indeed use notification-url to identify it. This for sure won't work for all repos tho as many do not support this endpoint.

[stof]

I would also expect this could be used by composer internally to skip some API calls, as my understanding is currently it just queries each repository for each package until one reports they have it, and so e.g. that'd probably mean every Drupal project results in a number of calls to the packagist repository that'll always fail/404 (since such projects will typically have a ton of Drupal modules)

I don't understand how this could be done: the lock file is the output of the dependency resolution. It cannot become a source of metadata to know where packages come from

[G-Rath]

This is kind of a made-up problem due to the way the Drupal 7/8 repos work, I don't think it affects others remotely as much.

I don't entirely disagree with that, but I still think it's useful information from an audit POV to be able to statically know where composer grabbed a particular package in the lock from (and ideally where it should grab it the next time), and one that happens to potentially help with the Drupal ecosystem.

It cannot become a source of metadata to know where packages come from

The lockfile is already a source of metadata to know which package to install right? I don't think it's a leap to also use it to answer the question of where a package is expected to be installed from - I'm not strictly saying that composer should blindly just use the repository field from the lockfile by itself e.g. it'd be completely fine (and probably best) if it rejected a lockfile that had a package from a repository that is not specified in composer.json.

I'm curious as to what happens currently, if I have repositories A and B and install package hello/world which is currently only on B; then later I do a fresh install but someone published a hello/world package to A that happens to reasonably overlap enough (e.g. has the same version, constraints are compatible, etc) - does composer just install A/hello/world?

I do see the setup examples on wpackagist include an only field in the configuration which I assume meant to help with this, and maybe drupal.org should be including that in their setup, but overall it feels to me like having the repository captured in the lockfile is a very cheap way of allowing some extra security even if it's not directly used by composer itself (though I would hope if it was present, composer would at least check and warn if its value changed on a second install)

[Seldaek]

I'd suggest you read up https://getcomposer.org/doc/articles/repository-priorities.md#repository-priorities to make sure you have a full understanding here.

I'm curious as to what happens currently, if I have repositories A and B and install package hello/world which is currently only on B; then later I do a fresh install but someone published a hello/world package to A that happens to reasonably overlap enough (e.g. has the same version, constraints are compatible, etc) - does composer just install A/hello/world?

On a fresh install, it will just install what is in the lock file, meaning it remains hello/world from B.

If you do an update of hello/world though, and it is now available from A then it will possibly update it from there. In practice this is almost never an issue tho because of the priority concept described in the link above.

I do see your point about audits tho, and I think this is an issue I'll keep in mind when looking at future metadata changes for sure, but I don't really want to do too many lock file format changes as that causes lots of noise in diffs, so probably will keep it for later and do it together with other things.

[G-Rath]

On a fresh install, it will just install what is in the lock file, meaning it remains hello/world from B.

How does it know to get it from B though, when the lockfile doesn't have that information?

I'm talking about a situation where we've got two repositories configured with no other settings about them so composer should be free to pull any package from either of them, and this is a completely fresh install say on a CI system with no prior vendor or even a cache.

[Seldaek]

The lock file has all the information needed to install the packages (source/dist URLs). There is no need to talk to the repo at install time.

[G-Rath]

Oh, have I somehow managed to not realize that composer is just downloading from e.g. https;//api.github.com/etc/etc and unzipping?! 🤦

While I feel a bit foolish now given how long I've been using composer and how relevant that is to this issue, I'm glad to have learnt about it - I always thought that composer was pulling the packages from the repository but if it's not then yeah I agree more with your initial comments.

I will leave you to decide if there would still be any value in having the repository in the lockfile, as I don't want to say now there's completely no benefit, but definitely far less useful outside of this Drupal/OSV challenge than I initially thought.

Thanks for taking the time to talk this through with me, and the work you've put in on composer ❤

[github-actions[bot]]

This issue has been automatically marked Stale and will be closed in 15 days if no further activity happens.

[G-Rath]

Ek I forgot to post on this when the bot first posted - I still think this information would be good to have in the lockfile, and it sounded like the maintainers agreed - should this remain open so that it can be actioned as part of the next major lockfile change?