Dependency details page

[daveverwer]

I spent some time today thinking about what we’d do with dependencies if we had unlimited resources and time! 😂 This is for discussion and won’t ever be implemented in one step, but I think there are some interesting features here not only to view dependency information, but for package discovery.

Package Page Metadata

We currently have the total number of dependencies listed in the package metadata section. We have some options for what we could display here:

1. Total number of dependencies (what we have now).
2. The number of package dependencies (see below for a definition).
3. The number of test-only dependencies (see below too).
4. The number of packages that depend on this package.
5. Any combination of 2, 3, and 4.

Note: I’m calling the dependencies that make their way into the app/project that’s being built “package dependencies” and referring to dependencies that are only used for testing “test-only” dependencies. If a package is used both in the app/project and in the tests, it should be counted as a package dependency. Test-only dependencies should be truly test-only.

After thinking about it, I think I would be in favour of two lines of metadata information:

• “This package depends on 4 other packages.”
• “5 packages depend on this package.”

We’ll need a bit of work on the icons next to these lines so that if both are displayed, they are well differentiated.

The first line would be visible as long as we were able to inspect dependency information and would say “This package has no package dependencies.” in that case, just as it does now. If there are no packages that depend on the package being viewed, the second line shouldn’t be included. I expect the number of packages that have other packages depend on them to be relatively small and this will keeps things clearer.

The number of test-only packages is important and is a factor people should consider when picking a dependency, but my gut feeling says what people are mainly concerned about is what packages will ship code along with their project.

“4 other packages” and “5 packages” would be links to the same page, described below. The links would also serve to highlight the important numbers.

Package Dependency Detail Page

Ultimately this page is all about displaying a list of packages, but there’s opportunity for more.

Most of what I have here depends on being able to identify dependencies as packages in the index. We may see easier ways to do this when the package registry changes arrive, but we’ll also need to cope with URLs as SPM will continue to support importing via direct URL. I think we could build something that normalises the URL then grabs the domain name (which we wouldn’t even really need to do, since everything we have right now is on GitHub) and repo/owner and I think we’d have something fairly reliable without too much fuss.

After a page title, I’d like to summarise the full dependency information in a sentence, which I know @finestructure will love. 😂 Something like this:

“[Package] has 4 package dependencies and 3 test-only dependencies. 2 packages depend on [Package].”

Dependency Metadata

I’d like to style top-level packages with richer metadata and then their dependencies with less metadata. Something like this:

More detail:

• Packages should show package name if we have it in the index, and repo name if we don’t.
• The version of the dependency should be shown highlighted in a lozenge.
• Top right will be either TEST-ONLY or PACKAGE DEPENDENCY.
• Description and other package list metadata are also displayed for top-level dependencies.
• The package metadata row for top-level dependencies should be a link to the package. The package name for tree dependencies also link to the package page.
• The tree comes off the metadata row showing just package name (or repo name if we don’t have the package) and the version it’s dependent on.

I didn’t cover it in this mockup, but I’d like to explore giving two links per package in this view. Obviously we will link to the package page from the dependencies, but I’d love to explore something that also allows a direct link to the dependency detail page (this page) for each dependency.

Controlling the List

I’d also like to allow this list to be filtered with a set of two controls:

• Toggle between top-level only and all-dependencies.
• Toggle between package dependencies and test-only dependencies.

This filtering could easily be done client-side or server-side, but it might make more sense to do it client-side. The controls should be fairly self-explanatory, but would give an easy way to visualise answers to some common questions about a package’s dependencies.

Page Layout

The rough layout of the page would be roughly this:

Finally, the sidebar. This would be a fairly simple list of links to packages that depend on this package. Not much more to say about that.

[daveverwer]

Soemthing I notice straight away is that the tree view of dependent packages would also need to show if a package was test-only.

[finestructure]

Sounds great!

One thing that keeps tripping me up when talking about dependencies is the bi-directional use of "depend" in "dependency/dependent package". I always need to think for a moment which way the arrow is pointing. Perhaps we could establish "is used by" or "is integrated by" as terminology to avoid "depend" showing up in both. That might make labelling a bit clearer.

I.e. we'd say

• "Is integrated by n packages."
• "Integrating packages"

to avoid "depend" when looking at it from the dependent package's point of view.

[daveverwer]

One thing that keeps tripping me up when talking about dependencies is the bi-directional use of "depend" in "dependency/dependent package". I always need to think for a moment which way the arrow is pointing. Perhaps we could establish "is used by" or "is integrated by" as terminology to avoid "depend" showing up in both. That might make labelling a bit clearer.

Yea this is a problem 😬 and is why I tried to make it explicit by including "this package" and "other packages" in the wording.

We could try to establish new terminology, but that feels maybe like just climbing a different hill 😂

[perezd]

One thing that keeps tripping me up when talking about dependencies is the bi-directional use of "depend" in "dependency/dependent package"

I've always found dependencies (us) vs dependents (them) somewhat straighforward, here's a good example of it in use: https://deps.dev/maven/com.google.protobuf%3Aprotobuf-java

[finestructure]

I've written up a note to track the backend implementation of this here: https://noteplan.co/n/DCE2A0BF-95BB-452E-B8E0-756E7E206B20

The reason I'm planning to keep it in that that note for now is that it's a sprawling task and it's easier to edit and track it in that document for now. I'll probably break it into GH issues once the actual tasks are more refined and all the "researchy" bits are done.

[finestructure]

There are at most eight queries we'll want to run, I believe. They are basically variants of two queries run with (=>) or without (→) test targets and for immediate (→) or including transient dependencies (→→):

• what other packages does package P depend on: (P → ?) | (P => ?) | (P →→ ?) | (P =>=> ?)
• what other packages depend on package P: (? → P) | (? => P) | (? →→ P) | (? =>=> P)

We have the following (existing and possibly future) datasets answering these queries:

1. versions.resolved_dependencies: (P =>=> ?), (? =>=> P)
2. "product dependency tree": (P → ?), (P →→ ?), (? → P), (? →→ P)
3. "full dependency tree": (P => ?), (P =>=> ?), (? => P), (? =>=> P) (this can be seen a superset of resolved_dependencies, which can be derived from the full tree)

The queries that dataset 3 enables over what 1 provides are:

• which test dependencies does P depend on directly: (P => ?)
• which packages depend on P as a test dependency directly: (? => P)

Test dependency relationship are probably only relevant in two regards:

• Software BOM, to determine security surface (? =>=> P)
• overall package usage stats (? =>=> P)

I can possibly see us wanting to include test dependencies if we "draw" a dependency tree (by ways of package links on the package page). But I wonder if we really want test dependencies there. If we do, that adds

• what packages does P use as a test dependency directly (P => ?)

This would require us to report the "full dependency tree". I can't quite see a use case for

• what packages use P as a test dependency directly (? => P)

but it also wouldn't matter because it would come for free if we need "what packages does P use as a test dependency directly", which surely is of more importance.

I think it's best to proceed by just adding dataset 2 for now. If we later determine we need the queries provided by dataset 3, we can either add it or consider replacing dataset 1.

[finestructure]

As discussed yesterday, dataset 3 isn't even viable for the full (P =>=> ?) or (? =>=> P) queries, at least not if we acquire it as planned, because we won't get test dependencies beyond direct dependencies.

Doing it recursively is going to be prohibitive, so we'll just have to stick with datasets 1 + 2.

Edit: that actually invalidates the queries of both datasets 1 and 3. Regardless, the primary use-case of any transient dependency queries is to assess exposure to security alerts and that's best covered by the actual Package.resolved surface, which dataset 1 covers.

There's just not that much to be gained by knowing which direct test target dependencies exist between packages.

[daveverwer]

Just pinging #1610 here as there was useful discussion related to this in https://github.com/SwiftPackageIndex/SwiftPackageIndex-Server/pull/1606#issue-1159576993 and https://github.com/SwiftPackageIndex/SwiftPackageIndex-Server/pull/1606#issuecomment-1060747909.

[finestructure]

Mostly done defining the schema and the queries.

Keeping this in notes again because it's not quite done and quite long.

[finestructure]

Cross-posted to the Swift forums

We've started drafting how we're going to improve the dependency information we track in the Swift Package Index. Here are the two issues to follow along if you're interested:

• https://github.com/SwiftPackageIndex/SwiftPackageIndex-Server/issues/1321
• https://github.com/SwiftPackageIndex/SwiftPackageIndex-Server/issues/1300

At a glance, I don't think these efforts would change the dependency count to one for swift-json. When we started laying this out, plugins didn't exist yet but even now it's not clear to me how we would exclude such dependencies.

As far as I can tell from looking at swift-json's manifest, there's nothing there that would tell us how it's being used.

For "test dependencies", this is a bit different. Here, we can figure out if dependencies are only used in test targets and exclude them.

We also want to be careful when excluding dependencies in general. For instance, even when we start distinguishing "product" and "test" dependencies, we'll still want to show the full number of dependencies for a package (in addition to just "product" dependencies).

This is because there are two different scenarios to consider:

1. How many dependencies do I need to deal with if I want to build my product, even if I'm willing to accept broken tests.
2. What's my exposure to supply-chain attacks against my build/dev process in general.

For instance, if you're willing to cut some corners, a low number or "product" dependencies will be helpful if a dependency update is causing build issues: you'll have a better to chance to at least be able to build your product.

However, if you want to know what your full exposure to anything happening with your build or dev process through malicious third-party dependencies is that number alone would be misleading.

I'm not sure what the best answer is for plugins. My feeling is that there are probably different groups of plugins:

1. source generators that generate input for a product
2. documentation generators
3. some other processing steps that don't affect the resulting binary or documentation

Group 1 feels like it should be a "product" dependency.

Group 2 could be either or. You could argue that if push comes to shove you can create a binary without updating the docs so it's not strictly a "product" dependency. However, if you're building for an SDK, doc updates are probably more tightly integrated and therefore the plugin could be regarded as a "product" dependency. It's a matter of policy really.

Group 3: I'm not sure what these would be exactly. Perhaps "swift-format"? That's a bit like docs: you can yank it if you need to build urgently but if it's your process then it's probably a "product" dependency.

I guess this is a long-winded way of saying: I'm not sure how we would determine what's part of the product and what isn't. Certainly from a supply-chain exposure perspective these should definitely be included, and I think the best default we could choose for the three groups I listed above would be to simply include them as "product" dependencies. It just so happens that that's pretty much the only thing we can do as well :)

I hope that makes sense!