Swift registry support

[watt]

It looks like there's currently no support for package dependencies resolved from a package registry. That is, packages defined with the id syntax:

dependencies: [
    .package(id: "mona.LinkedList", .upToNextMajor(from: "1.0.0")),
],

If Package.swift contains a dependency like this, the swift_update_pkgs task fails with this error:

gazelle: Identity could not be determined.

From a cursory look at the source it seems like registry dependencies aren't part of the data model. I'm not sure what additional changes would be necessary on top of that.

I'm guessing it might be a pain to build this out if one doesn't have access to a registry already, since there are no public Swift registries I am aware of. I'm happy to help test things or provide samples from a project resolved against a registry if it's useful.

[luispadron]

Is this something you'd want to implement and PR @watt? We can answer questions in here if needed.

[watt]

Is this something you'd want to implement and PR @watt? We can answer questions in here if needed.

I'm willing to try, though I will definitely need some guidance.

Aside from updating the data models that map to JSON files, any initial thoughts on what logic might need to be touched?

[luispadron]

I'm not super familiar with this part of the code so maybe @cgrindel could be more helpful.

I'd try updating this area first to support registry ID: https://github.com/cgrindel/rules_swift_package_manager/blob/main/gazelle/internal/swiftpkg/dependency.go and seeing how far we get. We likely just need to map into the package name which should be possible.

[cgrindel]

@watt Thanks for filing this issue. What is added to the Package.resolved after swift package update? Is it just a URL?

As a little background, I am currently working #924 which will radically change how external dependencies are downloaded. In essence, it relies on the information stored in the Package.resolved to download the external Swift packages. If the Package.resolved contains a URL for the resolved version, then supporting this should be fairly straightforward. 🤞

[watt]

What is added to the Package.resolved after swift package update? Is it just a URL?

No, unfortunately, it's frustratingly light on data.

For this Package.swift:

// swift-tools-version: 5.7

import PackageDescription

let package = Package(
    name: "my-project",
    dependencies: [
        .package(id: "apple.swift-collections", from: "1.1.0"),
    ]
)

I get this Package.resolved:

{
  "pins" : [
    {
      "identity" : "apple.swift-collections",
      "kind" : "registry",
      "location" : "",
      "state" : {
        "version" : "1.1.0"
      }
    }
  ],
  "version" : 2
}

To reconstruct the remote URL we could read the registries.json config and figure out which registry this comes from, but this would require recreating a fair amount of behavior of SwiftPM.

Alternatively, the dependency shows up in the build directory as well, and it might be easier to just use that.

[cgrindel]

Unfortunately, we can't use the files downloaded in the build directory. The repository rule needs to be download the package from the information that is checked into client's repository. I think that we will need to read the registries.json, per your suggestion.

At a minimum, I presume that the registries.json contains a URL. Does it embed any authentication information, as well?

[watt]

I drafted up an implementation in #1043. It'll surely need some work, but in the included example case, this works for me.

This draft hardcodes a registry URL rather than reading registries.json. I think it won't be too hard to read, but I wasn't sure if it would be more idiomatic to read the registry config early, during the swift_update_pkgs task, and generate build rules that look have a pre-resolved URL like this:

swift_registry_package(
    name = "swiftpkg_apple.swift_collections",
    dependencies_index = "@//:swift_packages_index.json",
    url = "https://artifactory.global.square/artifactory/api/swift/swift-test/apple/swift-collections/1.1.0.zip",
)

Or to read the registry config later, during build, and generate build rules that look like this:

swift_registry_package(
    name = "swiftpkg_apple.swift_collections",
    dependencies_index = "@//:swift_packages_index.json",
    id = "apple.swift-collections",
    version = "1.1.0",
)

The latter mirrors Package.swift more closely but the work involved will effectively be the same either way, I think? This draft generates rules like the former.

[watt]

At a minimum, I presume that the registries.json contains a URL.

Yes. The format is described here. I think picking a registry is a matter of merging the user config with project config, and then choosing either a scoped registry or the default.

Does it embed any authentication information, as well?

Nope. Mine is stored in ~/.netrc.

[cgrindel]

I wasn't sure if it would be more idiomatic to read the registry config early

As part of #924, we are moving away from generating Bazel stuff in swift_update_pkgs. We will want to read it later.

I like your second option. However, I think that it will be something closer to this:

swift_registry_package(
    name = "swiftpkg_apple_swift_collections",
    registry_json = "@//:registries.json",
    id = "apple.swift-collections",
    version = "1.1.0",
)

The repository rule can read the registries file look up the URL and perform the download. WDT?

[watt]

As part of #924, we are moving away from generating Bazel stuff in swift_update_pkgs. We will want to read it later.

Ah, OK. Does that mean the swift_update_pkgs task goes away entirely, and swift rules are generated dynamically on demand (if that's even possible in bazel)? Not that it affects this, just curious.

I like your second option. However, I think that it will be something closer to this:

swift_registry_package(
    name = "swiftpkg_apple_swift_collections",
    registry_json = "@//:registries.json",
    id = "apple.swift-collections",
    version = "1.1.0",
)

The repository rule can read the registries file look up the URL and perform the download. WDT?

Is @//:registries.json meant to be configurable, like you could reference a different JSON file? If we did that, I think it might differ from a bit from SwiftPM? SwiftPM normally looks in two places (and merges them):

• user config in ~/.swiftpm/configuration/registries.json
• project config in .swiftpm/configuration/registries.json

swift package has a --config-path option that seems like it might override the user path but I'm not sure you can alter the project config path.

[watt]

I pushed an update that reads the registry config from starlark. For now, it uses a hardcoded path of @//:.swiftpm/configuration/registries.json rather than making the config location configurable.