RFC: Add staging workflow for CI and human interoperability

[djsauble]

See the RFC

[dominykas]

Sorry, maybe I missed something, I know there's the RFC meetings, but can't find the calendar for them - when is this going to be discussed? Is it better to wait for the meeting or post the questions now?

[isaacs]

The registry will support a data structure like this:

{
  "stagedVersions": {
    "whatever": "other",
    "metadata": "we wanna",
    "put": "here",
    "versions": {
      "1.2.3": { "manifest": "data..." },
      "1.2.4": { "another": "staged manifest..." }
    }
  }
}

Discussion still to be had about what the dist field ought to look like in staged version manifests.

[ljharb]

Ideally there'd be more information than just "version", "package.json", and "tarball" - git sha, in particular, would be helpful?

[wesleytodd]

On the call we discussed the workflows around the tarballs and the package locks. The concern was that if you stage a package and then a user installs from the staging it would mean their tarball location moved and then they would have to do an update of their package lock after the package is promoted.

The reason I was concerned about this is because we talked about installing from the staged location, and I imagined that this would enable beta testers and other similar workflows. This was a mistake. Any workflow which relies on testing from the staging area should been done with pre-release versions and dist-tags.

I have changed my thoughts here and I think as this feature gets rolled out we should make sure it is clear that is not a supported use case. I would like to propose two additions:

1. The --stage flag on install only works when installing a single package
2. Installing from --stage is not reflected in the package.json or package-lock.json

I think these would help prevent accidental usage of this feature in a way which I had originally imagined which had problematic DX. You would still be able to run something like a CI test by running $ npm i --stage pkg, but it would not persist across installs or be locked accidentally.

[ljharb]

I don’t think it should be possible for anyone but an owner to install a staged build; and ideally with the restrictions Wes outlined.

[wesleytodd]

I thought about adding that, but I wonder if there is reason to let un-authed CI servers install it? So I could set my publish CI pipeline to look for a staged version to test against but let it do so unauthed so I didn't need a CI token for that part.

[ljharb]

If unauthed people can install a staged version, they can depend on it, and staging serves no purpose.

[wesleytodd]

They can depend on it by going strongly out of their way if we add the package.json changes. Seems fine to me if they would have to do npm i --stage pkg every build separate from npm i. But then this makes me thing we should for sure also specify that once promoted the --stage version tarball is no longer available.

[dominykas]

Some things we also discussed in the call, that should probably make it into the RFC (sorry for duplicating stuff already being discussed, just making sure we've got everything covered):

• for audit purposes, it should be possible to see both users - the one who staged the package and the one who promoted it
  • in the meeting we discussed that npm_user is the person who published, and a new field needs to be added for the person who promotes the package
• what happens with the lock files
  • can they remain unchanged after the package is promoted? I think we discussed that the answer to this one is probably "yes", i.e. if the client detects that the version was staged, but no longer is, and the integrity matches, then there is no need to modify the lock file?
  • if a different tarball is staged at the same version, then install needs to fail
  • is there a need for a new flag in the lock file or should the fact the tarball is staged be detectable from the URL or smth?
  • if you have a staged package (either by an explicit flag or detected via URL) in the lock file - is it ok to install without the --stage flag?
• what happens with the tarball URLs
  • does the URL change after the package is promoted?
  • can the URLs be aliased?
  • are the tarballs at the staged URL mutable?
  • caching is also relevant in the context of this discussion (for people using various proxies and replicating registries)

---

Mike also suggested we post the use cases for this feature. For me, right now, the only use case I do care about a lot is to have an area where I can put a tarball without 2FA, and then promote that tarball with 2FA. I don't particularly care if that tarball is installable, TBH - and maybe that's also an option for the MVP, just so we have something to start playing with?

[dominykas]

Installing from --stage is not reflected in the package.json or package-lock.json

Not sure that's possible, though? If package.json is not updated, then it will retain the semver range which might not match the staged version? Would you have to update the package.json beforehand? This would then be broken completely as npm i would never work?

I like where you're going with the --stage flag being required and only working explicitly with the package name - in that case it does make sense to not update the package-lock.json, but I wonder if that might not create even more confusion?

Would a restriction of --stage name not introduce issues with deduplication and/or transient deps? Not that these can't be solved, just pointing out the possible implications.

I also like the idea of only allowing the authorized people to use the staged version - if this can be done with the initial version, maybe we don't have to solve all of the tarball URL/package-lock problems, as the usage of the feature would be limited and could feed back into the ultimate decisions?

[ljharb]

I don’t think a staged version should ever be allowed to exist in a lock file, regardless of who dan install it.

[isaacs]

If you can install it, it belongs in a lockfile. There's no reason why we should not lock it once installed, as that can only result in build irregularities. And if you can't install it, why are we staging it?

Even if staging requires that someone go absurdly out of their way, and do npm install https://registry.npmjs.org/foo/-/staged/some/kind/of/uuid/foo-1.2.3.tgz, well, people will just do that. Let's not make people jump through dumb hoops. It's the equivalent of disabling copy/paste on a password field.

Requiring a login to install a staged public package is also not reasonable, and bloats the complexity for implementing this pretty significantly.

I'm not ready to throw the baby out with the bathwater here. Let's not steer so far away from a hypothetical DX problem without at least thinking through how to fix it, and establishing how much of a problem it really is, and providing gentler guardrails to help people avoid it.

Here's what I propose:

• When installing, if --include-staged is not provided, then staged versions are never considered. (This solves like 90% of the DX issue, because you have to opt in to using a staged package in the first place.)
• If --include-staged is provided, then staged packages are still lower priority than published packages. (This solves 90% of the rest; the chances of getting a staged package by accident are effectively nil, and a published version will always be used if possible.)
• When promoting, ensure that the dist.tarball path of the staged copy of the tarball continues to be valid and resolve to the final published URL of the tarball (possibly via a 301 redirect). (This solves most of what's left; if you lock a staged package that then gets promoted, nothing breaks.)
• When overwriting a staged version, the tarball url changes, and the previous staged tarball URL responds with ideally a 410 Gone, but may respond with a 404 Not Found. (And here we have our tiny fraction of cases which will actually affect DX in a negative way. But it's clear, you opted in, the error message says what to do, and you probably want to take note of this anyway.) A redirect is not appropriate in this case, because the integrity will be different, so anyone fetching from the previous location is going to get a harder to debug integrity error if we redirect it.

This is a lot simpler, lets the registry be in charge of what's what, and lets the CLI just do its current default behavior with the slight addition of checking packument.stagedVersions.versions for matches. (Ie, just a change to npm-pick-manifest, but no changes required to any other part of the system, at least for installation.)

The authorization is still just what it always is. No special auth for special parts of the packument. If you can publish, you can publish, if you can see it, you can install it.

The resulting data structure could look something like this:

{
  "name": "the-world",
  "dist-tags": {
    "latest": "1.2.3"
  },
  "versions": {
    "1.2.3": {
      "name": "the-world",
      "description:": "is a stage",
      "version": "1.2.3",
      "_npmUser": "publishing-stager",
      "_npmUserPromoted": "promoting-stager",
      "dist": {
        "tarball": "https://registry.npmjs.org/the-world/-/the-world-1.2.3.tgz",
        "integrity": "sha512-deadbeefcafebadholymolywowsuchstage"
      }
    }
  },
  "stagedVersions": {
    "log": [
      "maybe some kind of bookkeeping here?  what versions staged at which time, etc."
    ],
    "versions": {
      "1.2.4": {
        "dist": {
          "tarball": "https://registry.npmjs.org/the-world/-/staged/sha512-content-hash-url-safe-base64/the-world-1.2.4.tgz",
          "integrity": "sha512-content-hash-normal-base64"
        }
      }
    }
  }
}

The registry could then have a 301 redirect set up from any /:pkg/-/staged/:content-address/:file to the appropriate file. This wouldn't even have to be done explicitly for staged promotions; any file could be fetched this way, and it could be kept up to date with each file we store. And, since the url is deterministically different from the final artifact url, we can have different caching semantics around it. (Or not - the content-address is right in there, it's not like anything else could ever be at that URL.)

[isaacs]

If we wanted to go the extra mile for the user, Pacote could detect these 404 errors for tarballs, see that it's a staged tarball URL, re-fetch the packument and update the resolved value.

I start to worry a bit that that might be too magical, but otoh, maybe it's exactly the right amount of magical. In any event, we could explore that with this data and URL structure, and decide later whether it's worth doing.

[isaacs]

The net net on this would be:

• Use a staged package: npm install foo@2 --include-staged (as long as 2.x versions are all in staging, like if it's a beta or somethign, and 1.x is the published versions).
• When foo@2 is promoted, then npm install foo@2 --include-staged will pick up the promoted version.
• If the staged foo@2 depends on a staged bar@3, then ok, that'll Just Work, because you're in "include staged" mode.
• When the pkg is promoted and the resolved changes, the package lock doesn't break. (It might get slightly less canonical or cacheable, but then again, it might not. Maybe we want to have content-addresses in our tarball urls!)
• If the staged version is overwritten with a new staged version, then the build will break (unless Pacote is taught to just handle this, which may be worthwhile, and would be possible.)

I don't see any DX issues that justify restricting the utility of staged packages, but maybe I'm missing something.

[ljharb]

To me, the entire point of the feature is to be able to avoid committing to anything published - and therefore usable - prior to promotion. I want to be able to, for example, have every master build in CI generate a staged version, but have them all secret - ie, impossible to even view for non-owners - until an owner selects one to promote.

I would prefer to see "testing a staged version" as a non-goal, since that's what prereleases are for - the value here for me is "i don't want to let anyone but a human do npm publish, but i want to be able to do all the prep steps for publishing separately, and without 2FA" (which i believe is also the use case for conventional-release?)

If 2FA is not required to stage something, and staged version can be installed, then that seems like a pretty large potential security hole for anybody using --include-staged?

[isaacs]

In a case where you have a lot of packages that all work together, I'd really want to have 2 integration tests run before any publish is promoted from staging to release:

1. Run this module's tests against all published versions of the others in the set.
2. Run this module's tests against all staged versions of others in the set.

Otherwise, it's not really an integration test, because you're not testing the integration with current and next.

As for using prereleases for this, yeah, you can effectively get all of this with prereleases. But part of the goal of this feature is that it lets you minimize version churn (including prerelease version churn in git history). This allows for something in between "npm install github:user/foo#next" and "npm install foo@next". Of course there's already ways to do all of this already; there's ways to do everything already (including publishing from CI while still having TFA protection). They're just less convenient. I don't see why making staged publishes less convenient is a value. What harm comes from it?

If we make staged publishes effectively hidden, then we also make it intolerably friction-ful to have the registry decide to temporarily stage publishes that appear to be suspect. If the staged copy is available behind a flag, then the user is temporarily inconvenienced, but not completely blocked in that scenario.

If 2FA is not required to stage something, and staged version can be installed, then that seems like a pretty large potential security hole for anybody using --include-staged?

I don't think so. (a) It's opt-in. So, yes, you can shoot yourself in the foot by setting it in a .npmrc config or something, but you can also get into very similar trouble installing from a git repo. And (b), staged package versions would always be deprioritized, so even when opting in, you'd only get the staged copies if they're the only thing that satisfies the dep. (Allowing a monorepo to publish 100 packages to staging, verify that they all work together, fix whatever problems occur, and then promote them all together.)

[isaacs]

Also, 2FA is not required to publish today, so this isn't really much of a reduction in the security stance. Requiring 2FA for all publishes would be a big step, and I'd get ready for torches and pitchforks as a result.

[ljharb]

To clarify; I believe the desire (which i share) is for a package owner to require 2FA for promotion but not for staging, precisely so that it's convenient.

I guess I don't understand why "less version churn" is a value - https://twitter.com/izs/status/603936197941993476

[dominykas]

Just to clarify:

1. Someone installs a package with --include-staged
2. It gets added to a lock file
3. Someone else runs npm install for that lock file (the version is still staged)

Do they get an error (because the package is staged) or does everything proceed as if it's business as usual (because it's in the lock file and integrity matches)? Or do they get a confirmation step or at least a warning?

[isaacs]

@dominykas With the approach I'm proposing, the tarball url would get in the lockfile (default behavior today, in other words). The subsequent npm install user would get that staged version (if it's still available and has not been overridden), because fetching the tarball would be allowed. (npm does not fetch the packument unnecessarily if it can get everything it needs from the lockfile, so it has no way of knowing that it's a staged version; it's just a pre-resolved tarball url.)

This is required to be able to run headless CI jobs against staged versions of things, which is a use case I'd like to be able to support.

[isaacs]

I guess I don't understand why "less version churn" is a value - https://twitter.com/izs/status/603936197941993476

Yes, as we all know, there's no shortage of integers. But one of the use cases that has been brought up in reference to this feature is being able to more deterministically test and release projects that are composed of many different packages. Prerelease tags aren't an ideal fit for that use case.

[ljharb]

That use case seems somewhat in conflict between the use cases I described; perhaps we could make "it's installable" something that the stager can opt in to? like npm stage --installable or something? (preferably, not installable by default)

[wesleytodd]

I would prefer to see "testing a staged version" as a non-goal

As for using prereleases for this, yeah, you can effectively get all of this with prereleases.

This! I was wrong to think this was a good idea, this is what dist-tags and pre-release versions are for. The only exception to this is that a sanity check that the tarball the cli generated is what you expected (for example didn't accidentally .npmignore something your package needs). But this is achievable if only an owner can see/install it.

What harm comes from it?

The harm is that many other tools and workflows are coupled to today's behavior. We need to make sure we don't outright break integrations and things like registry proxies. Once end users can couple to staged versions it means they will start to have workflows which depend on it, and back-tracking later would be very difficult.

If we make staged publishes effectively hidden, then we also make it intolerably friction-ful to have the registry decide to temporarily stage publishes that appear to be suspect. If the staged copy is available behind a flag, then the user is temporarily inconvenienced, but not completely blocked in that scenario.

This is a use case I have not seen brought up here before. Are we sure this should be a goal? I like the idea in general, but I wonder if it could be served by an explicit feature for security warnings? Especially with the "install any staged package" behavior, if we used this as a work around it would mean folks opening the flood gates just to get a sip of water. For example, package-a has a auto-staged security warning which the user knows is invalid, so user does npm i --included-staged but didn't realize that pacakge-b also has one which is really a vulnerability. I like the idea, but I think we need to think through the ramifications before including it as a goal in this feature.

I guess I don't understand why "less version churn" is a value

Agreed, especially in pre-release versions.

"it's installable" something that the stager can opt in to? like npm stage --installable or something?

While this would get us out of the alignment bind we might be in today, I think this would be a mistake because it would let users opt into using this feature in a way which is less optimal than pre-release and dist-tags.

[ThisIsMissEm]

@evocateur I believe this RFC may help with the "one shot" publish issue we had in lerna, where we switched to using a tmp dist-tag.

[evocateur]

@ThisIsMissEm It's an interesting variation, to be sure. I think for the most part, dist-tags are still the way to go to ensure the unity of a given release. There is certainly tooling to improve in terms of recovery from partial releases, especially the "promotion" of the temporary tag once all the (previously) failed packages have been successfully published.

[wesleytodd]

I believe this RFC may help with the "one shot" publish issue we had in lerna, where we switched to using a tmp dist-tag.

I think this is specifically a design goal, to support this use case, although it is not called out as such in the RFC. Maybe it should be added?

[isaacs]

While there are some potential footguns avoided by adding the restriction that only package owners can load staged packages, it adds significant complexity on both the server and client implementations, and (depending on how it's implemented) potentially adds a few even more concerning footguns. I hope, in this message, to make it clear that is not feasible, so we can drop it from this discussion.

Let's say we do add the restriction that staged packages cannot be installed by non-owners. This can be implemented (as far as I can tell) in 5 different broad strategies, all of them with profound challenges.

1. Strictly a client-side restriction. The CLI just won't select packages for installation from the stagedVersions set unless the current user is in the maintainers list.
2. We exclude the stagedVersions object from the packument if the fetching user is not in the maintainers list.
3. We block the download of a staged tarball URL if the user is not in the maintainers list.
4. We use a different packument target entirely, and in order to download staged versions (even for inspection), one has to use a different packument endpoint.
5. Staged published artifacts do not exist in any packument at all, and some other out-of-band means would be implemented to find out about them. (This is really a special case of 4, where "other packument target" might be a completely different non-packument endpoint.)

(1) is, of course, not a real restriction in any sort of security sense, but that might be fine. Ie, a staged package isn't really securely hidden, let's say, just restricted by a handshake agreement that prevents the CLI from trying to fetch it unless the user is an owner. Leaving aside the question of whether other npm clients would enforce the same restriction, the bigger challenge comes about when we consider that the npm-pick-manifest module would have to know, at install time, your npm username. Without making a fetch to the registry, it doesn't know that. We only store an opaque bearer token that tells the server who you are. By design, for security, your npmrc file does not contain your username. In order to get the username from a bearer token, you have to hit the /-/whoami endpoint. (This way, npmrc files containing formerly-valid logins don't leak any information if the token is revoked.)

Having to hit a registry endpoint to be able to even select a package version for installation is pretty crummy. Authed requests are slower and more costly, so we try to minimize them as much as possible. Any unscoped packages bypass that part of the registry stack entirely, and the auth stack for scoped packages has a lot of shortcuts to quickly jump from a bearer token and action to an answer about whether that's allowed. What shows up in the maintainers list is a cached projection of the actual authorization set. (Also, we'd have to always fetch in fullMetadata mode, because corgis don't list maintainers.)

The fact that it's opt-in would help with the perf somewhat, but there's still the question of maintenance burden and complexity, since npm-pick-manifest now depends on more than just the packument as a discrete object. And once the staged package tarball url is in a lockfile, we'll have to track that it's staged (and, when doing that install, whether the user has the right to download that thing or not). It makes my head spin trying to work out the edge cases in this. If someone else wants to take it on in a fork or an alternative client, I'll be curious how it works out, but I'm not doing this in npm/cli. Too hard.

(2) imposes similar complexity, but on the server side. Remember that auth stack that we try to hit as little as possible? Now we need to keep 3 forms of the packument, and check the auth state on every packument request. I don't primarily work on that portion of the stack, but I'm confident that it would impose significant burden on us to do that, effectively making every package as costly to serve as a private package. (Which, on each individual fetch, is not a huge cost, but multiplied by ~70Bn package downloads a month, adds up quick.) So that's not gonna happen any time soon.

(3) suffers from essentially the same performance concern as (2), but on tarball urls instead of packument urls, which is potentially worse. The mitigating factor here that might make it less infeasible is that the staged tarball URLs themselves could have a discernible pattern (which is good for other reasons anyway), so we'd save having to make the check on every fetch. However, this one has the "even worse footgun" I alluded to. If you install something, and get a tarball url in your shrinkwrap, then someone else (who isn't a maintainer, or just isn't logged in) tries to download it, they get a 404. This is exactly the DX issue that got us down this road to begin with.

(4) and (5) -- ie, create a net new API surface for staged publishes -- are probably the least infeasible from an operational point of view, but also will take the longest to spec and deliver, and frankly, might never actually happen. Also, it's worth noting that "staged publishes that only a select few can install, from a completely different endpoint" is exactly what private registries provide. And, if the new registry endpoint for staged publishes isn't something you can npm install from (ie, option (5)), then it's pretty useless. The number one way to inspect a published npm package is to install it and run tests against it. (Ideally in CI, where a matrix of platform and engine versions can be tests in parallel.)

In all but (1), "staged publishes that only a select few can install" becomes a backdoor way to have a mix of private and public code at the same namespace entry, which can exist for both scoped and unscoped packages. If we were ever to offer something like that, we'd charge for it, because that's difficult to implement, costly to operate, and primarily serves the interests of corporate users. And (1) is a ton of work for me and my team that I'm not eager to have us do. Assuming that the public registry makes staged versions of public packages available to all, nothing's stopping any other registry implementation from doing this, of course. I'd love to have some data (or at least, anecdata) about how that works out. It might still not be a compelling enough case to get over the challenges at npm public registry scale, but it'd be more compelling than not having it, for sure.

I'm sorry. I don't think that's going to happen.

The good news is, just reducing the priority of staged versions below that of published versions solves these problems exceedingly simply, and gets rid of most of the hazards. And an implementation exists that supports it today, in the latest versions of npm-pick-manifest, pacote, and @npmcli/arborist. It pretty much impossible to get a staged version by accident. You have to both opt into considering staged versions, and there has to be no published version that satisfies the requested range. This prevents the "floodgates to get a sip of water" issue entirely, because those versions will never be chosen if there is any other option. (It's like deprecation, but stronger.)

If "testing a staged version" is a non-goal, then I don't think this is worth doing, honestly. Why would you ever stage something, if not to test it before committing to it being published? Isn't that the whole point?

Here's what I'd like to focus on, which I see as the main remaining obstacles to this:

• The dist.tarball url of staged packages. I like the content-address idea, could be long-lived even when the "real" artifact url is used moving forward.
• The handshake required to promote from staged to published. I'm thinking, I don't wanna have a situation where a bad actor stages a new thing right as you're about to promote it. The workflow has to be worked out there, though, because what you really want is to say "I want to promote this thing that I got 15 minutes ago", and you're unlikely to grab the integrity as you're installing it initially for inspection. Maybe provide the artifact itself and we check that it matches? What's the UX for "I'd like to start testing staged version XYZ, I'll promote it once I'm satisfied"?

[ljharb]

You tested it before staging it, so no, imo that's not the whole point - the point is to separate "prepare the publishable artifact and securely transmit it" from "propagate it publicly, which requires extra auth".

[isaacs]

You tested it before staging it, so no, imo that's not the whole point - the point is to separate "prepare the publishable artifact and securely transmit it" from "propagate it publicly, which requires extra auth".

But you're explicitly saying that "transmitting" it is not done securely. It doesn't require TFA to stage a package. (And, it's likely a token that far more people have access to.) Then, later, I provide TFA and a more restricted access token to promote it from staged to published (the secure step).

So I'm supposed to just put my stamp of approval on something without inspecting it first? That doesn't seem like a good idea.

[isaacs]

I believe this RFC may help with the "one shot" publish issue we had in lerna, where we switched to using a tmp dist-tag.

If I'm understanding you correctly, this is about the case where you have a bunch of packages you want to publish all together in a sync'ed version?

This would actually be slightly safer than using a temp dist-tag, because you wouldn't have the race condition period where mismatched versions are published and installable.

Granted, the version referenced by the latest dist-tag has long been prioritized, so if 1.2.3 is on latest, and you've soft-published 1.2.4 on the tmp dist-tag, then anyone fetching 1.2.x will get 1.2.3 and not 1.2.4. And, if you're not installing against latest, and it's a tmp dist-tag floating on top of next or something, then you won't have that coverage, but obviously way fewer people will be fetching that version range in the first place.

It does bring up an interesting point about the promotion flow, though. Should it be somehow transactional across multiple packages and versions? How much of a race condition is acceptable in this case? Even if there is a window of mismatch, it's going to be much smaller than with actually uploading a bunch of tarballs, so maybe it's fine?