crossplane icon indicating copy to clipboard operation
crossplane copied to clipboard
verified publisher icon crossplane

`TestDowngrade` E2E frequently failing

Open negz opened this issue 7 months ago • 14 comments

What happened?

This test is failing often:

                +e2e | === RUN   TestDowngrade/TestDowngrade/RequiredConfigurationIsHealthy
                +e2e |     pkg_test.go:495: Waiting 2m0s for Configuration.pkg.crossplane.io crossplane-e2e-depends-on-provider-nop to become HealthyPackageRevision, ActivePackageRevision...
                +e2e |     feature.go:361: - CONDITION: Configuration.pkg.crossplane.io crossplane-e2e-depends-on-provider-nop: Healthy=Unknown Reason=: "" (0001-01-01 00:00:00 +0000 UTC)
                +e2e |     feature.go:361: - CONDITION: Configuration.pkg.crossplane.io crossplane-e2e-depends-on-provider-nop: Healthy=Unknown Reason=UnknownPackageRevisionHealth: "" (2025-04-28 20:02:57 +0000 UTC)
                +e2e |     feature.go:361: - CONDITION: Configuration.pkg.crossplane.io crossplane-e2e-depends-on-provider-nop: Healthy=Unknown Reason=UnknownPackageRevisionHealth: cannot resolve package dependencies: missing dependencies: [crossplane-contrib/provider-nop] (2025-04-28 20:02:58 +0000 UTC)
                +e2e | W0428 20:04:51.368003    3567 warnings.go:70] v1 ComponentStatus is deprecated in v1.19+
                +e2e | W0428 20:05:04.120208    3567 warnings.go:70] ControllerConfig.pkg.crossplane.io/v1alpha1 is deprecated. Use DeploymentRuntimeConfig from pkg.crossplane.io/v1beta1 instead.
                +e2e |     pkg_test.go:495: resource did not have desired conditions: HealthyPackageRevision, ActivePackageRevision: context deadline exceeded:

https://app.buildpulse.io/@crossplane/crossplane shows it's failing ~25% of the time.

How can we reproduce it?

Run the package-dependency-upgrade E2E test a few times.

negz avatar Apr 28 '25 20:04 negz

When I run the test against the https://github.com/crossplane/crossplane/pull/6407 branch it looks like the provider is installing fine but the configuration thinks the dependency isn't satisfied and never becomes healthy.

# kubectl get provider
NAME                              INSTALLED   HEALTHY   PACKAGE                                                     AGE
crossplane-contrib-provider-nop   True        True      xpkg.crossplane.io/crossplane-contrib/provider-nop:v0.4.0   3m59s

# kubectl get configuration
NAME                                     INSTALLED   HEALTHY   PACKAGE                                                            AGE
configuration-downgrade                  True        Unknown   xpkg.upbound.io/crossplane/e2e-nested-configuration:v0.2.0-d       4m5s
crossplane-e2e-depends-on-provider-nop   True        Unknown   xpkg.crossplane.io/crossplane/e2e-depends-on-provider-nop:v0.3.1   4m

# kubectl describe configurationrevision
Name:         configuration-downgrade-e9c658bec1a2
Namespace:
Labels:       pkg.crossplane.io/package=configuration-downgrade
Annotations:  <none>
API Version:  pkg.crossplane.io/v1
Kind:         ConfigurationRevision
Metadata:
  Creation Timestamp:  2025-04-28T21:16:05Z
  Finalizers:
    revision.pkg.crossplane.io
  Generation:  1
  Owner References:
    API Version:           pkg.crossplane.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Configuration
    Name:                  configuration-downgrade
    UID:                   04675cb5-2a5a-4213-a86e-399bc0a6c7b8
  Resource Version:        782
  UID:                     1f97fc54-88a2-4c2e-b8b4-b7aba3b55cbf
Spec:
  Desired State:                  Active
  Ignore Crossplane Constraints:  false
  Image:                          xpkg.upbound.io/crossplane/e2e-nested-configuration:v0.2.0-d
  Package Pull Policy:            IfNotPresent
  Revision:                       1
  Skip Dependency Resolution:     false
Status:
  Conditions:
    Last Transition Time:  2025-04-28T21:16:19Z
    Message:               cannot resolve package dependencies: missing dependencies: [crossplane-contrib/provider-nop]
    Reason:                UnknownPackageRevisionHealth
    Status:                Unknown
    Type:                  Healthy
  Found Dependencies:      3
  Installed Dependencies:  2
Events:
  Type     Reason               Age                    From                                              Message
  ----     ------               ----                   ----                                              -------
  Warning  ResolveDependencies  6m36s (x3 over 6m37s)  packages/configurationrevision.pkg.crossplane.io  cannot resolve package dependencies: missing dependencies: [xpkg.crossplane.io/crossplane-contrib/provider-nop xpkg.crossplane.io/crossplane/e2e-depends-on-provider-nop]
  Warning  ResolveDependencies  6m32s (x2 over 6m32s)  packages/configurationrevision.pkg.crossplane.io  cannot resolve package dependencies: missing dependencies: [xpkg.crossplane.io/crossplane/e2e-depends-on-provider-nop]
  Warning  ResolveDependencies  51s (x8 over 6m24s)    packages/configurationrevision.pkg.crossplane.io  cannot resolve package dependencies: missing dependencies: [crossplane-contrib/provider-nop]


Name:         crossplane-e2e-depends-on-provider-nop-857d0a2b3c50
Namespace:
Labels:       pkg.crossplane.io/package=crossplane-e2e-depends-on-provider-nop
Annotations:  <none>
API Version:  pkg.crossplane.io/v1
Kind:         ConfigurationRevision
Metadata:
  Creation Timestamp:  2025-04-28T21:16:11Z
  Finalizers:
    revision.pkg.crossplane.io
  Generation:  1
  Owner References:
    API Version:           pkg.crossplane.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Configuration
    Name:                  crossplane-e2e-depends-on-provider-nop
    UID:                   b31beae8-8c42-481c-8da9-3c2ef72a0c03
  Resource Version:        745
  UID:                     b4e754f5-d907-4aa8-b45c-dc40e18e4b83
Spec:
  Desired State:                  Active
  Ignore Crossplane Constraints:  false
  Image:                          xpkg.crossplane.io/crossplane/e2e-depends-on-provider-nop:v0.3.1
  Package Pull Policy:            IfNotPresent
  Revision:                       1
  Skip Dependency Resolution:     false
Status:
  Conditions:
    Last Transition Time:  2025-04-28T21:16:13Z
    Message:               cannot resolve package dependencies: missing dependencies: [crossplane-contrib/provider-nop]
    Reason:                UnknownPackageRevisionHealth
    Status:                Unknown
    Type:                  Healthy
  Found Dependencies:      1
Events:
  Type     Reason               Age                   From                                              Message
  ----     ------               ----                  ----                                              -------
  Warning  ResolveDependencies  33s (x12 over 6m30s)  packages/configurationrevision.pkg.crossplane.io  cannot resolve package dependencies: missing dependencies: [crossplane-contrib/provider-nop]

Here's the lock:

# kubectl get -o yaml lock lock
apiVersion: pkg.crossplane.io/v1beta1
kind: Lock
metadata:
  creationTimestamp: "2025-04-28T21:15:58Z"
  finalizers:
  - lock.pkg.crossplane.io
  generation: 4
  name: lock
  resourceVersion: "744"
  uid: 57bd1dd2-850b-4c67-89b3-a0428e4461b4
packages:
- apiVersion: pkg.crossplane.io/v1
  dependencies:
  - constraints: <=v0.4.0
    package: xpkg.crossplane.io/crossplane-contrib/provider-nop
    type: Provider
  - constraints: v0.3.1
    package: xpkg.crossplane.io/crossplane/e2e-depends-on-provider-nop
    type: Configuration
  kind: Configuration
  name: configuration-downgrade-e9c658bec1a2
  source: xpkg.upbound.io/crossplane/e2e-nested-configuration
  version: v0.2.0-d
- apiVersion: pkg.crossplane.io/v1
  dependencies: []
  kind: Provider
  name: crossplane-contrib-provider-nop-a52ba8a7a054
  source: xpkg.crossplane.io/crossplane-contrib/provider-nop
  version: v0.4.0
- apiVersion: pkg.crossplane.io/v1
  dependencies:
  - apiVersion: pkg.crossplane.io/v1
    constraints: =v0.3.1
    kind: Provider
    package: crossplane-contrib/provider-nop
  kind: Configuration
  name: crossplane-e2e-depends-on-provider-nop-857d0a2b3c50
  source: xpkg.crossplane.io/crossplane/e2e-depends-on-provider-nop
  version: v0.3.1
status:
  conditions:
  - lastTransitionTime: "2025-04-28T21:16:07Z"
    reason: DependencyResolutionSucceeded
    status: "True"
    type: Resolved

Given the (almost) 100% failure rate on https://github.com/crossplane/crossplane/pull/6407 it'd appear that running with realtime compositions (or this PR specifically) breaks package dependencies somehow? I have no idea how - I wouldn't expect an interaction between the two features.

negz avatar Apr 28 '25 21:04 negz

Hid the above comment because I realized I was running the test without the right -test-suite flag, so downgrades weren't enabled.

negz avatar Apr 29 '25 01:04 negz

Just got this to fail locally. It looks like the provider is installed, but not in the lock...

# kubectl get provider.pkg
NAME                              INSTALLED   HEALTHY   PACKAGE                                                     AGE
crossplane-contrib-provider-nop   True        True      xpkg.crossplane.io/crossplane-contrib/provider-nop:v0.3.1   22m

# kubectl describe provider.pkg
Name:         crossplane-contrib-provider-nop
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  pkg.crossplane.io/v1
Kind:         Provider
Metadata:
  Creation Timestamp:  2025-04-29T01:05:55Z
  Generation:          27
  Resource Version:    1102
  UID:                 e993843b-e59b-4007-897c-3a19e1811327
Spec:
  Ignore Crossplane Constraints:  false
  Package:                        xpkg.crossplane.io/crossplane-contrib/provider-nop:v0.3.1
  Package Pull Policy:            IfNotPresent
  Revision Activation Policy:     Automatic
  Revision History Limit:         1
  Runtime Config Ref:
    API Version:               pkg.crossplane.io/v1beta1
    Kind:                      DeploymentRuntimeConfig
    Name:                      default
  Skip Dependency Resolution:  false
Status:
  Conditions:
    Last Transition Time:  2025-04-29T01:06:05Z
    Reason:                HealthyPackageRevision
    Status:                True
    Type:                  Healthy
    Last Transition Time:  2025-04-29T01:05:56Z
    Reason:                ActivePackageRevision
    Status:                True
    Type:                  Installed
  Current Identifier:      xpkg.crossplane.io/crossplane-contrib/provider-nop:v0.3.1
  Current Revision:        crossplane-contrib-provider-nop-761d51fed128
Events:
  Type     Reason                  Age                 From                                 Message
  ----     ------                  ----                ----                                 -------
  Warning  InstallPackageRevision  22m (x9 over 22m)   packages/provider.pkg.crossplane.io  current package revision health is unknown
  Warning  InstallPackageRevision  22m (x15 over 22m)  packages/provider.pkg.crossplane.io  current package revision is unhealthy
  Normal   InstallPackageRevision  22m                 packages/provider.pkg.crossplane.io  Successfully installed package revision

# kubectl get -o yaml lock lock
apiVersion: pkg.crossplane.io/v1beta1
kind: Lock
metadata:
  creationTimestamp: "2025-04-29T01:05:45Z"
  finalizers:
  - lock.pkg.crossplane.io
  generation: 51
  name: lock
  resourceVersion: "1103"
  uid: e55c9f6b-9495-49e5-9221-0ff16b52ecf3
packages:
- apiVersion: pkg.crossplane.io/v1
  dependencies:
  - constraints: <=v0.4.0
    package: xpkg.crossplane.io/crossplane-contrib/provider-nop
    type: Provider
  - constraints: v0.3.1
    package: xpkg.crossplane.io/crossplane/e2e-depends-on-provider-nop
    type: Configuration
  kind: Configuration
  name: configuration-downgrade-e9c658bec1a2
  source: xpkg.upbound.io/crossplane/e2e-nested-configuration
  version: v0.2.0-d
- apiVersion: pkg.crossplane.io/v1
  dependencies:
  - apiVersion: pkg.crossplane.io/v1
    constraints: =v0.3.1
    kind: Provider
    package: crossplane-contrib/provider-nop
  kind: Configuration
  name: crossplane-e2e-depends-on-provider-nop-857d0a2b3c50
  source: xpkg.crossplane.io/crossplane/e2e-depends-on-provider-nop
  version: v0.3.1
status:
  conditions:
  - lastTransitionTime: "2025-04-29T01:05:59Z"
    reason: DependencyResolutionSucceeded
    status: "True"
    type: Resolved

negz avatar Apr 29 '25 01:04 negz

This is interesting. I noticed provider-nop v0.3.1 existed and was the active revision. I edited it to add an annotation to kick off a reconcile, and it added itself to the lock.

Right after it added itself I saw this:

# kubectl get providerrevision
NAME                                           HEALTHY   REVISION   IMAGE                                                       STATE      DEP-FOUND   DEP-INSTALLED   AGE
crossplane-contrib-provider-nop-761d51fed128   True      2          crossplane-contrib/provider-nop:v0.3.1                      Active                                 39m
crossplane-contrib-provider-nop-a52ba8a7a054   True      1          xpkg.crossplane.io/crossplane-contrib/provider-nop:v0.4.0   Inactive                               39m

However soon after it's gone from the Lock again and I see this:

# kubectl get providerrevision
NAME                                           HEALTHY   REVISION   IMAGE                                                       STATE      DEP-FOUND   DEP-INSTALLED   AGE
crossplane-contrib-provider-nop-761d51fed128   True      2          xpkg.crossplane.io/crossplane-contrib/provider-nop:v0.3.1   Active                                 40m
crossplane-contrib-provider-nop-a52ba8a7a054   True      1          xpkg.crossplane.io/crossplane-contrib/provider-nop:v0.4.0   Inactive                               40m

Note the image changed - it's now xpkg.crossplane.io/....

negz avatar Apr 29 '25 01:04 negz

I added some logging:

--- a/internal/controller/pkg/revision/dependency.go
+++ b/internal/controller/pkg/revision/dependency.go
@@ -31,6 +31,7 @@ import (
        "sigs.k8s.io/controller-runtime/pkg/client"

        "github.com/crossplane/crossplane-runtime/pkg/errors"
+       "github.com/crossplane/crossplane-runtime/pkg/logging"
        "github.com/crossplane/crossplane-runtime/pkg/resource"

        pkgmetav1 "github.com/crossplane/crossplane/apis/pkg/meta/v1"
@@ -63,14 +64,16 @@ type PackageDependencyManager struct {
        client      client.Client
        newDag      dag.NewDAGFn
        packageType schema.GroupVersionKind
+       log         logging.Logger
 }

 // NewPackageDependencyManager creates a new PackageDependencyManager.
-func NewPackageDependencyManager(c client.Client, nd dag.NewDAGFn, pkgType schema.GroupVersionKind) *PackageDependencyManager {
+func NewPackageDependencyManager(c client.Client, nd dag.NewDAGFn, pkgType schema.GroupVersionKind, l logging.Logger) *PackageDependencyManager {
        return &PackageDependencyManager{
                client:      c,
                newDag:      nd,
                packageType: pkgType,
+               log:         l,
        }
 }

@@ -148,6 +151,12 @@ func (m *PackageDependencyManager) Resolve(ctx context.Context, meta pkgmetav1.P
        // to another registry)
        for _, lp := range lock.Packages {
                if self.Name == lp.Name && self.Type == lp.Type && self.Source != lp.Identifier() {
+                       m.log.Debug("Package with same name and type but different source exists in lock. Removing it.",
+                               "name", lp.Name,
+                               "type", ptr.Deref(lp.Type, "Unknown"),
+                               "old-source", lp.Identifier(),
+                               "new-source", lp.Source,
+                       )
                        if err := m.RemoveSelf(ctx, pr); err != nil {
                                return found, installed, invalid, err
                        }

...and I see this. It's repeated 31 times in total, but seems to eventually stabilize.

So it looks like at some point there's two revisions fighting to remove each other from the lock, because they think they're different sources when they're not (xpkg.crossplane.io is the default registry). Somehow this is ending up with neither revision being in the lock...

2025-04-29T02:31:15Z	DEBUG	crossplane	Package with same name and type but different source exists in lock. Removing it.	{"controller": "packages/providerrevision.pkg.crossplane.io", "name": "crossplane-contrib-provider-nop-761d51fed128", "type": "Unknown", "old-source": "crossplane-contrib/provider-nop", "new-source": "xpkg.crossplane.io/crossplane-contrib/provider-nop"}
2025-04-29T02:31:16Z	DEBUG	crossplane	Package with same name and type but different source exists in lock. Removing it.	{"controller": "packages/providerrevision.pkg.crossplane.io", "name": "crossplane-contrib-provider-nop-761d51fed128", "type": "Unknown", "old-source": "xpkg.crossplane.io/crossplane-contrib/provider-nop", "new-source": "crossplane-contrib/provider-nop"}
2025-04-29T02:31:17Z	DEBUG	crossplane	Package with same name and type but different source exists in lock. Removing it.	{"controller": "packages/providerrevision.pkg.crossplane.io", "name": "crossplane-contrib-provider-nop-761d51fed128", "type": "Unknown", "old-source": "crossplane-contrib/provider-nop", "new-source": "xpkg.crossplane.io/crossplane-contrib/provider-nop"}
2025-04-29T02:31:18Z	DEBUG	crossplane	Package with same name and type but different source exists in lock. Removing it.	{"controller": "packages/providerrevision.pkg.crossplane.io", "name": "crossplane-contrib-provider-nop-761d51fed128", "type": "Unknown", "old-source": "xpkg.crossplane.io/crossplane-contrib/provider-nop", "new-source": "crossplane-contrib/provider-nop"}
2025-04-29T02:31:19Z	DEBUG	crossplane	Package with same name and type but different source exists in lock. Removing it.	{"controller": "packages/providerrevision.pkg.crossplane.io", "name": "crossplane-contrib-provider-nop-761d51fed128", "type": "Unknown", "old-source": "crossplane-contrib/provider-nop", "new-source": "xpkg.crossplane.io/crossplane-contrib/provider-nop"}
2025-04-29T02:31:20Z	DEBUG	crossplane	Package with same name and type but different source exists in lock. Removing it.	{"controller": "packages/providerrevision.pkg.crossplane.io", "name": "crossplane-contrib-provider-nop-761d51fed128", "type": "Unknown", "old-source": "xpkg.crossplane.io/crossplane-contrib/provider-nop", "new-source": "crossplane-contrib/provider-nop"}
2025-04-29T02:31:21Z	DEBUG	crossplane	Package with same name and type but different source exists in lock. Removing it.	{"controller": "packages/providerrevision.pkg.crossplane.io", "name": "crossplane-contrib-provider-nop-761d51fed128", "type": "Unknown", "old-source": "crossplane-contrib/provider-nop", "new-source": "xpkg.crossplane.io/crossplane-contrib/provider-nop"}

negz avatar Apr 29 '25 02:04 negz

The PR that introduced the different-source check: https://github.com/crossplane/crossplane/pull/5946

negz avatar Apr 29 '25 02:04 negz

Not sure how this is happening...

If changing the source results in a new OCI digest (e.g crossplane-contrib/provider-nop:v0.4.0 -> xpkg.crossplane.io/crossplane-contrib/provider-nop:v0.3.1) that should result in two package revisions.

The package reconciler disables all active revisions before it sets a new revision active. So only one revision should ever be active:

https://github.com/crossplane/crossplane/blob/09e27bdb1e25c0f839a58685c2ff99768b75320d/internal/controller/pkg/manager/reconciler.go#L399-L403

The revision reconciler returns early (before it resolves dependencies) for inactive revisions:

https://github.com/crossplane/crossplane/blob/09e27bdb1e25c0f839a58685c2ff99768b75320d/internal/controller/pkg/revision/reconciler.go#L660

https://github.com/crossplane/crossplane/blob/09e27bdb1e25c0f839a58685c2ff99768b75320d/internal/controller/pkg/revision/reconciler.go#L843

Maybe there's a stale cache read going on somewhere? e.g.:

  • The package reconciler doesn't see the existing active revision yet when it creates a new active one
  • The revision reconciler is reconciling revisions that're actually inactive but appear active due to a stale cache read

negz avatar Apr 29 '25 03:04 negz

https://github.com/crossplane/crossplane/blob/09e27bdb1e25c0f839a58685c2ff99768b75320d/test/e2e/pkg_test.go#L488-L492

https://github.com/crossplane/crossplane/blob/09e27bdb1e25c0f839a58685c2ff99768b75320d/test/e2e/manifests/pkg/dependency-upgrade/downgrade/package/crossplane.yaml#L15-L18

https://explore.ggcr.dev/fs/xpkg.crossplane.io/crossplane/e2e-depends-on-provider-nop@sha256:e19cd5c9aa6701974c56aa819a9d784290366a353976264eb053b0160ab401b3/package.yaml

It looks like the configuration the (sometimes) failing test installs:

  • Depends directly on xpkg.crossplane.io/crossplane-contrib/provider-nop <= v0.4.0
  • Depends indirectly on crossplane-contrib/provider-nop == v0.3.1

Given both dependencies are created in rapid succession (they're both dependencies of the same configuration) I could easily imagine there's some cache race going on here.

negz avatar Apr 29 '25 03:04 negz

We could address the specific case this E2E test is hitting by teaching the check added in https://github.com/crossplane/crossplane/pull/5946 about the default registry, so that it knows xpkg.crossplane.io/a/b and a/b are the same package. I don't think that'd really solve the underlying bug though.

We shouldn't let two revisions fight to remove each other from the lock under any circumstance, e.g. if there was a brief window where mirror.acme.co/crossplane-contrib/provider-nop and crossplane-contrib/provider-nop` both thought they were the active package revision.

negz avatar Apr 29 '25 03:04 negz

Just got this to fail locally. It looks like the provider is installed, but not in the lock...

# kubectl get provider.pkg
NAME                              INSTALLED   HEALTHY   PACKAGE                                                     AGE
crossplane-contrib-provider-nop   True        True      xpkg.crossplane.io/crossplane-contrib/provider-nop:v0.3.1   22m

# kubectl describe provider.pkg
Name:         crossplane-contrib-provider-nop
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  pkg.crossplane.io/v1
Kind:         Provider
Metadata:
  Creation Timestamp:  2025-04-29T01:05:55Z
  Generation:          27
  Resource Version:    1102
  UID:                 e993843b-e59b-4007-897c-3a19e1811327
Spec:
  Ignore Crossplane Constraints:  false
  Package:                        xpkg.crossplane.io/crossplane-contrib/provider-nop:v0.3.1
  Package Pull Policy:            IfNotPresent
  Revision Activation Policy:     Automatic
  Revision History Limit:         1
  Runtime Config Ref:
    API Version:               pkg.crossplane.io/v1beta1
    Kind:                      DeploymentRuntimeConfig
    Name:                      default
  Skip Dependency Resolution:  false
Status:
  Conditions:
    Last Transition Time:  2025-04-29T01:06:05Z
    Reason:                HealthyPackageRevision
    Status:                True
    Type:                  Healthy
    Last Transition Time:  2025-04-29T01:05:56Z
    Reason:                ActivePackageRevision
    Status:                True
    Type:                  Installed
  Current Identifier:      xpkg.crossplane.io/crossplane-contrib/provider-nop:v0.3.1
  Current Revision:        crossplane-contrib-provider-nop-761d51fed128
Events:
  Type     Reason                  Age                 From                                 Message
  ----     ------                  ----                ----                                 -------
  Warning  InstallPackageRevision  22m (x9 over 22m)   packages/provider.pkg.crossplane.io  current package revision health is unknown
  Warning  InstallPackageRevision  22m (x15 over 22m)  packages/provider.pkg.crossplane.io  current package revision is unhealthy
  Normal   InstallPackageRevision  22m                 packages/provider.pkg.crossplane.io  Successfully installed package revision

kubectl get -o yaml lock lock

apiVersion: pkg.crossplane.io/v1beta1 kind: Lock metadata: creationTimestamp: "2025-04-29T01:05:45Z" finalizers:

  • lock.pkg.crossplane.io generation: 51 name: lock resourceVersion: "1103" uid: e55c9f6b-9495-49e5-9221-0ff16b52ecf3 packages:
  • apiVersion: pkg.crossplane.io/v1 dependencies:
    • constraints: <=v0.4.0 package: xpkg.crossplane.io/crossplane-contrib/provider-nop type: Provider
    • constraints: v0.3.1 package: xpkg.crossplane.io/crossplane/e2e-depends-on-provider-nop type: Configuration kind: Configuration name: configuration-downgrade-e9c658bec1a2 source: xpkg.upbound.io/crossplane/e2e-nested-configuration version: v0.2.0-d
  • apiVersion: pkg.crossplane.io/v1 dependencies:
    • apiVersion: pkg.crossplane.io/v1 constraints: =v0.3.1 kind: Provider package: crossplane-contrib/provider-nop kind: Configuration name: crossplane-e2e-depends-on-provider-nop-857d0a2b3c50 source: xpkg.crossplane.io/crossplane/e2e-depends-on-provider-nop version: v0.3.1 status: conditions:
    • lastTransitionTime: "2025-04-29T01:05:59Z" reason: DependencyResolutionSucceeded status: "True" type: Resolved

I think it's related with this comment.

ezgidemirel avatar Apr 29 '25 17:04 ezgidemirel

I think it's related with this comment.

@ezgidemirel FWIW it wasn't missing for a short period, and one of the revisions was definitely active. It was missing until I edited an annotation on the active revision to trigger a reconcile. Then the controller added it. That's in part why I opened https://github.com/crossplane/crossplane/pull/6415, to reconcile revisions when the lock changes.

negz avatar Apr 29 '25 19:04 negz

(From @ezgidemirel on Slack):

one suggestion I have is to normalize packages to the repository/image-name format within the Identifier() method and ensure that this method is used consistently across all DAG-related methods, like TraceNode. I didn’t get a chance to fully test this today, but I can take care of it tomorrow if you don’t have time to look into it.

That sounds similar to what I was thinking at https://github.com/crossplane/crossplane/issues/6413.

Do you mean repository/image-name like crossplane-contrib/provider-nop, or like xpkg.upbound.io/crossplane-contrib/provider-nop (with the registry)? Only the latter (with the registry) uniquely identifies a package. I've proposed changing this in https://github.com/crossplane/crossplane/issues/6018, but today if you strip the registry then compare just repo and image name you're losing information (the registry) that uniquely identifies a package.

So unless we choose to adopt https://github.com/crossplane/crossplane/issues/6018, today it'd be safer to always include the registry, and inject the default registry when it's omitted.

The other thing to consider is that as I mentioned in https://github.com/crossplane/crossplane/issues/6413, this could fix this failing E2E but it doesn't fix the underlying bug it uncovered - that two revisions of a package can be active at the same time. At least from the reconciler's perspective.

Put otherwise, if you copy xpkg.crossplane.io/crossplane-contrib/provider-nop:v1.2.0 to internal.acme.co/crossplane-contrib/provider-nop:v1.2.0 and then upgrade your package from one to the other, the revision reconciler could still enter this state where it thinks both are active and they fight to remove each other from the lock. It's an edge case, but still a problem.

negz avatar Apr 29 '25 20:04 negz

@negz at first, I thought we might not need registry information in the DAG, since installing the same package from multiple registries would lead to conflicts when acquiring CRD ownership. However, after reconsidering, I realized this approach introduces security concerns and is quite risky.

I’ve now propagated the default registry to the DAG and normalized package identifiers to the <registry>/<organization>/<repository> format, which resolved the issue in the e2e test.

That said, I encountered a problem when trying to install an image without a defined registry. The installation failed with the following log:

2025-04-30T18:42:57+03:00	DEBUG	crossplane	Package with same name and type but different source exists in lock. Removing it.	{"controller": "packages/providerrevision.pkg.crossplane.io", "name": "crossplane-contrib-provider-nop-761d51fed128", "type": "Unknown", "old-source": "xpkg.crossplane.io/crossplane-contrib/provider-nop", "new-source": "crossplane-contrib/provider-nop"}

As you’ve noticed, there’s a conflict here. I believe we have two options:

  1. Stop supporting automatic registry updates and require users to manually migrate between registries. This would involve pausing packages and performing some manual adjustments in the cluster to adopt the <registry>/<organization>/<repository> format as the canonical identifier.

  2. Assume the image is uniquely identified by <organization>/<repository>, and use the registry information only during installation and upgrades, not for ongoing identity checks.

ezgidemirel avatar Apr 30 '25 15:04 ezgidemirel

I think the automation from merging PR #6415 erroneously closed this issue.

This is a bunch of commits I made while debugging and attempting to fix https://github.com/crossplane/crossplane/issues/6413. I don't expect any of the commits to fix that issue

I think the text of "fix https://github.com/crossplane/crossplane/issues/6413" triggered the close here, but the rest of the context is saying it's not actually fixed 😂

jbw976 avatar May 21 '25 07:05 jbw976

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

github-actions[bot] avatar Aug 20 '25 01:08 github-actions[bot]