Trying to cancel an on going remote attesation crashes

[javirln]

The way I could reproduce it was:

• Start a remote attestation from a CI or somewhere else not local
• Grab the attestation id
• While the workflow is running, try to cancel it with chainloop attestation reset --trigger cancellation --attestation-id

Example of the error:

$ chainloop attestation reset --trigger cancellation --attestation-id 40c7b6bc-30cd-4a30-a05d-8b6a7462ad0d
ERR loading existing attestation error="failed to load crafting state: failed to read state: rpc error: code = Internal desc = server error"
ERR failed to load crafting state: failed to read state: rpc error: code = Internal desc = server error

[migmartri]

Start a remote attestation from a CI or somewhere else not local

what does it mean not local? what is it important?

[javirln]

Start a remote attestation from a CI or somewhere else not local

what does it mean not local? what is it important?

It's not that is important just the way I managed to ran into the issue :/

[migmartri]

While the workflow is running, try to cancel it with chainloop attestation reset --trigger cancellation --attestation-id

so does the attestation-id need to exist? Is a problem that the attestation has already been cancelled?

[migmartri]

Start a remote attestation from a CI or somewhere else not local

what does it mean not local? what is it important?

It's not that is important just the way I managed to ran into the issue :/

so you can't reproduce it locally?

[migmartri]

It would be nice if you could provide a full reproduction path, form init to crash

[javirln]

While the workflow is running, try to cancel it with chainloop attestation reset --trigger cancellation --attestation-id

so does the attestation-id need to exist? Is a problem that the attestation has already been cancelled?

It does need to exist, the attestation was still on going

[migmartri]

Did you try to cancel from another environment / CLI instance by any chance?

[javirln]

I managed to reproduce it locally:

1. Create an API token
2. Initialize an attestation using the remote state
3. Create another API token
4. With the second API token, try to cancel the on going attestation

$ chainloop --insecure org api-token create --name first-token
WRN API contacted in insecure mode
WRN Both user credentials and $CHAINLOOP_TOKEN set. Ignoring $CHAINLOOP_TOKEN.
┌─────────────┬─────────────┬─────────────────────┬────────────┬────────────┐
│ NAME        │ DESCRIPTION │ CREATED AT          │ EXPIRES AT │ REVOKED AT │
├─────────────┼─────────────┼─────────────────────┼────────────┼────────────┤
│ first-token │             │ 08 Jan 25 17:34 UTC │            │            │
└─────────────┴─────────────┴─────────────────────┴────────────┴────────────┘

Save the following token since it will not printed again:

 eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJvcmdfaWQiOiI5M2QwMjI3NS04NTNjLTRhZDYtOWQ2MC04ZjU2MmIxMjNmZDIiLCJvcmdfbmFtZSI6InJlYWQtb25seS1kZW1vIiwiaXNzIjoiY3AuY2hhaW5sb29wIiwiYXVkIjpbImFwaS10b2tlbi1hdXRoLmNoYWlubG9vcCJdLCJqdGkiOiIwMjk3NjhiNS0wYWQ2LTQwMzMtYWY1ZS03NTVjM2M0ODg2NWIifQ.qidxgiKe5Ie0cWVffcqW0206euNVJXikhNVGKY2-k5c

$ export CHAINLOOP_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJvcmdfaWQiOiI5M2QwMjI3NS04NTNjLTRhZDYtOWQ2MC04ZjU2MmIxMjNmZDIiLCJvcmdfbmFtZSI6InJlYWQtb25seS1kZW1vIiwiaXNzIjoiY3AuY2hhaW5sb29wIiwiYXVkIjpbImFwaS10b2tlbi1hdXRoLmNoYWlubG9vcCJdLCJqdGkiOiIwMjk3NjhiNS0wYWQ2LTQwMzMtYWY1ZS03NTVjM2M0ODg2NWIifQ.qidxgiKe5Ie0cWVffcqW0206euNVJXikhNVGKY2-k5c

$ chainloop --insecure attestation init --name reviews-pm --project one-project --replace --remote-state
Flag --name has been deprecated, please use --workflow instead
WRN API contacted in insecure mode
INF Attestation initialized! now you can check its status or add materials to it
┌────────────────┬──────────────────────────────────────┐
│ Initialized At │ 08 Jan 25 17:34 UTC                  │
├────────────────┼──────────────────────────────────────┤
│ Attestation ID │ 924c3c79-eeb0-4306-869c-297f2a3a3a76 │
│ Organization   │ read-only-demo                       │
│ Name           │ reviews-pm                           │
│ Project        │ one-project                          │
│ Version        │ none                                 │
│ Contract       │ one-project-reviews-pm (revision 1)  │
└────────────────┴──────────────────────────────────────┘

$ chainloop --insecure org api-token create --name second-token
WRN API contacted in insecure mode
WRN Both user credentials and $CHAINLOOP_TOKEN set. Ignoring $CHAINLOOP_TOKEN.
┌──────────────┬─────────────┬─────────────────────┬────────────┬────────────┐
│ NAME         │ DESCRIPTION │ CREATED AT          │ EXPIRES AT │ REVOKED AT │
├──────────────┼─────────────┼─────────────────────┼────────────┼────────────┤
│ second-token │             │ 08 Jan 25 17:34 UTC │            │            │
└──────────────┴─────────────┴─────────────────────┴────────────┴────────────┘

Save the following token since it will not printed again:

 eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJvcmdfaWQiOiI5M2QwMjI3NS04NTNjLTRhZDYtOWQ2MC04ZjU2MmIxMjNmZDIiLCJvcmdfbmFtZSI6InJlYWQtb25seS1kZW1vIiwiaXNzIjoiY3AuY2hhaW5sb29wIiwiYXVkIjpbImFwaS10b2tlbi1hdXRoLmNoYWlubG9vcCJdLCJqdGkiOiJlNDgwYjI1ZC0zM2Q0LTRkNDEtODNjZC01MjdjNDcxNDM3ZjEifQ.BbsQCzVYt8EDoXEzARW2HFn3HgYR9rlztOwlxgXBTKI

$ export CHAINLOOP_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJvcmdfaWQiOiI5M2QwMjI3NS04NTNjLTRhZDYtOWQ2MC04ZjU2MmIxMjNmZDIiLCJvcmdfbmFtZSI6InJlYWQtb25seS1kZW1vIiwiaXNzIjoiY3AuY2hhaW5sb29wIiwiYXVkIjpbImFwaS10b2tlbi1hdXRoLmNoYWlubG9vcCJdLCJqdGkiOiJlNDgwYjI1ZC0zM2Q0LTRkNDEtODNjZC01MjdjNDcxNDM3ZjEifQ.BbsQCzVYt8EDoXEzARW2HFn3HgYR9rlztOwlxgXBTKI

$ chainloop --insecure attestation reset --trigger cancellation --attestation-id 924c3c79-eeb0-4306-869c-297f2a3a3a76
WRN API contacted in insecure mode
ERR loading existing attestation error="failed to load crafting state: failed to read state: rpc error: code = Internal desc = server error"
ERR failed to load crafting state: failed to read state: rpc error: code = Internal desc = server error

And the controlplane logs:

2025-01-08T18:35:03.005+0100	INFO	{"kind": "server", "component": "grpc", "operation": "/controlplane.v1.AttestationStateService/Initialized", "args": "workflow_run_id:\"924c3c79-eeb0-4306-869c-297f2a3a3a76\"", "code": 0, "reason": "", "stack": "", "latency": 0.017220291}
2025-01-08T18:35:03.007+0100	INFO	{"msg": "[authN] processed credentials", "id": "e480b25d-33d4-4d41-83cd-527c471437f1", "type": "API-token"}
2025-01-08T18:35:03.021+0100	ERROR	{"component": "service", "msg": "failed to decrypt attestation state: incorrect passphrase"}
  github.com/go-kratos/kratos/contrib/log/zap/v2.(*Logger).Log
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/contrib/log/zap/[email protected]/zap.go:41
  github.com/go-kratos/kratos/v2/log.(*logger).Log
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/log/log.go:30
  github.com/go-kratos/kratos/v2/log.(*Filter).Log
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/log/filter.go:99
  github.com/go-kratos/kratos/v2/log.(*logger).Log
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/log/log.go:30
  github.com/go-kratos/kratos/v2/log.(*Helper).Error
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/log/helper.go:121
  github.com/chainloop-dev/chainloop/pkg/servicelogger.LogAndMaskErr
  /the-folder/projects/chainloop/backend/pkg/servicelogger/logger.go:57
  github.com/chainloop-dev/chainloop/app/controlplane/internal/service.handleUseCaseErr
  /the-folder/projects/chainloop/backend/app/controlplane/internal/service/service.go:158
  github.com/chainloop-dev/chainloop/app/controlplane/internal/service.(*AttestationStateService).Read
  /the-folder/projects/chainloop/backend/app/controlplane/internal/service/attestationstate.go:126
  github.com/chainloop-dev/chainloop/app/controlplane/api/controlplane/v1._AttestationStateService_Read_Handler.func1
  /the-folder/projects/chainloop/backend/app/controlplane/api/controlplane/v1/attestation_state_grpc.pb.go:187
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.NewGRPCServer.UnaryServerInterceptor.func2
  /the-folder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/protovalidate/protovalidate.go:38
  google.golang.org/grpc.getChainUnaryHandler.func1
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1211
  github.com/go-kratos/kratos/v2/transport/grpc.NewServer.(*Server).unaryServerInterceptor.func1.1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/transport/grpc/interceptor.go:35
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.NewSentryContext.func20.1
  /the-folder/projects/chainloop/backend/app/controlplane/internal/sentrycontext/sentry_context.go:41
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.WithAttestationContextFromAPIToken.func18.1
  /the-folder/projects/chainloop/backend/app/controlplane/internal/usercontext/apitoken_middleware.go:130
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.WithAttestationContextFromRobotAccount.func17.1
  /the-folder/projects/chainloop/backend/app/controlplane/internal/usercontext/robotaccount_middleware.go:63
  github.com/chainloop-dev/chainloop/app/controlplane/internal/usercontext/attjwtmiddleware.WithJWTMulti.func1.1
  /the-folder/projects/chainloop/backend/app/controlplane/internal/usercontext/attjwtmiddleware/attmiddleware.go:206
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.(*Builder).Build.selector.func25.1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/middleware/selector/selector.go:125
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.(*Builder).Build.selector.func24.1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/middleware/selector/selector.go:123
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.Prometheus.func5.1.1
  /the-folder/projects/chainloop/backend/app/controlplane/internal/usercontext/wrappers.go:45
  github.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).UnaryServerInterceptor.func3
  /the-folder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:108
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.Prometheus.func5.1
  /the-folder/projects/chainloop/backend/app/controlplane/internal/usercontext/wrappers.go:49
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.Server.func4.1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/middleware/logging/logging.go:34
  github.com/go-kratos/kratos/v2/middleware/recovery.Recovery.func2.1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/middleware/recovery/recovery.go:59
  github.com/go-kratos/kratos/v2/transport/grpc.NewServer.(*Server).unaryServerInterceptor.func1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/transport/grpc/interceptor.go:40
  google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1202
  github.com/chainloop-dev/chainloop/app/controlplane/api/controlplane/v1._AttestationStateService_Read_Handler
  /the-folder/projects/chainloop/backend/app/controlplane/api/controlplane/v1/attestation_state_grpc.pb.go:189
  google.golang.org/grpc.(*Server).processUnaryRPC
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1393
  google.golang.org/grpc.(*Server).handleStream
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1804
  google.golang.org/grpc.(*Server).serveStreams.func2.1
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1029
2025-01-08T18:35:03.022+0100	ERROR	{"kind": "server", "component": "grpc", "operation": "/controlplane.v1.AttestationStateService/Read", "args": "workflow_run_id:\"924c3c79-eeb0-4306-869c-297f2a3a3a76\"", "code": 500, "reason": "internal error", "stack": "error: code = 500 reason = internal error message = server error metadata = map[] cause = <nil>", "latency": 0.016550125}
  github.com/go-kratos/kratos/contrib/log/zap/v2.(*Logger).Log
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/contrib/log/zap/[email protected]/zap.go:41
  github.com/go-kratos/kratos/v2/log.(*logger).Log
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/log/log.go:30
  github.com/go-kratos/kratos/v2/log.(*Filter).Log
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/log/filter.go:99
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.Server.func4.1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/middleware/logging/logging.go:40
  github.com/go-kratos/kratos/v2/middleware/recovery.Recovery.func2.1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/middleware/recovery/recovery.go:59
  github.com/go-kratos/kratos/v2/transport/grpc.NewServer.(*Server).unaryServerInterceptor.func1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/transport/grpc/interceptor.go:40
  google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1202
  github.com/chainloop-dev/chainloop/app/controlplane/api/controlplane/v1._AttestationStateService_Read_Handler
  /the-folder/projects/chainloop/backend/app/controlplane/api/controlplane/v1/attestation_state_grpc.pb.go:189
  google.golang.org/grpc.(*Server).processUnaryRPC
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1393
  google.golang.org/grpc.(*Server).handleStream
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1804
  google.golang.org/grpc.(*Server).serveStreams.func2.1
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1029

[jiparis]

It makes sense, since the passphrase is generated from the workload token. We can read this in the code:

// In order to encrypt the state at rest, we will encrypt the state using a passphrase
// which comes in the request and is used to derive an encryption key.
// The passphrase that we'll use is the robot-account token that only the workload should have.

We should probably find a better way. But for sure, that error message should be more friendly.

[migmartri]

ok, that makes sense. The use of a different token was they key that was missing. We definitely need to handle this better but it's not that pressing.

@javier just so I understand if other users can run into this. Why are you using two API tokens, are you doing a mix of CI/local development?

[javirln]

Yes, that's exactly what I was doing. I was wondering if I could cancel an ongoing attestation locally. My use case was the following:

I had several pipelines that were not handled correctly, meaning the CI job failed but it didn't change the attestation status. So I thought about changing the state myself and mark them as failed. Since I didn't have access to the token used to initialized the attestation I used another one and then ran into the issue.