runtime icon indicating copy to clipboard operation
runtime copied to clipboard

Published 20 hours ago • verified publisher icon acorn-io

IAR + auto-upgrade - Not able to deploy app from images that was built locally and pushed to external registry and then signed.

Open sangee2004 opened this issue 1 year ago • 6 comments

Server - main

Steps to reproduce the problem:

  1. Enable IAR feature (acorn install --features image-allow-rules=true)
  2. Add IAR that allows all images that are signed by a gh key
kubectl describe imageallowrules -n acorn 
Name:         rule1
Namespace:    acorn
Labels:       <none>
Annotations:  <none>
API Version:  api.acorn.io/v1
Images:
  **
Kind:  ImageAllowRule
Metadata:
  Creation Timestamp:  2023-09-26T18:14:45Z
  Generation:          1
  Resource Version:    1359
  UID:                 f3b9c9c9-ee67-4b30-b024-6f7b8b3c418a-p
Signatures:
  Rules:
    Annotations:
      Match:
        Qatag:  ok
    Signed By:
      All Of:
        gh://***
Events:  <none>
  1. Build an push image. acorn build --push -t docker.io/sangeetha/mytestupgradebug:v0.0.1 .
  2. Sign image as per IAR from step2.
`acorn image sign sangeetha/mytestupgradebug:v0.0.1 --key ~/key -a qatag=ok`
  •  Signing Image sangeetha/mytestupgradebug:v0.0.1 (digest: index.docker.io/sangeetha/mytestupgradebug@sha256:e6948644671775a2657dcf646d7f267b53eb4c6f89033b1f0b8d774dd4f90cd8)
? Enter password for private key: *********
  ✔  Created signature sha256:1fbd2c7c7b518e10d782e21daad973234597b0af4fc6b5e01d679884a595254a
  1. Deploy app in autoupgrade format.
acorn run -n mytest docker.io/sangeetha/mytestupgradebug:v0.0.# 
  ✗  ERROR:  could not parse reference: docker.io/sangeetha/mytestupgradebug:v0.0.#
  1. Deploy app.
acorn run -n mytest docker.io/sangeetha/mytestupgradebug:v0.0.1 
  •  WARNING:  This application would like to use the image 'index.docker.io/sangeetha/mytestupgradebug:v0.0.1'.
                 This image is not trusted by any image allow rules in this project.
                 This could be VERY DANGEROUS to the cluster if you do not trust this
                 application. If you are unsure say no.

? Do you want to allow this app to use this (POTENTIALLY DANGEROUS) image? NO
  ✗  ERROR:  image <docker.io/sangeetha/mytestupgradebug:v0.0.1> is not allowed by any ImageAllowRule in this project
sangeethahariharan@Sangeethas-MacBook-Pro example1 % acorn images

Note - Once the local image entry is deleted, then we are able to deploy apps successfully

%acorn images
REPOSITORY                                   TAG       IMAGE-ID
index.docker.io/sangeetha/mytestupgradebug   v0.0.1    e69486446717

%acorn images rm e69486446717
Untagged index.docker.io/sangeetha/mytestupgradebug:v0.0.1
Deleted e6948644671775a2657dcf646d7f267b53eb4c6f89033b1f0b8d774dd4f90cd8

% acorn images                                                   
REPOSITORY   TAG       IMAGE-ID

% acorn run -n mytest docker.io/sangeetha/mytestupgradebug:v0.0.#
mytest
STATUS: ENDPOINTS[] HEALTHY[0] UPTODATE[0] waiting for image to satisfy auto-upgrade tag v0.0.#
STATUS: ENDPOINTS[] HEALTHY[0] UPTODATE[0] (container: mywebnew): pending
STATUS: ENDPOINTS[http://mywebnew-mytest-fd724ce7.local.oss-acorn.io] HEALTHY[0/1] UPTODATE[1] (container: mywebnew): ContainerCreating
┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
| STATUS: ENDPOINTS[http://mywebnew-mytest-fd724ce7.local.oss-acorn.io] HEALTHY[1] UPTODATE[1] OK |
└─────────────────────────────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
| STATUS: ENDPOINTS[http://mywebnew-mytest-fd724ce7.local.oss-acorn.io] HEALTHY[1] UPTODATE[1] OK |
└─────────────────────────────────────────────────────────────────────────────────────────────────┘

sangee2004 avatar Sep 26 '23 22:09 sangee2004

This seems to be fixed. I just tried it in my local runtime with the following IAR:

apiVersion: api.acorn.io/v1
kind: ImageAllowRule
metadata:
  name: iar-2186
  namespace: acorn
imageSelector:
  namePatterns:
    - "**"
  signatures:
    - signedBy:
        anyOf:
          - gh://iwilltry42
          - gh://***
      annotations:
        match:
          qatag: ok

This worked with a self-built image, with and without auto-upgrade and even with the image that you used above after I added your github account in the signatures settings. Note though, that due to other changes the command now has to specify the full registry: acorn run "docker.io/sangeetha/mytestupgradebug:v0.0.#"

iwilltry42 avatar Nov 01 '23 14:11 iwilltry42

@iwilltry42 I am still able to reproduce this issue in local cluster following the exact steps mentioned in the issue.

As mentioned in the issue , I will be able to deploy with the remote image as long as I delete the local image entry.

Tested with acorn version v0.9.1-16-g89212fea+89212fea

%acorn run -n mytest docker.io/sangeetha/mytestupgradebug:v1.0.# 
  ✗  ERROR:  could not parse reference: docker.io/sangeetha/mytestupgradebug:v1.0.#
 % acorn images
REPOSITORY                                   TAG       IMAGE-ID
index.docker.io/sangeetha/mytestupgradebug   v1.0.0    355d10ec575d
 % acorn run -n mytest docker.io/sangeetha/mytestupgradebug:v1.0.# 
  ✗  ERROR:  could not parse reference: docker.io/sangeetha/mytestupgradebug:v1.0.#
sangeethahariharan@Sangeethas-MacBook-Pro example1 % acorn run -n mytest docker.io/sangeetha/mytestupgradebug:v1.0.0
  •  WARNING:  This application would like to use the image 'index.docker.io/sangeetha/mytestupgradebug:v1.0.0'.
                 This image is not trusted by any image allow rules in this project.
                 This could be VERY DANGEROUS to the cluster if you do not trust this
                 application. If you are unsure say no.

? Do you want to allow this app to use this (POTENTIALLY DANGEROUS) image?  [Use arrows to move, type to filter]
> NO
  yes (this tag only)
  repository (all images in this repository)
  registry (all images in this registry)
  all (all images out there)
  ✗  ERROR:  interrupt: image [docker.io/sangeetha/mytestupgradebug:v1.0.0] is not allowed by any ImageAllowRule in this project
sangeethahariharan@Sangeethas-MacBook-Pro example1 % acorn images rm 355d10ec575d                                           
Untagged index.docker.io/sangeetha/mytestupgradebug:v1.0.0
Deleted 355d10ec575d223b9ddbca355d4ed981b6c9a76ab1d2a02bd1e9ea278b8c4165
sangeethahariharan@Sangeethas-MacBook-Pro example1 % acorn images                                                   
REPOSITORY   TAG       IMAGE-ID
sangeethahariharan@Sangeethas-MacBook-Pro example1 % acorn run -n mytest docker.io/sangeetha/mytestupgradebug:v1.0.# 
mytest
STATUS: ENDPOINTS[] HEALTHY[] UPTODATE[] 
STATUS: ENDPOINTS[] HEALTHY[0] UPTODATE[0] waiting for image to satisfy auto-upgrade tag v1.0.#
STATUS: ENDPOINTS[] HEALTHY[0] UPTODATE[0] waiting for image to satisfy auto-upgrade tag v1.0.#; could not parse reference: 
STATUS: ENDPOINTS[] HEALTHY[0] UPTODATE[0] (container: mywebnew): pending; could not parse reference: 
STATUS: ENDPOINTS[] HEALTHY[0/1] UPTODATE[1] (container: mywebnew): ContainerCreating
STATUS: ENDPOINTS[http://mywebnew-mytest-fd724ce7.local.oss-acorn.io] HEALTHY[0/1] UPTODATE[1] (container: mywebnew): ContainerCreating
STATUS: ENDPOINTS[http://mywebnew-mytest-fd724ce7.local.oss-acorn.io] HEALTHY[0/1] UPTODATE[1] (container: mywebnew): not ready
┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
| STATUS: ENDPOINTS[http://mywebnew-mytest-fd724ce7.local.oss-acorn.io] HEALTHY[1] UPTODATE[1] OK |
└─────────────────────────────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
| STATUS: ENDPOINTS[http://mywebnew-mytest-fd724ce7.local.oss-acorn.io] HEALTHY[1] UPTODATE[1] OK |
└─────────────────────────────────────────────────────────────────────────────────────────────────┘

sangee2004 avatar Nov 01 '23 20:11 sangee2004

Am I doing anything different here? (Neglecting the auto-upgrade part)

asciicast

iwilltry42 avatar Nov 02 '23 12:11 iwilltry42

@iwilltry42 Seems like when I sign sangeetha/testbug:v0.0.1 and try to deploy docker.io/sangeetha/testbug:v0.0.# , it works as expected as long as there is no local images (which is not signed).

Following scenario works as expected:

Make a version of image - sangeetha/testbug:v0.0.1 made available in docker hub that is still not signed ( I built and pushed this image from a different project).

From the current project , do the following

  1. IAR feature (acorn install --features image-allow-rules=true)
  2. Add IAR that allows all images that are signed by a gh key
kubectl describe imageallowrules -n acorn 
Name:         rule1
Namespace:    acorn
Labels:       <none>
Annotations:  <none>
API Version:  api.acorn.io/v1
Images:
  **
Kind:  ImageAllowRule
Metadata:
  Creation Timestamp:  2023-09-26T18:14:45Z
  Generation:          1
  Resource Version:    1359
  UID:                 f3b9c9c9-ee67-4b30-b024-6f7b8b3c418a-p
Signatures:
  Rules:
    Annotations:
      Match:
        Qatag:  ok
    Signed By:
      All Of:
        gh://***
Events:  <none>
  1. Try to deploy app using docker.io/sangeetha/testbug:v0.0.# . This prompts for allowing image which is as expected since we have not signed the image yet. Do not allow for the image to be used.
 acorn run -n mytest docker.io/sangeetha/testbug:v0.0.# 
  •  WARNING:  This application would like to use the image 'docker.io/sangeetha/testbug:v0.0.#'.
                 This image is not trusted by any image allow rules in this project.
                 This could be VERY DANGEROUS to the cluster if you do not trust this
                 application. If you are unsure say no.

? Do you want to allow this app to use this (POTENTIALLY DANGEROUS) image? NO
  ✗  ERROR:  image [docker.io/sangeetha/testbug:v0.0.#] is not allowed by any ImageAllowRule in this project

4.Sign the image using image name as sangeetha/testbug:v0.0.1

% acorn image sign sangeetha/testbug:v0.0.1 --key ~/<key> -a qatag=ok 
  •  Signing Image sangeetha/testbug:v0.0.1 (digest: index.docker.io/sangeetha/testbug@sha256:7d0d61cdb6be205c9ef3bd40a0ecc3eeec079e63cacf95c27c434dcf8760be8c)
? Enter password for private key: *********
  ✔  Created signature sha256:49f9193cbc6aeb794257027f4f7c741772cad0a8546a3eae691364f2c1dce3f2
  1. Deploy app using autoupgrade image format - docker.io/sangeetha/testbug:v0.0.#

This succeeds as expected.

% acorn run -n mytestautonew docker.io/sangeetha/testbug:v0.0.# 
mytestautonew
STATUS: ENDPOINTS[] HEALTHY[0] UPTODATE[0] waiting for image to satisfy auto-upgrade tag v0.0.#; could not parse reference: 
STATUS: ENDPOINTS[] HEALTHY[0] UPTODATE[0] (container: mywebnew): pending; could not parse reference: 
STATUS: ENDPOINTS[http://<Pending Ingress>] HEALTHY[0/1] UPTODATE[0] (container: mywebnew): updating
STATUS: ENDPOINTS[http://mywebnew-mytestautonew-b4cc21b7.local.oss-acorn.io] HEALTHY[0/1] UPTODATE[1] (container: mywebnew): ContainerCreating
STATUS: ENDPOINTS[http://mywebnew-mytestautonew-b4cc21b7.local.oss-acorn.io] HEALTHY[0/1] UPTODATE[1] (container: mywebnew): not ready
┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
| STATUS: ENDPOINTS[http://mywebnew-mytestautonew-b4cc21b7.local.oss-acorn.io] HEALTHY[1] UPTODATE[1] OK |
└────────────────────────────────────────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
| STATUS: ENDPOINTS[http://mywebnew-mytestautonew-b4cc21b7.local.oss-acorn.io] HEALTHY[1] UPTODATE[1] OK |
└────────────────────────────────────────────────────────────────────────────────────────────────────────┘

So the culprit in the use case mentioned in the bug seems to the presence of the local image that is not signed. That is the reason why I was able to deploy the app successfully once the local image was deleted.

sangee2004 avatar Nov 03 '23 00:11 sangee2004

  1. acorn build --push -t docker.io/iwilltry42/acorn:test-v0.0.1 .
  2. (change Acornfile) -> acorn build --push -t docker.io/iwilltry42/acorn:test-v0.0.2 .
  3. (change Acornfile) -> acorn build --push -t docker.io/iwilltry42/acorn:test-v0.0.3 . -> Ensure that we have 3 distinct images locally
  4. Ensure that there is a single IAR as per above
  5. acorn run docker.io/iwilltry42/acorn:test-v0.0.# -> Prompt as expected (not signed yet) -> Select No
  6. acorn image sign docker.io/iwilltry42/acorn:test-v0.0.2 --key <key> -a qatag=ok
  7. Check DockerHub -> NO signature was pushed
  8. Ensure that the signature is OK: acorn image verify docker.io/iwilltry42/acorn:test-v0.0.2 --key ~/.ssh/id_rsa.pub -a qatag=ok -> OK
  9. acorn run docker.io/iwilltry42/acorn:test-v0.0.# -> Prompt even though it shouldn't, since we have a signature locally -> Apparently it's only checking the remote signatures at this point
  10. Delete the signed image: acorn image rm docker.io/iwilltry42/acorn:test-v0.0.2
  11. acorn run "docker.io/iwilltry42/acorn:test-v0.0.#" -> Prompt as expected

Results/Questions:

a) should auto-upgrade logic still check local signatures, even though it's specifically looking on remote images? b) API-Server logs say, that it's looking for index.docker.io/iwilltry42/acorn:test-v0.0.1 (auto-prefixed index.), which will eventually still lead to a verification error, as it was signed at docker.io/..., lacking that index. prefix c) I didn't observe that deleting the local images fix it (since there is no signature in the remote registry after all). It should work as soon as we push a signature to the remote registry (by re-pushing the image after signing it)

iwilltry42 avatar Nov 13 '23 16:11 iwilltry42

@ibuildthecloud need your opinion here a) Should the check against the signedName signature annotation match docker.io/index.docker.io interchangeably? b) Should auto-upgrade logic still check local signatures, even though it's specifically looking on remote images?

iwilltry42 avatar Nov 21 '23 10:11 iwilltry42