CRC podman images are missing

[onderson]

Hi, i have a podman info output as below.

host:
  arch: amd64
  buildahVersion: 1.21.3
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.29-1.module+el8.4.0+11822+6cc1e7d7.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: ae467a0c8001179d4d0adf4ada381108a893d7ec'
  cpus: 12
  distribution:
    distribution: '"rhcos"'
    version: "4.10"
  eventLogger: file
  hostname: crc-m5qn9-master-0
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.18.0-305.45.1.el8_4.x86_64
  linkmode: dynamic
  memFree: 5401116672
  memTotal: 21029343232
  ociRuntime:
    name: runc
    package: runc-1.1.0-2.rhaos4.10.el8.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.0
      spec: 1.0.2-dev
      go: go1.17.5
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.4.0+11822+6cc1e7d7.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 0
  swapTotal: 0
  uptime: 44m 32.96s
registries:
  search:
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.6-1.module+el8.4.0+11822+6cc1e7d7.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.6
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.3
  Built: 1627570963
  BuiltTime: Thu Jul 29 15:02:43 2021
  GitCommit: ""
  GoVersion: go1.15.7
  OsArch: linux/amd64
  Version: 3.2.3

once i check Graphroot path i dont see any images in my localstorage and strangely the output is below.

ls -l /var/home/core/.local/share/containers/storage ls: /var/home/core/.local/share/containers/storage: No such file or directory

and for a locally build image, if i try to use skopeo it oddly fails.

skopeo copy containers-storage:localhost/sample:v1 docker://quay.io/user1/sample:latest
FATA[0000] Invalid source name containers-storage:localhost/sample:v1: chown /Users/burgekaraer/.local/share/containers/storage/vfs: operation not permitted

this is what i have got. ls -l /Users/burgekaraer/.local/share/containers/storage total 0 drwx------ 2 burgekaraer staff 64 11 Jun 03:34 mounts -rw-r--r-- 1 burgekaraer staff 0 11 Jun 03:34 storage.lock drwx------ 2 burgekaraer staff 64 11 Jun 03:34 tmp -rw-r--r-- 1 burgekaraer staff 0 11 Jun 03:34 userns.lock drwx------ 2 burgekaraer staff 64 11 Jun 03:34 vfs drwx------ 3 burgekaraer staff 96 11 Jun 03:49 vfs-containers drwx------ 3 burgekaraer staff 96 11 Jun 04:11 vfs-images

if i run

sudo chown root:wheel /Users/burgekaraer/.local/share/containers/storage/vfs

then skopeo is giving a different error.

skopeo copy containers-storage:localhost/sample:v1 docker://quay.io/osezgin/sample:latest FATA[0000] initializing source containers-storage:[vfs@/Users/burgekaraer/.local/share/containers/storage+/private/var/folders/k4/n90g79rd6vv0fl__ysg8_j280000gn/T/containers-user-501/containers]localhost/sample:v1: reference "[vfs@/Users/burgekaraer/.local/share/containers/storage+/private/var/folders/k4/n90g79rd6vv0fl__ysg8_j280000gn/T/containers-user-501/containers]localhost/sample:v1" does not resolve to an image ID: identifier is not an image

would anybody have any idea explanation etc to find out what's going on?

Another question is where podman is placing the built or downloaded images in local disk? How can i find that out?

Many thanks

[gbraad]

Which preset are you using? It seems you are running the OpenShift preset and using podman-env, right? As the info says:

distribution: '"rhcos"'

Note: Please make sure to provide the/'as much' information as asked in the template

[cfergeau]

ls -l /Users/burgekaraer/.local/share/containers/storage

Looks like you are using this on macOS. On macOS, podman really is podman-remote, and the actual work (image storage, ...) is done by the VM crc started. This means ls -l /var/home/core/.local/share/containers/storage ls: /var/home/core/.local/share/containers/storage: No such file or directory should be run inside the VM, I don't know where you run it.

[onderson]

Which preset are you using? It seems you are running the OpenShift preset and using podman-env, right? As the info says:

distribution: '"rhcos"'

Note: Please make sure to provide the/'as much' information as asked in the template

my config is as mentioned here

[onderson]

that means i should do skopeo from the vm? even though i am not sure how to connect to VM or if there is a way to copy from vm by running skopeo from host (my local) to remote. in theory there should be since skopeo is able to copy between two regs, i just dont know what to specify for the vm which is serving images..

[cfergeau]

that means i should do skopeo from the vm?

You haven't confirmed what you tried doing exactly, running skopeo from the VM would be one way yes. But we expose unix sockets on your host which can be used to talk to the podman instance using docker/podman http API, hopefully skopeo can use these directly?

[onderson]

i have built an image and wanted to copy that to public registry which was quay.io. it is experimental effort though but would be good to achieve.

[cfergeau]

Can you use podman push for this?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.