Slow performance when using AWS EFS as additional image store

[ankurmalhotra07]

Issue Description

Describe your issue Want to use additional image stores as explained in this guide to cache images and speed up builds. However, the use of these additional stores is slowing down builds even more.

Steps to reproduce the issue

Steps to reproduce the issue Provision EFS share in AWS Mount EFS share mount -t efs -o tls fs-123...:/ /var/lib/mycontainers Pull image using podman time podman --root /var/lib/mycontainers pull docker.io/amazoncorretto:latest

Describe the results you received

Describe the results you received Writing the image to the image store takes a significantly longer time. Note that copying the image directory using CP command to the EFS takes significantly less time. Therefore, I suspect that it has something to do with the interaction between podman and the EFS share.

Describe the results you expected

Describe the results you expected Write should be significant faster.

podman info output

+ podman info

host:

  arch: amd64

  buildahVersion: 1.33.3

  cgroupControllers:

  - cpuset

  - cpu

  - cpuacct

  - blkio

  - memory

  - devices

  - freezer

  - net_cls

  - perf_event

  - net_prio

  - hugetlb

  - pids

  cgroupManager: cgroupfs

  cgroupVersion: v1

  conmon:

    package: conmon-2.1.10-1.fc39.x86_64

    path: /usr/bin/conmon

    version: 'conmon version 2.1.10, commit: '

  cpuUtilization:

    idlePercent: 91.67

    systemPercent: 1.63

    userPercent: 6.7

  cpus: 8

  databaseBackend: sqlite

  distribution:

    distribution: fedora

    variant: container

    version: "39"

  eventLogger: file

  freeLocks: 2048

  hostname: 

  idMappings:

    gidmap: null

    uidmap: null

  kernel: 5.10.209-198.858.amzn2.x86_64

  linkmode: dynamic

  logDriver: k8s-file

  memFree: 18091565056

  memTotal: 65994149888

  networkBackend: netavark

  networkBackendInfo:

    backend: netavark

    dns:

      package: aardvark-dns-1.10.0-1.fc39.x86_64

      path: /usr/libexec/podman/aardvark-dns

      version: aardvark-dns 1.10.0

    package: netavark-1.10.3-1.fc39.x86_64

    path: /usr/libexec/podman/netavark

    version: netavark 1.10.3

  ociRuntime:

    name: crun

    package: crun-1.14.3-1.fc39.x86_64

    path: /usr/bin/crun

    version: |-

      crun version 1.14.3

      commit: 1961d211ba98f532ea52d2e80f4c20359f241a98

      rundir: /run/crun

      spec: 1.0.0

      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL

  os: linux

  pasta:

    executable: /usr/bin/pasta

    package: passt-0^20231230.gf091893-1.fc39.x86_64

    version: |

      pasta 0^20231230.gf091893-1.fc39.x86_64

      Copyright Red Hat

      GNU General Public License, version 2 or later

        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>

      This is free software: you are free to change and redistribute it.

      There is NO WARRANTY, to the extent permitted by law.

  remoteSocket:

    exists: false

    path: /run/podman/podman.sock

  security:

    apparmorEnabled: false

    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT

    rootless: false

    seccompEnabled: true

    seccompProfilePath: /usr/share/containers/seccomp.json

    selinuxEnabled: false

  serviceIsRemote: false

  slirp4netns:

    executable: /usr/bin/slirp4netns

    package: slirp4netns-1.2.2-1.fc39.x86_64

    version: |-

      slirp4netns version 1.2.2

      commit: 0ee2d87523e906518d34a6b423271e4826f71faf

      libslirp: 4.7.0

      SLIRP_CONFIG_VERSION_MAX: 4

      libseccomp: 2.5.3

  swapFree: 0

  swapTotal: 0

  uptime: 27h 29m 35.00s (Approximately 1.12 days)

  variant: ""

plugins:

  authorization: null

  log:

  - k8s-file

  - none

  - passthrough

  - journald

  network:

  - bridge

  - macvlan

  - ipvlan

  volume:

  - local

registries:

  search:

  - registry.fedoraproject.org

  - registry.access.redhat.com

  - docker.io

  - quay.io

store:

  configFile: /etc/containers/storage.conf

  containerStore:

    number: 0

    paused: 0

    running: 0

    stopped: 0

  graphDriverName: overlay

  graphOptions:

    overlay.ignore_chown_errors: "true"

    overlay.imagestore: /var/lib/mycontainers

    overlay.mount_program:

      Executable: /usr/bin/fuse-overlayfs

      Package: fuse-overlayfs-1.12-2.fc39.x86_64

      Version: |-

        fusermount3 version: 3.16.1

        fuse-overlayfs: version 1.12

        FUSE library version 3.16.1

        using FUSE kernel interface version 7.38

    overlay.mountopt: nodev,fsync=0

  graphRoot: /var/lib/containers/storage

  graphRootAllocated: 549743210496

  graphRootUsed: 60121776128

  graphStatus:

    Backing Filesystem: xfs

    Native Overlay Diff: "false"

    Supports d_type: "true"

    Supports shifting: "true"

    Supports volatile: "true"

    Using metacopy: "false"

  imageCopyTmpDir: /var/tmp

  imageStore:

    number: 0

  runRoot: /var/run/containers/storage

  transientStore: false

  volumePath: /var/lib/containers/storage/volumes

version:

  APIVersion: 4.9.0

  Built: 1706090847

  BuiltTime: Wed Jan 24 10:07:27 2024

  GitCommit: ""

  GoVersion: go1.21.6

  Os: linux

  OsArch: linux/amd64

  Version: 4.9.0

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

• Fuse driver already installed - https://github.com/kuberenetes-learning-group/fuse-device-plugin/blob/master/README_EN.md
• This is what storage.conf looks like:

[storage]
driver = "overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"

[storage.options]
additionalimagestores = ["/var/lib/mycontainers"]

[storage.options.overlay]
ignore_chown_errors = "true"
mount_program = "/usr/bin/fuse-overlayfs"
mountopt = "nodev,fsync=0"

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[giuseppe]

[storage.options.overlay] ignore_chown_errors = "true" mount_program = "/usr/bin/fuse-overlayfs" mountopt = "nodev,fsync=0"

why are you using fuse-overlayfs?

You can use native overlay with additional image stores

[giuseppe]

sorry, closed by mistake

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.