node.js 16 build on UBI8 is very slow and fails in GH Actions with ECONNRESET and too many open files errors.

[jmontleon]

/kind bug

Description:

I am experiencing a problem trying to build node.js project on UBI8 using podman and qemu-user-static, that does not occur with UBI9. https://github.com/konveyor/tackle2-ui/actions/runs/4168992831/jobs/7216425953#step:2:1067 https://github.com/konveyor/tackle2-ui/actions/runs/4174604421/jobs/7228394767#step:2:1077

I am able to reproduce the painfully long builds I see there locally on Fedora with qemu-user-static-7.0.0-13.fc37.x86_64. I turned up logging on the first node install command to silly and it looks like just before downloading dependencies should start CPU ramps up to 100% and it does nothing for a very long time. Locally the too many files and ECONNRESET errors do not reproduce, but the arm64 build still takes several hours.

Steps to reproduce the issue:

1. clone https://github.com/konveyor/tackle2-ui
2. DOCKER_BUILDKIT=1 podman build -f Dockerfile --build-arg ENV=PROD --manifest multiarch-ui-test:latest --platform linux/arm64 .

Describe the results you received: Extremely slow builds, in GH action it eventually fails with an error about too many open files.

Describe the results you expected: Sane build time and successful result

Environment: GH Action using Ubunut 22.04 and mostly reproduced on Fedora 37 with qemu-user-static-7.0.0-13.fc37.x86_64

$ podman version
Client:       Podman Engine
Version:      4.4.1
API Version:  4.4.1
Go Version:   go1.19.5
Built:        Thu Feb  9 05:58:53 2023
OS/Arch:      linux/amd64

Additional information optionally: UBI9 works fine both locally and in GH Actions: https://github.com/jmontleon/tackle2-ui/actions/runs/4175222062/jobs/7229841318

We can likely do this to get past the situation, though we weren't in a hurry to do it immediately. I thought I'd try here to see if there is a known issue we can work around, else what we can do to troubleshoot/debug further.