🐛[BUG]: Too many open files error in ahmed_body_mgn example

[naruhikot]

Version

24.04

On which installation method(s) does this occur?

Docker

Describe the issue

Converted docker container to apptainer. When I ran training script, I got "too many open files" error. See the trimmed log bellow.

Minimum reproducible example

apptainer run --nv --bind `pwd`,$USER_DIR $CONTAINER_IMAGE bash -c "python train.py"

Relevant log output

...
OSError: [Errno 24] Too many open files: '/usr/local/lib/python3.10/dist-packages/pandas/core/api.py'
OSError: [Errno 24] Too many open files: '/usr/local/lib/python3.10/dist-packages/pandas/core/algorithms.py'
OSError: [Errno 24] Too many open files: '/usr/local/lib/python3.10/dist-packages/pandas/core/config_init.py'
...

Environment details

Environment location: TSUBAME4.0 (https://www.t4.gsic.titech.ac.jp/en)
ulimit -n is set to 16384 on this system