sylabs
Target of /etc/localtime symlink in container should not be overwritten by host localtime content
Before you report an issue...
- [x] Have you searched for a duplicate report?
- [x] Have you replicated the problem on the latest release?
- [x] Have you checked the latest documentation?
- [x] Are you using an up-to-date system?
Only issues that can be replicated on the latest release, or development branch, of SingularityCE will be investigated and fixed. The open source project does not maintain long-term stable branches or fix bugs in prior versions. If you require LTS support then please see the Sylabs website.
Version of Singularity
What version of Singularity are you using? Run:
kinow@ranma:~/Development/golang/workspace/singularity$ singularity --version
singularity-ce version 4.3.0+77-g2244f30fe
Describe the bug
This bug is similar to this bug in Apptainer, https://github.com/apptainer/apptainer/issues/1868, which was fixed in this pull request, https://github.com/apptainer/apptainer/pull/2102
We (cc @oloapinivad) have a container for data analysis that is built with Docker, using a Ubuntu base image, and some extra layers/tools/etc.. The final container is used to run Python code that uses libraries like Numpy, Pandas, Xarray, etc., and Matplotlib to create a plot. The input data is normally data from climate models executed on HPCs.
These containers are executed within these HPCs: LUMI, in Finland with Singularity 4.1.3, and MareNostrum5 in Spain using Singularity 3.11.5, and 4.1.5.
We noticed that one of these plots was being produced with an incorrect year, and identified that it was due to an issue with the timezone. In LUMI, the timezone is set to Europe/Helsinki, and in MareNostrum5 to Europe/Madrid.
We tried forcing the timezone to be UTC so that the Python scripts could be correctly executed, and even though the TZ variable is set, Python was still getting the EEST timezone for Helsinki. Upon inspection, we realized that /usr/share/zoneinfo/Etc/UTC had the exact same content (md5sum) as the Europe/Helsinki file (NOTE: that's inside the container, in the host computer these are distinct files).
So even forcing the TZ=UTC variable in the container, it still used the Etc/UTC file and loaded the incorrect timezone, so Python (and its time module & glibc calls) got into a very confusing state, where some calls/libs would return UTC, but in the end datetime and matplotlib modules would use UTC with EEST values.
We reported the issue to matplotlib and confirmed the issue was not in matplotlib, https://github.com/matplotlib/matplotlib/issues/30136
And I confirmed that it works on my laptop with Apptainer 1.3.4. The fix in Apptainer was done in 1.3.1, in April 2024. It seems like their fix was simply avoids replacing UTC when that's not a symbolic link (probably something around this part of the code/fix https://github.com/apptainer/apptainer/pull/2102/files#diff-8fe2e788783d673a9a5754cc5b40a58e965dd209aea9027670ec808d31758c82R1843-R1859).
I think the options we have now in our project are a) to try to run operations with --no-mount /etc/localtime and hope it doesn't break anything, b) ask operation to install apptainer and adjust our workflows to drop Singularity CE, or c) install a newer version with the fix.
To Reproduce
Steps to reproduce the behavior:
I simplified our Dockerfile down to what I think is the minimum required to reproduce this:
FROM ubuntu:22.04
# Install common dependencies
RUN export DEBIAN_FRONTEND=noninteractive && \
apt-get --yes update && \
apt-get --yes upgrade && \
apt-get --yes --no-install-recommends install \
bash \
binutils \
tzdata \
&& \
apt-get --yes clean && \
apt-get --yes autoremove && \
rm -rf /var/lib/apt/lists/*
CMD ["/bin/bash"]
docker build --load -t test .singularity build test.sif docker-daemon://testsingularity shell --cleanenv test.sif
As I am in Europe/Madrid time zone, now I can verify that the UTC file is wrong (I used 4.3.0+77-g2244f30fe to produce the output below with the test.sif file):
# Same hash
Singularity> md5sum /usr/share/zoneinfo/Europe/Madrid
491ee8e91dc29f30301542bbb391548e /usr/share/zoneinfo/Europe/Madrid
Singularity> md5sum /usr/share/zoneinfo/Etc/UTC
491ee8e91dc29f30301542bbb391548e /usr/share/zoneinfo/Etc/UTC
# `localtime` points to `UTC`
Singularity> ls -lah /etc/localtime
lrwxrwxrwx 1 root root 27 Jun 6 15:13 /etc/localtime -> /usr/share/zoneinfo/Etc/UTC
# But UTC file is not really UTC
Singularity> strings /usr/share/zoneinfo/UTC
TZif2
WEST
WEMT
CEST
TZif2
WEST
WEMT
CEST
CET-1CEST,M3.5.0,M10.5.0/3
Now some applications might work, when they handle the time zone conversion in a different way, but those that read the information in the file or use syscalls that do that, will get a confusing result where UTC range/offset will not really match.
There won't be any errors, but the data produced will be wrong/inconsistent.
Expected behavior
The file /usr/share/zoneinfo/Etc/UTC contains the zone information for UTC, and not the value from the localtime from the host computer.
OS / Linux Distribution
Which Linux distribution are you using? Is it up-to-date?
# LUMI
$ cat /etc/os-release
NAME="SLES"
VERSION="15-SP5"
VERSION_ID="15.5"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp5"
DOCUMENTATION_URL="https://documentation.suse.com/"
# MareNostrum5
NAME="Red Hat Enterprise Linux"
VERSION="9.2 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.2"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.2 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.2"
# My laptop (works with apptainer, fails with latest Singularity from source)
PRETTY_NAME="Ubuntu 24.10"
NAME="Ubuntu"
VERSION_ID="24.10"
VERSION="24.10 (Oracular Oriole)"
VERSION_CODENAME=oracular
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=oracular
LOGO=ubuntu-logo
Installation Method
Sorry, I don't have details about the HPC modules. They are managed with Lua module, and installed by HPC operations. I believe they are simply checking out the sources and compiling onto the HPC, perhaps installing some of the dependencies directly from their source/repo, or from the HPC vendor if necessary/available.
To reproduce it locally, I used 4.3.1 tag, and also main, following the installation steps from this doc, https://github.com/sylabs/singularity/blob/2244f30fe0f9d52cd0f1d8f689bb442b2ac2ed7a/INSTALL.md
Additional context
NA
This seems like a valid bug fix to import. However...
(a) We need to think about where the fix is applied, i.e. possibly only importing into the main branch, for the next minor version rather than including in a patch release. It is likely that there are users / workflows (inadvertently or otherwise) depending on the fact that Singularity is effectively 'forcing' a local timezone where UTC should be used. We try to avoid breaking anything like this in patch releases where a workaround is available for the underlying issue.
(b) we need to handle the underlay approach that Apptainer is specifically calling out as unsupported.
For now, I would recommend not mounting /etc/localtime via the --no-mount flag as you suggest.
I have the same issue, so I'll be sharing similar details here.
After building, the container's UTC definition is replaced by the local timezone (EST). One example of a downstream effect of this bug is in python-dateutil where calling dateutil.tz.gettz("UTC") incorrectly behaves like local time instead of UTC.
Here's a minimal recipe to reproduce:
Bootstrap: library
From: ubuntu:22.04
%post
apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends tzdata
Build and run these inside the container:
# UTC appears configured (empty values omitted)
$ debconf-show tzdata
tzdata/Areas: Etc
tzdata/Zones/Etc: UTC
$ cat /etc/timezone
Etc/UTC
$ readlink /etc/localtime
/usr/share/zoneinfo/Etc/UTC
# This produces wrong UTC epoch times
$ date --date "1970-01-01T00:00:00" +%s
18000
$ TZ=UTC date --date "1970-01-01T00:00:00" +%s
18000
$ TZ=EST date --date "1970-01-01T00:00:00" +%s
18000
$ TZ=GMT+0 date --date "1970-01-01T00:00:00" +%s
0
# The UTC definition has local timezone content
$ zdump -v -c 1970,1971 UTC | head
UTC -9223372036854775808 = NULL
UTC -9223372036854689408 = NULL
UTC Sun Apr 26 06:59:59 1970 UT = Sun Apr 26 01:59:59 1970 EST isdst=0 gmtoff=-18000
UTC Sun Apr 26 07:00:00 1970 UT = Sun Apr 26 03:00:00 1970 EDT isdst=1 gmtoff=-14400
UTC Sun Oct 25 05:59:59 1970 UT = Sun Oct 25 01:59:59 1970 EDT isdst=1 gmtoff=-14400
UTC Sun Oct 25 06:00:00 1970 UT = Sun Oct 25 01:00:00 1970 EST isdst=0 gmtoff=-18000
UTC 9223372036854689407 = NULL
UTC 9223372036854775807 = NULL
# Note the md5sum of the UTC definition
$ find /usr/share/zoneinfo -type f -exec md5sum {} + | awk -v ref="$(md5sum /etc/localtime | awk '{print $1}')" '$1 == ref'
71b8c43aff041d1523e4922a74530810 /usr/share/zoneinfo/Etc/UTC
Exit the container, and note that its UTC definition content matches the host's local timezone definition
$ find /usr/share/zoneinfo -type f -exec md5sum {} + | awk -v ref="$(md5sum /etc/localtime | awk '{print $1}')" '$1 == ref'
71b8c43aff041d1523e4922a74530810 /usr/share/zoneinfo/America/Montreal
Changing the symbolic link into a hard link at the end of the recipe is another workaround that still allows mounting the host's timezone:
[ -L /etc/localtime ] && ln -f $(realpath /etc/localtime) /etc/localtime
It's wise to bind /etc/timezone too so that it's consistent with /etc/localtime. The debconf values for tzdata should probably also reflect the host's timezone, but I can't think of a simple solution for that.
