[Bug]: mySQLTestcontainer can't chown /etc/mysql/conf.d folder inside rootless podman container

[JoVanMontfort]

Module

Core

Testcontainers version

1.18.3

Using the latest Testcontainers version?

Yes

Host OS

RHEL8

Host Arch

x86

Docker version

podman version 4.2.0

What happened?

mysqltestcontainer-cant-chown-etc-mysql-conf-d-folder-inside-rootless-podman-container

Relevant log output

org.testcontainers.containers.ContainerLaunchException: Container startup failed for image mysql:8.0.33 at com.xyzcorp.StudentServiceCopyTest.testMySQL8(StudentServiceCopyTest.java:61) Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception at com.xyzcorp.StudentServiceCopyTest.testMySQL8(StudentServiceCopyTest.java:61) Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container at com.xyzcorp.StudentServiceCopyTest.testMySQL8(StudentServiceCopyTest.java:61) Caused by: com.github.dockerjava.api.exception.InternalServerErrorException:  Status 500: {"cause":"error during bulk transfer for copier.request{Request:\"PUT\", Root:\"/\", preservedRoot:\"/home/jvanmont/.local/share/containers/storage/overlay/297e95cb55db40d82aa6e025772f7e05a009d64f5563ac3421e34592209e8627/merged\", rootPrefix:\"/home/jvanmont/.local/share/containers/storage/overlay/297e95cb55db40d82aa6e025772f7e05a009d64f5563ac3421e34592209e8627/merged\", Directory:\"/\", preservedDirectory:\"/home/jvanmont/.local/share/containers/storage/overlay/297e95cb55db40d82aa6e025772f7e05a009d64f5563ac3421e34592209e8627/merged\", Globs:[]string{}, preservedGlobs:[]string{}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string(nil), ExpandArchives:false, ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:false, Rename:map[string]string(nil), NoDerefSymlinks:false, IgnoreUnreadable:false, NoCrossDevice:false}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), DefaultDirOwner:(*idtools.IDPair)(nil), DefaultDirMode:(*fs.FileMode)(nil), ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, IgnoreXattrErrors:false, IgnoreDevices:false, NoOverwriteDirNonDir:false, NoOverwriteNonDirDir:false, Rename:map[string]string(nil)}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*fs.FileMode)(nil)}, RemoveOptions:copier.RemoveOptions{All:false}}: copier: put: error setting ownership of \"/etc/mysql/conf.d\" to 1980042590:1980042590: lchown /etc/mysql/conf.d: invalid argument","message":"error during bulk transfer for copier.request{Request:\"PUT\", Root:\"/\", preservedRoot:\"/home/jvanmont/.local/share/containers/storage/overlay/297e95cb55db40d82aa6e025772f7e05a009d64f5563ac3421e34592209e8627/merged\", rootPrefix:\"/home/jvanmont/.local/share/containers/storage/overlay/297e95cb55db40d82aa6e025772f7e05a009d64f5563ac3421e34592209e8627/merged\", Directory:\"/\", preservedDirectory:\"/home/jvanmont/.local/share/containers/storage/overlay/297e95cb55db40d82aa6e025772f7e05a009d64f5563ac3421e34592209e8627/merged\", Globs:[]string{}, preservedGlobs:[]string{}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string(nil), ExpandArchives:false, ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:false, Rename:map[string]string(nil), NoDerefSymlinks:false, IgnoreUnreadable:false, NoCrossDevice:false}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), DefaultDirOwner:(*idtools.IDPair)(nil), DefaultDirMode:(*fs.FileMode)(nil), ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, IgnoreXattrErrors:false, IgnoreDevices:false, NoOverwriteDirNonDir:false, NoOverwriteNonDirDir:false, Rename:map[string]string(nil)}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*fs.FileMode)(nil)}, RemoveOptions:copier.RemoveOptions{All:false}}: copier: put: error setting ownership of \"/etc/mysql/conf.d\" to 1980042590:1980042590: lchown /etc/mysql/conf.d: invalid argument","response":500}

Additional Information

mysqltestcontainer-cant-chown-etc-mysql-conf-d-folder-inside-rootless-podman-container

[eddumelendez]

Can you try using latest podman version?

[JoVanMontfort]

Can we use already podman 4.3.1 in RHEL-8 ... https://access.redhat.com/discussions/6984773#:~:text=The%20current%20podman%20version%20in,1.

[JoVanMontfort]

podman v 4.4.1 doesn't solve the problem

[guss77]

I also have the same issue, and testing with Podman 4.6.0 we still get this.

[magnuspedro]

I could fix the issue by overriding the configure method of the MySQLContainer file and removing the lchown execution. It's not a permanent fix, but MySQL will run.

public class MysqlTestContainerConfig extends MySQLContainer<MysqlTestContainerConfig> {

    private static final String MYSQL_ROOT_USER = "root";

    MysqlTestContainerConfig(String image) {
        super(image);
    }

    @Override
    protected void configure() {
        addEnv("MYSQL_DATABASE", databaseName);
        if (!MYSQL_ROOT_USER.equalsIgnoreCase(username)) {
            addEnv("MYSQL_USER", username);
        }
        if (password != null && !password.isEmpty()) {
            addEnv("MYSQL_PASSWORD", password);
            addEnv("MYSQL_ROOT_PASSWORD", password);
        } else if (MYSQL_ROOT_USER.equalsIgnoreCase(username)) {
            addEnv("MYSQL_ALLOW_EMPTY_PASSWORD", "yes");
        } else {
            throw new ContainerLaunchException("Empty password can be used only with the root user");
        }
        setStartupAttempts(3);
    }
}

[magnuspedro]

@eddumelendez apologize for my ignorance. Could you clarify why the MySQL container is creating a volume for the /etc/mysql/conf.d config?

[JoVanMontfort]

@magnuspedro how to use this configuration class? Spring config, ... ?

[jsloop42]

This issue is happening for us with Ubuntu 22.04 with Clojure and kaocha where we are using testcontainer for running tests locally that spins up MySQL. It however works with Gitlab pipeline. This is really blocker for us. The same setup works on a mac.

I have tried overriding the MySQLContainer from Clojure side using proxy but it is not working. I don't get any error at this time, but the tests get stuck. I am also not able trace any logs because I couldn't find a way to enable printing logs in kaocha.

Kindly requesting your input on how to solve this. Thanks.

[magnuspedro]

@jsloop42 Which version of docker are you using on Ubuntu vs Mac?

[magnuspedro]

@JoVanMontfort You will use the MysqlTestContainerConfig as the container starter.

[jsloop42]

@jsloop42 Which version of docker are you using on Ubuntu vs Mac?

On macOS, the docker version is:

Docker version 26.1.4, build 5650f9b

On Ubuntu, the docker version is:

Docker version 27.0.3, build 7d4bcd8

[kiview]

@jsloop42 This issue is about Podman, but you are reporting here different Docker versions. If you are using Docker and see a similar issue, please open a dedicated issue for it.

For Podman users, does this issue still persist with recent versions of Testcontainers and Podman? If yes, that means that likely the copy API behaves differently for Podman and an issue should be created with the Podman team.