withFileSystemBind does not mount any files

[tmspgl]

trafficstars

Kubedock Version 0.9.2

Host OS MacOS

Host Arch ARM

Environment: Local Cluster with k3d: Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.10+k3s1", GitCommit:"471f5eb3dbfeaee6b6dd6ed9ab4037c10cc39680", GitTreeState:"clean", BuildDate:"2022-02-24T23:38:19Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/arm64"} Gitlab CI/CD pipeline k8s cluster: Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.9", GitCommit:"b631974d68ac5045e076c86a5c66fba6f128dc72", GitTreeState:"clean", BuildDate:"2022-01-19T17:45:53Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

Steps to Reproduce:

• Crate a new Container
• Add a withFileSystemBind of any folder
• Start the container

Expected:

• Folder is successfully bound to container and files are available

Actual:

• The folder was not mounted.

Additional Information: The argument "--pre-archive" needs to be set for kubedock. Otherwise no copy/mount did work at all.

We tried mounting and copying files to the container with all available possibilities. None of them is working on it's own. Meaning if you use any of the following methods no file is ever mounted or copied.

• .withFileSystemBind(...)
• .withCopyToContainer(...)
• .withCopyFileToContainer(...)
• .withClasspathResourceMapping(...)

However when combining multiple of these Methods they partly work. Whenever the method ".withFileSystemBind(...)" is present for the container, all other copies work as expected. But the filesystembind itself does not. In this case there is a config map as well as the mounted files present in the container.

Following shows the describe for the started alpinecontainer.

With all 4 methods called:

	Mounts:
		/config2 from pfiles (rw,path="fbc1ac88b427356b61892951faa93e54")
		/config3/redis.conf from pfiles (rw,path="c7f85a5709e2ec3a35488d25ef0bdde7")
		/config4/redis.conf from pfiles (rw,path="377f7586930fe9b2b041e64a12d5e76e")
		/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r4f4t (ro)
	Volumes:
		pfiles:
			Type: ConfigMap (a volume populated by a ConfigMap)
			Name: e363d58d2cf8
			Optional:  false

The same output without the .withFileSystemBind BUT still we the other 3 copies (no volume present afterwards):

	Mounts:
		/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jn4c8 (ro)

Used example test:

@Test
@SuppressWarnings("unchecked")
void shouldBindDirectory() {
    String configPath = Paths.get("").toAbsolutePath() + "/build/it-config/data";
    File file = new File(configPath + "/redis.conf");
    GenericContainer test = new GenericContainer(DockerImageName.parse("alpine:3.15"))
        .withLogConsumer(new Slf4jLogConsumer(getLogger(this.getClass().getName())))
        .withFileSystemBind(configPath, "/config", BindMode.READ_ONLY)
        .withCopyToContainer(Transferable.of(file.getAbsolutePath()), "/config2/redis.conf")
        .withCopyFileToContainer(MountableFile.forHostPath(file.getAbsolutePath()), "/config3/redis.conf")
        .withClasspathResourceMapping("redis.conf", "/config4/redis.conf", BindMode.READ_ONLY)
        .withCommand("sh", "-c", "sleep 5 && echo started && tail -f /dev/null")
        .waitingFor(Wait.forLogMessage(".*started.*", 1).withStartupTimeout(Durations.ONE_MINUTE));
    test.start();
    try {
        String basePath = test.execInContainer("ls", "/").getStdout();
        log.info(basePath);
        String foo1 = test.execInContainer("ls", "/config").getStdout();
        String foo2 = test.execInContainer("ls", "/config2/").getStdout();
        String foo3 = test.execInContainer("ls", "/config3").getStdout();
        String foo4 = test.execInContainer("ls", "/config4").getStdout
        log.info("TEST withFileSystemBind: " + foo1);
        log.info("TEST withCopyToContainer: " + foo2);
        log.info("TEST withCopyFileToContainer: " + foo3);
        log.info("TEST withClasspathResourceMapping: " + foo4);
    } catch (IOException | InterruptedException e) {
        throw new RuntimeException(e);
    }
}

[tmspgl]

This does only occur when executing in gitlab-ci pipeline or for testing purposes in a local cluster. Running it locally in docker works as expected.

[joyrex2001]

I am trying to reproduce as well (using minikube), but I can't. Both with and without the --pre-archive all tests produce the expected output. Is the instance of kubedock able to directly access the files, or folders? What I have seen e.g. in similar issues, is that kubedock is running in a sidecar, but the actual folders that are to be bound are not available in that sidecar; maybe it is something similar?

[joyrex2001]

Did some more extensive testing and the only way to reproduce is when the volumes which are to be bound, are not directly accessible by kubedock. The other working methods in your tests are methods that copy the files from the running test system via the rest api, and don't require direct access to the filesystem.

See also the tekton example here, where the source folder is also mounted in the sidecar, making the folders directly accessible for kubedock as well.

[tmspgl]

I will check out the example and provide you with more information if needed. Had no time yet to dig into.

[mausch]

I stepped through the kubedock code when called from testcontainers.net I inspected the raw request contents and noticed that the mounts are defined in HostConfig.Mounts but the kubedock struct for HostConfig only supports Binds not Mounts: https://github.com/joyrex2001/kubedock/blob/8b1f7ce3d08386cbd4f20cdc13ca9c4119f9a766/internal/server/routes/docker/types.go#L39-L44

Not sure if this is also what testcontainers java does but I think kubedock should support Mounts since IIRC it's the more recent and feature-rich API?

[jonesbusy]

I stepped through the kubedock code when called from testcontainers.net I inspected the raw request contents and noticed that the mounts are defined in HostConfig.Mounts but the kubedock struct for HostConfig only supports Binds not Mounts: https://github.com/joyrex2001/kubedock/blob/8b1f7ce3d08386cbd4f20cdc13ca9c4119f9a766/internal/server/routes/docker/types.go#L39-L44

Not sure if this is also what testcontainers java does but I think kubedock should support Mounts since IIRC it's the more recent and feature-rich API?

I'm not sure if I have similar issue. But only happen with testcontainers .NET and not java or node. So something different on implementation.

The point is that expect my container (ryuk) to be created with sidecars when it's not. So it could be possible the socket bind created differently on docker API.

Dind setup works great for Java and Node.

[joyrex2001]

@mausch @jonesbusy I added support for mounts in hostconfig, that fixed ryuk support for dotnet at my end.

[mausch]

Nice! I'll give it a try in a few days, thanks!