Sysbox Data Root Configuration Issues

[matthewparkinsondes]

The following Sysbox Data Root issues were observed with Sysbox-CE 0.4.0.

• specifying a Data Root of /data/sysbox creates the standard 5x directories under both /var/lib/sysbox and /data/sysbox
• it is expected that neither /var/lib/sysbox nor its subdirectories would be created in this scenario
• this becomes confusing when confirming which directories Sysbox is using
• it also makes it more difficult to tell whether Sysbox is successfully cleaning up the Data Root directory

---

• using systemctl start sysbox ignores config changes in /lib/systemd/system/sysbox.service
• whereas systemctl restart sysbox does pick up config changes
• both after firstly performing systemctl daemon-reload

---

• specifying a Data Root which is a symlink produces the following error when a System container is launched

docker: Error response from daemon: OCI runtime create failed: container_linux.go:393: starting container process caused: process_linux.go:607: container init caused: standard_init_linux.go:82: validating cwd caused: rootfs_linux.go:1142: cwd /data/sysbox/shiftfs/b158cc46-6d84-4c73-94df-36f5b8517ac6 is not container's rootfs /var/lib/sysbox/shiftfs/b158cc46-6d84-4c73-94df-36f5b8517ac6 caused:: unknown.
ERRO[0006] error waiting for container: context canceled

• seems restarting Sysbox again allowed the System container to launch successfully in one test which is now not able to be reproduced
• the symlink was from /var/lib/sysbox to /data/sysbox
• the config is kept in original condition (ie. specifies --version only and no --data-root specified on the startup line)
• using a symlink works successfully in Sysbox-CE 0.3.0 (ie. the above issues are not observed)
• the workaround in Sysbox-CE 0.4.0 is to configure --data-root to directly point to the target directory, rather than the source directory of the symlink

[ctalledo]

Hi @matthewparkinsondes,

Thanks for reporting the problems.

Regarding the first problem:

specifying a Data Root of /data/sysbox creates the standard 5x directories under both /var/lib/sysbox and /data/sysbox.

I don't see this; I see Sysbox creates the directory under /data/sysbox (as it should), and does not create it under /var/lib/sysbox.

However, I do see that Sysbox does not cleanup the data root directory when stopped. For example, if data-root is /var/lib/sysbox and you stop Sysbox, the contents of that directory are removed but the directory itself remains. This is a bug and we will fix it very soon.

I wonder if this bug explains your observation in the following way: you start Sysbox with data-root set to /var/lib/sysbox (the default), then you stop Sysbox, modify the data-root to /data/sysbox, and start Sysbox again. In this case, the /var/lib/sysbox directory will remain since it was not removed by Sysbox when it was stopped.

[ctalledo]

Regarding:

using systemctl start sysbox ignores config changes in /lib/systemd/system/sysbox.service

I don't see this either. For example, this sequence works without problem:

1. Start Sysbox with its default data-root (/var/lib/sysbox).

2. Stop Sysbox (systemctl stop sysbox)

3. Modify the sysbox-mgr data-root to /data/sysbox (i.e., via file /lib/systemd/system/sysbox-mgr.service)

4. Reload the systemd unit files (systemctl daemon-reload)

5. Start Sysbox (systemctl start sysbox)

After step (5), Sysbox creates the data-root dir in /data/sysbox (as expected).

[ctalledo]

Regarding:

specifying a Data Root which is a symlink produces the following error when a System container is launched

Given that sysbox-mgr has the --data-root option to configure the Sysbox data-root, a symlink seems unnecessary. I am not sure this is something we want to support this going forward as it introduces unnecessary complexity (some parts of the Sysbox code now need to evaluate if the data-root is a symlink or not and react accordingly).

Is there a strong use case that I am missing here?

[matthewparkinsondes]

We notice sysbox-mgr is mentioned in the ExecStart line within the following two files.

/lib/systemd/system/sysbox.service /lib/systemd/system/sysbox-mgr.service

The first with --version, and the second with --log /var/log/sysbox-mgr.log.

We have been configuring the Data Root by removing --version from the first file and adding --data-root /data/sysbox.

We have been leaving the second file as is (it appears we should have been applying the Data Root configuration to this file instead).

Is this a valid configuration option, and what is the correct technique for configuring the Data Root location (given there are two different configurations specified in these files for starting sysbox-mgr)?

[matthewparkinsondes]

Regarding the symlink:

• using a symlink works successfully in Sysbox-CE 0.3.0 (ie. the above issues are not observed)

This looks to be a regression in the latest version.

The symlink behaviour is similar to Docker configuration. A symlink can be supplied in the standard /var/lib/docker location, which points to a larger dedicated disk.

Similarly, configuring Sysbox with a symlink allows the standard /var/lib/sysbox location to be used, while pointing to a larger dedicated disk.

This provides a simple reference point for locating the associated data stores.

As the example in the previous comment potentially demonstrates, it also avoids the need to delve into specific system configuration files to setup a dedicated data root directory.

[ctalledo]

Hi @matthewparkinsondes ,

We have been configuring the Data Root by removing --version from the first file and adding --data-root /data/sysbox.

Oh I see, that explains your observations; the --data-root option must currently be set via the sysbox-mgr.service unit file, not the sysbox.service file. The former actually starts the sysbox-mgr sub-service, while the latter serves as a wrapper that waits for the sub-services (sysbox-fs.service and sysbox-mgr.service) to start. The latter also invokes sysbox-mgr --version and sysbox-fs --version simply as a way to log the version info in that service's journal log, nothing more. It does not actually start those sub-services (that's done by systemd itself).

Apologies for the confusion. We have documentation about this in the Sysbox User Guide's configuration chapter, though I can see further clarification is needed in that doc. I will add that soon.

Going forward, we plan to add a Sysbox configuration file to capture all configuration options, so users will not need to touch the systemd services. This will be done soon (likely before the next release).

[ctalledo]

Regarding:

This looks to be a regression in the latest version.

Could be, but I don't think it's something we meant to formally support in Sysbox 0.3.0 either.

I see your points on why it's useful to support the symlink. I'll take a closer look again at the effort required, but it's likely a low priority item at this point.

[ctalledo]

Also: apart from the sysbox-mgr's --data-root option, note that sysbox-fs also sets up a FUSE mountpoint which by default is located at /var/lib/sysboxfs. This can also be changed via the --mountpoint option in the sysbox-fs systemd unit file.

[ctalledo]

Apologies for the confusion. We have documentation about this in the Sysbox User Guide's configuration chapter, though I can see further clarification is needed in that doc. I will add that soon.

Doc clarifications were added; see here:

https://github.com/nestybox/sysbox/blob/master/docs/user-guide/configuration.md

[matthewparkinsondes]

Thanks, this has helped.

[leojonathanoh]

i agree with @ctalledo. Even if symlink of /var/lib/sysbox was supported, it is best practice to explicitly specify the data-dir as with docker daemon.json.

{
    "graph": "/mnt/docker"
}

So sysbox should really have some config file to make this easier, as with #220