Docker container creates all files owned by root

[wtanksleyjr]

Describe the bug My log shows that it's not finding /libation/appsettings.json, which is odd because it's not supposed to be looking there. It's supposed to be looking in /config/Settings.json, given how the Docker script is written. (I admit I'm speculating.)

To Reproduce Here's my docker-compose yml (unfortunately this editor flattened it, never seen it do that before!):

version: "2"
services:
libation:
image: rmcrackan/libation
restart: unless-stopped
user: 1000:1000
volumes:
- /home/william/au/sources/downloads/Audible:/audiobooks
- /home/william/au/libation/config/:/config
- /home/william/au/libation/db/:/db

Expected behavior I expected it to load my Settings.json file; instead it tries and fails to load a nonexistent file inside the container, that I can't map to.

Screenshots n/a

Platform Ubuntu Docker-Compose

Log Files I really don't know how to get Docker log files.... Here's the backtrace.

2023-01-06 07:02:23: Liberating books
libation_1        | Unhandled exception. System.UnauthorizedAccessException: Access to the path '/libation/appsettings.json' is denied.
libation_1        |  ---> System.IO.IOException: Permission denied
libation_1        |    --- End of inner exception stack trace ---
libation_1        |    at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
libation_1        |    at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Func`4 createOpenException)
libation_1        |    at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Func`4 createOpenException)
libation_1        |    at System.IO.File.OpenHandle(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
libation_1        |    at System.IO.File.WriteToFile(String path, FileMode mode, String contents, Encoding encoding)
libation_1        |    at LibationFileManager.Configuration.getLibationFilesSettingFromJson() in /Source/LibationFileManager/Configuration.LibationFiles.cs:line 71
libation_1        |    at LibationFileManager.Configuration.get_LibationFiles() in /Source/LibationFileManager/Configuration.LibationFiles.cs:line 25
libation_1        |    at AudibleUtilities.AudibleApiStorage.get_AccountsSettingsFile() in /Source/AudibleUtilities/AudibleApiStorage.cs:line 10
libation_1        |    at AudibleUtilities.AudibleApiStorage.EnsureAccountsSettingsFileExists() in /Source/AudibleUtilities/AudibleApiStorage.cs:line 15
libation_1        |    at AppScaffolding.LibationScaffolding.RunPostConfigMigrations(Configuration config) in /Source/AppScaffolding/LibationScaffolding.cs:line 76
libation_1        |    at LibationCli.Setup.Initialize() in /Source/LibationCli/Setup.cs:line 24
libation_1        |    at LibationCli.Program.Main(String[] args) in /Source/LibationCli/Program.cs:line 29
libation_1        |    at LibationCli.Program.<Main>(String[] args)
libation_1        | ./libation/liberate.sh: line 66:    20 Aborted                 (core dumped) /libation/LibationCli liberate
libation_1        | 2023-01-06 07:02:23: Sleeping for 30m

[rmcrackan]

I'm not a linux person. Did your PR fix this?

[wtanksleyjr]

I only modified the DEB script, not Docker. I haven't tested the new Docker.

[rmcrackan]

I mean, they both start with D don't they?

So does: d'oh!

[pixil98]

@wtanksleyjr didn't realize you already had an issue for this. Can you try following the instruction at https://github.com/rmcrackan/Libation/blob/master/Documentation/Docker.md and see if it works correctly?

[wtanksleyjr]

Sure thing! Sorry, I've been struggling with a cold/flu/something. Hopefully will get time tonight, we'll see if I can stay awake.

-Wm

On Mon, Jan 23, 2023 at 11:36 AM pixil98 @.***> wrote:

@wtanksleyjr https://github.com/wtanksleyjr didn't realize you already had an issue for this. Can you try following the instruction at https://github.com/rmcrackan/Libation/blob/master/Documentation/Docker.md and see if it works correctly?

— Reply to this email directly, view it on GitHub https://github.com/rmcrackan/Libation/issues/439#issuecomment-1400875069, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7H6MTD2ONUGES4YQQGRTWT3MSDANCNFSM6AAAAAATSZEWMY . You are receiving this because you were mentioned.Message ID: @.***>

[rmcrackan]

@wtanksleyjr any luck on this?

[wtanksleyjr]

Oh my! I just got over the flu, so today is a perfect day to do this. I'm starting right now.

-Wm

On Sun, Feb 5, 2023 at 5:42 AM rmcrackan @.***> wrote:

@wtanksleyjr https://github.com/wtanksleyjr any luck on this?

— Reply to this email directly, view it on GitHub https://github.com/rmcrackan/Libation/issues/439#issuecomment-1417878502, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7H6IZAZNC7SI7BT2SE2LWV6U6BANCNFSM6AAAAAATSZEWMY . You are receiving this because you were mentioned.Message ID: @.***>

[wtanksleyjr]

OK, so ... there are a few problems. The little one is that you're using sudo to run as root, which really shouldn't be the case; this is an audiobook downloader, and shouldn't need root permissions. The big problem is that even when I don't run as root, the payload creates all files using root's account, so you wind up having to use sudo anyhow.

I tried running like this (to run a single fetch, I just bought something at the new sale):

docker run -d -v ~/Test:/config -v ~/Test/Books:/data -e SLEEP_TIME='-1' --name libation --restart=no rmcrackan/libation

It worked, I got the file; but the resulting file is owned by root. So I have to sudo chown to fix that.

I'm not sure at all how to fix this kind of thing. I know Docker can handle it, but I don't know how.

[wtanksleyjr]

I think I found how to fix this -- I dug around, and the option to use seems to be --user $(id -u):$(id -g). That tells Docker to map the internal user to the given external user, and those variable expansions run Linux commands that are replaced with the correct UID and GID.

On the other hand, that clearly won't work on Windows, and I don't know what to do there.

So the command that works is:

docker run -d -v ~/Test:/config -v ~/Test/Books:/data -e SLEEP_TIME='-1' --name libation --restart=no --user $(id -u):$(id -g) rmcrackan/libation

Notice this is not run with sudo.

[rmcrackan]

@wtanksleyjr Thanks for looking into this. Glad you're feeling better

@pixil98 Can you verify? This is not in my wheelhouse

[pixil98]

The sudo is there in the example because I didn't want to assume that the user had added themselves to the docker group. I didn't want to end up owning docker install/config support. That looks like a good way to run the docker image as the current user and group ids. There is also a directive that can be put in the Dockerfile that will set default user and group ids that we should consider. I'm not sure what down stream effects that would have. The liberate.sh script currently makes some assumptions that the user it's running as it root and those would need to be updated. I don't recall if there is a directive for the CLI that lets us pass in a config path rather than assuming it's at ~/Libation, if not that would help here.

@wtanksleyjr if you want to tackle this I'm willing to review the pull request. Heads up that I'll be leaving for vacation in a week and won't be back until mid March.

[wtanksleyjr]

1. Unfortunately I'm not going to be using the Docker ... although I like doing that in general, in this case I just can't make it work for me, and don't know enough to figure it out ... and the .DEB install works for me.

2. I don't like running as root, though. I think people who install Docker should always add themselves to the group; it's simply a standard part of the installation. I don't think we should recommend doing otherwise.

3. I suspect we've got a "good enough" with the recommendation to run it with the "--user" addition. People who don't remember how to run with a user will remember; people who don't want to will hopefully have a good enough reason to (shudder) run as root.

4. Possibly we shouldn't use the currently provided script. Running a docker with its own built-in cron is kind of a strange thing to do. There are IMO better ways to do scheduled running, mainly using the system you're actually using to do the scheduling. Additionally, it's unfortunate that the main way to run it is a rather inflexible script; it would be better to just let the user have full access to the CLI, as is done with other Docker packages (for example, I use grpcurl through a Docker package, in order to let me spoof the DNS using Docker's tools).

5. Likely if we do move forward with this, we should make an archive that includes everything the user needs except their account files, particularly generating useful config files. Like how the DEB install does it. All you should have to do is copy in the account file, and make sure everything's in the right place.

-Wm

On Mon, Feb 13, 2023 at 9:29 AM pixil98 @.***> wrote:

The sudo is there in the example because I didn't want to assume that the user had added themselves to the docker group. I didn't want to end up owning docker install/config support. That looks like a good way to run the docker image as the current user and group ids. There is also a directive that can be put in the Dockerfile that will set default user and group ids that we should consider. I'm not sure what down stream effects that would have. The liberate.sh script currently makes some assumptions that the user it's running as it root and those would need to be updated. I don't recall if there is a directive for the CLI that lets us pass in a config path rather than assuming it's at ~/Libation, if not that would help here.

@wtanksleyjr https://github.com/wtanksleyjr if you want to tackle this I'm willing to review the pull request. Heads up that I'll be leaving for vacation in a week and won't be back until mid March.

— Reply to this email directly, view it on GitHub https://github.com/rmcrackan/Libation/issues/439#issuecomment-1428360342, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7H6OGCCONAFY5PWXLVUTWXJVN5ANCNFSM6AAAAAATSZEWMY . You are receiving this because you were mentioned.Message ID: @.***>

[rmcrackan]

@wtanksleyjr Thanks for the update. Should we just close out this ticket?

[wtanksleyjr]

Probably - but perhaps comment in the documentation that Docker could use some work and we welcome advice on how to do it right.

-Wm

On Thu, Feb 16, 2023 at 4:49 AM rmcrackan @.***> wrote:

@wtanksleyjr https://github.com/wtanksleyjr Thanks for the update. Should we just close out this ticket?

— Reply to this email directly, view it on GitHub https://github.com/rmcrackan/Libation/issues/439#issuecomment-1433038615, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7H6K5D5E2CC4ASRONICTWXYO4RANCNFSM6AAAAAATSZEWMY . You are receiving this because you were mentioned.Message ID: @.***>

[robflate]

Apologies adding to a closed thread but I'm struggling to get the docker image to map to my local user. My compose file is;

  libation:
    image: rmcrackan/libation
    container_name: libation
    restart: unless-stopped
    volumes:
      - /libation/opt/libation/config:/config
      - /libation/opt/libation/books:/data
    environment:
      SLEEP_TIME: 30m
    user: 1000:1000

This results in;

2023-09-13 11:12:53: Starting
2023-09-13 11:12:53: Sleep time is set to 30m
2023-09-13 11:12:53: Linking config directory to the Libation config directory
ln: failed to create symbolic link '/root/Libation': Permission denied
2023-09-13 11:12:53: Scanning accounts
Unhandled exception. System.UnauthorizedAccessException: Access to the path '/Libation' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.FileSystem.CreateDirectory(String fullPath, UnixFileMode unixCreateMode)
   at System.IO.Directory.CreateDirectory(String path)
   at FileManager.PersistentDictionary..ctor(String filepath, Boolean isReadOnly) in /Source/FileManager/PersistentDictionary.cs:line 30
   at LibationFileManager.Configuration.get_LibationFiles() in /Source/LibationFileManager/Configuration.LibationFiles.cs:line 32
   at AudibleUtilities.AudibleApiStorage.get_AccountsSettingsFile() in /Source/AudibleUtilities/AudibleApiStorage.cs:line 10
   at AudibleUtilities.AudibleApiStorage.EnsureAccountsSettingsFileExists() in /Source/AudibleUtilities/AudibleApiStorage.cs:line 15
   at AppScaffolding.LibationScaffolding.RunPostConfigMigrations(Configuration config) in /Source/AppScaffolding/LibationScaffolding.cs:line 83
   at LibationCli.Setup.Initialize() in /Source/LibationCli/Setup.cs:line 20
   at LibationCli.Program.Main(String[] args) in /Source/LibationCli/Program.cs:line 65
   at LibationCli.Program.<Main>(String[] args)
./libation/liberate.sh: line 66:    12 Aborted                 (core dumped) /libation/LibationCli scan
2023-09-13 11:12:55: Liberating books

If I remove user: 1000:1000, everything works but the files are owned by root.

Any advice appreciated. Thanks.

[pixil98]

I think there are a few things going wrong and there's not much that can be done without an MR to make some changes to how the docker image works.

The first problem is that it's trying to setup a symlink from /config to where Libation expects the configuration to be, in the user's home directory. It's currently hard coded to be root's home directory. This could be changed to use the current user's home directory, or ideally, we'd just pass in a parameter to the CLI telling it where the config directory is. This would need to be changed in https://github.com/rmcrackan/Libation/blob/master/Docker/liberate.sh and possibly have a new CLI command added.

The second problem is that it looks like the /libation folder isn't readable by your user. We create that folder in the Dockerfile, https://github.com/rmcrackan/Libation/blob/master/Dockerfile, on line 19 via the COPY command. We should just add a USER declaration in there to be a non-root user by default and then provide the COPY command with a --chown flag to set the user correctly on it.

[wtanksleyjr]

Maybe it would be easier if we didn't include the libation.sh script, and instead included a zipfile containing the basic config that the user is expected to customize and then map to /config (probably the zip would have same files the Debian installer copies into ~/Libation). Anyhow, the user can unzip it, revealing a docker-compose.yml file and a folder containing the config; they can then add a soft-link to their download space at the folder pointed to by the docker-compose, and then copy their own accounts.json file from a machine with graphics support, optionally with other changes for the other config file that has the application settings.

They can then do docker-compose up -d, and then use docker-compose exec libationcontainer LibationCli ... to run its commands. If the user wants to do crontabs (like libation.sh does now) they can do that from their own machine. That way all the files that Libation needs to write to are actually mapped to user folders, nothing sits inside the container except immutable stuff.

I should try again to see if I can actually do this ... last time I tried I didn't know enough about either docker or how Libation actually works, I've learned a lot since due to the excellent Debian installer.

[pixil98]

I think the unfriendliness of the config is a real problem, but is separate from the non-root issue. Something I've seen done on other containers is to have an init parameter that creates a basic config the first time they're run, then users could go edit that config in the mapped volume and run it again. Another option could be to support at least basic config via envvars and either construct our own config if one isn't provided or ideally the CLI would allow them to be specified as switches (or maybe libation just reads the envvars directly?).

[wtanksleyjr]

True, but I was referring to making all of the writable files exist only on user drive mappings, not inside the container. That way the permissions don't matter, and the script doesn't do this kind of copying.

On Thu, Sep 14, 2023 at 12:56 PM pixil98 @.***> wrote:

I think the unfriendliness of the config is a real problem, but is separate from the non-root issue. Something I've seen done on other containers is to have an init parameter that creates a basic config the first time they're run, then users could go edit that config in the mapped volume and run it again. Another option could be to support at least basic config via envvars and either construct our own config if one isn't provided or ideally the CLI would allow them to be specified as switches (or maybe libation just reads the envvars directly?).

— Reply to this email directly, view it on GitHub https://github.com/rmcrackan/Libation/issues/439#issuecomment-1720057356, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7H6JGN4UETRB23ZD4NPLX2NOORANCNFSM6AAAAAATSZEWMY . You are receiving this because you were mentioned.Message ID: @.***>

[pixil98]

Ah sure, I'm not sure moving away from the entry script is the right move as users would need to know where to mount the directory, it will be in the home folder of the user the docker image is running as internally. A simple fix of updating the script to link it to ~/Libation instead of /root/Libation would probably suffice and keep the mount directory consistent. Longer term I'd rather have the ability to tell the CLI where the libation directory is via switch. @rmcrackan how hard would it be to add a --config switch to the cli?

[rmcrackan]

In theory not difficult. In practice I've been too busy this summer to do anything and I'm kinda burned out on personal projects in general, hence my absence lately. Not sure about @Mbucari 's availability and/or interest.

[wtanksleyjr]

Ah sure, I'm not sure moving away from the entry script is the right move as users would need to know where to mount the directory, it will be in the home folder of the user the docker image is running as internally.

This is a problem created by placing the config directory in a home folder. Make it a fixed mount point like /config/ and the problem goes away. The user knows where to point the mount point to, and the config's permissions are equal to the running user's (because it's a mount point). And of course appsettings.json is the place to put /config.

[Mbucari]

In theory not difficult. In practice I've been too busy this summer to do anything and I'm kinda burned out on personal projects in general, hence my absence lately. Not sure about @Mbucari 's availability and/or interest.

I'm a little burned out at the moment too, and I've also got a pretty full plate from now through October 1st.

[pixil98]

I haven't looked at Libation in long enough that I forgot all about appsettings.json. It wouldn't be hard to set some values in it using jq in the docker build. Could also take some elements in as env vars to further customize it at run time.

[robflate]

Did this ever get solved?

[pixil98]

I don’t think so, it’s been on my list of things to do, but it’s pretty far down.

[muchtall]

In my use case, I am trying to have Libation dump the files directly into a directory that is shared by Nextcloud, which is also resident on the same docker host. In this case, the UID and GID are set to 33, so I have to have Libation run as that UID/GID to ensure it continues to write the files with the same permissions as the existing, Nextcloud-created files.

This is what my docker-compose.yml looks like:

services:
    libation:
        image: rmcrackan/libation
        restart: always
        user: 33:33
        volumes:
            - ./config/:/config
            - ./config/:/Libation  # Needed to avoid a crash on LibationCli startup due to an unwritable /Libation dir
            - /path/to/my/Nextcloud/Libation/dir/:/data
        environment:
            - SLEEP_TIME=10m  # Refresh libraries interval
            - UID=33
            - GID=33

So far this seems to be working fine for me. YMMV. I'm sure there's a better fix to the unwritable /Libation dir issue above.

[robflate]

Are you sure the UID and GID environment variables are actually present in the image? I couldn't find them anywhere in the code.

[pixil98]

Are you sure the UID and GID environment variables are actually present in the image? I couldn't find them anywhere in the code. Though it won’t hurt anything to set them.

They aren't currently available in this image.

[pixil98]

I have a prototype that runs as 1001:1001. I've uploaded it to docker hub, available at pixil/libation:latest.

I've tested deploying it in a fresh kubernetes cluster and (after the tedious manual config) it downloads books as 1001:1001. It also correctly fails to read a database or config owned by root. I tried using a securityContext to override the user to something else and it fails to run.

@muchtall, @wtanksleyjr, @robflate - Can you try my image and make sure it's working in your use cases? If so I'll clean it up a bit and submit a PR.

Since I'm pretty close to a solution, I'm opening this issue back up so it's easier to find and so I get the satisfaction of closing it.