splunk user uid/gid

[tod-uma]

So, I'm trying to do more splunk app development and with that I'm trying to get a better setup going. I am following this page: https://www.splunk.com/en_us/blog/tips-and-tricks/hands-on-lab-sandboxing-with-splunk-with-docker.html

This all sounds great. I can experiment/rebuild anytime I want. However, I'm running into an issue. When I use the -v flag on docker to mount my local app directory to the container, the ansible scripts in the container chown it the the container's splunk uid/gid. This changed the permissions on the host so I can no longer edit the files as my user.

I could be wrong, but it looks like the splunk uid/gid is set in the Dockerfile and the user is created when the the image is built. I've tried using a namespace, but that doesn't seem to fix the issue. (I'm kinda new to docker still and may be doing it wrong) It seems like I need the splunk user to be created at container run time, not at image build time.

Does that seem correct? If so, I could make a pull-request to the splunk-ansible repo to make it able to add the splunk user when the ansible scripts run and use a variable for the uid/gid. Then we could specify the splunk uid/gid on the docker run command.

[nwang92]

I believe this might be more related to how uids/gids and permissions map between host and container. In the Docker images, we do create a static splunk user with some arbitrary UID. This is mostly for security compliance and to ensure that Splunk won't run as some privileged user.

But in your case, since you're developing locally, I don't see why you can't change those settings are runtime. This might require some tweaks to the docker CLIs in that blog, but if you're on Linux and you're running as the root user, you can run:

$ docker run -d -P --user root -e SPLUNK_START_ARGS=--accept-license -e SPLUNK_GROUP=root -e SPLUNK_USER=root -e SPLUNK_PASSWORD=helloworld splunk/splunk:latest

This might get a little trickier if you're on Docker for Mac - not sure if there's any extra layer of user-mapping done by the VM powering that, but I can test that out for you to find something that works if you'd like.

[tod-uma]

I don't think switching splunk to run as root will fix it. With userns-remap on, I get the following error setting SPLUNK_USER and SPLUNK_GROUP to root (the UMS_app is the app I'm developing and mounting using the -v flag to docker):

TASK [splunk_common : Update Splunk directory owner] ***************************
fatal: [localhost]: FAILED! => {
    "changed": false,
    "gid": 65534,
    "group": "nobody",
    "mode": "0775",
    "owner": "nobody",
    "path": "/opt/splunk/etc/apps/UMS_app",
    "size": 8,
    "state": "directory",
    "uid": 65534
}

MSG:

chown failed: [Errno 1] Operation not permitted: '/opt/splunk/etc/apps/UMS_app'

If I don't have userns-remap enabled, then the UMS_app directory gets chown'ed to root and my local user can't edit the files. This is the same problem I have without userns-remap but having splunk run as the splunk user in the container. the directory gets chown'ed to the splunk uid/gid in the container which my host system doesn't know about.

[nwang92]

I think I see what you're saying. I actually don't see this on Docker for Mac at all (I don't explicitly set userns-remap in the daemon, so I'm guessing this may be disabled out of the box?). With an app in my local filesystem such as:

splunk_app_example/
    default/
        app.conf
    metadata/
        default.meta

I can run this without any problems, and without changes to permissions on my host filesystem.

docker run -v "$(pwd)/splunk_app_example:/opt/splunk/etc/apps/splunk_app_example" -e SPLUNK_START_ARGS=--accept-license -e SPLUNK_PASSWORD=helloworld -P -d splunk/splunk:latest

But I believe this behavior might be special to Mac due to the local Docker VM behind the scenes to enable containers to run. My guess is you're seeing the problem on straight Linux? Let me pull up a Linux VM and tinker around and see if I can workaround this.

[tod-uma]

Yes, I'm running a linux host.

If I download the docker-splunk repo and edit splunk/common-files/Dockerfile to use my user's UID/GID then rebuild the image, it works as I'd want. I'm able to edit files in the app via my host's vim setup as I'd like and it's immediately reflected in the docker container.

[nwang92]

Just posting an update: I think I've got your scenario reproduced, but I seem to be hitting the same issues/misunderstanding how to use the userns-remap feature?

Following a mix between https://docs.docker.com/engine/security/userns-remap/ and https://success.docker.com/article/user-namespace-runtime-error, I've got dockerd running with the correct configurations in place:

[root@mi-shc-sh2 ~]# cat /etc/docker/daemon.json
{
  "userns-remap": "dockerusr"
}
[root@mi-shc-sh2 ~]# sudo -u dockerusr bash
[dockerusr@mi-shc-sh2 apps]$ cat /etc/subuid
dockerusr:231072:65536
[dockerusr@mi-shc-sh2 apps]$ cat /etc/subgid
dockerusr:231072:65536
[dockerusr@mi-shc-sh2 apps]$ ls -ltrah
total 4.0K
drwxr-xr-x   4 dockerusr dockerusr   35 Jun 17 22:41 splunk_app_example
drwxr-xr-x   3 dockerusr dockerusr   31 Jun 17 22:50 .
drwxrwxrwt. 13 root      root      4.0K Jun 17 23:00 ..

My thinking here is at this point, the uid of dockerusr should be mapped to the equivalent of the 0 uid/root user from within any container created. Therefore, ensuring that Splunk Enterprise runs as root and the default user of the container is also root should preserve the same uid from host::container all the way through:

docker run -it -P -e SPLUNK_START_ARGS=--accept-license \
                             -e SPLUNK_PASSWORD=helloworld \
                             -e SPLUNK_USER=root \
                             -e SPLUNK_GROUP=root \
                             -u root -v "$(pwd):/opt/splunk/etc/apps/" \
                             splunk/splunk:latest

But this apparently doesn't work as I expect. I am reaching the same error as you; adding a -e ANSIBLE_VERBOSITY=3 to the above CLI does tell me that it's trying to configure the permissions as root:

TASK [splunk_common : Update Splunk directory owner] **********************************************************************************************************************************************
task path: /opt/ansible/roles/splunk_common/tasks/change_splunk_directory_owner.yml:2
The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
  File "/tmp/ansible_file_payload_Nat4zB/ansible_file_payload.zip/ansible/module_utils/basic.py", line 1010, in set_owner_if_different
    os.lchown(b_path, uid, -1)
fatal: [localhost]: FAILED! => {
    "changed": false,
    "gid": 65534,
    "group": "nobody",
    "invocation": {
        "module_args": {
            "_diff_peek": null,
            "_original_basename": null,
            "access_time": null,
            "access_time_format": "%Y%m%d%H%M.%S",
            "attributes": null,
            "backup": null,
            "content": null,
            "delimiter": null,
            "directory_mode": null,
            "follow": true,
            "force": false,
            "group": "root",
            "mode": null,
            "modification_time": null,
            "modification_time_format": "%Y%m%d%H%M.%S",
            "owner": "root",
            "path": "/opt/splunk",
            "recurse": true,
            "regexp": null,
            "remote_src": null,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "src": null,
            "state": "directory",
            "unsafe_writes": null
        }
    },
    "mode": "0755",
    "owner": "nobody",
    "path": "/opt/splunk/etc/apps",
    "size": 31,
    "state": "directory",
    "uid": 65534
}

MSG:

chown failed: [Errno 1] Operation not permitted: '/opt/splunk/etc/apps'

[jbreed]

Any updates on this? Attempting to get this over to my unraid server directly from this image without having to make modifications to it. UNRaid requires volumes to be set the user to 99 and the group to 100, which would be awesome to be able to pass in the ID used for the accounts to align up with the infrastructure permissions.

I attempted to run it simply with -u root and changing the splunk user/group to root, but it failed on the following: TASK [splunk_common : Start Splunk via CLI] ************************************ fatal: [localhost]: FAILED! => { "changed": false, "cmd": [ "/opt/splunk/bin/splunk", "start", "--accept-license", "--answer-yes",

With the traceback pointing to the following: `An unforeseen error occurred:

    Exception: <class 'FileNotFoundError'>, Value: [Errno 2] No such file or directory: '/opt/splunk/etc/users'`

After failing, looking at this path this is what is listed:

bash-4.4# cd /opt/splunk/etc/ bash-4.4# ls -l total 103 drwxrwxrwx 1 1000 sudo 512 Aug 27 08:07 anonymizer drwxrwxrwx 1 1000 sudo 512 Sep 4 05:44 apps drwxrwxrwx 1 1000 sudo 512 Sep 30 22:24 auth -r-xr-xr-x 1 1000 sudo 57 Aug 27 07:47 copyright.txt -r-xr-xr-x 1 1000 sudo 8182 Aug 27 07:47 datetime.xml drwxrwxrwx 1 1000 sudo 512 Aug 27 08:07 deployment-apps drwxrwxrwx 1 1000 sudo 512 Aug 27 08:07 disabled-apps -rwxrwxrwx 1 1000 sudo 2759 Aug 27 07:50 findlogs.ini drwxrwxrwx 1 1000 sudo 512 Aug 27 08:07 init.d drwxrwxrwx 1 1000 sudo 512 Sep 30 22:24 licenses -r-xr-xr-x 1 1000 sudo 498 Aug 27 07:49 log-btool-debug.cfg -r-xr-xr-x 1 1000 sudo 526 Aug 27 07:49 log-btool.cfg -r-xr-xr-x 1 1000 sudo 256 Aug 27 07:49 log-cmdline-debug.cfg -r-xr-xr-x 1 1000 sudo 2340 Aug 27 07:49 log-cmdline.cfg -r-xr-xr-x 1 1000 sudo 5676 Aug 27 07:49 log-debug.cfg -r-xr-xr-x 1 1000 sudo 555 Aug 27 07:50 log-dfs-search.cfg -r-xr-xr-x 1 1000 sudo 549 Aug 27 07:50 log-dfs.cfg -r-xr-xr-x 1 1000 sudo 2606 Aug 27 07:50 log-searchprocess.cfg -r-xr-xr-x 1 1000 sudo 1171 Aug 27 07:50 log-utility.cfg -rwxrwxrwx 1 1000 sudo 21954 Aug 27 07:50 log.cfg drwxrwxrwx 1 1000 sudo 512 Aug 27 08:07 master-apps drwxrwxrwx 1 1000 sudo 512 Aug 27 08:07 modules drwxrwxrwx 1 1000 sudo 512 Sep 30 22:24 myinstall drwxrwxrwx 1 1000 sudo 512 Sep 30 22:24 openldap -rwxrwxrwx 1 1000 sudo 164 Sep 30 22:24 passwd -r-xr-xr-x 1 1000 sudo 1836 Aug 27 07:47 prettyprint.xsl -rwxrwxrwx 1 1000 sudo 1116 Aug 27 08:08 regid.2001-12.com.splunk-Splunk-Enterprise.swidtag -r-xr-xr-x 1 1000 sudo 21719 Aug 27 07:47 searchLanguage.xml drwxrwxrwx 1 1000 sudo 512 Aug 27 08:07 shcluster -rwxrwxrwx 1 1000 sudo 1725 Aug 27 08:07 splunk-enttrial.lic -rwxrwxrwx 1 1000 sudo 720 Sep 30 22:24 splunk-launch.conf -rwxrwxrwx 1 1000 sudo 720 Aug 27 08:07 splunk-launch.conf.default -r-xr-xr-x 1 1000 sudo 70 Aug 27 07:50 splunk.version drwxrwxrwx 1 1000 sudo 512 Sep 30 22:22 system bash-4.4#

Looking on dockerhub splunk/splunk, I'm guessing the this is being set statically not giving us the option to pass UID and GID. Layer 16: |2 GID=41812 UID=41812 /bin/sh -c groupadd -r -g ${GID} ${SPLUNK_GROUP} && useradd -r -m -u ${UID} -g ${GID} -s /bin/bash ${SPLUNK_USER} && chmod 755 /sbin/updateetc.sh