Grafana issue on Synology NAS

[jaydkay]

Hi,

I tried setting up the Powerwall-Dashboard with Docker on a Synology NAS. I'm not sure if this is even supposed to work, but the setup script is running fine and three containers seem to start up okay. Only grafana keeps shutting down with the following entries in the docker log:

GF_PATHS_DATA='/var/lib/grafana' is not writable. You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migration-from-a-previous-version-of-the-docker-container-to-5-1-or-later mkdir: cannot create directory '/var/lib/grafana/plugins': Permission denied

Any idea what is causing this or how to get grafana running?

Best regards, Jochen

[jasonacox]

Hi @jaydkay - When grafana starts, it attempts to install the required plugins as indicated by the error. It uses user 1000 group 1000 for this directory. The link you pasted talks about this as being a user permission issue. Some ideas I have:

Is it possible you already had grafana (older version) installed? Try removing it and re-install:

# stop and remove
docker stop grafana
docker rm grafana

# reinstall
docker-compose -f powerwall.yml up -d

I have seen permission error if you run thesetup.sh as root. If you did this, try to remove the Powerwall-Dashboard folder and reinstall using the local user. You will need to make sure your local user has docker permissions:

# Add your user to docker group
sudo usermod -aG docker $USER

If that wasn't the case, you could try to change the permission manually:

# run a shell in grafana interactively
docker exec -ti --user root grafana /bin/bash
ls -la /var/lib/grafana/
chown 1000:1000 /var/lib/grafana/

[jaydkay]

Dear @jasonacox,

thanks for your quick reply. I tried following your suggestions, alas, no success so far...

Is it possible you already had grafana (older version) installed? Try removing it and re-install:

No, it's a completely new installation from scratch. In fact, it's my first contact with docker.

I have seen permission error if you run thesetup.sh as root. If you did this, try to remove the Powerwall-Dashboard folder and reinstall using the local user. You will need to make sure your local user has docker permissions:

Yes, I needed to start ./setup.sh via sudo - it seems that on Synology DSM, docker is typically not available for the normal user. Command usermod also doesn't exist, but there are corresponding commands, so was able to add the current user to the docker group and start docker with a standard user account. Unfortunately, the error remained the same. Additionally, now also Telegraf refuses to start with the following log entry:

I! Using config file: /etc/telegraf/telegraf.conf E! [telegraf] Error running agent: Error loading config file /etc/telegraf/telegraf.conf: open /etc/telegraf/telegraf.conf: permission denied

Finally, I also tried your final suggestion:

If that wasn't the case, you could try to change the permission manually:

That didn't work either, because the container already refuses to start, so the command you suggested complains about grafana not yet running (but it never will):

Error response from daemon: Container ac20100690faff31b7c2df6bcb6fa9079eea9a496efdbbcf2fed6b8144731aae is restarting, wait until the container is running

If you have any more suggestions, please let me know. Thanks a lot again for your help!

[jasonacox]

Since you originally used sudo the file permission are likely in bad shape. I suggest removing the Powerwall-Dashboard directory and reinstall from scratch now that you have your local user in the docker group:

# remove the old install
rm -fr Powerwall-Dashboard

# git clone or pull down the repository again
git clone https://github.com/jasonacox/Powerwall-Dashboard.git

# run setup
cd Powerwall-Dashboard
./setup.sh

Thanks for the feedback on this. Hopefully we can figure this out for anyone else using a Synology NAS to host their dashboard. :)

[jaydkay]

Dear @jasonacox,

after a few more tries and errors, I guess I finally got it working! Here are the steps that were required - at least in my case:

Log in via SSH to Synology DSM. Create a usergroup docker: sudo synogroup --add docker

Change ownership of docker to this group: sudo chown root:docker /var/run/docker.sock

Make the user a member of the newly created group: sudo synogroup --member docker <user>

Finally, edit file powerwall.yml and change the line

user: "1000:1000"

in the section grafana to the uid of the user on the Synology. I got that by running id while logged in via SSH.

After this, I was able to issue ./setup.sh, finish setting it up and all containers are starting up.

So far it seems to be working, although I didn't do much more beyond starting it up, yet.

[jasonacox]

Awesome!! Nice job @jaydkay ! Thanks for posting the helpful instructions. I'll reference that in the troubleshooting tips.

[BuongiornoTexas]

Hi @jasonacox, @jaydkay,

I've just run into either the same problem or a closely related one while setting up Powerwall-Dashboard under a rootless docker. I was getting the same messages as Jochen GF_PATHS_DATA='/var/lib/grafana' is not writable. and mkdir: cannot create directory '/var/lib/grafana/plugins': Permission denied.

It took a while for me to track it down - as far as I can tell for the rootless version, it comes down to the fact that grafana runs with user:group ids of 472:0 in the rootless container, and this results in name space mapping issues when creating/writing files on the host machine (this explanation isn't quite right, but is probably close enough to the truth allow resolution of the problem).

I found two possible solutions:

1. The first solution is probably safe in a rootless system, but I really wouldn't recommend it as a) only probably safe and b) if you ever switch to a normal docker, it becomes very not safe. It's simple - run the rootless container as "root" by changing the grafana user line in powerwall.yml to user: "0:0". This works as docker rootless maps root to the user running the docker installation. Again, I would really not recommend this approach - I'm just including it as someone else may find this step useful in their own troubleshooting.

2. The second solution is somewhat cleaner, although I'm pretty sure this is still just a work around for a limitation on name space mapping for rootless docker.

• Change the grafana user line in powerwall.yml to user: "472:0" (you may be able to delete it altogether - haven't tried this).

• Get the subuid entry for the user xxxx running docker from /etc/subuid on the host machine. In my case it is xxxx:100000:65536 - from this, we can calculate the uid mapping of the container uid grafana user to a subuser user id on the host machine as: first number in the subuid (100000) + 471 (0 offset of the grafana uid) - so for me this is 100471.

• Change the ownership of the grafana directory in Powerwall-Dashboard on the host machine to this subuid:

  sudo chown 100471 ./grafana (Fails without sudo, but this is a subuid in the xxxx user name space, so has the same privileges as xxxx - effectively no change in permissions)

Running ls -ld ./grafana should result in:

drwxr-xr-x 6 100471 xxxx 4096 Sep 21 10:54 ./grafana (where xxxx is the username running docker - same as the subuid file).

• Kill the current container and restart.

docker stop grafana
docker rm grafana
docker compose -f powerwall.yml up -d

I hope this saves someone else a day or so of digging in the future. Picking up docker this way was definitely a baptism by fire.

Finally, @jasonacox, thanks again for the time you have put into a great monitoring tool!

[jasonacox]

Thanks for this great information, @BuongiornoTexas ! Is this on a Synology NAS or something else?

[BuongiornoTexas]

Is this on a Synology NAS or something else?

Raspberry Pi 4B - so quite the journey getting docker running at all even without dealing with getting it rootless and the grafana permissions issue.

Dropped in here as the error is identical to Jochen's and almost certainly related, but if you want to include it as a tip, I'd put it in a section on grafana permissions and/or rootless docker.

[jasonacox]

Thanks @BuongiornoTexas !!!- I'm running on a Raspberry Pi as well (Raspbian GNU/Linux 11 bullseye). Can you cat /etc/os-release for anyone following this?

For my setup, I had to run something like this:

# Install docker
curl -fsSL https://get.docker.com/ -o get-docker.sh
sudo sh get-docker.sh
sudo apt install -y docker-compose

# Add your user to docker group
sudo usermod -aG docker $USER

# Set docker to start on boot
sudo systemctl enable docker.service
sudo systemctl enable containerd.service

# install docker-compose
sudo pip3 install docker-compose

I wonder if I can add notes or setup.sh instructions to help.

[BuongiornoTexas]

Like you, I'm on bullseye (64 bit).

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian

I think the difference between our installs is that I'm running docker as a non-privileged (rootless) user. So I don't have a docker group at all.

The process for setup is a bit messier, but works pretty well. It's detailed at: https://docs.docker.com/engine/security/rootless/

The steps I took were:

• Use apt -qq list yyy to check for the following pre-req packages, and if not, installed them.

  uidmap
  dbus-user-session
  fuse-overlayfs
  slirp4netns (0.4 or later)
  

• Run the Without packages install script curl -fsSL https://get.docker.com/rootless | sh. At which point docker engine is installed.

• Added the two environment lines at the end of the script to my .bashrc:

 export PATH=/home/username/bin:$PATH
 export DOCKER_HOST=unix:///run/user/1000/docker.sock

• Installed v2 of docker compose manually per the process at https://docs.docker.com/compose/install/linux/ (there are binaries for arm64 on github now, aarch64 file identifier from uname -m!) :

  DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
  mkdir -p $DOCKER_CONFIG/cli-plugins
  curl -SL https://github.com/docker/compose/releases/download/v2.11.0/docker-compose-linux-aarch64 -o $DOCKER_CONFIG/cli-plugins/docker-compose
  

I think everything was running after this.

[jasonacox]

Thanks @BuongiornoTexas ! This is a great 'get starting' for anyone wanting to run docker as a non-privileged (rootless) user. I'll capture this in the troubleshooting section for anyone else wanting to do the same.

Thank you!

[BuongiornoTexas]

No probs. One more step I forgot - I created a file called docker-compose in ~/bin containing:

docker compose "$@"

And ran chown +x ~/bin/docker-compose. An alias that emulates v1 with v2.

Alternatively, users can just run docker compose manually.

[metaton8086]

I had the same permission error on a Pi4 running Ubuntu. The only way I could fix it was to change the grafana user line in powerwall.yml to user: "0:0", as noted by @BuongiornoTexas.

[BuongiornoTexas]

The only way I could fix it was to change the grafana user line in powerwall.yml to user: "0:0",

Is this on a standard docker install? If so, something doesn't sound quite right there - the problem I had was because docker was missing some of the privileges from a standard install (I think you have effectively given your grafana instance root privileges in the docker container, which shouldn't be needed at all).

One quick thought is your user id 1000? If not, that would break the setup.

[metaton8086]

Yes, it is standard. My user id is 1001. I change powerwall.yml to user: "1001", killed the current container and reinstalled. It is now working as expected.

[BuongiornoTexas]

I change powerwall.yml to user: "1001", killed the current container and reinstalled. It is now working as expected.

Excellent. You may want to make the powerwall.yml entry 1001:1001 so that it also has the correct group permissions.

[marcbaier]

Hi all - i am also trying to install this on my Synology NAS. This is what I have done so far:

• I created a shared folder called Powerwall
• Followed the instructions above:
• Log in via SSH to Synology DSM. Create a usergroup docker: sudo synogroup --add docker

Change ownership of docker to this group: sudo chown root:docker /var/run/docker.sock

Make the user a member of the newly created group: sudo synogroup --member docker

• installed from github: git clone https://github.com/jasonacox/Powerwall-Dashboard.gi

• ran the ./setup.sh from the directory Powerwall-Dashboard:

• Backupadmin@BackupStation:/volume1/powerwall/Powerwall-Dashboard$ ./setup.sh

Now I am getting this error back:

Powerwall Dashboard (v2.8.0) - SETUP

ERROR: docker is not available or not runnning. This script requires docker, please install and try again.

Docker is running on the Synology. The user "BackupAdmin" is a member of the "docker" group:

Is there anything else I can check to figure out what the issue is ?

Thanks so much for your help.

[jasonacox]

HI @marcbaier - I hope our community can chime in on this. I guess I need to add a "Synology NAS" to my test suite (currently MacOS, Ubuntu Linux, Win 11 WSL and Raspberry Pi). 😄 What model are you using?

ERROR: docker is not available or not runnning.

What do you get when you run docker ps or docker info?

[marcbaier]

I get this when running docker ps: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": dial unix /var/run/docker.sock: connect: permission denied

docker info: Client: Context: default Debug Mode: false

Server: ERROR: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": dial unix /var/run/docker.sock: connect: permission denied

[marcbaier]

I am trying to run this on a DS1513+ running DSM7.1.1. I could also try it on a newer DS1019+ but I would presume that I would get the same issue there as well.

[jasonacox]

Thanks @marcbaier - that helps a lot! Your local user does not have permission to run docker. Check the permissions of the socket:

ls -l /var/run/docker.sock

Also, just to confirm, you did run these?

sudo synogroup --add docker
sudo chown root:docker /var/run/docker.sock
sudo synogroup --member docker $USER

DS1513+ running DSM7.1.1.

Thanks! I'll see if I can find one.

[marcbaier]

This is what i get back when I run the docker.sock command above: srw-rw---- 1 root docker 0 Feb 4 09:02 /var/run/docker.sock

[jasonacox]

Interesting. That should work if your local user is part of the docker group. I have seen it "not take" until I log out and log in again or reboot the host (on Linux, not a Synology). You can also try this (a common 'fix' mentioned on the docker helps):

sudo chmod 666 /var/run/docker.sock

[marcbaier]

That worked ! Now I was able to install the whole ./setup.sh. I am now stuck at the grafana setup. The IP Address of my Synology Nas is 192.168.86.58 and I wanted to use the browser on my iMac to access: http://192.168.86.58:9000, but I am getting an "this site can't be reached" error.

[marcbaier]

looks like grafana is not running. I went and looked into the docker install and this is what I see:

[marcbaier]

When I try to restart grafana I am getting these errors, as I think others on here also got: GF_PATHS_DATA='/var/lib/grafana' is not writable. You may have issues with file permissions, more information here: http://docs.grafana.org/ mkdir: cannot create directory '/var/lib/grafana/plugins': Permission denied

[jaydkay]

Did you follow the hints in my post at the beginning of this thread? https://github.com/jasonacox/Powerwall-Dashboard/issues/22#issuecomment-1110099800

This part in particular:

Finally, edit file powerwall.yml and change the line

user: "1000:1000"

in the section grafana to the uid of the user on the Synology. I got that by running id while logged in via SSH.

After this, I was able to issue ./setup.sh, finish setting it up and all containers are starting up.

You probably need to stop, delete and re-create the containers after doing this change to take effect.

[marcbaier]

I did follow your instructions and replaced the uid with mine. however, I did not delete and re-create the containers. How would I do that ? directly in docker and then run the ./setup.sh again ?

[marcbaier]

ok so i did the following:

• stop and delete all the containers
• check the powerwall.yml file and change the value to 1029
• (uid=1029(Backupadmin) gid=100(users) groups=100(users),101(administrators),65536(docker))
• executed ./setup.sh again
• in the docker app on the synology I still see the grafana and telegraf containers restarting.... so i guess nothing has changed.

[marcbaier]

ah - think i found the issue. I did not correct the user: value in the grafana section... stupid me. Now that I've done that, the containers are not restarting anymore and I am not getting an error message either. thanks so much