Unable to get Proxmox list widget to work

[ImChet]

Discussed in https://github.com/Lissy93/dashy/discussions/1315

^{Originally posted by ImChet September 14, 2023} Error:

An error occurred, see the logs for more info.

Unable to fetch data

Here is the code block for my proxmox-list:

sections:
  - name: Server Management
    icon: ''
    items:
      - title: xxxxxxxxxx
        description: ''
        icon: hl-proxmox
        url: xxxxxxxxxx
        target: newtab
        id: xxxxxxxx
      - title: xxxxxxxxx
        icon: hl-proxmox
        url: xxxxxxxxxxx
        target: newtab
        id: xxxxxxxx
      - title: xxxxxxxx
        icon: hl-truenas
        url: xxxxxxxxxx
        target: newtab
        id: xxxxxxx
    displayData:
      sortBy: default
      rows: 1
      cols: 1
      collapsed: false
      hideForGuests: true
    widgets:
    - type: proxmox-lists
      useProxy: true
      options:
        cluster_url: https://x.x.x.x:xxxx
        user_name: root@pam
        token_name: dashy
        token_uuid: xxxxxxxxxxxxxxxxxxxxxxxx
```</div>

[liss-bot]

If you're enjoying Dashy, consider dropping us a ⭐
_{🤖 I'm a bot, and this message was automated}

[ilium007]

Searched and found this issue before creating the same. No VM data is listed. If I put an intentionally wrong value in for node the widget throws an error so I can only assume that with the correct value and no error on screen it is gettign data back from the API but not parsing it correctly.

[ilium007]

This is the JSON from the constructed API call in the browser:

{"data":[{"mem":7728077790,"diskread":0,"maxmem":8589934592,"cpu":0.0041496411882829,"status":"running","diskwrite":0,"pid":2374150,"disk":0,"name":"omv01","netin":0,"vmid":103,"cpus":2,"netout":0,"uptime":22400,"maxdisk":34359738368},{"netin":0,"disk":0,"name":"debian01","diskwrite":0,"status":"running","pid":791159,"cpu":0.0117877760698244,"maxdisk":34359738368,"netout":0,"cpus":4,"uptime":364263,"vmid":101,"mem":5502137380,"maxmem":8589934592,"diskread":0},{"mem":0,"maxmem":4294967296,"diskread":0,"name":"windows11","disk":0,"netin":0,"cpu":0,"status":"stopped","diskwrite":0,"uptime":0,"cpus":2,"netout":0,"maxdisk":34359738368,"vmid":102}]}

[nortuzar]

Same problem here, fresh install of dashy and couldnt make this widget work. Anyway im enjoying dashy very much

[wnaude]

Hi, I also struggled at first, but then I changed the permissions of the API token in the datacenter web gui. Unticked the privilege separation., then the "show permissions" section got populated where it was empty before. I am a noob, so I must still go rtfm on the detail of what it means and what the risks are. Just dropping the note incase it might help someone.

[lumper5]

No luck here getting this widget to connect. I had hopes that the "privilege separation" would work (it was the solution to a similar problem in homepage and Heimdall). I wonder what the trick is to get this to authenticate?

[Lissy93]

I don't think I'm going to be much help here, as I didn't build the Proxmox widget, and I don't have a Proxmox instance to test with. But what does the error message in the browser logs say? (here's how to check)

[CrazyWolf13]

@alayham Could you take over here?

[alayham]

Gladly. I have a video that shows how to install a development environment for Dashy and install the widget to fix a bug in it. I also added documentation to handle the common errors I faced with the widget. please look at my video here: https://www.youtube.com/watch?v=6H9j2OFr69U&list=PLZLlJrtYGHany2Mvz-Q59SPeWp88iESU1&index=8

The documentation for the errors is in the repo, but not visible on the documentation site. https://github.com/Lissy93/dashy/blob/master/docs/widgets.md#proxmox-lists

Suggestions for users:

1. Make sure you use Dashy in production mode.
2. Make sure PVE certificate is trusted by nodejs on the host that runs Dashy.

[alayham]

@ilium007 @ImChet @nortuzar @wnaude Were you able to resolve the issue with the information above?

[CrazyWolf13]

Hi @alayham Sadly I could not resolve the issue with the info provided above and in the docs.

Sadly the part how to extract the CA is compledtely missing, did you forget that?!

By manually clicking "view" on the root ca I could see the CA key, I copy pasted it to a file and did the export command with my path, sadly even after a reboot and rebuild of dashy I'm still getting the error code: axios request failed with status code 500

Is it really neccessary for everyone to extract that certificate?

[alayham]

Yes, It is necessary unless the Proxmox VE uses a trusted certificate. The reason is that Nodejs has a list of trusted certificates authority compiled inside the binary, and will only accept them for security. We have the ability to add to them using the following process: 1- Visit the certificates page of your PVE, you should see at lease two certificates, one ends with -ca.pem, click on it. 2- click on View Certificate 3- click on Raw certificate

4- Copy everything in the text area below View Certificate and paste it into a file called pve-root-ca.pem 5- Move the file to your Dashy host. Either: 6- start dashy with NODE_EXTRA_CA_CERTS=<path of the copied file> yarn start or 6- A a line to the file .bash_profile in the home directory of the user that run dashy containing: export NODE_EXTRA_CA_CERTS=<path of the copied file> and restart the shell session for the export to be active, then start Dashy using yarn start You can watch my video about setting up Dashy using Ansible, or read this article about the subject.

[CrazyWolf13]

@alayham Thanks for the guide! I set everything up according to your guide above, but I cannot seem to get it working.

Does Dashy have to have a certificate and run with https /ssl ? Because currently mine runs in http mode.

Also is it okay to use the IP instead of a domain name for the cluster url?

Here a little screenshot.

My config:

  - type: proxmox-lists
    useProxy: true
    options:
      cluster_url: https://10.50.20.10:8006
      user_name: root@pam
      token_name: dashy
      token_uuid: {token}

Thanks.-

[alayham]

Timeout is usually a firewall issue. Can you access the cluster from the Dashy host using curl? try curl https://10.50.20.10:8006 from the dashy host. Install curl if missing.

[CrazyWolf13]

Timeout is usually a firewall issue. Can you access the cluster from the Dashy host using curl? try curl https://10.50.20.10:8006 from the dashy host. Install curl if missing.

Seems to work just fine when pinging and curl works if I add -k (think that was to ignore self signed cert) curl -k https://10.50.20.10:8006

[alayham]

Troubleshooting integrations is tricky, and to help you I need to identify the root cause of the issue. Let's try first to identify whether the cause is in the Dashy host, the PVE host, or the network.

Since you need to use -k, this means that the PVE CA certificate is not trusted on the Dashy host. Can you try to trust it? the steps to trust a ca certificate depend on the operating system and the browser. I can advise you if needed but I need to know the OS on the dashy host.

After trusting the PVE root certificate, we can do the following on the dashy host, not on your machine. It is important to run the command from the Dashy host using the Dashy user, not root.

replace YOURAPITOKEN with your token in the following command after dashy= and before the single quote, then run run the command on the dashy host and put the response here.

curl  -H 'Authorization: PVEAPIToken=root@pam!dashy=YOURAPITOKEN' https://10.50.20.10:8006/api2/json/nodes/

[CrazyWolf13]

replace YOURAPITOKEN with your token in the following command after dashy= and before the single quote, then run run the command on the dashy host and put the response here.

Hi, Thank you!

Yes as stated above, dashy is not running in SSL mode and Proxmox currently only has it's self signed cert.

Host of Dashy is a raspberrypi running raspbian.

Here the result which seems that you guess with the cert not being trusted is true;

pi@raspberrypi:~ $ curl  -H 'Authorization: PVEAPIToken=root@pam!dashy=MYAPITOKEN' https://10.50.20.10:8006/api2/json/nodes/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Again when using-k switch it seems to output the data, is there any way to get such a -k option in the widget as well? Maybe it would also be worth to include all those steps you describe in the docs, as I'm pretty sure others will run into this as well :)

[alayham]

My first suggestion is to trust the PVE root certificate on the PI. Here are some ideas from StackOverflow: https://raspberrypi.stackexchange.com/questions/76419/entrusted-certificates-installation

how do you start Dashy? can you paste the command to start Dashy here?

We can update the documentation after we solve your issue.

[CrazyWolf13]

My first suggestion is to trust the PVE root certificate on the PI. Here are some ideas from StackOverflow: https://raspberrypi.stackexchange.com/questions/76419/entrusted-certificates-installation

how do you start Dashy? can you paste the command to start Dashy here?

We can update the documentation after we solve your issue.

I use a "standard" Docker installation, so I added the ca the following way: Create the ca file on host, map it with a bind volume to the container, verify it's there.

Then add the ENV variable: Rebuild and restart dashy, notice the wdiget is still not loading.

I unsure where I have to add this cert, in dashy, in the container, in docker or in the host? I guess the container? Found this thread: https://unix.stackexchange.com/questions/464484/install-self-signed-certificate-to-alpine-linux#:~:text=%2Fetc%2Fssl%2Fcerts%2Fca%2Dcertificates.

So I edited the bind mount to point to /usr/local/share/ca-certificates/proxmox_ca.pem and removed the node env variable. The file /etc/ssl/certs/ca-certificates.crt now contains the cert, however dashy widget is still not working.

[alayham]

In this case your Dashy host is not the PI, it is the docker engine. There are many things that can go wrong with such setup, and it is not easy to troubleshoot. I will do my best.

1- Can you open a shell to the Dashy container and try to curl the PVE api? 2- Inside that shell, can you do cat /app/public/proxmox_ca.pem to make sure the cert is mounted, and compare it to the original cert on the PVE. 3- In that shell, can you access the outside world, like curl GitHub or the Dashy site?

You might need to install curl in the docker container.

[CrazyWolf13]

In this case your Dashy host is not the PI, it is the docker engine. There are many things that can go wrong with such setup, and it is not easy to troubleshoot. I will do my best.

1- Can you open a shell to the Dashy container and try to curl the PVE api? 2- Inside that shell, can you do cat /app/public/proxmox_ca.pem to make sure the cert is mounted, and compare it to the original cert on the PVE. 3- In that shell, can you access the outside world, like curl GitHub or the Dashy site?

You might need to install curl in the docker container.

1- Yes, but it still gives me the same error:

curl: (60) SSL: no alternative certificate subject name matches target host name '10.50.20.10'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

2- Yes as stated above, the CA is there and contains the same cert, it's even in the /etc/ssl/certs/ca-certificates.crt included

3- Yes I can access outside world from dashy container

[alayham]

I think the docker image is built using the Dockerfile in the root directory of the repo https://github.com/Lissy93/dashy/blob/master/Dockerfile It seems to build Dashy using node16, and run it using node20. I am not sure how that will impact the code.

It is not enough to put the cert in the container, it needs to be trusted, and you need to trust it every time you start the container unless you change the image. Besides, It seems that Dashy uses a base image of Nodejs that is based on Alpine Linux. On Alpine trusting the certificate is different from Raspian. Here is a StackOverflow link: https://superuser.com/questions/1430089/how-to-add-a-self-signed-ssl-certificate-to-linux-ubuntu-alpine-trust-store By the bottom of the answer you see how to trust a cert in Alpine. Notice that the cert needs to be in PEM format, and to have a .crt extension.

After you trust the CA certificate, can you curl the PVE API again from inside the container?

Also, Can you execute the following command inside the container: set | grep "/app/public/proxmox_ca.pem" and paste the output here?

[CrazyWolf13]

I don't really think the other engine is a problem, as I could also not get it to work on a debian build from source machine.

So I have now two bind mounts, one to /usr/local/share/ca-certificates/proxmox.crt, which is the same ca file just with crt extension, the other goes inside the /app/public

installing update-ca-certificates, installs fine but I do not get any output when run.

Do I have to specify the ip of the cert anywhere? as curl states t could not find an external cert for that ip, do I have to somewhere write that ip?

Here the output:

/app # set | grep "/app/public/proxmox_ca.pem"
NODE_EXTRA_CA_CERTS='/app/public/proxmox_ca.pem' 

[alayham]

installing update-ca-certificates, installs fine but I do not get any output when run.

You should see 1 cert added. If you do not see it then the cert is not added. Try deleting the PVE CA cert, running update-ca-certificates, then add the PVE CA cert, and update again. If still you do not see a log of 1 added, them maybe the cert file is corrupted, make sure it is identical to the original. Depending on the system, you might need to add a new line at the end of the file. If you are using windows to copy the file, you need to set the newline character to the Linux new line, not the Windows new line.

This StackOverflow answer shows how to list installed certificates on Alpine. https://unix.stackexchange.com/questions/637545/listing-installed-certificates-in-alpine

You need to make sure the cert is installed correctly and to test to curl the PVE API from the container shell without issues. Not because that solves the node problem but because it is easier to troubleshoot certificate issues this way.

Once we have a certificate that we know to work, we can see why NodeJs does not accept it.

I could also not get it to work on a debian build from source machine

I can help you a lot more in this setup, and my video and article are exactly about this scenario. In the video I show how the cert did not work, and how I fixed it.

[CrazyWolf13]

Thanks for the help, but sadly no matter what I do I cannot get update-ca-certificates to give me any output.

Installed exactly as shown in the url but I do not get an output. Also i can't really find anything about the no output part on the web.

Also is there really no way to get an "allow insecure" flag?

currently this is an extremely complicated with mounting binding and installing things inside a container which will be gone on reboot etc.

[alayham]

Also is there really no way to get an "allow insecure" flag?

Nodejs has its trusted certificates compiled inside the binary. I don't know if we can configure it to ignore certificate trust, and I would not advise you to do that.

Thanks for the help, but sadly no matter what I do I cannot get update-ca-certificates to give me any output.

Most probably the problem is in the CA certificate file. Let me know if you want to keep working on this until you solve your problem.

[CrazyWolf13]

Also is there really no way to get an "allow insecure" flag?

Nodejs has its trusted certificates compiled inside the binary. I don't know if we can configure it to ignore certificate trust, and I would not advise you to do that.

Thanks for the help, but sadly no matter what I do I cannot get update-ca-certificates to give me any output.

Most probably the problem is in the CA certificate file. Let me know if you want to keep working on this until you solve your problem.

I'd love to get it working, as the widget looks awesome !

Well if there would be an issue with the file shouldn't there still at least be some output like 0 added or so?

[alayham]

My approach to this issue is the following: 1- Confirm that the certificate works, and for this I wanted curl to recognize it because it is a lot easier to troubleshoot. Since we can not confirm that is working on the system, let's try to add it manually to curl. Here is the command, replace YOURCACERTFILE with the path of the PEM cert, and YOURAPITOKEN with the API token:

curl --cacert YOURCACERTFILE  -H 'Authorization: PVEAPIToken=root@pam!dashy=YOURAPITOKEN' https://10.50.20.10:8006/api2/json/nodes/ 

When we know the CA cert works, we can see why NodeJS does not recognize it.

[CrazyWolf13]

Sadly still fails. The more I read this, the more I think it has to do with the name, it specifically says it has not found any certificate that matched 10.50.20.10, however neither my cert name nor inside the cert there is the ip specified, where would I have to specify the IP?

curl: (60) SSL: no alternative certificate subject name matches target host name '10.50.20.10'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

[alayham]

Aha, This is very good news. Now we know the error. This means the CA certificate works, but you are calling the server with a domain name that does not exist in the Subject Alternative Names field of your server certificate. I do not have this issue because I use my own certificate authority and I run my own DNS service in my devlab. Here is an example of what I have in my server:

You can see this if you inspect the certificate, and you can see it also in the PVE web admin when you click on the node, and select certificates:

To solve your issue, you need to: 1- Determine the Subject Alternative names of your server's certificate. You can do that by either inspecting the server certificate, or checking the certificates page in your PVE. 2. Make sure those DNS names resolve to the server IP in your network: You can try passing those names to the nslookup command, and the command should return the server IP. 3. Make sure those DNS names resolve to the server IP inside the Docker container. Here you might need to replace the IP with a name that resolves in the previous curl command. 4. Update your widget in the Dashy conf to use any name in the SAN field that resolves to the server IP, rebuild Dashy, and let me know if you get an error.