grafana-image-renderer
grafana-image-renderer copied to clipboard
grafana
Feature Request: Allow setting CA file or CA path for the renderer
Referring to https://github.com/grafana/grafana/issues/19842:
Currently there does not seem to be an option to make the renderer work with custom certificates on the Grafana server except configuring the plugin to ignore HTTPS errors altogether (e.g. by setting GF_RENDERER_PLUGIN_IGNORE_HTTPS_ERRORS=true in Grafana's environment at startup).
A much cleaner option would be a configuration that allows setting a custom CA for the server, preferably by pointing it to a CA directory, e.g. /etc/pki/tls/certs for RHELoid distributions, or to a specific CA file containing one or more trusted certificates, by a similar environment variable or the like.
Please try doing something along these lines, e.g. import a self-signed CA ssl certificate into the linux version of google chrome. Still think if you have a self signed cert for localhost it will never work in regular Chrome/Chromium, but I may be wrong.
I actually tried something like that in one of my first attempts to fix the problem, see https://github.com/grafana/grafana/issues/19842#issuecomment-542706966.
Following the guidelines in your link, I slightly modfied my second approach (the specific certificate for the Grafana user) to grant even more trust to the certificates, system-wide as well as user specific, in old and new nssdb format, with the root cert and the sub-ca cert just to be really sure I didn't miss anything:
certutil -d sql:/etc/pki/nssdb -A -n hbr-dev-subca01 -t TCP,TCP,TCP -i /etc/pki/tls/certs/hbr-dev-subca01.crt.pem
certutil -d sql:/etc/pki/nssdb -A -n hbr-dev-root-ca -t TCP,TCP,TCP -i /etc/pki/tls/certs/hbr-dev-root-ca.crt.pem
certutil -d /etc/pki/nssdb -A -n hbr-dev-subca01 -t TCP,TCP,TCP -i /etc/pki/tls/certs/hbr-dev-subca01.crt.pem
certutil -d /etc/pki/nssdb -A -n hbr-dev-root-ca -t TCP,TCP,TCP -i /etc/pki/tls/certs/hbr-dev-root-ca.crt.pem
certutil -d sql:/usr/share/grafana/.pki/nssdb -A -n hbr-dev-subca01 -t TCP,TCP,TCP -i /etc/pki/tls/certs/hbr-dev-subca01.crt.pem
certutil -d sql:/usr/share/grafana/.pki/nssdb -A -n hbr-dev-root-ca -t TCP,TCP,TCP -i /etc/pki/tls/certs/hbr-dev-root-ca.crt.pem
certutil -d /usr/share/grafana/.pki/nssdb -A -n hbr-dev-subca01 -t TCP,TCP,TCP -i /etc/pki/tls/certs/hbr-dev-subca01.crt.pem
certutil -d /usr/share/grafana/.pki/nssdb -A -n hbr-dev-root-ca -t TCP,TCP,TCP -i /etc/pki/tls/certs/hbr-dev-root-ca.crt.pem
Checking the success of the operation reveals the following status:
[root@master1 grafana]# certutil -d sql:/etc/pki/nssdb -L
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
hbr-dev-root-ca CT,C,C
hbr-dev-subca01 CT,C,C
[root@master1 grafana]# certutil -d /etc/pki/nssdb -L
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
hbr-dev-subca01 CT,C,C
hbr-dev-root-ca CT,C,C
[root@master1 grafana]# certutil -d sql:/usr/share/grafana/.pki/nssdb -L
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
hbr-dev-root-ca CT,C,C
hbr-dev-subca01 CT,C,C
[root@master1 grafana]# certutil -d /usr/share/grafana/.pki/nssdb -L
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
hbr-dev-root-ca CT,C,C
hbr-dev-subca01 CT,C,C
So both relevant CA certificates are in all potential nssdb instances, with all possible trust levels (the link you sent is effectively overdoing it a bit, resp. trying to apply trust levels that don't work, but never mind :-)). If any of these settings were efficient it should work now.
Ah yes, just to make sure the certificates themselves work, are the same the server uses, and are properly installed in the NSS databases:
[root@master1 grafana]# openssl s_client -connect 192.168.106.101:3443 -showcerts
CONNECTED(00000003)
depth=2 C = DE, ST = Hessen, [...], OU = HBR DEV Root CA
verify return:1
depth=1 C = DE, ST = Hessen, [...], OU = HBR DEV SubCA 01
verify return:1
depth=0 C = DE, ST = Hessen, [...], CN = grafana.xxx.xxx
verify return:1
---
Certificate chain
0 s:/C=DE/ST=Hessen/[...]/CN=grafana.xxx.xxx
i:/C=DE/ST=Hessen/[...]/OU=HBR DEV SubCA 01
-----BEGIN CERTIFICATE-----
MIIG2jCCBMKgAwIBAgIUMgrJGgTdxzIrMn62yD26YHlYp+UwDQYJKoZIhvcNAQEL
BQAwgYQxCzAJBgNVBAYTAkRFMQ8wDQYDVQQIDAZIZXNzZW4xFDASBgNVBAcMC0Jh
[...]
OVMYrxlAqDuKauqLPM9dvS2xthaYfE30NU80MKQUvnNGgiZJ9nRqIfwlQp45wotp
hUjwhgBW/GIRiyvBU30LgwZTrf3NA/SdqxweLT5cGQzeOVnBQ34cyEo+sTDpKxmN
yPgk6xGx8q7fRdz7FEKOYSNdauWhnwNTZZrHThC+
-----END CERTIFICATE-----
1 s:/C=DE/ST=Hessen/[...]/OU=HBR DEV SubCA 01
i:/C=DE/ST=Hessen/[...]/OU=HBR DEV Root CA
-----BEGIN CERTIFICATE-----
MIIFujCCA6KgAwIBAgIUFrsUndqtaf4SUA42htpLHqurylYwDQYJKoZIhvcNAQEL
BQAwgYMxCzAJBgNVBAYTAkRFMQ8wDQYDVQQIDAZIZXNzZW4xFDASBgNVBAcMC0Jh
[...]
0aqc+fTvsK7EBlTBYyCrNNrd68Jd30KaMAw3Tw9CP5LyLdXaaoK7PH08kvb2JbSg
7yHQq+UfA1WNyTRTn8yvvysZEyLKK8CRtUNSU323
-----END CERTIFICATE-----
---
Server certificate
subject=/C=DE/ST=Hessen/[...]/CN=grafana.xxx.xxx
issuer=/C=DE/ST=Hessen/[...]/OU=HBR DEV SubCA 01
---
No client certificate CA names sent
---
SSL handshake has read 3523 bytes and written 887 bytes
---
[...]
Verify return code: 0 (ok)
---
[root@master1 ~]# openssl x509 -noout -fingerprint -hash -text | egrep -A1 '(Subject Alter|Subject:|Fingerprint)'
-----BEGIN CERTIFICATE-----
MIIG2jCCBMKgAwIBAgIUMgrJGgTdxzIrMn62yD26YHlYp+UwDQYJKoZIhvcNAQEL
BQAwgYQxCzAJBgNVBAYTAkRFMQ8wDQYDVQQIDAZIZXNzZW4xFDASBgNVBAcMC0Jh
[...]
hUjwhgBW/GIRiyvBU30LgwZTrf3NA/SdqxweLT5cGQzeOVnBQ34cyEo+sTDpKxmN
yPgk6xGx8q7fRdz7FEKOYSNdauWhnwNTZZrHThC+
-----END CERTIFICATE-----
SHA1 Fingerprint=46:D9:EE:7F:A6:2C:85:D8:EA:48:BF:D5:39:9E:5C:74:1E:C6:D7:F2
084a673f
--
Subject: C=DE, ST=Hessen, [...], CN=grafana.xxx.xxx
Subject Public Key Info:
--
X509v3 Subject Alternative Name:
DNS:master1.xxx.xxx, IP Address:192.168.106.101, DNS:master2.xxx.xxx, IP Address:192.168.106.102, IP Address:192.168.106.52, DNS:grafana.xxx.xxx
[root@master1 ~]# openssl x509 -noout -fingerprint -hash -text | egrep -A1 '(Subject Alter|Subject:|Fingerprint)'
-----BEGIN CERTIFICATE-----
MIIFujCCA6KgAwIBAgIUFrsUndqtaf4SUA42htpLHqurylYwDQYJKoZIhvcNAQEL
BQAwgYMxCzAJBgNVBAYTAkRFMQ8wDQYDVQQIDAZIZXNzZW4xFDASBgNVBAcMC0Jh
[...]
0aqc+fTvsK7EBlTBYyCrNNrd68Jd30KaMAw3Tw9CP5LyLdXaaoK7PH08kvb2JbSg
7yHQq+UfA1WNyTRTn8yvvysZEyLKK8CRtUNSU323
-----END CERTIFICATE-----
SHA1 Fingerprint=16:33:EC:24:27:FB:AB:86:56:A0:91:A2:62:97:8C:82:66:8E:D7:63
95023bec
--
Subject: C=DE, ST=Hessen, [...], OU=HBR DEV SubCA 01
Subject Public Key Info:
[root@master1 ~]# certutil -d /etc/pki/nssdb -L -n hbr-dev-root-ca | grep -A1 'Fingerprint (SHA1)'
Fingerprint (SHA1):
E1:1B:7A:6A:88:1F:78:5C:CB:E4:30:D5:A5:6C:0D:5E:A1:DC:CE:F2
[root@master1 ~]# certutil -d /etc/pki/nssdb -L -n hbr-dev-subca01 | grep -A1 'Fingerprint (SHA1)'
Fingerprint (SHA1):
16:33:EC:24:27:FB:AB:86:56:A0:91:A2:62:97:8C:82:66:8E:D7:63
I hope I didn't miss anything.
So now I restarted the two Grafana instances, and tried again to use the renderer:
t=2019-12-04T12:39:22+0000 lvl=eror msg="Render request failed" logger=rendering error=map[] url="https://192.168.106.101:3443/d-solo/zxDJxNaZk/graphite-metrics?orgId=1&refresh=1m&from=1575438321300&to=1575459921300&var-Host=master1&panelId=4&width=1000&height=500&tz=Europe%2FBerlin&render=1" timestamp=0001-01-01T00:00:00.000Z
t=2019-12-04T12:39:22+0000 lvl=eror msg="Rendering failed." logger=context userId=1 orgId=1 uname=admin error="Rendering failed: Error: net::ERR_CERT_AUTHORITY_INVALID at https://192.168.106.101:3443/d-solo/zxDJxNaZk/graphite-metrics?orgId=1&refresh=1m&from=1575438321300&to=1575459921300&var-Host=master1&panelId=4&width=1000&height=500&tz=Europe%2FBerlin&render=1"
t=2019-12-04T12:39:22+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/render/d-solo/zxDJxNaZk/graphite-metrics status=500 remote_addr=192.168.106.101 time_ms=310 size=1722 referer="https://grafana.xxx-xxx/d/zxDJxNaZk/graphite-metrics?orgId=1&refresh=1m"
I think we can be fairly certain that this is not an issue of host name vs. certificate subject/SAN, as the error message explicitly says ERR_CERT_AUTHORITY_INVALID, so the renderer does not seem to pick up the custom CA certificates from anywhere I put them to. Which also makes it fairly sure to assume that it does not use the NSS library, at least not in a standard way.
I must admit I'm out of my wits at the moment.
I must admit I'm out of my wits at the moment.
Not quite, as it turned out. It's a permissions problem as it seems. I'll work that out and update this issue as soon as I can come up with precise instructions.
So, i think I got it fixed.
The following actions provided the minimal fix for my setup that still works:
[root@master1 ~]# [ -d ~grafana/.pki/nssdb ] || mkdir -p /usr/share/grafana/.pki/nssdb
[root@master1 ~]# certutil -d sql:/usr/share/grafana/.pki/nssdb -A -n hbr-dev-root-ca -t C -i /etc/pki/tls/certs/hbr-dev-root-ca.crt.pem
[root@master1 ~]# chown -R grafana: /usr/share/grafana/.pki/nssdb
Root certificate alias and file name obviously need to be adapted.
The most important part is that the NSS database needs to be owned by the grafana user, or at least to have write permission for that user (it also works when the files have the Grafana group assigned and are group-writable). Somewhat strange. certutilby default creates the database files with permissions 0600, so having the Grafana user as their owner is probably the most straightforward solution.
Given the fact that it is possible to implant custom certificates in the way I described above I suggest changing this from a "Feature Request" to a "Documentation Improvement Request".
Is there anything I can do?
Should probably be added to https://grafana.com/docs/grafana/latest/administration/image_rendering/#troubleshooting. Care to contribute changes needed?
@peteeckel do you happen to know if your instructions would work in the alpine docker image for the rendering service?
Sorry, I can't test it at the moment, but I would expect the situation to be exactly the same - The renderer uses chromium as its engine, chromium uses NSS, and the instructions describe how to add a trusted CA to the NSS cert store.
The only obstacles that come to my mind are that the certutil tool might me missing from the Docker image, or that the chromium instance on the renderer Image might be running under a different user.
Hello,
under Ubuntu 20.04, due to the bug https://bugs.launchpad.net/ubuntu/+source/nss/+bug/1647285 the solution to import with certutil does not work depending of the situation.
In my case, I have a root and an intermediate certificate authorities to import and I never make it works. Here another solution that works much better (it comes from here https://superuser.com/questions/437330/how-do-you-add-a-certificate-authority-ca-to-ubuntu):
sudo apt-get install -y p11-kit libnss3
sudo mv /usr/lib/x86_64-linux-gnu/nss/libnssckbi.so /usr/lib/x86_64-linux-gnu/nss/libnssckbi.so.bak
sudo ln -s /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so /usr/lib/x86_64-linux-gnu/nss/libnssckbi.so
sudo update-ca-certificates
This way finally solved the issue with ERR_CERT_AUTHORITY_INVALID
Hello,
I had the same issue with the net::ERR_CERT_AUTHORITY_INVALID error message.
I tried to create the nssdb inside /usr/share/grafana/.pki/nssdb with my custom CA only and grafana-image-rendere failed to connect my Grafana instance..
However, adding my custom CA certificate in (it's a RHEL box) /usr/share/pki/ca-trust-source/anchors/ and launching update-ca-trust (without creating the nssdb db): I was able to see my custom CA in the trust list command output.
Note 1: Chrome or Chromium may throw you the following error message even if you had the correct CA to be system wide :
error="Error: net::ERR_CERT_COMMON_NAME_INVALID at https:// in my case, after digging, it seems that Chrome needs to have SAN entries for your certificate (one for your FQDN server, and one for your domain).
Once done, I was able to retrieve my panel in PNG.
Best regards,
Nikoos
