grafana-image-renderer icon indicating copy to clipboard operation
grafana-image-renderer copied to clipboard

Published 20 hours ago • verified publisher icon grafana

401 (Unauthorized) remote render service when running as remote docker container

Open Bashere1 opened this issue 2 years ago • 20 comments

What happened: I am running grafana via container utilizing google compute engine. https://cloud.google.com/compute/docs/containers/deploying-containers

In another separate google compute engine I am running the remote render image via container using the same model. However, when I try to generate a "Direct link rendered image" render fails.

I noticed in the renderer logs that we are getting a 401 unauthorized to the callback url. {"level":"info","message":"HTTP Server started, listening at http://redacted_ip:8081"} {"msg":"Failed to load resource: the server responded with a status of 401 (Unauthorized)","url":"http://grafana.mydomain.com:3000/d-solo/gL-UXHm7z/mydashboard?orgId=1&from=1628254952332&to=1628276552332&panelId=2&width=1000&height=500&tz=America%2FChicago&render=1","level":"error","message":"Browser console error"} {"url":"/render?deviceScaleFactor=1.000000&domain=grafana.mydomain.com&encoding=&height=500&renderKey=REDACTED&timeout=60&timezone=America%2FChicago&url=http%3A%2F%2Fgrafana.mydomain.com%3A3000%2Fd-solo%2FgL-UXHm7z%2Fmydashboard%3ForgId%3D1%26from%3D1628254952332%26to%3D1628276552332%26panelId%3D2%26width%3D1000%26height%3D500%26tz%3DAmerica%252FChicago%26render%3D1&width=1000","stack":"TimeoutError: waiting for function failed: timeout 60000ms exceeded\n at new WaitTask (/usr/src/app/node_modules/puppeteer/lib/DOMWorld.js:388:34)\n at DOMWorld.waitForFunction (/usr/src/app/node_modules/puppeteer/lib/DOMWorld.js:303:16)\n at Frame.waitForFunction (/usr/src/app/node_modules/puppeteer/lib/FrameManager.js:402:32)\n at Page.waitForFunction (/usr/src/app/node_modules/puppeteer/lib/Page.js:805:33)\n at Browser. (/usr/src/app/build/browser/browser.js:195:24)\n at Generator.next ()\n at fulfilled (/usr/src/app/build/browser/browser.js:5:58)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)","level":"error","message":"Request failed"}

As expected if I test the URL from my browser while logged in the above url loads successfully. I should also clarify that I have setup LDAP auth for the grafana instance. I've also attempted to set GF_AUTH_ANONYMOUS_ENABLED=true as well with no luck.

What you expected to happen:

Grafana to render images using remote render service

How to reproduce it (as minimally and precisely as possible):

  1. Deploy grafana via docker instance
  2. Deploy grafana image render via docker instance on alternate compute
  3. Attempt to share direct link with rendered images

Anything else we need to know?:

Environment:

  • Grafana Image Renderer version: latest
  • Grafana version: latest
  • Installed plugin or remote renderer service: remote renderer service
  • OS Grafana Image Renderer is installed on: COS 89 LTS
  • User OS & Browser: Chrome v91
  • Others:

EnvironmentVariables GF_RENDERING_SERVER_URL: http://REDACTED:8081/render GF_RENDERING_CALLBACK_URL: http://grafana.mydomain.com:3000 GF_AUTH_ANONYMOUS_ENABLED: true GF_LOG_FILTERS: rendering:debug

Bashere1 avatar Aug 06 '21 20:08 Bashere1

Hello! Thanks for reaching us, do you also have the logs from the Grafana server?

AgnesToulet avatar Aug 31 '21 09:08 AgnesToulet

I'm hitting a similar Issue today

K8s env but docker images under the hood.

A grafana server & grafana-render with the vars setup like the one on your the docker docs.

401 Unauthorized

Partial grafana debug logs follows

t=2022-09-09T17:42:45+0000 lvl=info msg="New state change" logger=alerting.resultHandler ruleId=8 newState=ok prev state=no_data
t=2022-09-09T17:42:45+0000 lvl=info msg=Rendering logger=rendering renderer=http path="d-solo/h5sY0munk/server-monitoring-general?orgId=1&panelId=22"
t=2022-09-09T17:42:45+0000 lvl=dbug msg="calling remote rendering service" logger=rendering renderer=http url="http://grafana-renderer:8081/render?deviceScaleFactor=1.000000&domain=grafana&encoding=&height=500&renderKey=[.....]&timeout=15&timezone=&url=http%3A%2F%2Fgrafana%3A3000%2Fgrafana%2Fd-solo%2Fh5sY0munk%2Fserver-monitoring-general%3ForgId%3D1%26panelId%3D22%26render%3D1&width=1000"
t=2022-09-09T17:42:45+0000 lvl=eror msg="Remote rendering request failed" logger=rendering renderer=http error="401 Unauthorized"
t=2022-09-09T17:42:45+0000 lvl=eror msg="Failed to render and upload alert panel image." logger=alerting.notifier ruleId=8 error="Remote rendering request failed. 401: 401 Unauthorized"
t=2022-09-09T17:42:45+0000 lvl=eror msg="Could not generate Telegram message with inline image." logger=alerting.notifier.telegram err="open : no such file or directory"

Grafana render log entry:

{"level":"error","message":"::ffff:10.1.15.92 - - [09/Sep/2022:18:14:15 +0000] \"GET /render?deviceScaleFactor=1.000000&domain=grafana&encoding=&height=500&renderKey=[.....]&timeout=15&timezone=&url=http%3A%2F%2Fgrafana%3A3000%2Fgrafana%2Fd-solo%2Fh5sY0munk%2Fserver-monitoring-general%3ForgId%3D1%26panelId%3D22%26render%3D1&width=1000 HTTP/1.1\" 401 74 \"-\" \"Grafana/7.3.3\"\n"}

stdevPavelmc avatar Sep 09 '22 17:09 stdevPavelmc

I had the same problem. Try changing the version of Grafana Image Renderer and try again.

yinyicao avatar Sep 20 '22 01:09 yinyicao

I had the same problem. Try changing the version of Grafana Image Renderer and try again.

3.5.0 version works for me (grafana version is 9.0.5)

romcheck avatar Sep 23 '22 19:09 romcheck

Even for me 3.5.0 worked with Grafana 8.3.3 version

kisshore avatar Sep 28 '22 11:09 kisshore

I had the same problem. Try changing the version of Grafana Image Renderer and try again.

works for me !! grafana version 8.4.2

Codecaver avatar Sep 29 '22 09:09 Codecaver

I had the same issue - I was already on the latest version of image renderer, solution for me was to upgrade Grafana from v9.1.0 to v9.1.6

yetkinergun avatar Sep 29 '22 18:09 yetkinergun

Same here with Grafana 7.5.5 and image renderer "lastest" or "3.6.1". Downgrading image renderer to 3.5.0 worked for me.

carlosrodfern avatar Oct 07 '22 19:10 carlosrodfern

lastest is not working....

debu99 avatar Oct 11 '22 12:10 debu99

Same. I was using "latest" tag for grafana and image renderer. I started getting the 401 errors. I followed the instructions here to make sure a token was properly setup. But I am still getting 401 messages in the image renderer docker logs and images wont render. I rolled back to 3.5 and it's working again but now I suspect I am vulnerable.

KevinGage avatar Oct 17 '22 14:10 KevinGage

It's worth pointing out that 3.6.0 contains this change which has a requirement on Grafana >=8.3.11 so if you have an older version of Grafana you'll see 401 failures on 3.6.0+/latest.

tomelliff avatar Nov 11 '22 11:11 tomelliff

Grafana 8.5.1 + Image rendered 3.6.2 - does not work (Error: Unauthorized request). Probably, grafana-image-renderer v. 3.6.2 (which is the latest at the moment) is broken OR incompatible with Grafana 8 and below. https://hub.docker.com/r/grafana/grafana-image-renderer/tags

If these versions are supposed to be incompatible, this requires a clear error message on that.

dmitryp-orca avatar Nov 23 '22 08:11 dmitryp-orca

@dmitryp-orca see https://grafana.com/blog/2022/08/30/security-release-new-versions-of-grafana-and-grafana-image-renderer-with-a-high-severity-security-fix-for-cve-2022-31176/ - in the 8.5 series would need at least 8.5.11

tyrken avatar Dec 02 '22 09:12 tyrken

Hi i have same issue here.

I have Grafana: 9.4.3 and Grafana Image Renderer: 3.6.4. I use Image Renderer like standalone app: node build/app.js server --port=8081 In my grafana.ini:

[unified_alerting.screenshots]
capture = true

[plugin.grafana-image-renderer]
rendering_ignore_https_errors = true
rendering_args = --no-sandbox,--disable-setuid-sandbox,--disable-dev-shm-usage,--disable-accelerated-2d-canvas,--disable-gpu,--window-size=1280x758,--ignore-certificate-errors,--ignore-certificate-errors-spki-list
auth_token = token_string (same as renderer_token)

[rendering]
server_url = http://192.168.171.53:8081/render
callback_url = http://192.168.171.53:3000
server_domain = 192.168.171.53
renderer_token = token_string (same as auth_token)

After start the Image renderer i was got this:

{"level":"error","message":"::ffff:192.168.171.53 - - [28/Mar/2023:19:56:01 +0000] \"GET /render?deviceScaleFactor=1.000000&domain=192.168.171.53&encoding=&height=500&renderKey=xXCCVR82sEGgIhVGtzro8EbtcPdwhX2s&timeout=10&timezone=&url=http%3A%2F%2F192.168.171.53%3A3000%2Fd-solo%2FZjraN-fGz%2F21_test_ph%3Ffrom%3Dnow-1h%26orgId%3D1%26panelId%3D20%26to%3Dnow%26render%3D1&width=1000 HTTP/1.1\" 401 74 \"-\" \"Grafana/9.4.3\""}
{"level":"error","message":"Request failed","stack":"Error: Unauthorized request
   at /home/atb/grafana-image-renderer/build/service/middlewares.js:27:30
   at Layer.handle [as handle_request] (/home/atb/grafana-image-renderer/node_modules/express/lib/router/layer.js:95:5 undefined)  
   at trim_prefix .trim_prefix (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:328:13 undefined)   at /home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:286:9
   at Function.process_params (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:346:12 undefined)
   at next .next (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:280:10 undefined) 
   at jsonParser .jsonParser (/home/atb/grafana-image-renderer/node_modules/body-parser/lib/types/json.js:110:7 undefined)
   at Layer.handle [as handle_request] (/home/atb/grafana-image-renderer/node_modules/express/lib/router/layer.js:95:5 undefined)  
   at trim_prefix .trim_prefix (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:328:13 undefined)   
   at /home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:286:9","url":"/render?deviceScaleFactor=1.000000&domain=192.168.171.53&encoding=&height=500&renderKey=ilFSQne8WdyAC4KuzbwXLz5jhh6r2PwJ&timeout=10&timezone=&url=http%3A%2F%2F192.168.171.53%3A3000%2Fd-solo%2FZjraN-fGz%2F21_test_ph%3Ffrom%3Dnow-1h%26orgId%3D1%26panelId%3D68%26to%3Dnow%26render%3D1&width=1000"}

If I add console.log(config) in the middleware.js, i see in console - authToken: '-' (Image renderere in standalone mode dosn't get Grafana config ENV and Settings?) Okay, what if you comment out the check of the token at all. I was get:

{"err":"TimeoutError: Navigation timeout of 10000 ms exceeded
    at LifecycleWatcher._LifecycleWatcher_createTimeoutPromise (/home/atb/grafana-image-renderer/node_modules/puppeteer/lib/cjs/puppeteer/common/LifecycleWatcher.js:167:12)","level":"error","message":"Error while trying to prepare page for screenshot","url":"http://192.168.171.53:3000/d-solo/A-sK4r-Mk/20_test_ec?from=now-1h&orgId=1&panelId=80&to=now&render=1"}
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGTERM listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGHUP listeners added to [process]. Use emitter.setMaxListeners() to increase limit

btw same thing if i remove auth_token from grafana.ini and add - to renderer_token

At the same time, URLs that cannot be reached by GIR open in the browser without problems. What am I doing wrong? How do I get GIR to work with Grafana and get it to see settings.

sv-hmelevsky avatar Mar 28 '23 20:03 sv-hmelevsky

Hi i have same issue here.

I have Grafana: 9.4.3 and Grafana Image Renderer: 3.6.4. I use Image Renderer like standalone app: node build/app.js server --port=8081 In my grafana.ini:

[unified_alerting.screenshots]
capture = true

[plugin.grafana-image-renderer]
rendering_ignore_https_errors = true
rendering_args = --no-sandbox,--disable-setuid-sandbox,--disable-dev-shm-usage,--disable-accelerated-2d-canvas,--disable-gpu,--window-size=1280x758,--ignore-certificate-errors,--ignore-certificate-errors-spki-list
auth_token = token_string (same as renderer_token)

[rendering]
server_url = http://192.168.171.53:8081/render
callback_url = http://192.168.171.53:3000
server_domain = 192.168.171.53
renderer_token = token_string (same as auth_token)

After start the Image renderer i was got this:

{"level":"error","message":"::ffff:192.168.171.53 - - [28/Mar/2023:19:56:01 +0000] \"GET /render?deviceScaleFactor=1.000000&domain=192.168.171.53&encoding=&height=500&renderKey=xXCCVR82sEGgIhVGtzro8EbtcPdwhX2s&timeout=10&timezone=&url=http%3A%2F%2F192.168.171.53%3A3000%2Fd-solo%2FZjraN-fGz%2F21_test_ph%3Ffrom%3Dnow-1h%26orgId%3D1%26panelId%3D20%26to%3Dnow%26render%3D1&width=1000 HTTP/1.1\" 401 74 \"-\" \"Grafana/9.4.3\""}
{"level":"error","message":"Request failed","stack":"Error: Unauthorized request
   at /home/atb/grafana-image-renderer/build/service/middlewares.js:27:30
   at Layer.handle [as handle_request] (/home/atb/grafana-image-renderer/node_modules/express/lib/router/layer.js:95:5 undefined)  
   at trim_prefix .trim_prefix (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:328:13 undefined)   at /home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:286:9
   at Function.process_params (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:346:12 undefined)
   at next .next (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:280:10 undefined) 
   at jsonParser .jsonParser (/home/atb/grafana-image-renderer/node_modules/body-parser/lib/types/json.js:110:7 undefined)
   at Layer.handle [as handle_request] (/home/atb/grafana-image-renderer/node_modules/express/lib/router/layer.js:95:5 undefined)  
   at trim_prefix .trim_prefix (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:328:13 undefined)   
   at /home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:286:9","url":"/render?deviceScaleFactor=1.000000&domain=192.168.171.53&encoding=&height=500&renderKey=ilFSQne8WdyAC4KuzbwXLz5jhh6r2PwJ&timeout=10&timezone=&url=http%3A%2F%2F192.168.171.53%3A3000%2Fd-solo%2FZjraN-fGz%2F21_test_ph%3Ffrom%3Dnow-1h%26orgId%3D1%26panelId%3D68%26to%3Dnow%26render%3D1&width=1000"}

If I add console.log(config) in the middleware.js, i see in console - authToken: '-' (Image renderere in standalone mode dosn't get Grafana config ENV and Settings?) Okay, what if you comment out the check of the token at all. I was get:

{"err":"TimeoutError: Navigation timeout of 10000 ms exceeded
    at LifecycleWatcher._LifecycleWatcher_createTimeoutPromise (/home/atb/grafana-image-renderer/node_modules/puppeteer/lib/cjs/puppeteer/common/LifecycleWatcher.js:167:12)","level":"error","message":"Error while trying to prepare page for screenshot","url":"http://192.168.171.53:3000/d-solo/A-sK4r-Mk/20_test_ec?from=now-1h&orgId=1&panelId=80&to=now&render=1"}
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGTERM listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGHUP listeners added to [process]. Use emitter.setMaxListeners() to increase limit

btw same thing if i remove auth_token from grafana.ini and add - to renderer_token

At the same time, URLs that cannot be reached by GIR open in the browser without problems. What am I doing wrong? How do I get GIR to work with Grafana and get it to see settings.

That work form me. I think you have come kind of connection problem between yout render and the grafana server

pelandrun avatar Apr 28 '23 19:04 pelandrun

oding=&height=500&

Was this resolved? getting the same issue

ccruz2 avatar May 16 '23 03:05 ccruz2

the same issue， grafana 9.1.2 ， image renderer 3.7.0

kwenzh avatar Aug 07 '23 03:08 kwenzh

Hello! @ccruz2 @kwenzh Can you share your Grafana and image renderer setup please?

It seems like a lot of issues in this thread comes from the security update that happened in August 2022. Please ensure you have correctly set up your shared token between Grafana and the image renderer (as explained here: https://grafana.com/blog/2022/08/30/security-release-new-versions-of-grafana-and-grafana-image-renderer-with-a-high-severity-security-fix-for-cve-2022-31176/#:~:text=by%20this%20vulnerability.-,Solutions%20and%20mitigations,-All%20Grafana%20installations). Please note that if you are running the image renderer as a plugin (installed in your Grafana folder), you need to add an auth_token config key in the [plugin.grafana-image-renderer] section of the Grafana config file. But if you run the image renderer as a separate server, you need to set it as an environment variable (AUTH_TOKEN or in the image renderer configuration file (https://github.com/grafana/grafana-image-renderer/blob/master/default.json#L21). In both cases, you will need to also set it In the rendering section of the Grafana configuration file, in renderer_token.

AgnesToulet avatar Oct 16 '23 09:10 AgnesToulet

I had the same problem. Try changing the version of Grafana Image Renderer and try again.

it's work! grafana 8.4.2 the image render use 3.4.2

minibear2333 avatar Mar 06 '24 08:03 minibear2333

I've tested grafana-image-renderer v3.10.4 with

  • Grafana v8.2.7: not work (error="remote rendering request failed, status code: 401, status: 401 Unauthorized")
  • Grafana v9.5.1: work!
  • Grafana v11.0.0: not work (error="remote rendering request failed, status code: 401, status: 401 Unauthorized")

[Updated 2024/05/16] When I try to upgrade from

  • v8.2.7 not work --(directly upgrade to)--> v11.0.0 not work
  • v8.2.7 not work --> v9.5.1 work! --> v10.4.3 work! --> v11.0.0 work!

yioda avatar May 15 '24 09:05 yioda

Same problem I only managed to make it work with Grafana v9.5.1 and image-renderer v3.10.4 as mentionned in @yioda message

emmanuelbertho avatar Jun 13 '24 07:06 emmanuelbertho

Same issue for me with k8s and 2 containers in a pod with grafana 11.0.0 and renderer 3.11.0

logger=rendering renderer=http level=debug msg="calling remote rendering service" url=http://localhost:8081/render/version
logger=rendering renderer=http level=error msg="Failed to send request to remote rendering service" error="Get \"http://localhost:8081/render/version\": dial tcp [::1]:8081: connect: connection refused"

  containers:
    - name: grafana
      image: grafana/grafana:11.0.0
      ports:
        - name: grafana
          containerPort: 3000
      env:
        - name: GF_FEATURE_TOGGLES_ENABLE
          value: nestedFolders
        - name: GF_AUTH_BASIC_ENABLED
          value: "false"
        - name: GF_RENDERING_SERVER_URL
          value: http://localhost:8081/render
        - name: GF_RENDERING_CALLBACK_URL
          value: http://localhost:3000/
        - name: GF_LOG_FILTERS
          value: rendering:debug
        - name: GF_RENDERING_RENDERER_TOKEN
          value: test
    - name: renderer
      image: grafana/grafana-image-renderer:3.11.0
      ports:
      - name: renderer
        containerPort: 8081
      env:
        - name: AUTH_TOKEN
          value: test
        - name: GF_PLUGIN_AUTH_TOKEN
          value: test

florentsorel avatar Jun 19 '24 11:06 florentsorel