error="Rendering failed: TypeError: input.on is not a function"

[TryTryAgain]

What happened: Rendering fails

What you expected to happen: Rendering to succeed

How to reproduce it (as minimally and precisely as possible): Start the Grafana server, choose a panel and click "Direct link rendered image" and it says Grafana / Server Error... Rendering failed... Check the Grafana server logs for more details.

And the /var/log/grafana/grafana.log contains:

t=2020-02-21T19:19:18+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=127.0.0.1:3001 protocol=http subUrl=/grafana socket=
t=2020-02-21T19:19:18+0000 lvl=info msg="Renderer plugin started" logger=plugins.backend pluginId=grafana-image-renderer grpcHost=127.0.0.1 grpcPort=37269.000 chromeBin=/var/lib/grafana/plugins/grafana-image-renderer/chrome-linux/chrome ignoreHTTPSErrors=false
t=2020-02-21T19:19:33+0000 lvl=info msg=Rendering logger=rendering renderer=plugin path="d-solo/IEfxMsQWz/new-dashboard-copy-testing?orgId=1&from=1582139966395&to=1582312766395&panelId=2&width=1000&height=500&tz=America%2FChicago"
t=2020-02-21T19:19:33+0000 lvl=dbug msg="calling renderer plugin" logger=rendering renderer=plugin req="url:\"http://127.0.0.1:3001/d-solo/IEfxMsQWz/new-dashboard-copy-testing?orgId=1&from=1582139966395&to=1582312766395&panelId=2&width=1000&height=500&tz=America%2FChicago&render=1\" width:1000 height:500 timeout:60 timezone:\"America/Chicago\" filePath:\"/var/lib/grafana/png/cATAqusr6z6czMWWu7D7.png\" renderKey:\"b6TaN0tB2KdQS2a9LiYo8wqt37Zc6hXM\" domain:\"127.0.0.1\" "
t=2020-02-21T19:19:33+0000 lvl=eror msg="Render request failed" logger=plugins.backend pluginId=grafana-image-renderer url="http://127.0.0.1:3001/d-solo/IEfxMsQWz/new-dashboard-copy-testing?orgId=1&from=1582139966395&to=1582312766395&panelId=2&width=1000&height=500&tz=America%2FChicago&render=1" error="TypeError: input.on is not a function"
t=2020-02-21T19:19:33+0000 lvl=eror msg="Rendering failed." logger=context userId=3 orgId=1 uname=my.user error="Rendering failed: TypeError: input.on is not a function"
t=2020-02-21T19:19:33+0000 lvl=eror msg="Request Completed" logger=context userId=3 orgId=1 uname=my.user method=GET path=/render/d-solo/IEfxMsQWz/new-dashboard-copy-testing status=500 remote_addr=100.110.250.244 time_ms=15 size=1730 referer="https://ourgitlabinstance.com/grafana/d/IEfxMsQWz/new-dashboard-copy-testing?orgId=1"

Anything else we need to know?: It looks like this is likely being caused by the fact that I am using nginx to redirect https://ourgitlabinstance.com/grafana to 127.0.0.1:3001 with proxypass and the rendering request has a path of path=/render/... which may be the problem? ...because in my case it would need to be path=grafana/render/? I'm not sure where remote_addr is getting populated but maybe it would work if it was the loopback?

Environment:

• Grafana Image Renderer version: "version": "1.0.10"
• Grafana version: v6.6.2 (3fa63cfc34)
• Installed plugin or remote renderer service: Installed plugin, also tried standalone node.
• OS Grafana Image Renderer is installed on: Same server as grafana
• User OS & Browser: CentOS 7, Chromium

[marefr]

You have to save the dashboard first.

[TryTryAgain]

The dashboard was saved, just tried again with a previously saved and unmodified dashboard as well. @marefr What made you think that was the reason for it not working?

[marefr]

Sorry, thought I read /new-dashboardurl.

[marefr]

Please include your Grafana config (replace any sensitive information), specifically interested in your server and rendering settings together with your reverse proxy setup. I think you have to configure the callback_url, see https://grafana.com/docs/grafana/latest/installation/configuration/#callback-url

[TryTryAgain]

@marefr, ah, that's understandable as it was called /new-dashboard-copy-test :)

grafana.ini included below (I've removed all default/unmodified, and I included commented server_url and callback_url of which I did try out with no change):

#################################### Server ####################################
[server]
http_addr = 127.0.0.1
http_port = 3001
root_url = https://ourgitlabinstance.com/grafana

#################################### Session ####################################
[session]
cookie_secure = false
cookie_samesite = disabled

#################################### Security ####################################
[security]
allow_embedding = true

#################################### Users ###############################
[users]
allow_org_create = true
auto_assign_org = true
default_theme = dark

#################################### GitLab Auth ##########################
[auth.gitlab]
enabled = true
allow_sign_up = true
client_id = [REDACTED]
client_secret = [REDACTED]
scopes = api, read_user, openid
auth_url = https://ourgitlabinstance.com/oauth/authorize
token_url = https://ourgitlabinstance.com/oauth/token
api_url = https://ourgitlabinstance.com/api/v4

#################################### SMTP / Emailing ##########################
[smtp]
enabled = true
host = smtp.ourmailserver.com:25
user = some_user
password = secretpassword
skip_verify = true
from_address = [email protected]
from_name = Our-Grafana
ehlo_identity = ourdomain.com

[emails]
welcome_email_on_sign_up = true

#################################### Logging ##########################
[log]
filters = rendering:debug

#################################### Internal Grafana Metrics ##########################
[metrics]
enabled           = true
interval_seconds  = 10

#################################### External image storage ##########################
[rendering]
# Options to configure external image rendering server like https://github.com/grafana/grafana-image-renderer
;server_url = http://127.0.0.1:8081/render
;callback_url = https://ourgitlabinstance.com/grafana/

[panels]
# here for to support old env variables, can remove after a few months
enable_alpha = false
disable_sanitize_html = true
GF_PANELS_DISABLE_SANITIZE_HTML=true

[marefr]

If you can browse http://127.0.0.1:3001/d-solo/IEfxMsQWz/new-dashboard-copy-testing?orgId=1&from=1582139966395&to=1582312766395&panelId=2 then config should be all good and you have a problem with the image renderer/chromium.

• Have you installed centos dependencies listed here?
• Please try other versions of the plugin, like 1.0.7, 1.0.8, 1.0.9 or lower
• If you have a local install of chromium you can try and use that to verify if the bundled chromium is the problem, see https://grafana.com/docs/grafana/latest/administration/image_rendering/#custom-chrome-chromium

[TryTryAgain]

@marefr I can navigate to the d-solo resulting image, although in my case I have to remove an added /render/ that comes between my subdir//grafana and the /d-solo/.

I have installed all dependencies: yum install [...] results in [...] already installed and latest version Nothing to do.

I will try a variety of versions, have not yet. I'll report back on that...

...but, it works without issue as expected when using custom-chrome-chromium!

[marefr]

@TryTryAgain v1.0.11 is released which enables you to enable verbose logging, for plugin thru environment variable see https://github.com/grafana/grafana-image-renderer/blob/master/README.md#environment-variables. Please try enabling that and report back Grafana server log output. Thanks

[TryTryAgain]

@marefr

tail -f /var/log/grafana/grafana.log with bundled chromium, not working:

[log]
filters = rendering:debug

...and

GF_RENDERER_PLUGIN_VERBOSE_LOGGING=true

enabled...

t=2020-03-24T01:14:44+0000 lvl=info msg="Initializing TracingService" logger=server
t=2020-03-24T01:14:44+0000 lvl=info msg="Initializing UsageStatsService" logger=server
t=2020-03-24T01:14:44+0000 lvl=info msg="Initializing CleanUpService" logger=server
t=2020-03-24T01:14:44+0000 lvl=info msg="Initializing NotificationService" logger=server
t=2020-03-24T01:14:44+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server
t=2020-03-24T01:14:44+0000 lvl=info msg="Initializing Stream Manager"
t=2020-03-24T01:14:44+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=127.0.0.1:3001 protocol=http subUrl=/grafana socket=
t=2020-03-24T01:14:45+0000 lvl=info msg="Renderer plugin started" logger=plugins.backend pluginId=grafana-image-renderer grpcHost=127.0.0.1 grpcPort=34432.000 chromeBin=/var/lib/grafana/plugins/grafana-image-renderer/chrome-linux/chrome ignoreHTTPSErrors=false
t=2020-03-24T01:15:51+0000 lvl=info msg=Rendering logger=rendering renderer=plugin path="d-solo/SiI1Ltjmz/test-copy?orgId=1&from=1569287737875&to=1585012537875&var-Account=CloudWatch-AccountA&var-test=All&panelId=2&width=1000&height=500&tz=America%2FChicago"
t=2020-03-24T01:15:51+0000 lvl=dbug msg="calling renderer plugin" logger=rendering renderer=plugin req="url:\"http://127.0.0.1:3001/d-solo/SiI1Ltjmz/test-copy?orgId=1&from=1569287737875&to=1585012537875&var-Account=CloudWatch-AccountA&var-test=All&panelId=2&width=1000&height=500&tz=America%2FChicago&render=1\" width:1000 height:500 timeout:60 timezone:\"America/Chicago\" filePath:\"/var/lib/grafana/png/E65yoAOZv9OchGfjMzj4.png\" renderKey:\"1eyPuMdaUvsVek6iaCgytac2ci6IfIM3\" domain:\"127.0.0.1\" "
t=2020-03-24T01:15:51+0000 lvl=eror msg="Render request failed" logger=plugins.backend pluginId=grafana-image-renderer url="http://127.0.0.1:3001/d-solo/SiI1Ltjmz/test-copy?orgId=1&from=1569287737875&to=1585012537875&var-Account=CloudWatch-AccountA&var-test=All&panelId=2&width=1000&height=500&tz=America%2FChicago&render=1" error="TypeError: input.on is not a function"
t=2020-03-24T01:15:51+0000 lvl=eror msg="Rendering failed." logger=context userId=4 orgId=1 uname=some.admin error="Rendering failed: TypeError: input.on is not a function"
t=2020-03-24T01:15:51+0000 lvl=eror msg="Request Completed" logger=context userId=4 orgId=1 uname=some.admin method=GET path=/render/d-solo/SiI1Ltjmz/test-copy status=500 remote_addr=10.10.25.24 time_ms=33 size=1730 referer="https://ourgrafanadomain.com/grafana/d/SiI1Ltjmz/test-copy?orgId=1"
t=2020-03-24T01:15:52+0000 lvl=info msg="Renderer plugin started" logger=plugins.backend pluginId=grafana-image-renderer grpcPort=39344.000 chromeBin=/var/lib/grafana/plugins/grafana-image-renderer/chrome-linux/chrome ignoreHTTPSErrors=false grpcHost=127.0.0.1

tail -f /var/log/grafana/grafana.log with custom chromium, working:

t=2020-03-24T01:17:06+0000 lvl=info msg="Initializing TracingService" logger=server
t=2020-03-24T01:17:06+0000 lvl=info msg="Initializing UsageStatsService" logger=server
t=2020-03-24T01:17:06+0000 lvl=info msg="Initializing CleanUpService" logger=server
t=2020-03-24T01:17:06+0000 lvl=info msg="Initializing NotificationService" logger=server
t=2020-03-24T01:17:06+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server
t=2020-03-24T01:17:06+0000 lvl=info msg="Initializing Stream Manager"
t=2020-03-24T01:17:06+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=127.0.0.1:3001 protocol=http subUrl=/grafana socket=
t=2020-03-24T01:17:06+0000 lvl=info msg="Renderer plugin started" logger=plugins.backend pluginId=grafana-image-renderer grpcHost=127.0.0.1 grpcPort=46028.000 chromeBin=/usr/bin/chromium-browser ignoreHTTPSErrors=false
t=2020-03-24T01:17:19+0000 lvl=info msg=Rendering logger=rendering renderer=plugin path="d-solo/SiI1Ltjmz/test-copy?orgId=1&from=1569287830720&to=1585012630720&var-Account=CloudWatch-AccountA&var-test=All&panelId=2&width=1000&height=500&tz=America%2FChicago"
t=2020-03-24T01:17:19+0000 lvl=dbug msg="calling renderer plugin" logger=rendering renderer=plugin req="url:\"http://127.0.0.1:3001/d-solo/SiI1Ltjmz/test-copy?orgId=1&from=1569287830720&to=1585012630720&var-Account=CloudWatch-AccountA&var-test=All&panelId=2&width=1000&height=500&tz=America%2FChicago&render=1\" width:1000 height:500 timeout:60 timezone:\"America/Chicago\" filePath:\"/var/lib/grafana/png/xAzP5Tgk2wR40Aj7GeA2.png\" renderKey:\"lJ3nz7FVPiVON2WAKi2XQExiYNxLhF3C\" domain:\"127.0.0.1\" "