Image renderer plugin 3.4.2 appears to be incompatible with Grafana 9.0.3 Alpine Docker image

[chennin]

What happened: Grafana 9.0.3 in Alpine Docker image fails to start when Grafana Image Renderer is enabled. The latest version of the plugin right now is 3.4.2.

9.0.2 works. Ubuntu Docker image works. Alpine image is 3.15.4 in case of both 9.0.2 and 9.0.3.

How to reproduce it (as minimally and precisely as possible):

cd $(mktemp -d)
wget https://raw.githubusercontent.com/grafana/grafana/acf54905eeeb92cd692b006c1819fc3ecd623361/packaging/docker/custom/Dockerfile
docker build --build-arg GF_INSTALL_IMAGE_RENDERER_PLUGIN=true -t grafana-image .
docker run --rm  -e GF_LOG_LEVEL=debug grafana-image

Anything else we need to know?:

Logs:

logger=plugin.grafana-image-renderer t=2022-07-16T14:19:12.035711493Z level=debug msg="Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /var/lib/grafana-plugins/grafana-image-renderer/plugin_start_linux_amd64)"
logger=plugin.grafana-image-renderer t=2022-07-16T14:19:12.039483486Z level=debug msg="Error relocating /var/lib/grafana-plugins/grafana-image-renderer/plugin_start_linux_amd64: gnu_get_libc_version: symbol not found"
logger=plugin.grafana-image-renderer t=2022-07-16T14:19:12.039670258Z level=debug msg="Error relocating /var/lib/grafana-plugins/grafana-image-renderer/plugin_start_linux_amd64: __register_atfork: symbol not found"
logger=plugin.grafana-image-renderer t=2022-07-16T14:19:12.039767921Z level=debug msg="Error relocating /var/lib/grafana-plugins/grafana-image-renderer/plugin_start_linux_amd64: __strdup: symbol not found"
logger=plugin.grafana-image-renderer t=2022-07-16T14:19:12.039943081Z level=debug msg="Error relocating /var/lib/grafana-plugins/grafana-image-renderer/plugin_start_linux_amd64: setcontext: symbol not found"
logger=plugin.grafana-image-renderer t=2022-07-16T14:19:12.040066454Z level=debug msg="Error relocating /var/lib/grafana-plugins/grafana-image-renderer/plugin_start_linux_amd64: makecontext: symbol not found"
logger=plugin.grafana-image-renderer t=2022-07-16T14:19:12.040506122Z level=debug msg="Error relocating /var/lib/grafana-plugins/grafana-image-renderer/plugin_start_linux_amd64: backtrace: symbol not found"
logger=plugin.grafana-image-renderer t=2022-07-16T14:19:12.040525338Z level=debug msg="Error relocating /var/lib/grafana-plugins/grafana-image-renderer/plugin_start_linux_amd64: getcontext: symbol not found"
logger=plugin.grafana-image-renderer t=2022-07-16T14:19:12.040530167Z level=debug msg="Error relocating /var/lib/grafana-plugins/grafana-image-renderer/plugin_start_linux_amd64: __libc_stack_end: symbol not found"
logger=server t=2022-07-16T14:19:12.040532281Z level=error msg="Stopped background service" service=*rendering.RenderingService reason="Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
logger=secrets t=2022-07-16T14:19:12.040560965Z level=debug msg="Grafana is shutting down; stopping..."

...
*rendering.RenderingService run error: Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.
logger=server t=2022-07-16T14:19:12.15968329Z level=debug msg="Stopped background service" service=*updatechecker.PluginsService reason="context canceled"
logger=server t=2022-07-16T14:19:12.159732342Z level=error msg="Server shutdown" error="*rendering.RenderingService run error: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."

The ld-linux-x86-64.so.2 and the symbol not found errors also appear in 9.0.2 and the plugin works, so I guess those are red herrings and we need a new plugin release?

Environment:

• Grafana Image Renderer version: 3.4.2
• Grafana version: 9.0.3
• Installed plugin or remote renderer service: Plugin
• OS Grafana Image Renderer is installed on: Ubuntu 22.04

[chennin]

Unfortunately not fixed with plugin 3.5.0

[chennin]

And still an issue with Grafana 9.0.4

[simPod]

Same issue with grafana version https://github.com/grafana/grafana/commit/77421907b77d92bcb5d43b42304c218da952cbb2 and renderer 3.5

[ampelio]

Same issue grafana:latest (9.05), renderer 3.5.

[joao1996f]

With grafana 9.0.6 and renderer 3.5.0 I have the same issue.

[joao1996f]

With grafana 9.0.7 and renderer 3.5.0 I don't have this issue anymore.

[spinillos]

Maybe is related to this?: https://github.com/grafana/grafana-image-renderer/issues/320#issuecomment-1044602134

image-renderer is not working inside Grafana Docker image and you should create it in a separate image. I was taking a look about the changes inside Grafana and I didn't find anything that we could change by mistake.

[chennin]

I don't think so. image-renderer does work in a custom Grafana image built with Alpine (first option in that comment; also what the first comment on this issue does -- note the Dockerfile provided by Grafana Labs), except for with Grafana versions 9.0.3 through 9.0.6 inclusive.

[vqmanh]

when I install plugin image renderer => I can't start container grafana. did something go wrong?

[GSA27]

When you start Grafana what is the error in the logs? @vqmanh

[vqmanh]

hello, i try to use a lot of version, but it's not working

[vqmanh]

[vqmanh]

when I removed grafana-image-renderer, I could start grafana container

[iameli]

Anyone know of a way to diagnose this further? "Unrecognized remote plugin message: \n\n" is pretty unhelpful.

[jianali]

I also face the problem, anyone can help me?

[jianali]

I also face the problem, anyone can help me?

i have find some reasons about this problem; first, you should set the log level to debug, and you will find the detail error in start log; for example, my problem is about the Glibc or libC++ version is too low. second, according to the debug error, may be you can find some solutions.

my OS is suse, too old; And i turn to centos7.6, the problem is solved. because update the C or C++ library is too hard and have some risk for environment.

hope can hep for you; @iameli @chennin @joao1996f

[akamensky]

Still an issue with grafana 9.3.2 and plugin 3.6.3 when running Grafana as Docker image and installing plugin using CLI grafana-cli plugins install grafana-image-renderer:

logger=server t=2023-01-25T01:47:27.085648023Z level=error msg="Stopped background service" service=*rendering.RenderingService reason="Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
logger=plugin.grafana-image-renderer t=2023-01-25T01:47:27.085910739Z level=warn msg="plugin failed to exit gracefully"

Edit: actually found that in the readme it already says for Docker setup to run this as separate container. So guess my case is different.

[Davenury]

Still an issue with grafana 9.3.2 and plugin 3.6.3 when running Grafana as Docker image and installing plugin using CLI grafana-cli plugins install grafana-image-renderer:

logger=server t=2023-01-25T01:47:27.085648023Z level=error msg="Stopped background service" service=*rendering.RenderingService reason="Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
logger=plugin.grafana-image-renderer t=2023-01-25T01:47:27.085910739Z level=warn msg="plugin failed to exit gracefully"

@akamensky did you manage to resolve this problem? I have the same versions of grafana and image renderer and I'm facing the same issue.

[akamensky]

@Davenury Actually yes. For Docker setup we run this plugin in another container and configured Grafana to connect to it. That's in the readme of this project (I should've updated my comment).

[Davenury]

@akamensky could you provide more details :sweat_smile: I've been also trying to make it work from the grafana helm chart but I receive mutliple errors from renderer (also renderer eats 10GB of RAM, so I suspected a memory leak)?

[akamensky]

@Davenury

could you provide more details

I just read documentation as described in readme file. I suggest you check that too perhaps? We are using Docker so no idea about running it in k8s or other systems using helm charts. I think documentation or devs of this project are better placed to help you.

also renderer eats 10GB of RAM

I am not a dev of this project so I can't say for memory leaks. But it is literally Chrome browser, so 10GB is quite normal for that IMO

[AgnesToulet]

Tested with latest versions (Grafana 9.4.7 and image renderer 3.6.4) and it's working now.

@Davenury You can find a simple docker-compose example here and more complex examples in the parent folder. If you still have trouble running the image renderer as a separate container, please open a new issue with your current configuration.

[vbichov]

faced this issue with grafana 9.5.3 with all the latest versions of plugins. alpine uses musl instead of glibc and the issue seems to be that running this command apk --no-cache upgrade resulted in libc reverted to using musl

ether remove that line or force link glibc again with

ln -sf /usr/glibc-compat/lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2

this should solve the issue