Overview / Metrics / Radiator not working

[cartera93]

When we select "All environments" the "Overview", "Metrics", and "Radiator" pages return this error

Internal Server Error The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

This error is also returned by the "Metrics" page with any specific environment selected but the "Overview" and "Radiator" pages work for those.

We are running puppetboard 2.1.2 in a docker container with pypuppetdb 2.1.0 and PuppetDB 6.9.1

[bgibson710]

We downgraded puppetdb to 6.9.0 and things again work as they should. Seems like the changes for the new API aren't fully in place even though the change logs seem to indicate they should be? Is there something we're missing?

[shoddyguard]

Similar issue here but only with Metrics not working - downgrading PuppetDB to 6.9.0 also fixed this issue for me.

Some syslog goodness in case it helps:

Apr 27 00:45:40 vagrant docker-puppetboard[704]: 10.0.2.2 - - [27/Apr/2020:00:45:40 +0000] "GET /metrics HTTP/1.1" 500 290 "http://127.0.0.1:8080/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/$
Apr 27 00:45:40 vagrant docker-puppetboard[704]: ERROR:puppetboard.core:Exception on /metrics [GET]
Apr 27 00:45:40 vagrant docker-puppetboard[704]: Traceback (most recent call last):
Apr 27 00:45:40 vagrant docker-puppetboard[704]:   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
Apr 27 00:45:40 vagrant docker-puppetboard[704]:     response = self.full_dispatch_request()
Apr 27 00:45:40 vagrant docker-puppetboard[704]:   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
Apr 27 00:45:40 vagrant docker-puppetboard[704]:     rv = self.handle_user_exception(e)
Apr 27 00:45:40 vagrant docker-puppetboard[704]:   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
Apr 27 00:45:40 vagrant docker-puppetboard[704]:     reraise(exc_type, exc_value, tb)
Apr 27 00:45:40 vagrant docker-puppetboard[704]:   File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
Apr 27 00:45:40 vagrant docker-puppetboard[704]:     raise value
Apr 27 00:45:40 vagrant docker-puppetboard[704]:   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
Apr 27 00:45:40 vagrant docker-puppetboard[704]:     rv = self.dispatch_request()
Apr 27 00:45:40 vagrant docker-puppetboard[704]:   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
Apr 27 00:45:40 vagrant docker-puppetboard[704]:     return self.view_functions[rule.endpoint](**req.view_args)
Apr 27 00:45:40 vagrant docker-puppetboard[704]:   File "/usr/src/app/puppetboard/app.py", line 864, in metrics
Apr 27 00:45:40 vagrant docker-puppetboard[704]:     metrics_domains = get_or_abort(puppetdb.metric)
Apr 27 00:45:40 vagrant docker-puppetboard[704]:   File "/usr/src/app/puppetboard/utils.py", line 99, in get_or_abort
Apr 27 00:45:40 vagrant docker-puppetboard[704]:     return func(*args, **kwargs)
Apr 27 00:45:40 vagrant docker-puppetboard[704]:   File "/usr/local/lib/python3.8/site-packages/pypuppetdb/api.py", line 927, in metric
Apr 27 00:45:40 vagrant docker-puppetboard[704]:     raise DoesNotComputeError(res['error'])
Apr 27 00:45:40 vagrant docker-puppetboard[704]: pypuppetdb.errors.DoesNotComputeError: java.lang.Exception : No access from client 172.17.0.2 allowed

[oldNoakes]

We ran into this and found this documentation: https://puppet.com/docs/puppetserver/latest/metrics-api/v2/metrics_api.html#configuring-jolokia which seemed to align with the errors we were seeing on the puppetdb side (access not allowed for client).

That said, we were unable to make this work - if anyone else gets this working and can provide a working example, please could you post? Is this where we should be looking?

[oldNoakes]

ok - we found a solution for us. Not terribly well documented but the breadcrumbs are all there.

1. puppetdb uses a different jolokia-access.xml file than puppetserver
2. We had to copy the configuration we had from /etc/puppetlabs/puppetserver/conf.d/metrics.conf to /etc/puppetlabs/puppetdb/conf.d/metrics.conf (created the conf.d folder along the way as it didn't exist

File looks as follows: /etc/puppetlabs/puppetdb/conf.d/metrics.conf

# settings related to metrics
metrics: {
    # a server id that will be used as part of the namespace for metrics produced
    # by this server
    server-id: puppet.prod.net
    registries: {
        puppetserver: {
            # specify metrics to allow in addition to those in the default list
            #metrics-allowed: ["compiler.compile.production"]

            reporters: {
                # enable or disable JMX metrics reporter
                jmx: {
                    enabled: true
                }
                # enable or disable Graphite metrics reporter
                graphite: {
                    enabled: true
                }
            }
        }
    }

    # this section is used to configure settings for reporters that will send
    # the metrics to various destinations for external viewing
    reporters: {
        graphite: {
            # graphite host
            host: "localhost"
            # graphite metrics port
            port: 2003
            # how often to send metrics to graphite
            update-interval-seconds: 5
        }
    }
    metrics-webservice: {
        jolokia: {
            # Enable or disable the Jolokia-based metrics/v2 endpoint.
            # Default is true.
            # enabled: false

            # Configure any of the settings listed at:
            #   https://jolokia.org/reference/html/agents.html#war-agent-installation
            servlet-init-params: {
                # Specify a custom security policy:
                #  https://jolokia.org/reference/html/security.html
                policyLocation: "file:///etc/puppetlabs/puppetdb/jolokia-access.xml"
                debug: "true"
                allowErrorDetails: "false"
            }
        }
    }
}

and then added the following file and contents: /etc/puppetlabs/puppetdb/jolokia-access.xml

<?xml version="1.0" encoding="utf-8"?>

<restrict>
  <remote>
    <host>127.0.0.1</host>
    <host>a.b.c.d</host>
  </remote>

  <commands>
    <command>read</command>
    <command>list</command>
    <command>version</command>
    <command>search</command>
  </commands>

  <mbean>
    <name>puppetlabs.puppetdb.population:name=num-nodes</name>
    <attribute mode="read">Value</attribute>
    <operation>objectName</operation>
  </mbean>
</restrict>

where a.b.c.d is the ip of the server running puppetboard.

Restarted puppetdb and everything now works - hope this helps someone out there.

[bgibson710]

I can confirm that the information provided above fixed our issue, and everything is working as expected after making these changes. We probably just need the documentation updated to include these changes for the v2 API.

[pjamenaja]

Thanks a lot guys this helped me solved the issue.

My puppetdb and puppetboard are in the same pod so I set the host in /etc/puppetlabs/puppetdb/jolokia-access.xml to 0.0.0.0/0 for simplicity.

      <remote>
        <host>0.0.0.0/0</host>
      </remote>

[raphink]

Note: we're still seeing this with PuppetDB 7, and it seems to be because there is now an auth.conf for PuppetDB which prevents unauthenticated requests to /metrics.

[gdubicki]

Can someone please provide a PR to add the required info to the app docs? @oldNoakes perhaps?

[blodone]

in /etc/puppetlabs/puppetdb/conf.d/auth.conf we had to set allow: * to allow-unauthenticated: true

for name: "puppetlabs puppetdb metrics"

restart puppetdb and puppetboard does not show Forbidden 403 anymore

[jcpunk]

For folks using the containers, I've opened https://tickets.puppetlabs.com/browse/PDB-5522 to give a place for this change to be done automatically.