Explore exposing monitoring Data with JMX and the Google Cloud Agent

[ludoch]

[gregw]

To start the discussion, I'll give a bit of background to the Jetty JMX mechanism.

It can be enabled by adding the jmx module to jetty (--module=jmx or --add-to-startd=jmx or ask and I'll add to the docker images).

Jetty's architecture is essentially a collection of beans rooted the the Server instance which extends our ContainerLifeCycle. With JMX enabled, any bean added to the Server or any of the bean containers it contains will be inspected to see if an MBean can be created for it. We indicate MBeans with our @ManagedObject class annotation, so that any instance or derived instance will be seen as a managed object.

Individual fields/methods can then be annotated with @ManagedAttribute/@ManagedOperation which also provides some documentation available via JMX.

All core Jetty classes are instrumented in this way, so you can see what connectorrs/handlers a server contains and what servlets the ServletHandler contains etc. etc. Every lifecycle object can be seen to be starting/started/stopping/stopped etc and key configuration (eg port) is also exposed.

Note that this is read/write, so via JMX you can call connector.stop() connector.setPort(8888) connector.start() to change the port. Some configuration can even be changed without stop/start (eg size of the thread pool).

So while you may wish to hide the write side of these MBeans, the read side is very good for observing structure, configuration and state.

The we also have a statistics (enabled with --module=stats or --add-to-startd=stats) which collects request/connections statistics and makes them available via JMX through http://download.eclipse.org/jetty/stable-9/apidocs/org/eclipse/jetty/server/handler/StatisticsHandler.html and http://download.eclipse.org/jetty/stable-9/apidocs/org/eclipse/jetty/server/ConnectorStatistics.html

We could enable these all the time, or with the new images in the jetty93 branch the --modules=jmx,stats can be passed in with the docker CMD