Report memory usage and GC stats to Stackdriver Monitoring

[meltsufin]

[pettyjamesm]

Pretty much everything in the stackdriver JVM plugin should be sent. What's the lift on making that happen? From the stackdriver docs, they make it sound like you just have to drop in the collectd plugin and wire up the JMX port.

[meltsufin]

Do you have a link to the documentation you're referring to? Which Stackdriver JVM plugin?

[pettyjamesm]

https://cloud.google.com/monitoring/agent/plugins/jvm

[meltsufin]

Thanks! Have you tried enabling it using a custom Dockerfile based on this runtime and JAVA_USER_OPTS?

[pettyjamesm]

I played around with trying to get it to work, but couldn't- to the best of my knowledge it's not possible without special support from the AppEngine flexible environment setup on the physical instances. I want to underscore that this is "to the best of my knowledge", but here's what seems to be the case:

1. The stackdriver monitoring agent (collectd, specifically) runs outside of the docker container on the host itself
2. That collectd instance would need the collectd plugin configuration added to it's plugin directory, and since that directory exists on the native host there is no way to achieve that via a customer Dockerfile alone
3. Even with that plugin configured (which I tried by SSH'ing into a debug instance), the jvm plugin uses JMX to connect to the JVM being monitored, and therefore needs to be able to locate shared library (libjvm.so) for which the only reasonable place to find that is inside the container's JVM installation directory.

The rest of the setup should be fairly mechanical, setting JAVA_USER_OPTS to allow JMX connections on a specified port isn't a big deal- it's just that there doesn't seem to be any way to work around the issues above to bridge the containerized JVM to the host's collectd agent without some support from the host's setup.

[meltsufin]

Thanks for a detailed analysis. I think you're right on all the points. It basically comes down to the fact that the JVM Monitoring plugin is not supported on App Engine flexible VMs. So, it really is out of scope for this project because it's isolated to the gaeapp container in the VM.

[pettyjamesm]

Is there somewhere I can go other than GCP Support to track down getting stackdriver support? There seems to be a big documentation gap between stackdriver and app engine flexible. Is there a component somewhere responsible for bridging the App Engine Flexible compute instance to container that this kind of feature makes sense? For example, how are custom logs from the container picked up by the stackdriver logging agent?

[meltsufin]

I would recommend GCP Support as the best path to get visibility for your request.