Stream-JMX

NOTE: tnt4j-stream-jmx version 2.x migrated to Java 11 and bumped SLF4J from 1.7.x to 2.x.

Latest Java 8 compliant tnt4j-stream-jmx version is 1.18.x.

NOTE: Stream-JMX version 0.14 migrated logger to log4j2. In case you have changed default Stream-JMX logger configuration using previous Stream-JMX versions, please check new logger configuration file log4j2.xml.

Major `Log4j12` to `Log4j2` migration notes to consider

Log4j2 supports configuration defined in XML, JSON, YAML and properties formats (NOTE: Log4j2 XML and properties formats differs from ones used by Log4j12). Previous Log4j12 log4j.properties file defined configuration shall be migrated into one of these new configuration definition formats.
Log4j2 changed configuration file definition System property name from log4j.configuration to log4j2.configurationFile.
See Log4j 2.x migration guide and Log4j 2.x configuration reference for more details
Log4j2 used asynchronous logging techniques noticeably increases application performance comparing to Log4j12. See Log4j 2.x Asynchronous Logging Performance report as reference.

A Lightweight framework to stream and monitor JMX metrics. (Formerly known as PingJMX)

Stream JMX metrics to:

Central monitoring server
File, socket, log4j
User defined destination

These metrics can be used to monitor health, performance and availability of your JVMs and applications. Use Stream-JMX to embed a monitoring agent within your application and monitor memory, GC activity, CPU as well as user defined MBeans.

Here is what you can do with Stream-JMX:

Periodic JVM heartbeat
Monitor memory utilization, GC activity, memory leaks
High/Low, normal vs. abnormal CPU usage
Monitor threading, runtime and other JVM performance metrics
Monitor standard and custom MBean attributes
Conditional actions based on MBean attribute values
Conditional streaming based on custom filters
Application state dumps on VM shut-down for diagnostics

Why Stream-JMX

Stream-JMX provides and easy, lightweight and secure way to stream and monitor JMX metrics from within java runtime containers.

Stream JMX metrics out of the JVM container (vs. polling from outside/remote)
Makes it easy to monitor farms of JVMs, application servers
Reduce cyber-security risk: No need to enable remote JMX, SSL, security, ports, firewalls
Integration with monitoring tools for alerting, proactive monitoring (AutoPilot M6)
Integration with cloud analytics tools (https://www.meshiq.com via JESL)
Integration with log4j, slf4j, jKoolCloud (via TNT4J event sinks)
Embedded application state dump framework for diagnostics
Easily build your own extensions, monitors

NOTE: JESL provides a way to stream events generated by Stream-JMX to meshIQ. For more information on JESL visit: JESL project on GitHub.

Using Stream-JMX

It is simple, do one of the following:

run Stream-JMX as a -javaagent
attach Stream-JMX as agent to running JVM
connect Stream-JMX over JMXConnector to locally running JVM or remote JMX service

embed Stream-JMX code into your application, by using this Maven dependency:

    <dependency>
        <groupId>com.jkoolcloud.tnt4j.stream</groupId>
        <artifactId>tnt4j-stream-jmx-core</artifactId>
        <version>2.1.1</version>
    </dependency>

NOTE: Running Stream-JMX as -javaagent, attaching agent to running JVM or connecting over JMXConnector to locally running JVM or remote JMX service over RMI connection can be invoked without changing your application code.

NOTE: When you want to sample JMX inter-process communication way (providing -connect or -attach mode -vm parameter value as PID/process name), then Stream-JMX runner and sampled process VMs MUST be:

built by same vendor (e.g. IBM, Oracle, OpenJDK)
of same architecture (x86, x64)

Stream-JMX runner VM version higher or equal compared to sampled VMs version - e.g. trying to sample JVM 10+ using JVM 8 may produce exceptions like this:

java.io.IOException: Non-numeric value found - int expected
  at sun.tools.attach.HotSpotVirtualMachine.readInt(HotSpotVirtualMachine.java:299)
  at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:63)
  at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:79)
  at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:103)
  at com.sun.tools.attach.VirtualMachine.loadAgent(VirtualMachine.java:540)

check if JDK attach tooling is enabled and bound:
- for Java versions prior 9, check which JDK tools.jar is referred over LIBPATH variable, it also must match runner JVM
- for Java versions 9+, check if jdk.attach module is enabled over --add-opens and --add-exports arguments:
```
java --add-exports=jdk.attach/sun.tools.attach=ALL-UNNAMED --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED ...
```

If JVM vendors, architecture or versions does not match conditions above - then only way to collect JMX samples is using -connect mode defining JMXConnector (over RMI) URL.

NOTE: when running stream-jmx sometimes it may appear some Unsupported type exceptions in console/log, e.g.:

Failed to sample:
 ojbName=java.lang:type=MemoryPool,name=PS Eden Space,
 info=javax.management.MBeanAttributeInfo[description=UsageThresholdExceeded, name=UsageThresholdExceeded, type=boolean, read-only, isIs, descriptor={openType=javax.management.openmbean.SimpleType(name=java.lang.Boolean), originalType=boolean}],
 exclude=false,
 ex=javax.management.RuntimeMBeanException: java.lang.UnsupportedOperationException: Usage threshold is not supported
javax.management.RuntimeMBeanException: java.lang.UnsupportedOperationException: Usage threshold is not supported
    ... 25 more
Caused by: java.lang.UnsupportedOperationException: Usage threshold is not supported
    ... 27 more

It is OK, since some MBeans do not provide some attributes depending on Operating System or other run environment setup. If you ran JConsole you'll get values Unavailable for such attributes painted in red (e.g. see java.lang -> MemoryPool -> PS Eden Space attribute UsageThresholdExceeded). These Unsupported exceptions are shown on first iteration of sampling for user to know that all MBean attributes where accessed.

Common command line arguments

Arguments definition:

-vm: - virtual machine descriptor. It can be PID or JVM process name fragment. * value is wildcard to pick all found VMs, running on local machine.
-ao: - agent options string using ! symbol as delimiter. Options format: mbean-filter!exclude-filter!sample-ms!init-delay-ms!batch-size
- mbean-filter - MBean include name filter defined using object name pattern: domainName:keysSet. NOTE: Multiple filters can be defined using ; as delimiter: domainName1:keysSet1;domainName2:keysSet2;domainName3:keysSet3. Default is *:*.
- exclude-filter - MBean exclude name filter defined using object name pattern: domainName:keysSet. NOTE: Multiple filters can be defined using ; as delimiter: domainName1:keysSet1;domainName2:keysSet2;domainName3:keysSet3. Optional, default is ``.
- sample-ms - MBeans sampling rate in milliseconds. Optional, Default is 30sec.,
- init-delay-ms - MBeans sampling initial delay in milliseconds. Optional, by default it is equal to sample-ms value.
- batch-size - number of sampled MBeans to post over single package. Optional, default is -1 (unlimited).
-cp: - JMX connection parameter string using = symbol as delimiter. Defines only one parameter, to define more than one use this argument multiple times. Argument format: -cp:key=value. See Java SE monitoring and management documentation for more details.
-slp: - sampler parameter string using = symbol as delimiter. Defines only one parameter, to define more than one use this argument multiple times. Argument format: -slp:key=value
- trace - flag indicating whether the sample listener should print trace entries to print stream. Default value - false.
- forceObjectName - flag indicating to forcibly add objectName attribute if such is not present for a MBean. Default value - false.
- addStatisticMetadata - flag indicating to add J2EE metadata entries as attributes for a MBean. Default value - true.
- compositeDelimiter - delimiter used to tokenize composite/tabular type MBean properties keys. Default value - \.
- useObjectNameProperties - flag indicating to copy MBean ObjectName contained properties into sample snapshot properties. Default value - true.
- excludeOnError - flag indicating to auto-exclude failed to sample attributes. Default value - false.
- excludedAttributes - list of user chosen attribute names (may have wildcards * and ?) to exclude, pattern: attr1,attr2,...,attrN@MBean1_ObjectName;...;attr1,attr2,...,attrN@MBeanN_ObjectName.
-sp: - sampler system property string using = symbol as delimiter. Defines only one system property, to define more than one use this argument multiple times. Argument format: -sp:key=value.
-ssl - flag indicating to disable SSL verification in case SSL is misconfigured: expired, self-signed, or domain mismatching certificate.

Running Stream-JMX as `-javaagent`

Command line to run

Executable OS shell run script files bin/stream-jmx.bat or bin/stream-jmx.sh are dedicated to do the job.

Command line to run stream-jmx as JVM agent looks like this:

java -javaagent:tnt4j-stream-jmx-core-all.jar="*:*!30000" -Dtnt4j.config=tnt4j.properties -Dlog4j2.configurationFile=log4j2.xml -classpath "tnt4j-stream-jmx.jar;lib/*" your.class.name your-args

The options are -javaagent:tnt4j-stream-jmx-core-all.jar="mbean-filter!sample-time-ms", classpath must include tnt4j-stream-jmx jar files as well as locations of log4j and tnt4j configuration files. See JMX Sampling Agent sampler options for details.

NOTE: in case your Java runtime is version 9+, add these Java command line arguments:

--add-exports=java.base/jdk.internal.loader=ALL-UNNAMED --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED

Attaching Stream-JMX to running JVM

Command line to run

Executable OS shell run script files bin/stream-jmx-attach.bat or bin/stream-jmx-attach.sh are dedicated to do the job.

Command line to attach local JVM process JMX looks like this:

java -Dtnt4j.config=.\config\tnt4j.properties -Dlog4j2.configurationFile=.\config\log4j2.xml -classpath "tnt4j-stream-jmx-core-all.jar;lib/*" com.jkoolcloud.tnt4j.stream.jmx.SamplingAgent -attach -vm:activemq -ap:tnt4j-stream-jmx-core-all.jar -ao:*:*!10000

System properties -Dxxxxx defines Stream-JMX configuration. For details see Stream-JMX configuration .

SamplingAgent arguments -attach -vm:activemq -ap:tnt4j-stream-jmx-core-all.jar -ao:*:*!10000 states:

-attach - defines that SamplingAgent shall be attached to running JVM process
-vm:activemq - is JVM descriptor. In this case it is running JVM name fragment activemq. But it also may be JVM process identifier - PID. Mandatory argument.
-ap:tnt4j-stream-jmx-core-all.jar - is agent library name. If it is class path - then only name should be sufficient. In any other case define full or relative path, e.g., ..\build\tnt4j-stream-jmx\tnt4j-stream-jmx-2.1.1\tnt4j-stream-jmx-core-all.jar. Mandatory argument.
-ao:*:*!10000 - is JMX sampler options stating to include all MBeans and schedule sampling every 10 seconds. Sampler options are optional - default value is *:*!30000. Initial sampler delay can be configured by adding numeric parameter *:*!30000!1000 defining initial sampler delay as 1 second. Default sampler delay value is equal to sampling period value. See JMX Sampling Agent sampler options for details.
-slp: - any JMX sampler configuration property. See Program arguments used for details.
-sp: - any system property used by sampler. See System properties used for details.

NOTE: arguments and properties defined running SamplingAgent.main is forwarded to SamplingAgent agent attached to JVM process.

NOTE: if you get exception com.sun.tools.attach.AgentInitializationException: Agent JAR loaded but agent failed to initialize while running stream-jmx in -attach mode it is most likely that some stream-jmx used classes are not resolved in attach JVM class path. Pay attention to use tnt4j-stream-jmx-core-[VERSION]-all.jar as agent lib to have all dependent classes available within one jar. If you are trying to attach application implementing J2EE API, use tnt4j-stream-jmx-j2ee-[VERSION]-all.jar. Same approach should be followed when attaching WAS and Liberty instances - use -all.jar for matching product API.

Coding into API

You can attach SamplingAgent to JVM from your custom API by calling SamplingAgent.attach(String,String,String) method.

Sample attach call:

try {
  SamplingAgent.attach("activemq", "tnt4j-stream-jmx-core-all.jar", "*:*!10000");
} catch (Exception exc) {
  exc.printStackTrace();
}

Connecting Stream-JMX to local or remote JMX service

Command line to run

Executable OS shell run script files bin/stream-jmx-connect.bat or bin/stream-jmx-connect.sh are dedicated to do the job:

stream-jmx-connect script has 4 parameters:

process id or service URI (required)
agent options for MBeans include/exclude filter and sampling interval (optional, . value sets default)
service identifier for the process/service being monitored (optional, . value sets default)
sampling agent arguments (optional, . value sets default)

MS Windows

rem using URL
/bin/stream-jmx-connect.bat service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi
rem using URL with connection parameters
/bin/stream-jmx-connect.bat service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi . . -ul:admin -up:admin -cp:java.naming.security.authentication=simple -cp:java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory
rem using process name part
/bin/stream-jmx-connect.bat activemq
rem using pid
/bin/stream-jmx-connect.bat 1553

*nix

# using URL
./bin/stream-jmx-connect.sh service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi
# using URL with connection parameters
./bin/stream-jmx-connect.sh service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi . . -ul:admin -up:admin -cp:java.naming.security.authentication=simple -cp:java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory
# using process name part
./bin/stream-jmx-connect.sh activemq
# using pid
./bin/stream-jmx-connect.sh 1553

To connect to local JVM process

Command line to connect local JVM process JMX looks like this:

java -Dtnt4j.config=.\config\tnt4j.properties -Dlog4j2.configurationFile=.\config\log4j2.xml -classpath "tnt4j-stream-jmx-core.jar;lib/*" com.jkoolcloud.tnt4j.stream.jmx.SamplingAgent -connect -vm:activemq -ao:*:*!*:dummy!10000

System properties -Dxxxxx defines Stream-JMX configuration. For details see Stream-JMX configuration .

SamplingAgent arguments -connect -vm:activemq -ao:*:*!*:dummy!10000 states:

-connect - defines that SamplingAgent shall connect to running JVM process over JMXConnector (RMI) connection.
-vm:activemq - is JVM descriptor. In this case it is running JVM name fragment activemq. But it also may be JVM process identifier - PID. Mandatory argument.
-ao:*:*!*:dummy!10000 - is JMX sampler options stating to include all MBeans, exclude all dummy MBeans and schedule sampling every 10 seconds. Sampler options are optional - default value is *:*!30000. Initial sampler delay can be configured by adding numeric parameter *:*!30000!1000 defining initial sampler delay as 1 second. Default sampler delay value is equal to sampling period value. See JMX Sampling Agent sampler options for details.
-slp: - any JMX sampler configuration property. See Program arguments used for details.
-sp: - any system property used by sampler. See System properties used for details.

To connect to JMX service over URL

Command line to connect remote JMX service looks like this:

java -Dtnt4j.config=.\config\tnt4j.properties -Dlog4j2.configurationFile=.\config\log4j2.xml -classpath "tnt4j-stream-jmx-core.jar;lib/*" com.jkoolcloud.tnt4j.stream.jmx.SamplingAgent -connect -vm:service:jmx:<JMX_URL> -ul:admin -up:admin -ao:*:*!!10000 -cri:30 -cp:java.naming.security.authentication=simple -cp:java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory

System properties -Dxxxxx defines Stream-JMX configuration. For details see Stream-JMX configuration .

SamplingAgent arguments -connect -vm:service:jmx:<JMX_URL> -ul:admin -up:admin -ao:*:*!!10000 -cri:30 -cp:java.naming.security.authentication=simple -cp:java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory states:

-connect - defines that SamplingAgent shall connect to running JMX service over JMXConnector (RMI) connection.
-vm:service:jmx:<JMX_URL> - is JMX service URL to use for connection. Mandatory argument. Full URL may be like service:jmx:rmi://127.0.0.1/stub/rO0ABXN9AAAAAQAlamF2YXgubWFuYWdlbWVudC5yZW1vdGUucm1pLlJNSVNlcnZlcnhyABdqYXZhLmxhbmcucmVmbGVjdC5Qcm94eeEn2iDMEEPLAgABTAABaHQAJUxqYXZhL2xhbmcvcmVmbGVjdC9JbnZvY2F0aW9uSGFuZGxlcjt4cHNyAC1qYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN0SW52b2NhdGlvbkhhbmRsZXIAAAAAAAAAAgIAAHhyABxqYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN002G0kQxhMx4DAAB4cHc2AAtVbmljYXN0UmVmMgAACzE3Mi4xNi42Ljg2AADPWKO5DJD/bZIhG9aBuwAAAVo8DdAkgAEAeA== .
-ul:admin - is user login. In this case it is admin. User login argument is optional.
-up:admin - is user password. In this case it is admin. User password argument is optional.
-ao:*:*!!10000 - is JMX sampler options stating to include all MBeans and schedule sampling every 10 seconds. Sampler options are optional - default value is *:*!30000. Initial sampler delay can be configured by adding numeric parameter *:*!30000!1000 defining initial sampler delay as 1 second. Default sampler delay value is equal to sampling period value. See JMX Sampling Agent sampler options for details.
-cp:java.naming.security.authentication=simple -cp:java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory - is JMX connector parameters definitions in properties format key=value. JMX connector parameters are optional and can be defined multiple times - as many as there are required JMX connector parameters. See Java API Context class documentation for available properties naming. NOTE: If you are using some API extending JNDI, check documentation if it provides some additional connection configuration properties.
-cri:30 - is connection retry interval in seconds. In this case it is 30sec between connect retry attempts. Connection retry interval is optional - default value is 10sec. Special values are:
- 0 indicates no delay between repeating connect attempts.
- -1 indicates no repeating connect attempts shall be made at all and application has to stop on first failed attempt to connect.
-slp: - any JMX sampler configuration property. See Program arguments used for details.
-sp: - any system property used by sampler. See System properties used for details.

NOTE:

URI of remote RMI service (e.g., to connect remote Kafka service) may require additional / chars:

-vm:service:jmx:rmi:///jndi/rmi:///[HOST]:[PORT]/jmxrmi

e.g.:

-vm:service:jmx:rmi:///jndi/rmi:///172.16.6.35:2181/jmxrmi

Remote end may require to enable JMX listening port in service configuration to accept connections (e.g., to connect remote Kafka service):
```
export JMX_PORT=${JMX_PORT:-9999}
```
Tomcat and Kafka does not provide J2EE implementation, thus you need only stream-jmx core jar in classpath when sampling Tomcat/Kafka metrics over JMX. Executable OS shell run script files uses only core as MODULE_SET variable value:
- bin/stream-jmx-connect.bat
```
set MODULE_SET=core
```
- bin/stream-jmx-connect.sh
```
MODULE_SET=("core")
```
See Enabling Tomcat JMX Remote to enable remote JMX access of Tomcat server instance. If your Tomcat server is behind firewall - also refer to JMX Remote Lifecycle Listener on how to set up such Tomcat server instance to be accessible from outside.
See ActiveMQ JMX how to enable remote ActiveMQ JMX access.

Multiple VMs monitoring over single stream run

Multiple local JVM processes monitoring

If sampling agent finds multiple VM's matching provided JVM descriptor, it will bind and sample JMX beans from all these VM's, e.g.:

./bin/stream-jmx-connect.sh kafka

sampler will collect JMX beans for all local machine running VM's containing kafka token within VM process name.

Multiple VM's monitoring configuration using external file

Dedicated SamplingAgent arguments:

-f: - defines path for external configuration file, containing set of VM connection definitions.

You can configure multiple VM's connections using external configuration file. Sample configuration file is ./config/connections.cfg. The syntax for that file is:

##############################################################################################################################################################
#            VM connection string                      #    Agent options   #  User name    #    Password       #                Source addition             #
##############################################################################################################################################################
 service:jmx:rmi:///jndi/rmi://localhost:9995/jmxrmi       *:*!!60000           admin             admin           SERVICE=@bean:org.apache.ZooKeeperService:name0=*/?ClientPort#SERVER=@bean:java.lang:type=Runtime/?Name
 service:jmx:rmi:///jndi/rmi://localhost:9996/jmxrmi       *:*!!60000           admin             admin           SERVICE=@bean:java.lang:type=Runtime/?Name#SERVER=@bean:kafka.server:id=?,type=app-info#DATACENTER=@bean:kafka.server:type=KafkaServer,name=ClusterId/?Value
 service:jmx:rmi:///jndi/rmi://localhost:9997/jmxrmi       *:*!!60000           admin             admin           SERVICE=@bean:java.lang:type=Runtime/?Name#SERVER=@bean:kafka.server:type=app-info,id=?#DATACENTER=@bean:kafka.server:type=KafkaServer,name=ClusterId/?Value
##############################################################################################################################################################

To run JMX samples streaming for multiple VM's define in external configuration file use stream-jmx-connect-file-config.bat (.sh) files, e.g.: *nix

./bin/stream-jmx-connect-file-config.sh ./config/connections.cfg

MS Windows

\bin\stream-jmx-connect-file-config.bat \config\connections.cfg

Connections configuration using stanza divided properties file

When connections configuration gets too complex to be defined using tokenized lines, there is possibility to define connections using properties divided into stanzas. One stanza defines one VM descriptor (connection) properties, like this:

##############################################################################################################################################################
{
    zk.vm:                  service:jmx:rmi:///jndi/rmi://localhost:9995/jmxrmi
    zk.vm.user:             admin
    zk.vm.pass:             admin
    zk.vm.reconnect.sec:    10
    zk.agent.options:       java.lang:*!!60000
    zk.source.fqn:          SERVICE=@bean:org.apache.ZooKeeperService:name0=*/?ClientPort#SERVER=@bean:java.lang:type=Runtime/?Name
}
##############################################################################################################################################################
{
    kafka.vm:               kafka:zk://127.0.0.1:2181
    kafka.vm.user:          admin
    kafka.vm.pass:          admin
    kafka.vm.reconnect.sec: 10
    kafka.agent.options:    java.lang:*!!60000
    kafka.source.fqn:       SERVICE=@bean:java.lang:type=Runtime/?Name#SERVER=@bean:kafka.server:id=?,type=app-info#DATACENTER=@bean:kafka.server:type=KafkaServer,name=ClusterId/?Value
}
##############################################################################################################################################################
{
    solr.vm:                 solr:zk://172.16.6.208:2181
    solr.vm.user:            admin
    solr.vm.pass:            admin
    solr.vm.reconnect.sec:   10
    solr.agent.options:      java.lang:*!!5000
    solr.source.fqn:         SERVER=@bean:solr:dom1=core,dom2=?,dom3=*,reporter=*,category=*,scope=core,name=*
    solr.custom.urlPattern:  service:jmx:rmi:///jndi/rmi://{0}:{1,number,######}/jmxrmi
    solr.custom.port:        18983
}
##############################################################################################################################################################

There are three VM descriptors defined - zk, kafka and solr. As you see first property key token (delimited using .) means VM descriptor name. You can name it freely, like solrGroup1, myZooKeeeper or kafkaCluster1. Rest is used to define particular VM descriptor properties:

vm or vm.url - defines VM or (VM orchestrator instance) URL.
vm.user - defines VM connection used user name.
vm.pass or vm.password - defines VM connection used user password.
vm.reconnect.sec - defines period in seconds to reconnect if VM connection gets interrupted. Negative value means do not try to reconnect.
agent.options - defines agent sampling options.
source.fqn - defines TNT4J Source FQN values mapping used by VM descriptor.
custom - defines group of additional custom VM descriptor properties:
- urlPattern - defines JMX service URL pattern to be filled in by ZooKeeper orchestrated VMs resolver. Default value - service:jmx:rmi:///jndi/rmi://{0}:{1,number,######}/jmxrmi, where {0} is replaced with JMX service runner machine host name/IP and {1} is replaced with JMX service runner machine port number.
- port - defines JMX service port number to be used by ZooKeeper orchestrated VMs resolver. Some services (like Apache Solr) does not provide JMX service port to ZooKeeper, so it is needed to define it manually. Apache Kafka for instance - does provide JMX connection port, so there is no need to define it additionally - it is picked automatically from ZooKeeper node provided data.

You can use variable expressions within vm or vm.url property definition. Array values are delimited using ; symbol. For example:

    zk.vm:                 service:jmx:rmi:///jndi/rmi://${vm.host}:${vm.port}/jmxrmi
    zk.vm.host:            172.16.6.208
    zk.vm.port:            9995;9996;9997

will produce 3 VM definitions by combining defined single vm.host value and array of 3 vm.port values:

    service:jmx:rmi:///jndi/rmi://172.16.6.208:9995/jmxrmi
    service:jmx:rmi:///jndi/rmi://172.16.6.208:9996/jmxrmi
    service:jmx:rmi:///jndi/rmi://172.16.6.208:9997/jmxrmi

Yet if size of both vm.host and vm.port values are same, like this:

    zk.vm:                 service:jmx:rmi:///jndi/rmi://${vm.host}:${vm.port}/jmxrmi
    zk.vm.host:            172.16.6.208;172.16.6.210;172.16.6.212
    zk.vm.port:            9995;9996;9997

then it will produce 3 VM definitions by mapping array values 1:1 as this:

    service:jmx:rmi:///jndi/rmi://172.16.6.208:9995/jmxrmi
    service:jmx:rmi:///jndi/rmi://172.16.6.210:9996/jmxrmi
    service:jmx:rmi:///jndi/rmi://172.16.6.212:9997/jmxrmi

Another way to get same result would be as this:

    zk.vm:                 service:jmx:rmi:///jndi/rmi://${vm.address}/jmxrmi
    zk.vm.address:         172.16.6.208:9995;172.16.6.210:9996;172.16.6.212:9997

Multiple VM's monitoring notes for Source

See TNT4J Source fields configuration as base Source fields configuration reference.

Multiple VM's monitoring requires additional data Source configuration when you want to distinguish where JMX data came from.

When all monitored VM's can uniquely identify itself using same SourceFQN by using dynamic MBean property value (common when monitoring same service on different machines), e.g.

    source.factory.SERVICE: @bean:org.apache.activemq:type=Broker,brokerName=localhost/?BrokerId
    source.factory.SERVER: @bean:java.lang:type=Runtime/?Name
    source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GENERIC=?

then no additional source configuration is required.

But when your services ecosystem is widely distributed running different services on different machines (or multiple different services on same machine), every service identifies itself over different MBean property. In this case you can configure source RootFQN in tnt4j.properties and define additional SourceFQN path fragment for every monitored VM. This way final VM MBeans sample will have SourceFQN made by Stream-JMX by concatenating tnt4j.properties defined RootFQN with that additional VM SourceFQN path fragment.

For example, when tnt4j.properties defines RootFQN as:

    source.factory.GENERIC: Streams
    source.factory.DATACENTER: HQDC
    source.factory.SERVER: @bean:java.lang:type=Runtime/?Name
    source.factory.RootFQN: SERVER=?#DATACENTER=?#GENERIC=?

and having VMs configured this way:

##############################################################################################################################################################
#            VM connection string                      #    Agent options   #  User name    #    Password       #                Source addition             #
##############################################################################################################################################################
 service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi        *:*!!60000          admin           admin              SERVICE=@bean:org.apache.activemq:type=Broker,brokerName=localhost/?BrokerId
 service:jmx:rmi:///jndi/rmi://192.168.1.1:9998/jmxrmi      *:*!!30000          admin           admin              SERVICE=@bean:kafka.server:id=?,type=app-info 
##############################################################################################################################################################

may build such SourceFQNs for VMs:

localhost - `SERVICE=ID:PC-NAME-52295-1538060220413-0:1#SERVER=2453@PC-NAME#DATACENTER=HQDC
192.168.1.1 - `SERVICE=3SERVER=2457@boxName#DATACENTER=HQDC

NOTE: VM additional SourceFQN path must be defined fallowing same rules as in TNT4J config:

SourceType1=value1#SourceType2=value2#...#SourceTypeN=valueN

ZooKeeper orchestrated VMs access

Stream-JMX has ability to resolve JMX service connections for VMs orchestrated by ZooKeeper. In this case you'll need:

Use tnt4j-stream-jmx-zk module dependency (for development) or use built jar of this module in your JVM classpath (for deployment). Also see bin/stream-jmx-connect.bat (.sh) files for commented out variable MODULE_SET definition combinations.
Use predefined VM descriptor prefixes to initiate VMs JMX connection resolution over running ZooKeeper service instance:
- kafka:zk://[ip]:[port] - for Apache Kafka VMs access. Stream-JMX will monitor /broker/ids node for registered brokers and will connect to all JMX enabled (exposing JMX port not equal -1) brokers.
- solr:zk://[ip]:[port] - for Apache Solr VMs access. Stream-JMX will monitor /live_nodes node for registered Solr node instances. NOTE: Solr does not expose JMX port, so you need to specify JMX port number as Other options column value port=[port number].

NOTE: ZooKeeper service instance JMX sampling does not start automatically defining VM descriptors for Apache Kafka or Apache Solr. In case you want to sample JMX of referred ZooKeeper instance, you have to define ZooKeeper runner VM dedicated JMX service URL.

NOTE: When Stream-JMX binds to ZooKeeper node, it monitors for changes under that node. So if new brokers gets registered/unregistered, Stream-JMX connects/disconnects JMX services of these brokers on runtime.

Sample configuration:

############################################################################################################################################################################################################################################
#            VM connection string                         #    Agent options        #  User name    #    Password       #                Source addition                                                        #      Custom options      #
############################################################################################################################################################################################################################################
 service:jmx:rmi:///jndi/rmi://localhost:9995/jmxrmi        *:*!!60000                 admin            admin              SERVICE=@bean:org.apache.ZooKeeperService:name0=*/?ClientPort#SERVER=@bean:java.lang:type=Runtime/?Name
 kafka:zk://127.0.0.1:2181                                  *:*!!60000                 .                .                  SERVICE=@bean:java.lang:type=Runtime/?Name#SERVER=@bean:kafka.server:id=?,type=app-info#DATACENTER=@bean:kafka.server:type=KafkaServer,name=ClusterId/?Value
 solr:zk://172.16.6.208:2181                                java.lang:*!!5000          .                .                  SERVER=@bean:solr:dom1=core,dom2=?,dom3=*,reporter=*,category=*,scope=core,name=*         port=18983
############################################################################################################################################################################################################################################

Having first line we define that we want to sample JMX of ZooKeeper VM where Kafka brokers registers.

Second line we initiate Stream-JMX use ZooKeeper instance to collect JMX service connections for all registered Kafka brokers.

In third line we initiate Stream-JMX use ZooKeeper instance to collect JMX service connections for all registered Solar node instances. Note, we also define Solr JMX connection port there - port=18983.

NOTE: if some configuration token needs to be omitted (like some user names or passwords in our configuration), you can use . symbol as configuration token placeholder.

How to migrate Source definitions from single to multi VM sampling

When you where sampling JMX running as many instances of Stream-JMX as there were monitored JVMs, you where defining complete SourceFQN within source.factory.RootFQN property of Stream-JMX TNT4J configuration.

When monitoring multiple JVMs running single instance of Stream-JMX, source.factory.RootFQN property should define only part of SourceFQN common for all monitored JVMs. Remaining (JVM unique) part of SourceFQN shall be defined in external JVM connections configuration file (e.g. config/connections.cfg). But this also implies that if SourceFQN tokens refers unique values resolved from JMX data (e.g, over @bean: reference), it is enough to have only source.factory.RootFQN property of Stream-JMX TNT4J configuration to uniquely distinguish JVMs.

Consider this Stream-JMX run scenario to monitor 3 instances of service runner VMs:

One Stream-JMX instance per monitored VM

you've been having such TNT4J source configuration:

    source.factory.SERVICE: $sjmx.serviceId
    source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GENERIC=?

running 3 instances of Stream-JMX

  ./bin/stream-jmx-connect.sh service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi -ul:admin -up:admin -sp:sjmx.serviceId=Broker-0
  ./bin/stream-jmx-connect.sh service:jmx:rmi:///jndi/rmi://localhost:9998/jmxrmi -ul:admin -up:admin -sp:sjmx.serviceId=Broker-1
  ./bin/stream-jmx-connect.sh service:jmx:rmi:///jndi/rmi://localhost:9997/jmxrmi -ul:admin -up:admin -sp:sjmx.serviceId=Broker-2

One Stream-JMX instance per 3 monitored VMs

you've been having such TNT4J source configuration:

   source.factory.RootFQN: SERVER=?#DATACENTER=?#GENERIC=?

configuring VM connections in connections.cfg file

##############################################################################################################################################################
#            VM connection string                      #    Agent options   #  User name    #    Password       #                Source addition             #
##############################################################################################################################################################
 service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi        *:*!!60000          admin             admin             SERVICE=Broker-0
 service:jmx:rmi:///jndi/rmi://localhost:9998/jmxrmi        *:*!!60000          admin             admin             SERVICE=Broker-1 
 service:jmx:rmi:///jndi/rmi://localhost:9997/jmxrmi        *:*!!60000          admin             admin             SERVICE=Broker-2
##############################################################################################################################################################

running Stream-JMX

  ./bin/stream-jmx-connect-file-config.sh connections.cfg

Connecting remote WebSphere Application Server (WAS)

Additions needed to run SamplingAgent connected to remote WAS machine can be found in executable OS shell run script files bin/stream-jmx-connect-was.bat or bin/stream-jmx-connect-was.sh. It contains these major configuration additions:

WAS environment setup configuration
appending LIBPATH variable with WAS libs
adding WAS specific JMX sampler options
defining WAS JMXConnector parameters
and adding them as additional java command parameters

NOTE: When you have IBM JVM running on client side and getting authentication or naming related exceptions, you have to:

Alter sas.client.props by setting com.ibm.CORBA.validateBasicAuth=false

Add additional SamplingAgent arguments:

    -ul:[your WAS user]
    -up:[your WAS user pass]
    -cp:java.naming.factory.initial=com.ibm.websphere.naming.WsnInitialContextFactory
    -cp:java.naming.factory.url.pkgs=com.ibm.ws.naming
    -cp:java.naming.provider.url=corbaloc:iiop:localhost:2809/WsnAdminNameService

Connecting remote WebLogic (version 12c) server instance

Additions needed to run SamplingAgent connected to remote WebLogic machine can be found in executable OS shell run script files bin/stream-jmx-connect.bat or bin/stream-jmx-connect.sh. It contains these major configuration additions:

WebLogic environment setup configuration in setDomainEnv.sh/setDomainEnv.cmd file, exposing JMX access port and security options like:

.bat/cmd

set JAVA_OPTIONS=%JAVA_OPTIONS% -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8788 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false

.sh

JAVA_OPTIONS="${JAVA_OPTIONS} -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8788 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
export JAVA_OPTIONS

appending LIBPATH variable with WebLogic client libs from ${WL_INSTALL_DIR}/wlserver/server/lib/:

wlclient.jar
wljmxclient.jar
javax.javaee-api.jar

For example:

.bat/cmd

set WL_HOME=C:\Oracle\Middleware\Oracle_Home
set WL_CLIENT_LIBS=%WL_HOME%\wlserver\server\lib\wlclient.jar;%WL_HOME%\wlserver\server\lib\wljmxclient.jar;%WL_HOME%\wlserver\server\lib\javax.javaee-api.jar
set LIBPATH=%LIBPATH%;%RUNDIR%..\lib\*;%WL_CLIENT_LIBS%

.sh

WL_HOME="/opt/Oracle/Middleware/Oracle_Home"
WL_CLIENT_LIBS="$WL_HOME/wlserver/server/lib/wlclient.jar:$WL_HOME/wlserver/server/lib/wljmxclient.jar:$WL_HOME/wlserver/server/lib/javax.javaee-api.jar"
LIBPATH="$LIBPATH:$SCRIPTPATH/../lib/*:$WL_CLIENT_LIBS"

enable j2ee module use:

.bat/cmd
```
set MODULE_SET=core j2ee
```
.sh
```
MODULE_SET=("core" "j2ee")
```

Add additional SamplingAgent arguments:

    -cp:jmx.remote.protocol.provider.pkgs=weblogic.management.remote
    # Add user credentials if needed
    -cp:java.naming.security.principal=user
    -cp:java.naming.security.credentials=password
    # Optional naming parameters
    #-cp:java.naming.factory.initial=weblogic.jndi.WLInitialContextFactory
    #-cp:java.naming.provider.url=t3://localhost:7001

WebLogic 12c provides these JNDI Names for WebLogic MBean Servers:

weblogic.management.mbeanservers.domainruntime - Domain Runtime MBean Server
weblogic.management.mbeanservers.runtime - Runtime MBean Server
weblogic.management.mbeanservers.edit - Edit MBean Server

You can find your WebLogic server exposed JMX Connectors in console output:

<May 10, 2019, 10:57:11,147 AM EEST> <Warning> <JMX> <BEA-149512> <JMX Connector Server started at service:jmx:iiop://172.16.6.56:7001/jndi/weblogic.management.mbeanservers.runtime.>
<May 10, 2019, 10:57:11,823 AM EEST> <Warning> <JMX> <BEA-149512> <JMX Connector Server started at service:jmx:iiop://172.16.6.56:7001/jndi/weblogic.management.mbeanservers.domainruntime.>
<May 10, 2019, 10:57:11,853 AM EEST> <Warning> <JMX> <BEA-149512> <JMX Connector Server started at service:jmx:iiop://172.16.6.56:7001/jndi/weblogic.management.mbeanservers.edit.>

JMX Service connection URL for WebLogic is like this:

service:jmx:iiop://172.16.6.56:7001/jndi/weblogic.management.mbeanservers.runtime

And complete set of stream-jmx program arguments:

-connect
-vm:service:jmx:iiop://172.16.6.56:7001/jndi/weblogic.management.mbeanservers.edit
-ao:*:*!!10000
-cp:jmx.remote.protocol.provider.pkgs=weblogic.management.remote
-cp:java.naming.security.principal=weblogic
-cp:java.naming.security.credentials=password

For more information, see Programming WebLogic JNDI, WebLogic API Environment class documentation and Accessing WebLogic Server MBeans with JMX.

Coding into API

You can connect SamplingAgent to JVM from your custom API by calling SamplingAgent.connect(String,String) method.

Sample connect call for local JVM:

try {
   SamplingAgent.newSamplingAgent().connect("activemq", "*:*!!10000");
} catch (Exception exc) {
  exc.printStackTrace();
}

Sample connect call for remote JVM:

try {
   SamplingAgent.newSamplingAgent().connect("service:jmx:iiop://172.16.6.40:2809/jndi/JMXConnector", "*:*!!10000");
} catch (Exception exc) {
  exc.printStackTrace();
}

Sample connect call for remote JVM defining username and password:

try {
   SamplingAgent.newSamplingAgent().connect("activemq", "admin", "admin", "*:*!!10000");
} catch (Exception exc) {
  exc.printStackTrace();
}

or defining map of connection configuration parameters:

try {
  Map<String, Object> connParams = new HashMap<String, Object>();
  connParams.put("javax.net.ssl.trustStore", "/your/path/to/truststore.jks");
  connParams.put("javax.net.ssl.trustStorePassword", "truststore_pwd");
   SamplingAgent.newSamplingAgent().connect("activemq", "*:*!!10000", connParams);
} catch (Exception exc) {
  exc.printStackTrace();
}

Sampling local process runner JVM

Command line to run

java -Dtnt4j.config=.\config\tnt4j.properties -Dlog4j2.configurationFile=.\config\log4j2.xml -classpath "tnt4j-stream-jmx-core.jar;lib/*" com.jkoolcloud.tnt4j.stream.jmx.SamplingAgent -local -ao:*:*!*:dummy!10000

System properties -Dxxxxx defines Stream-JMX configuration. For details see Stream-JMX configuration .

SamplingAgent arguments -connect -vm:activemq -ao:*:*!*:dummy!10000 states:

-local - defines that SamplingAgent shall run sampling on local process runner JVM.
-ao:*:*!*:dummy!10000 - is JMX sampler options stating to include all MBeans, exclude all dummy MBeans and schedule sampling every 10 seconds. Sampler options are optional - default value is *:*!30000. Initial sampler delay can be configured by adding numeric parameter *:*!30000!1000 defining initial sampler delay as 1 second. Default sampler delay value is equal to sampling period value. See JMX Sampling Agent sampler options for details.
-slp: - any JMX sampler configuration property. See Program arguments used for details.
-sp: - any system property used by sampler. See System properties used for details.

Coding into API

You can run SamplingAgent for local process runner JVM from your custom API by calling SamplingAgent.sampleLocalVM(String,boolean) method.

Sample sampleLocalVM call for a local process runner JVM:

try {
  SamplingAgent.sampleLocalVM("*:*!!10000", true);
} catch (Exception exc) {
  exc.printStackTrace();
}

Running Stream-JMX as servlet

Generic servlet page features

Top of the page is dedicated to list Stream-JMX used properties.

Change properties as you like in input field and press Sumbit button to apply changes. JMX sampler service will restart and changes will apply instantly without restating servlet or web container.

tnt4j.properties file is displayed bellow. You can change that online. Press Submit TNT4.config to update configuration. JMX sampler service will restart.

Bellow you can see Stream-JMX console output. Refresh page or click Update button to update console contents.

WebSphere Application Server (WAS)

Build Stream-JMX and find opt/tnt4j-stream-jmx-was-*.ear in distribution build directory or archives produced.

Navigate Applications->New application and choose New Enterprise Application. Choose file you have built and install Stream-JMX application you usually do, or fallowing IBM's instruction. Keep in mind context path.
Start the application.

By default, Stream-JMX connects to local WAS runner JVM using com.ibm.websphere.management.AdminServiceFactory.getMBeanFactory().

Navigate to http://localhost:9080/YOUR_CHOOSEN_CONTEXT_PATH/. TNT4J-Stream-JMX configuration/monitoring page should be displayed. See Generic servlet page features on what is available there.

Stream-JMX logging on WAS can be performed by WAS Admin console:

Troubleshooting -> [your server instance] -> Change log detail levels -> select Configuration/Runtime tab -> Components and Groups -> Components -> expand All components -> expand com.jkoolcloud.tnt4j.* -> select com.jkoolcloud.tnt4j.stream.jmx* node -> scroll to the tree bottom and select level you would like to use.
Troubleshooting -> [your server instance] -> Change log detail levels -> select Configuration/Runtime tab -> into text area paste : com.jkoolcloud.tnt4j.stream.jmx.*=finer (change finer to the level you would like to use).

See Generic servlet page features.

WebSphere Liberty Server

Build Stream-JMX and find opt/tnt4j-stream-jmx-liberty-war-*.war in distribution build directory or archives produced.

There is a simple Liberty server.xml configuration required to run Stream-JMX servlet:

<?xml version="1.0" encoding="UTF-8"?>
<server description="new server">

    <!-- Enable features -->
    <featureManager>
        <feature>webProfile-7.0</feature>
        <feature>restConnector-1.0</feature>
        <feature>localConnector-1.0</feature>
        <feature>monitor-1.0</feature>
        <feature>j2eeManagement-1.1</feature>
    </featureManager>

    <!-- To access this server from a remote client add a host attribute to the following element, e.g. host="*" -->
    <httpEndpoint id="defaultHttpEndpoint"
                  httpPort="9080"
                  httpsPort="9443"
                  host="*"/>

    <!-- Automatically expand WAR files and EAR files -->
    <applicationManager autoExpand="true"/>

    <webContainer deferServletLoad="false"/>

    <auth-method>BASIC</auth-method>

    <basicRegistry id="basic" realm="default">
        <user name="Admin" password="admin" />
        <user name="JMXManager" password="jmxAdmin" />
    </basicRegistry>

    <administrator-role>
        <user>Admin</user>
    </administrator-role>

    <!-- Configure Stream-JMX logging level, e.g. to get DEBUG entries -->
    <logging traceSpecification="*=info:com.jkoolcloud.tnt4j.stream.jmx.*=finer" />

    <application contextRoot="tnt-jmx" location="tnt4j-stream-jmx-liberty-war-2.1.1.war" type="war" id="tnt-jmx" name="tnt-jmx">
        <application-bnd>
            <security-role name="StreamJmxManager">
                <user name="JMXManager" />
                <run-as userid="JMXManager" />
            </security-role>
        </application-bnd>
    </application>
</server>

By default, Stream-JMX connects to local Liberty runner JVM using java.lang.management.ManagementFactory.getPlatformMBeanServer().

Navigate to http://localhost:9080/YOUR_CHOOSEN_CONTEXT_PATH/. TNT4J-Stream-JMX configuration/monitoring page should be displayed. See Generic servlet page features on what is available there.

Embed Stream-JMX code into your application

Coding

    // obtain SamplerFactory instance
    SamplerFactory factory = DefaultSamplerFactory.getInstance();
    // create an instance of the sampler that will sample mbeans
    Sampler sampler = factory.newInstance();
    // schedule collection (ping) for given MBean filter and 30000 ms sampling period
    Map<String, Object> samplerCfg = new HashMap<>(3);
    samplerCfg.put(SampleHandler.CFG_INCLUDE_FILTER, Sampler.JMX_FILTER_ALL);
    samplerCfg.put(Sampler.CFG_SAMPLING_PERIOD, 30000);
    sampler.setSchedule(samplerCfg).run();

NOTE: setSchedule(..).run() sequence must be called to run the schedule. setSchedule(..) just sets the scheduling parameters, run() executes the schedule.

To schedule metric collection for a specific MBean server:

    // obtain SamplerFactory instance
    SamplerFactory factory = DefaultSamplerFactory.getInstance();
    // create an instance of the sampler that will sample mbeans
    Sampler sampler = factory.newInstance(ManagementFactory.getPlatformMBeanServer());
    // schedule collection (ping) for given MBean filter and 30000 ms sampling period
    Map<String, Object> samplerCfg = new HashMap<>(3);
    samplerCfg.put(SampleHandler.CFG_INCLUDE_FILTER, Sampler.JMX_FILTER_ALL);
    samplerCfg.put(Sampler.CFG_SAMPLING_PERIOD, 30000);
    sampler.setSchedule(samplerCfg).run();

Stream-JMX supports inclusion and exclusion filters. To schedule metric collection for a specific MBean server and exclude certain MBeans: (Exclusion filters are applied after inclusion filters)

    // obtain SamplerFactory instance
    SamplerFactory factory = DefaultSamplerFactory.getInstance();
    // create an instance of the sampler that will sample mbeans
    Sampler sampler = factory.newInstance(ManagementFactory.getPlatformMBeanServer());
    // schedule collection (ping) for given MBean filter and 30000 ms sampling period
    Map<String, Object> samplerCfg = new HashMap<>(3);
    samplerCfg.put(SampleHandler.CFG_INCLUDE_FILTER, Sampler.JMX_FILTER_ALL);
    samplerCfg.put(SampleHandler.CFG_EXCLUDE_FILTER, "mydomain:*");
    samplerCfg.put(Sampler.CFG_SAMPLING_PERIOD, 30000);
    sampler.setSchedule(samplerCfg).run();

Below is an example of how to sample all registered mbean servers:

    // obtain SamplerFactory instance
    SamplerFactory factory = DefaultSamplerFactory.getInstance();
    // find other registered mbean servers
    ArrayList<MBeanServer> mlist = MBeanServerFactory.findMBeanServer(null);
    Map<String, Object> samplerCfg = new HashMap<>(3);
    samplerCfg.put(SampleHandler.CFG_INCLUDE_FILTER, Sampler.JMX_FILTER_ALL);
    samplerCfg.put(Sampler.CFG_SAMPLING_PERIOD, 30000);
    for (MBeanServer server: mlist) {
        Sampler jmxp = factory.newInstance(server);
        jmxp.setSchedule(samplerCfg).run();
    }

Alternatively, Stream-JMX provides a helper class SamplingAgent that lets you schedule sampling for all registered MBeanServer instances.

     SamplingAgent.newSamplingAgent().sample(Sampler.JMX_FILTER_ALL, Sampler.JMX_FILTER_NONE, 60000, TimeUnit.MILLISECONDS);

NOTE: Sampled MBean attributes and associated values are stored in a collection of Snapshot objects stored within Activity instance. Current Activity instance can be obtained via AttributeSample passed when calling listeners such as AttributeCondition, SampleListener. Snapshots can be accessed using Activity.getSnapshots() method call.

NOTE: Sampled output is written to underlying tnt4j event sink configured in tnt4j.properties file. Sink destinations could be a file, socket, log4j, user defined event sink implementations.

For more information on TNT4J and tnt4j.properties see TNT4J Wiki.

Command line to run

Example below runs SamplingAgent helper class as a standalone java application with a given MBean filter "*:*", sampling period in milliseconds (10000), and time to run in milliseconds (60000):

java -Dlog4j2.configurationFile=.\config\log4j2.xml -classpath "*;lib/*" com.jkoolcloud.tnt4j.stream.jmx.SamplingAgent "*:*" "" 10000 60000

Stream-JMX configuration

Stream-JMX has configuration properties allowing to configure JMX sampler. It is possible to configure same stream JMX sampler parameter using System property or program argument. Depending on sampling environment used, in some cases it is easier to configure it using one approach or another. When both definitions available first System property defined value is assigned and then program argument value after.

JMX sampler configuration properties are:

forceObjectName - flag indicating to forcibly add objectName attribute if such is not present for a MBean. Default value - false.
addStatisticMetadata - flag indicating to add J2EE metadata entries as attributes for a MBean. Default value - true.
compositeDelimiter - delimiter used to tokenize composite/tabular type MBean properties keys. Default value - \;
useObjectNameProperties - flag indicating to copy MBean ObjectName contained properties into sample snapshot properties. Default value - true.
excludeOnError - flag indicating to auto-exclude failed to sample attributes. Default value - false.
excludedAttributes - list of user chosen attribute names (may have wildcards * and ?) to exclude, pattern: attr1,attr2,...,attrN@MBean1_ObjectName;...;attr1,attr2,...,attrN@MBeanN_ObjectName. Default value - ``.

See System properties used how to configure Stream-JMX using system properties. See Program arguments used how to configure Stream-JMX using program arguments.

System properties used

To define system property for application you can use common JVM argument -Dkey=value or SamplingAgent program argument -sp:key=value.

General use:

tnt4j.config - defines TNT4J properties file path. Example: -Dtnt4j.config=".\config\tnt4j.properties"
log4j2.configurationFile - defines stream-jmx logging used LOG4J properties file path. Example: -Dlog4j2.configurationFile="file:./config/log4j2.xml"
com.jkoolcloud.tnt4j.stream.jmx.agent.forceObjectName - defines whether to forcibly add objectName attribute if such is not present for a MBean. Default value - false. Example: -Dcom.jkoolcloud.tnt4j.stream.jmx.agent.forceObjectName=true
com.jkoolcloud.tnt4j.stream.jmx.agent.addStatisticMetadata - defines whether to add J2EE statistic metadata entries as attributes for a MBean. Default value - true. Example: -Dcom.jkoolcloud.tnt4j.stream.jmx.agent.addStatisticMetadata=false
com.jkoolcloud.tnt4j.stream.jmx.agent.compositeDelimiter - defines delimiter used to tokenize composite/tabular type MBean properties keys. Default value - \. Example: -Dcom.jkoolcloud.tnt4j.stream.jmx.agent.compositeDelimiter=.
com.jkoolcloud.tnt4j.stream.jmx.agent.useObjectNameProperties - defines whether to copy MBean ObjectName contained properties into sample snapshot properties. Default value - true. Example: -Dcom.jkoolcloud.tnt4j.stream.jmx.agent.useObjectNameProperties=false
sjmx.serviceId - defines stream-jmx service identifier used by TNT4J SourceFQN to distinguish monitored application instance. Example: -Dsjmx.serviceId=broker-0 or -sp:sjmx.serviceId=broker-0
com.jkoolcloud.tnt4j.stream.jmx.agent.excludeOnError - defines whether to auto-exclude failed to sample attributes. Default value - false. Example: -Dcom.jkoolcloud.tnt4j.stream.jmx.agent.excludeOnError=true
com.jkoolcloud.tnt4j.stream.jmx.agent.excludedAttributes - defines list of user chosen attribute names (may have wildcards * and ?) to exclude, pattern: attr1,attr2,...,attrN@MBean1_ObjectName;...;attr1,attr2,...,attrN@MBeanN_ObjectName. Default value - ``. Example: -Dcom.jkoolcloud.tnt4j.stream.jmx.agent.excludedAttributes=javaVersion,javaVendor@WebSphere:mbeanIdentifier=cells*,*
com.jkoolcloud.tnt4j.stream.jmx.sampler.factory- defines class name of SamplerFactory class to be used by stream. Default value - com.jkoolcloud.tnt4j.stream.jmx.factory.DefaultSamplerFactory. Example: -Dcom.jkoolcloud.tnt4j.stream.jmx.sampler.factory=com.jkoolcloud.tnt4j.stream.jmx.impl.WASSamplerFactory
tnt4j.stream.log.filename - defines name of stream log file. Default value - ./logs/tnt4j-stream-jmx.log. Example: -Dtnt4j.stream.log.filename=./logs/tnt4j-stream-jmx_broker0.log
tnt4j.activities.log.filename - defines name of streamed activities log file. Default value - ./logs/tnt4j-stream-jmx_samples.log. Example: -Dtnt4j.activities.log.filename=./logs/tnt4j-stream-jmx_broker0_samples.log

Changes between versions:

Prior to version 0.7 stream-jmx was writing logging messages to System.out/err print streams. Since version 0.7 logging is performed over TNT4J Log Sink to slf4j-log4j12. Logger configuration is defined in ./config/log4j2.xml file. If you where using system property -Dcom.jkoolcloud.tnt4j.stream.jmx.agent.trace=true prior to 0.7 version, this is now configured over logger log level, by setting it to DEBUG value, e.g:
```
<Logger name="com.jkoolcloud.tnt4j.stream.jmx" level="DEBUG"/>
```

NOTE: when defining configuration file properties over file: URI notation (e.g. configurationFile for log4j or logback) on Windows change backslashes \ to slash / symbols for URI to get successfully parsed.

NOTE: when running multiple parallel instances of stream-jmx (e.g. collecting Kafka broker metrics using dedicated stream instance), it is recommended to define different log file names (using system properties tnt4j.stream.log.filename and tnt4j.activities.log.filename) for every stream-jmx ran JVM. Using same log file on different VMs may arise logging conflicts when VM ran logger holds different last log line references, resulting log entries to be written in random order. When monitoring multiple VM's over single stream-jmx instance, only one logger instance is ran (as there is one JVM running), and no logging conflicts shall occur.

Program arguments used

To define stream JMX sampler configuration property use program argument-slp:. One argument defines one property. To define multiple properties use as many argument definitions as there are required properties. For example:

-slp:forceObjectName=true
-slp:addStatisticMetadata=false
-slp:compositeDelimiter=.
-slp:useObjectNameProperties=false
-slp:excludeOnError=true
-slp:excludedAttributes=java*@WebSphere:mbeanIdentifier=cells*,*

JMX Sampling Agent sampler options

Agent options are defined using format: mbean-filter!exclude-filter!sample-ms!init-delay-ms!batch-size

mbean-filter - MBean include name filter defined using object name pattern: domainName:keysSet. NOTE: Multiple filters can be defined using ; as delimiter: domainName1:keysSet1;domainName2:keysSet2;domainName3:keysSet3.
exclude-filter - MBean exclude name filter defined using object name pattern: domainName:keysSet. NOTE: Multiple filters can be defined using ; as delimiter: domainName1:keysSet1;domainName2:keysSet2;domainName3:keysSet3.
sample-ms - MBeans sampling rate in milliseconds.
init-delay-ms - MBeans sampling initial delay in milliseconds. Optional, by default it is equal to sample-ms value.
batch-size - number of sampled MBeans to post over single package. Optional, default is -1 (unlimited).

Default sampling agent options value is: *:*!!30000!30000!-1

TNT4J Source fields configuration

Stream-JMS has couple additional features in comparison with basic TNT4J when building Source RootFQN value:

Has two dedicated value placeholders sjmx.serverAddress and sjmx.serverName to resolve remotely sampled machine IP address and host name. To map these sampled machine IP/host name values into source field, put @ symbol before placeholder name in source field definition. In this case tnt4j.properties configuration would be like this:

; Remote machine IP address
...
source.factory.GENERIC: Streams
source.factory.DATACENTER: HQDC
source.factory.SERVICE: broker-01
source.factory.SERVER: @sjmx.serverAddress
source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GENERIC=?
...

; Remote machine host name
...
source.factory.GENERIC: Streams
source.factory.DATACENTER: HQDC
source.factory.SERVICE: broker-01
source.factory.SERVER: @sjmx.serverName
source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GENERIC=?
...

Can resolve values for source fields from JMX MBean attributes. Two items are required to use in tnt4j.properties configuration to enable this feature:
1. Set source.factory to com.jkoolcloud.tnt4j.stream.jmx.source.JMXSourceFactoryImpl
2. Put MBean attribute descriptor having prefix @bean: as source field value. There are two types of MBean attribute descriptors:
  - Mbean attribute value descriptor pattern is "@bean:MBean_ObjectName/?AttributeName. It also allows use of wildcard symbols *, e.g.: @bean:org.apache.ZooKeeperService:name0=*/?ClientPort resolving ClientPort value for first available (if more than one is running on same JVM, but usually it should be only one) ZooKeeper service instance.
  - MBean key property descriptor pattern is @bean:MBean_ObjectName:PropertyKey=?(,OtherProperties) (part within () is optional), e.g.: @bean:kafka.server:id=?,type=app-info.
3. Since MBean initialization may take some time on server side (usually it may happen after some server component restart) MBean attribute value may not be resolvable at that time. To overcome such case, you can define MBean value resolution retry interval in seconds (property RetryIntervalSec) and maximum retry attempts count (property MaxRetryAttempts).
then putting all together, tnt4j.properties configuration will be like this:
```
...
source.factory: com.jkoolcloud.tnt4j.stream.jmx.source.JMXSourceFactoryImpl
source.factory.GENERIC: Streams
source.factory.DATACENTER: HQDC
source.factory.SERVICE: @bean:org.apache.activemq:type=Broker,brokerName=localhost/?BrokerId
source.factory.SERVER: @bean:JMImplementation:type=MBeanServerDelegate/?MBeanServerId
source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GENERIC=?
source.factory.RetryIntervalSec: 1
source.factory.MaxRetryAttempts: 10
...
```

Stream-JMX event data formatters

com.jkoolcloud.tnt4j.stream.jmx.format.FactNameValueFormatter - this class provides key/value formatting for tnt4j activities, events and snapshots. The output format follows the following format:

"OBJ:name-value-prefix1\name1=value1,name-value-prefix1\name2=value2,....,name-value-prefixN\nameN=valueN"

Sample output:

OBJ:Streams\0,
0\HQDC\samtis\com.jkoolcloud.tnt4j.stream.jmx.impl.PlatformJmxSampler\Activities,
Self\location=0,
0,
Self\level=INFO,
Self\id.count=0,
Self\pid=7660,
Self\tid=63,
Self\snap.count=34,
Self\elapsed.usec=9989365,
JMImplementation:type\MBeanServerDelegate\MBeanServerId=samtis_1486645461070,
JMImplementation:type\MBeanServerDelegate\SpecificationName=Java Management Extensions,
JMImplementation:type\MBeanServerDelegate\SpecificationVersion=1.4,
JMImplementation:type\MBeanServerDelegate\SpecificationVendor=Oracle Corporation,
JMImplementation:type\MBeanServerDelegate\ImplementationName=JMX,
JMImplementation:type\MBeanServerDelegate\ImplementationVersion=1.8.0_121-b13,
JMImplementation:type\MBeanServerDelegate\ImplementationVendor=Oracle Corporation,
org.apache.activemq:brokerName\localhost!type\Broker\Uptime=1 minute,
org.apache.activemq:brokerName\localhost!type\Broker\MemoryPercentUsage=0,
org.apache.activemq:brokerName\localhost!type\Broker\Persistent=true,
org.apache.activemq:brokerName\localhost!type\Broker\DataDirectory=D:\tmp\apache-activemq-5.14.3\data,
org.apache.activemq:brokerName\localhost!type\Broker\VMURL=vm://localhost,
org.apache.activemq:brokerName\localhost!type\Broker\TempLimit=42844577792,
org.apache.activemq:brokerName\localhost!type\Broker\MemoryLimit=668309914,
org.apache.activemq:brokerName\localhost!type\Broker\StoreLimit=42846849471,
org.apache.activemq:brokerName\localhost!type\Broker\MaxMessageSize=1024,
org.apache.activemq:brokerName\localhost!type\Broker\AverageMessageSize=1024,
org.apache.activemq:brokerName\localhost!type\Broker\TotalDequeueCount=0,
org.apache.activemq:brokerName\localhost!type\Broker\TotalConsumerCount=0,
org.apache.activemq:brokerName\localhost!type\Broker\StatisticsEnabled=true,
org.apache.activemq:brokerName\localhost!type\Broker\JobSchedulerStoreLimit=0,
org.apache.activemq:brokerName\localhost!type\Broker\MinMessageSize=1024,
org.apache.activemq:brokerName\localhost!type\Broker\JobSchedulerStorePercentUsage=0,
org.apache.activemq:brokerName\localhost!type\Broker\StorePercentUsage=0,
org.apache.activemq:brokerName\localhost!type\Broker\TempPercentUsage=0,
org.apache.activemq:brokerName\localhost!type\Broker\CurrentConnectionsCount=0,
org.apache.activemq:brokerName\localhost!type\Broker\TotalMessageCount=0,
org.apache.activemq:brokerName\localhost!type\Broker\TotalConnectionsCount=0,
org.apache.activemq:brokerName\localhost!type\Broker\TotalEnqueueCount=1,
org.apache.activemq:brokerName\localhost!type\Broker\TotalProducerCount=0,
org.apache.activemq:brokerName\localhost!type\Broker\TransportConnectors=[openwire=tcp://samtis:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600,
 amqp=amqp://samtis:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600,
 mqtt=mqtt://samtis:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600,
 stomp=stomp://samtis:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600,
 ws=ws://samtis:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600],
org.apache.activemq:brokerName\localhost!type\Broker\BrokerVersion=5.14.3,
org.apache.activemq:brokerName\localhost!type\Broker\UptimeMillis=62787,
org.apache.activemq:brokerName\localhost!type\Broker\BrokerName=localhost,
org.apache.activemq:brokerName\localhost!type\Broker\Slave=false,
org.apache.activemq:brokerName\localhost!type\Broker\BrokerId=ID:samtis-52769-1486645462150-0:1,
SampleContext\noop.count=0,
SampleContext\sample.count=6,
SampleContext\total.error.count=6,
SampleContext\total.exclude.count=204,
SampleContext\mbean.count=37,
SampleContext\condition.count=0,
SampleContext\listener.count=1,
SampleContext\total.action.count=0,
SampleContext\total.metric.count=3582,
SampleContext\last.metric.count=595,
SampleContext\sample.time.usec=10054,
SampleContext\listener.exclude.set.count=6,
SampleContext\listener.trace.mode=false,
...

NOTE: Entries are not sorted, sequence is same as returned by activity contained snapshots map entries and snapshot properties iterators.

com.jkoolcloud.tnt4j.stream.jmx.format.FactPathValueFormatter - this class provides key/value formatting for tnt4j activities, events and snapshots. The output format follows the following format:

"OBJ:object-path1\name1=value1,object-path1\name2=value2,....,object-pathN\nameN=valueN""

Sample output:

OBJ:Streams\0,
0\HQDC\samtis\com.jkoolcloud.tnt4j.stream.jmx.impl.PlatformJmxSampler\Activities,
Self\location=0,
0,
Self\level=INFO,
Self\id.count=0,
Self\pid=3136,
Self\tid=24,
Self\snap.count=34,
Self\elapsed.usec=9885208,
JMImplementation\MBeanServerDelegate\ImplementationName=JMX,
JMImplementation\MBeanServerDelegate\ImplementationVendor=Oracle Corporation,
JMImplementation\MBeanServerDelegate\ImplementationVersion=1.8.0_121-b13,
JMImplementation\MBeanServerDelegate\MBeanServerId=samtis_1486998422874,
JMImplementation\MBeanServerDelegate\SpecificationName=Java Management Extensions,
JMImplementation\MBeanServerDelegate\SpecificationVendor=Oracle Corporation,
JMImplementation\MBeanServerDelegate\SpecificationVersion=1.4,
SampleContext\condition.count=0,
SampleContext\last.metric.count=633,
SampleContext\listener.count=1,
SampleContext\listener.exclude.set.count=6,
SampleContext\listener.trace.mode=true,
SampleContext\mbean.count=37,
SampleContext\noop.count=0,
SampleContext\sample.count=2,
SampleContext\sample.time.usec=72856,
SampleContext\total.action.count=0,
SampleContext\total.error.count=6,
SampleContext\total.exclude.count=66,
SampleContext\total.metric.count=1272,
org.apache.activemq\Broker\localhost\AverageMessageSize=1024,
org.apache.activemq\Broker\localhost\BrokerId=ID:samtis-63754-1486998424125-0:1,
org.apache.activemq\Broker\localhost\BrokerName=localhost,
org.apache.activemq\Broker\localhost\BrokerVersion=5.14.3,
org.apache.activemq\Broker\localhost\CurrentConnectionsCount=0,
org.apache.activemq\Broker\localhost\DataDirectory=D:\tmp\apache-activemq-5.14.3\data,
org.apache.activemq\Broker\localhost\DurableTopicSubscribers=[],
org.apache.activemq\Broker\localhost\DynamicDestinationProducers=[],
org.apache.activemq\Broker\localhost\InactiveDurableTopicSubscribers=[],
org.apache.activemq\Broker\localhost\JMSJobScheduler=null,
org.apache.activemq\Broker\localhost\JobSchedulerStoreLimit=0,
org.apache.activemq\Broker\localhost\JobSchedulerStorePercentUsage=0,
org.apache.activemq\Broker\localhost\MaxMessageSize=1024,
org.apache.activemq\Broker\localhost\MemoryLimit=668309914,
org.apache.activemq\Broker\localhost\MemoryPercentUsage=0,
org.apache.activemq\Broker\localhost\MinMessageSize=1024,
org.apache.activemq\Broker\localhost\Persistent=true,
org.apache.activemq\Broker\localhost\QueueProducers=[],
org.apache.activemq\Broker\localhost\QueueSubscribers=[],
org.apache.activemq\Broker\localhost\Queues=[],
org.apache.activemq\Broker\localhost\Slave=false,
org.apache.activemq\Broker\localhost\StatisticsEnabled=true,
org.apache.activemq\Broker\localhost\StoreLimit=42845775469,
org.apache.activemq\Broker\localhost\StorePercentUsage=0,
org.apache.activemq\Broker\localhost\TempLimit=42842071040,
org.apache.activemq\Broker\localhost\TempPercentUsage=0,
org.apache.activemq\Broker\localhost\TemporaryQueueProducers=[],
org.apache.activemq\Broker\localhost\TemporaryQueueSubscribers=[],
org.apache.activemq\Broker\localhost\TemporaryQueues=[],
org.apache.activemq\Broker\localhost\TemporaryTopicProducers=[],
org.apache.activemq\Broker\localhost\TemporaryTopicSubscribers=[],
org.apache.activemq\Broker\localhost\TemporaryTopics=[],
org.apache.activemq\Broker\localhost\TopicProducers=[],
org.apache.activemq\Broker\localhost\TopicSubscribers=[],
org.apache.activemq\Broker\localhost\Topics=[org.apache.activemq:type=Broker,
brokerName=localhost,
destinationType=Topic,
destinationName=ActiveMQ.Advisory.MasterBroker],
org.apache.activemq\Broker\localhost\TotalConnectionsCount=0,
org.apache.activemq\Broker\localhost\TotalConsumerCount=0,
org.apache.activemq\Broker\localhost\TotalDequeueCount=0,
org.apache.activemq\Broker\localhost\TotalEnqueueCount=1,
org.apache.activemq\Broker\localhost\TotalMessageCount=0,
org.apache.activemq\Broker\localhost\TotalProducerCount=0,
org.apache.activemq\Broker\localhost\TransportConnectors=[amqp=amqp://samtis:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600,
 mqtt=mqtt://samtis:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600,
 ws=ws://samtis:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600,
 openwire=tcp://samtis:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600,
 stomp=stomp://samtis:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600],
org.apache.activemq\Broker\localhost\Uptime=13 minutes,
org.apache.activemq\Broker\localhost\UptimeMillis=822599,
org.apache.activemq\Broker\localhost\VMURL=vm://localhost,
...

NOTE: Entries are sorted by key alphanumeric ordering and key representation is more common to be used for, e.g., tree model construction to represent JMX structure more like JConsole does.

com.jkoolcloud.tnt4j.format.LevelingJSONFormatter - this class provides JSON formatting for tnt4j activities, events and snapshots. Difference from com.jkoolcloud.tnt4j.format.JSONFormatter is only facts payload is present in produced JSON without major part of TNT4J metadata.

Sample output:

{
    "source": "com.jkoolcloud.tnt4j.stream.jmx.impl.PlatformJmxSampler",
    "source-ssn": "tnt4j-stream-jmx",
    "type": "ACTIVITY",
    "snapshots": [
        {
            "type": "SNAPSHOT",
            "name": "JMImplementation:type=MBeanServerDelegate",
            "properties": {
                "ImplementationName": "JMX",
                "ImplementationVendor": "Oracle Corporation",
                "ImplementationVersion": "1.8.0_152-b16",
                "MBeanServerId": "samtis_1517919369871",
                "SpecificationName": "Java Management Extensions",
                "SpecificationVendor": "Oracle Corporation",
                "SpecificationVersion": "1.4",
                "type": "MBeanServerDelegate"
            }
        },
        {
            "type": "SNAPSHOT",
            "name": "SampleContext",
            "properties": {
                "condition.count": 0,
                "last.metric.count": 5688,
                "listener.compositeProperty.delimiter": "\\",
                "listener.count": 1,
                "listener.exclude.set.count": 0,
                "listener.forceObjectName.mode": false,
                "listener.silence.set.count": 6,
                "listener.trace.mode": true,
                "listener.useObjectNameProperties.mode": true,
                "mbean.count": 636,
                "noop.count": 0,
                "sample.count": 1,
                "sample.time.usec": 452552,
                "total.action.count": 0,
                "total.error.count": 6,
                "total.exclude.count": 0,
                "total.metric.count": 5688
            }
        },
        {
            "type": "SNAPSHOT",
            "name": "Self",
            "properties": {
                "elapsed.usec": 10442872,
                "id.count": 0,
                "level": "INFO",
                "pid": 13388,
                "snap.count": 637,
                "tid": 1
            }
        },
        {
            "type": "SNAPSHOT",
            "name": "java.lang:name=Code Cache,type=MemoryPool",
            "properties": {
                "CollectionUsage": null,
                "CollectionUsageThreshold": "<unsupported>",
                "CollectionUsageThresholdCount": "<unsupported>",
                "CollectionUsageThresholdExceeded": "<unsupported>",
                "CollectionUsageThresholdSupported": false,
                "MemoryManagerNames": "[CodeCacheManager]",
                "Name": "Code Cache",
                "ObjectName": "java.lang:type=MemoryPool,name=Code Cache",
                "PeakUsage.committed": 16384000,
                "PeakUsage.init": 2555904,
                "PeakUsage.max": 251658240,
                "PeakUsage.used": 16255488,
                "Type": "NON_HEAP",
                "UsageThreshold": 0,
                "UsageThresholdCount": 0,
                "UsageThresholdExceeded": false,
                "UsageThresholdSupported": true,
                "Usage.committed": 16384000,
                "Usage.init": 2555904,
                "Usage.max": 251658240,
                "Usage.used": 16062016,
                "Valid": true,
                "name": "Code Cache",
                "type": "MemoryPool"
            }
        },
        {
            "type": "SNAPSHOT",
            "name": "java.lang:name=CodeCacheManager,type=MemoryManager",
            "properties": {
                "MemoryPoolNames": "[Code Cache]",
                "Name": "CodeCacheManager",
                "ObjectName": "java.lang:type=MemoryManager,name=CodeCacheManager",
                "Valid": true,
                "name": "CodeCacheManager",
                "type": "MemoryManager"
            }
        },
        {
            "type": "SNAPSHOT",
            "name": "java.lang:name=Compressed Class Space,type=MemoryPool",
            "properties": {
                "CollectionUsage": null,
                "CollectionUsageThreshold": "<unsupported>",
                "CollectionUsageThresholdCount": "<unsupported>",
                "CollectionUsageThresholdExceeded": "<unsupported>",
                "CollectionUsageThresholdSupported": false,
                "MemoryManagerNames": ["Metaspace Manager"],
                "Name": "Compressed Class Space",
                "ObjectName": "java.lang:type=MemoryPool,name=Compressed Class Space",
                "PeakUsage.committed": 4325376,
                "PeakUsage.init": 0,
                "PeakUsage.max": 1073741824,
                "PeakUsage.used": 4108024,
                "Type": "NON_HEAP",
                "UsageThreshold": 0,
                "UsageThresholdCount": 0,
                "UsageThresholdExceeded": false,
                "UsageThresholdSupported": true,
                "Usage.committed": 4325376,
                "Usage.init": 0,
                "Usage.max": 1073741824,
                "Usage.used": 4108024,
                "Valid": true,
                "name": "Compressed Class Space",
                "type": "MemoryPool"
            }
        },
        ...
    ]
}

NOTE: Entries are sorted by snapshot name and property key alphanumeric ordering.

MBeans collected data aggregations

When JMX MBeans attribute values gets collected into Activity snapshots (one activity per sampling iteration and one snapshot per MBean), it is possible to aggregate attribute values. Aggregations of individual MBean attributes can be performed into already Activity contained snapshot (append some additional attributes for MBean) or make a new snapshot containing these aggregated attribute values (create kind of new virtual MBean).

Aggregations configuration is defined using JSON format. Default aggregations configuration file path is config/aggregations.json. System property to set custom aggregations configuration file path to use is com.jkoolcloud.tnt4j.stream.jmx.aggregations.config.

High level aggregations configuration is like this:

[
    {
        aggregator1 config
    },
    {
        aggregator2 config
    },
    ...
    {
        aggregatorN config
    }
]

In general, it is a list of aggregator configurations. Aggregations manager is able to load that list and initiate aggregator instance by aggregator configuration defined aggregator type class. Aggregator itself is responsible to load rest of configuration over interface method com.jkoolcloud.tnt4j.stream.jmx.aggregations.ActivityAggregator.configure(Map<String, ?> cfg) throws IllegalArgumentException implementation.

Available aggregator implementations:

com.jkoolcloud.tnt4j.stream.jmx.aggregations.SnapshotAggregator - picks individual MBean attribute values from Activity contained snapshots and puts them into existing or new snapshot.

Aggregator configuration schema is:

aggregatorId - aggregator identifier, to identify aggregator instance in the logs, e.g. when fails to load configuration or performs some aggregation actions. Optional, if not defined then made of class name and sequence index.
type - aggregator implementation class to use. Required.
enabled - flag indicating if aggregator configuration shall be loaded and used. Optional, default value -true.
ignoreEmpty - flag indicating whether to ignore aggregator created empty snapshots. Optional, default value -false.
snapshots - aggregator implementation specific configuration array defining values aggregation into snapshots. Required:
- name - aggregation snapshot name RegEx string. It can be Activity contained snapshot name (to append properties) or any other (to create new snapshot and add it to Activity). To define Activity contained snapshot name use ObjectName notation syntax, see attribute beanId as a sample. Required.
- category - aggregation snapshot category RegEx string. Optional, if aggregation snapshot name has it defined like this category:beanId or sets default value jmx.aggregated in any other case if ommited.
- enabled - flag indicating if snapshot aggregation shall be used. Optional, default value -true.
- properties - list of snapshot properties to aggregate and store MBean attribute values. Required:
  - beanId - property bound bean identifier, it can have variable expression like varName=? or object name pattern. To define beanId use ObjectName notation syntax. When ommited, aggregations target snapshot is used to resolve property values. Optional.
  - attribute - property bound bean attribute name to get value, it can have variable expression like ${attrName1}-${attrName2}. Required. NOTE: when all variables resolve null value, aggregated value also gets null.
  - default - defines default value(s) mapping for attribute defined variables or unresolved aggregated value in general. Optional:
    - can define default aggregated value when all variables resolve null values:
```
"default": "-"
```
    - can define mapping for individual variables and/or unresolved aggregated value:
```
"attribute": "${attrName1}-${attrName2}-${attrName3}",
"default": {
    "attrName1": 0,
    "attrName2": "NA",
    "": "-",
    ">>": "UNRESOLVED"
}
```
      - key "" defines fallback default value to be applied for any unmapped variable.
      - key ">>" defines default aggregated value (when attribute referenced value gets unresolved or all variables resolve null values).
  - name - property (snapshot property) name, it can have variable expression like ${varName}. Required.
  - where - set of property used variable definitions. Optional:
    - [variable name] - variable name
    - [variable values] - variable values string delimited (if variable has multiple values) by | symbol
  - transparent - flag indicating if this property definition produces transient snapshot property. Optional, default value - false.

Sample aggregator configuration may be like that:

{
    "aggregatorId": "MLSnapshotsAggregator",
    "type": "com.jkoolcloud.tnt4j.stream.jmx.aggregations.SnapshotAggregator",
    "enabled": true,
    "snapshots": [
        {
            "name": "kafka.aggregated:KafkaStatsML",
            "enabled": true,
            "properties": [
                {
                    "beanId": "kafka.server:name=UnderReplicatedPartitions,type=ReplicaManager",
                    "attribute": "Value",
                    "name": "UnderReplicatedPartitions"
                },
                {
                    "beanId": "kafka.server:name=IsrShrinksPerSec,type=ReplicaManager",
                    "attribute": "MeanRate",
                    "name": "IsrShrinksPerSec"
                },
                {
                    "beanId": "kafka.server:name=IsrExpandsPerSec,type=ReplicaManager",
                    "attribute": "MeanRate",
                    "name": "IsrExpandsPerSec"
                },
                {
                    "beanId": "kafka.controller:name=ActiveControllerCount,type=KafkaController",
                    "attribute": "Value",
                    "name": "ActiveControllerCount"
                },
                {
                    "beanId": "kafka.controller:name=OfflinePartitionsCount,type=KafkaController",
                    "attribute": "Value",
                    "name": "OfflinePartitionsCount"
                },
                {
                    "beanId": "kafka.controller:name=LeaderElectionRateAndTimeMs,type=ControllerStats",
                    "attribute": "Mean",
                    "name": "LeaderElectionRateAndTimeMs"
                },
                {
                    "beanId": "kafka.controller:name=UncleanLeaderElectionsPerSec,type=ControllerStats",
                    "attribute": "MeanRate",
                    "name": "UncleanLeaderElectionsPerSec"
                },
                {
                    "beanId": "kafka.network:name=TotalTimeMs,request=?,type=RequestMetrics",
                    "where": {
                        "request": "Produce|FetchConsumer|FetchFollower"
                    },
                    "attribute": "Mean",
                    "name": "${request}-TotalTimeMs"
                },
                {
                    "beanId": "kafka.server:delayedOperation=?,name=PurgatorySize,type=DelayedOperationPurgatory",
                    "where": {
                        "delayedOperation": "Produce|Fetch"
                    },
                    "attribute": "Value",
                    "name": "${delayedOperation}-PurgatorySize"
                },
                {
                    "beanId": "kafka.server:name=BytesInPerSec,type=BrokerTopicMetrics",
                    "attribute": "MeanRate",
                    "name": "BytesInPerSec"
                },
                {
                    "beanId": "kafka.server:name=BytesOutPerSec,type=BrokerTopicMetrics",
                    "attribute": "MeanRate",
                    "name": "BytesOutPerSec"
                },
                {
                    "beanId": "kafka.network:name=RequestsPerSec,request=?,type=RequestMetrics,*",
                    "where": {
                        "request": "Produce|FetchConsumer|FetchFollower"
                    },
                    "attribute": "Count",
                    "name": "${request}-RequestsPerSec"
                }
            ]
        },
        {
            "name": ".*",
            "category": ".*",
            "enabled": true,
            "properties": [
                {
                    "beanId": "kafka.server:type=KafkaServer,name=ClusterId",
                    "attribute": "Value",
                    "name": "kafkaClusterId",
                    "transparent": true
                },
                {
                    "beanId": "kafka.server:type=app-info,id=*",
                    "attribute": "id",
                    "name": "kafkaBrokerId",
                    "transparent": true
                },
                {
                    "attribute": "${kafkaBrokerId}-${kafkaClusterId}",
                    "name": "kafkaNode"
                }
            ]
        }
    ]
}

Where do the streams go?

Stream-JMX streams all collected metrics based on a scheduled interval via TNT4J event streaming framework. All streams are written into TNT4J event sinks defined in tnt4j.properties file which is defined by -Dtnt4j.config=tnt4j.properties property.

To meshIQ: (Requires JESL libraries)

Below is an example of TNT4J stream configuration writing collected JMX samples to jKoolCloud server, formatted as JSON:

;Stanza used for Stream-JMX sources
{
    source: com.jkoolcloud.tnt4j.stream.jmx
    source.factory: com.jkoolcloud.tnt4j.stream.jmx.source.JMXSourceFactoryImpl
    source.factory.GENERIC: Streams
    source.factory.GEOADDR: New York
    source.factory.DATACENTER: YourDC
    source.factory.SERVICE: $sjmx.serviceId
    source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GEOADDR=?#GENERIC=?
    source.factory.RootSSN: tnt4j-stream-jmx
    source.factory.RetryIntervalSec: 1
    source.factory.MaxRetryAttempts: 10

    tracker.factory: com.jkoolcloud.tnt4j.tracker.DefaultTrackerFactory
    dump.sink.factory: com.jkoolcloud.tnt4j.dump.DefaultDumpSinkFactory
    event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.BufferedEventSinkFactory
    event.sink.factory.PooledLoggerFactory: com.jkoolcloud.tnt4j.sink.impl.PooledLoggerFactoryImpl

    ; Event Sink configuration for streaming to jKoolCloud
    ; NOTE: Requires JESL libraries (http://nastel.github.io/JESL/)
    event.sink.factory.EventSinkFactory: com.jkoolcloud.jesl.tnt4j.sink.JKCloudEventSinkFactory
    event.sink.factory.EventSinkFactory.LogSink: file:./logs/stream-jmx_activities.json
    event.sink.factory.EventSinkFactory.Url: https://stream.meshiq.com
    event.sink.factory.EventSinkFactory.Token: YOUR-ACCESS-TOKEN
    event.formatter: com.jkoolcloud.tnt4j.format.JSONFormatter

    ; Configure default sink filter based on level and time (elapsed/wait)
    event.sink.factory.Filter: com.jkoolcloud.tnt4j.filters.EventLevelTimeFilter
    event.sink.factory.Filter.Level: TRACE

    tracking.selector: com.jkoolcloud.tnt4j.selector.DefaultTrackingSelector
    tracking.selector.Repository: com.jkoolcloud.tnt4j.repository.FileTokenRepository
}

To socket

Below is an example of TNT4J stream configuration writing collected JMX samples to a socket event sink com.jkoolcloud.tnt4j.sink.impl.SocketEventSinkFactory, formatted by com.jkoolcloud.tnt4j.stream.jmx.format.FactNameValueFormatter:

;Stanza used for Stream-JMX sources
{
    source: com.jkoolcloud.tnt4j.stream.jmx
    source.factory: com.jkoolcloud.tnt4j.stream.jmx.source.JMXSourceFactoryImpl
    source.factory.GENERIC: Streams
    source.factory.GEOADDR: New York
    source.factory.DATACENTER: YourDC
    source.factory.SERVICE: $sjmx.serviceId
    source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GEOADDR=?#GENERIC=?
    source.factory.RootSSN: tnt4j-stream-jmx
    source.factory.RetryIntervalSec: 1
    source.factory.MaxRetryAttempts: 10

    tracker.factory: com.jkoolcloud.tnt4j.tracker.DefaultTrackerFactory
    dump.sink.factory: com.jkoolcloud.tnt4j.dump.DefaultDumpSinkFactory
    event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.BufferedEventSinkFactory
    event.sink.factory.PooledLoggerFactory: com.jkoolcloud.tnt4j.sink.impl.PooledLoggerFactoryImpl

    event.sink.factory.EventSinkFactory: com.jkoolcloud.tnt4j.sink.impl.SocketEventSinkFactory
    ; If socket sent data should not be logged anywhere else
    event.sink.factory.EventSinkFactory.LogSink: null
    ; If socket sent data should be logged to file
    #event.sink.factory.EventSinkFactory.LogSink: file:./logs/tnt4j-stream-jmx_samples_socket.log
    event.sink.factory.EventSinkFactory.Host: localhost
    event.sink.factory.EventSinkFactory.Port: 6060
    ; NOTE: DO NOT define "event.formatter" property value if there is no need for custom formatter.
    ;       SamplerFactory will take care to set appropriate one for a context.
    #event.formatter: com.jkoolcloud.tnt4j.format.JSONFormatter
    ; If JMX attributes should be formatted as JMX object names
    event.formatter: com.jkoolcloud.tnt4j.stream.jmx.format.FactNameValueFormatter
    ; If JMX attributes should be formatted as JMX object paths
    #event.formatter: com.jkoolcloud.tnt4j.stream.jmx.format.FactPathValueFormatter
    ; If JMX attributes should be formatted as JMX object paths for IBM WAS and Liberty
    #event.formatter: com.jkoolcloud.tnt4j.stream.jmx.format.SLIFactPathValueFormatter
    ; Whether to add Activity/Event context data snapshot 'Self'
    #event.formatter.AddSelfSnapshot: false
    ; Whether to add AutoPilot fact value type prefixes for fact names
    #event.formatter.AddAPValueTypePrefix: true
    ; Mapping of attribute key string symbol replacements
    #event.formatter.KeyReplacements: " "->"_" "\""->"'" "/"->"%" "="->"\\" ","->"!" "\\\\"->"\\"
    ; Mapping of attribute value string symbol replacements
    #event.formatter.ValueReplacements: "\r"->"\\r" "\n"->"\\n" ";"->"|" ","->"|" "["->"{(" "]"->")}" "\""->"'"
    ; Definitions ObjectName attributes sets used when building path: ';' is level set delimiter and ',' is set attribute names delimiter
    #event.formatter.PathLevelAttributes:  domain; type; name, brokerName; service, connector, destinationType; instanceName, connectorName, destinationName
    ; Defines JMX sample attribute key suffix to be added when duplicate keys for "branch" and "leaf" nodes are found.
    ; NOTE: AP does not allow to have same name for "branch" and "leaf" nodes at same tree level
    #event.formatter.DuplicateKeySuffix: ___

    ; Configure default sink filter based on level and time (elapsed/wait)
    event.sink.factory.Filter: com.jkoolcloud.tnt4j.filters.EventLevelTimeFilter
    event.sink.factory.Filter.Level: TRACE

    tracking.selector: com.jkoolcloud.tnt4j.selector.DefaultTrackingSelector
    tracking.selector.Repository: com.jkoolcloud.tnt4j.repository.FileTokenRepository
}

To file

Below is an example of TNT4J stream configuration writing collected JMX samples to a log file MyStream.log, formatted by com.jkoolcloud.tnt4j.stream.jmx.format.FactNameValueFormatter:

;Stanza used for Stream-JMX sources
{
    source: com.jkoolcloud.tnt4j.stream.jmx
    source.factory: com.jkoolcloud.tnt4j.stream.jmx.source.JMXSourceFactoryImpl
    source.factory.GENERIC: Streams
    source.factory.GEOADDR: New York
    source.factory.DATACENTER: YourDC
    source.factory.SERVICE: $sjmx.serviceId
    source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GEOADDR=?#GENERIC=?
    source.factory.RootSSN: tnt4j-stream-jmx
    source.factory.RetryIntervalSec: 1
    source.factory.MaxRetryAttempts: 10

    tracker.factory: com.jkoolcloud.tnt4j.tracker.DefaultTrackerFactory
    dump.sink.factory: com.jkoolcloud.tnt4j.dump.DefaultDumpSinkFactory
    event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.BufferedEventSinkFactory
    event.sink.factory.PooledLoggerFactory: com.jkoolcloud.tnt4j.sink.impl.PooledLoggerFactoryImpl

    event.sink.factory.EventSinkFactory: com.jkoolcloud.tnt4j.sink.impl.FileEventSinkFactory
    event.sink.factory.EventSinkFactory.FileName: ./MyStream.log
    ; NOTE: DO NOT define "event.formatter" property value if there is no need for custom formatter.
    ;       SamplerFactory will take care to set appropriate one for a context.
    #event.formatter: com.jkoolcloud.tnt4j.format.JSONFormatter
    ; If JMX attributes should be formatted as JMX object names
    event.formatter: com.jkoolcloud.tnt4j.stream.jmx.format.FactNameValueFormatter
    ; If JMX attributes should be formatted as JMX object paths
    #event.formatter: com.jkoolcloud.tnt4j.stream.jmx.format.FactPathValueFormatter
    ; If JMX attributes should be formatted as JMX object paths for IBM WAS and Liberty
    #event.formatter: com.jkoolcloud.tnt4j.stream.jmx.format.SLIFactPathValueFormatter
    ; Whether to add Activity/Event context data snapshot 'Self'
    #event.formatter.AddSelfSnapshot: false
    ; Whether to add AutoPilot fact value type prefixes for fact names
    #event.formatter.AddAPValueTypePrefix: true
    ; Mapping of attribute key string symbol replacements
    #event.formatter.KeyReplacements: " "->"_" "\""->"'" "/"->"%" "="->"\\" ","->"!" "\\\\"->"\\"
    ; Mapping of attribute value string symbol replacements
    #event.formatter.ValueReplacements: "\r"->"\\r" "\n"->"\\n" ";"->"|" ","->"|" "["->"{(" "]"->")}" "\""->"'"
    ; Definitions ObjectName attributes sets used when building path: ';' is level set delimiter and ',' is set attribute names delimiter
    #event.formatter.PathLevelAttributes:  domain; type; name, brokerName; service, connector, destinationType; instanceName, connectorName, destinationName
    ; Defines JMX sample attribute key suffix to be added when duplicate keys for "branch" and "leaf" nodes are found.
    ; NOTE: AP does not allow to have same name for "branch" and "leaf" nodes at same tree level
    #event.formatter.DuplicateKeySuffix: ___

    ; Configure default sink filter based on level and time (elapsed/wait)
    event.sink.factory.Filter: com.jkoolcloud.tnt4j.filters.EventLevelTimeFilter
    event.sink.factory.Filter.Level: TRACE

    tracking.selector: com.jkoolcloud.tnt4j.selector.DefaultTrackingSelector
    tracking.selector.Repository: com.jkoolcloud.tnt4j.repository.FileTokenRepository
}

And even more

You can also write your own custom event sinks (HTTPS, HTTP, etc.) and your own stream formatters without having to change Stream-JMX code or your application. TNT4J comes with a set of built-in event sink implementations such as:

com.jkoolcloud.tnt4j.logger.Log4JEventSinkFactory -- log4j
com.jkoolcloud.tnt4j.sink.impl.BufferedEventSinkFactory -- buffered sink
com.jkoolcloud.tnt4j.sink.impl.FileEventSinkFactory - standard log file
com.jkoolcloud.tnt4j.sink.impl.SocketEventSinkFactory -- socket (tcp/ip)
com.jkoolcloud.tnt4j.sink.impl.NullEventSinkFactory -- null (empty)

Streaming to multiple endpoints

Consider one of most common cases when Stream-JMX produced samples shall appear on both AutoPilot and jKool at same time. To achieve that, BroadcastingEventSinkFactory shall be used and have two sinks bound: for AutoPilot endpoint - ap (SocketEventSinkFactory) and for jKool endpoint - jkool (JKCloudEventSinkFactory). This way same JMX sample will be sent to AutoPilot over configured socket having sample data formatted by FactNameValueFormatter and to jKool repo defined by access token having sample data formatted by JSONFormatter.

;Stanza used for Stream-JMX sources
{
    source: com.jkoolcloud.tnt4j.stream.jmx
    source.factory: com.jkoolcloud.tnt4j.stream.jmx.source.JMXSourceFactoryImpl
    source.factory.GENERIC: Streams
    source.factory.GEOADDR: New York
    source.factory.DATACENTER: YourDC
    source.factory.SERVICE: $sjmx.serviceId
    source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GEOADDR=?#GENERIC=?
    source.factory.RootSSN: tnt4j-stream-jmx
    source.factory.RetryIntervalSec: 1
    source.factory.MaxRetryAttempts: 10

    tracker.factory: com.jkoolcloud.tnt4j.tracker.DefaultTrackerFactory
    dump.sink.factory: com.jkoolcloud.tnt4j.dump.DefaultDumpSinkFactory
    event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.BroadcastingEventSinkFactory
    event.sink.factory.BroadcastSequence: ap,jkool

    event.sink.factory.EventSinkFactory.ap: com.jkoolcloud.tnt4j.sink.impl.SocketEventSinkFactory
    ; If socket sent data should not be logged anywhere else
    event.sink.factory.EventSinkFactory.ap.LogSink: null
    ; If socket sent data should be logged to file
    #event.sink.factory.EventSinkFactory.ap.LogSink: file:./logs/tnt4j-stream-jmx_samples_socket.log
    event.sink.factory.EventSinkFactory.ap.Host: localhost
    event.sink.factory.EventSinkFactory.ap.Port: 6060
    event.sink.factory.EventSinkFactory.ap.Formatter: com.jkoolcloud.tnt4j.stream.jmx.format.FactNameValueFormatter
    #event.sink.factory.EventSinkFactory.ap.Formatter.AddSelfSnapshot: false
    #event.sink.factory.EventSinkFactory.ap.Formatter.AddAPValueTypePrefix: true
    event.sink.factory.EventSinkFactory.ap.Formatter.KeyReplacements: " "->"_" "\""->"'" "/"->"%" "="->"\\" ","->"!" "\\\\"->"\\"
    event.sink.factory.EventSinkFactory.ap.Formatter.ValueReplacements: "\r"->"\\r" "\n"->"\\n" ";"->"&" ","->"&" "["->"{(" "]"->")}" "\""->"'"

    event.sink.factory.EventSinkFactory.jkool: com.jkoolcloud.jesl.tnt4j.sink.JKCloudEventSinkFactory
    event.sink.factory.EventSinkFactory.jkool.LogSink: file:./logs/stream-jmx_activities.json
    event.sink.factory.EventSinkFactory.jkool.Url: https://stream.meshiq.com
    event.sink.factory.EventSinkFactory.jkool.Token: YOUR-ACCESS-TOKEN
    event.sink.factory.EventSinkFactory.jkool.Formatter: com.jkoolcloud.tnt4j.format.JSONFormatter

    ; Configure default sink filter based on level and time (elapsed/wait)
    event.sink.factory.Filter: com.jkoolcloud.tnt4j.filters.EventLevelTimeFilter
    event.sink.factory.Filter.Level: TRACE

    tracking.selector: com.jkoolcloud.tnt4j.selector.DefaultTrackingSelector
    tracking.selector.Repository: com.jkoolcloud.tnt4j.repository.FileTokenRepository
}

Auto-generating application state dump

Stream-JMX is utilizing TNT4J state dump capability to generate application state dumps

(1) Dump on VM shut-down:

java -Dtnt4j.dump.on.vm.shutdown=true -Dtnt4j.dump.provider.default=true -Dtnt4j.dump.folder=./ ...

(2) Dump on uncaught thread exceptions:

java -Dtnt4j.dump.on.exceptionn=true -Dtnt4j.dump.provider.default=true -Dtnt4j.dump.folder=./ ...

-Dtnt4j.dump.folder=./ specifies the destination folder where dump (.dump) files will be created (default is current working directory).

By default, Stream-JMX will generate dumps with the following info:

Java Properties Dump -- PropertiesDumpProvider
Java Runtime Dump -- MXBeanDumpProvider
Thread Stack Dump -- ThreadDumpProvider
Thread Deadlock Dump -- ThreadDumpProvider
Logging Statistics Dump -- LoggerDumpProvider

You may create your own dump providers and handlers

Overriding default `SamplerFactory`

SamplerFactory instances are used to generate Sampler implementation for a specific runtime environment. Stream-JMX supplies sampler and sampler factories for standard JVMs, J2EE, JBoss, IBM WebSphere Application Server, IBM WebSphere Liberty server. You may want to override default SamplerFactory with your own or an alternative by specifying:

java -Dcom.jkoolcloud.tnt4j.stream.jmx.sampler.factory=com.jkoolcloud.tnt4j.stream.jmx.impl.PlatformSamplerFactory ...

SamplerFactory is used to generate instances of the underlying sampler implementations (objects that provide sampling of underlying mbeans).

// return default or user defined SamplerFactory implementation
SamplerFactory factory = DefaultSamplerFactory.getInstance();
...

Managing Sample Behavior

Stream-JMX provides a way to intercept sampling events such as pre, during a post for each sample run and control sample behavior. See SampleListener interface for more details. Applications may register more than one listener per Sampler. Each listener is called in registration order.

In addition to intercepting sample events, applications may want to control how one or more attributes are sampled and whether each sample is reported/logged. See example below:

// return default or user defined SamplerFactory implementation
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance();
Map<String, Object> samplerCfg = new HashMap<>(3);
samplerCfg.put(SampleHandler.CFG_INCLUDE_FILTER, Sampler.JMX_FILTER_ALL);
samplerCfg.put(Sampler.CFG_SAMPLING_PERIOD, 30000);
sampler.setSchedule(samplerCfg).addListener(new MySampleListener())).run();

Below is a sample of what MySampleListener may look like:

class MySampleListener implements SampleListener {
    @Override
    public void getStats(SampleContext context, Map<String, Object> stats) {
        // add your own stats to the map
    }

    @Override
    public void register(SampleContext context, ObjectName oName) {
        System.out.println("Register mbean: " + oName + ", mbean.server=" + context.getMBeanServer());
    }

    @Override
    public void unregister(SampleContext context, ObjectName oName) {
        System.out.println("Unregister mbean: " + oName + ", mbean.server=" + context.getMBeanServer());
    }

    @Override
    public void pre(SampleContext context, Activity activity) {
        // called once per sample, beginning of each sample
        // set activity to NOOP to disable further sampling
        // no other attribute will be sampled during current sample
        if (some-condition) {
            activity.setType(OpType.NOOP);
        }
    }

    @Override
    public void pre(SampleContext context, AttributeSample sample) {
        // called once before attribute is sampled
        // set exclude to true to skip sampling this attribute
        sample.excludeNext(sample.getAttributeInfo().isReadable());
    }

    @Override
    public void post(SampleContext context, AttributeSample sample) {
        // called once after attribute is sampled
        Object value = sample.get();
    }

    @Override
    public void post(SampleContext context, Activity activity) {
        // called once per sample, end of each sample
        // set activity to NOOP to disable sampling reporting
        if (some-condition) {
            activity.setType(OpType.NOOP);
        }
    }

    @Override
    public void error(SampleContext context, Throwable ex) {
        // called once for every exception that occurs not associated with a sample
        ex.printStackTrace();
    }

    @Override
    public void error(SampleContext context, AttributeSample sample) {
        // called once for every exception that occurs during each sample
        Throwable ex = sample.getError();
        ex.printStackTrace();
    }
}

Conditions and Actions

Stream-JMX allows you to associate conditions with user defined actions based on values of MBean attributes on each sampling interval. For example, what if you wanted to set up an action when a specific MBean attribute exceeds a certain threshold?

Stream-JMX AttributeCondition and AttributeAction interfaces allow you to call your action at runtime every time a condition is evaluated to true. See example below:

// return default or user defined SamplerFactory implementation
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance();
// create a condition when ThreadCount > 100
AttributeCondition myCondition = new SimpleCondition("java.lang:type=Threading", "ThreadCount", 100, ">");
// schedule collection (ping) for given MBean filter and 30000 ms sampling period
Map<String, Object> samplerCfg = new HashMap<>(3);
samplerCfg.put(SampleHandler.CFG_INCLUDE_FILTER, Sampler.JMX_FILTER_ALL);
samplerCfg.put(Sampler.CFG_SAMPLING_PERIOD, 30000);
sampler.setSchedule(samplerCfg).register(myCondition, new MyAttributeAction()).run();

Below is a sample of what MyAttributeAction may look like:

public class MyAttributeAction implements AttributeAction {
    @Override
    public Object action(SampleContext context, AttributeCondition cond, AttributeSample sample) {
        Activity activity = sample.getActivity();
        // obtain a collection of all sampled metrics
        Collection<Snapshot> metrics = activity.getSnapshots();
        System.out.println("MyAction called with value=" + sample.get()
            + ", age.usec=" + sample.ageUsec()
            + ", count=" + metrics.size());
        return null;
    }
}

How to Build TNT4J-Stream-JMX

Modules

(M) marked modules are mandatory
(O) marked modules are optional
(U) marked modules are utility modules (e.g., performs compiled assemblies packaging)

Modules list:

Core (M) - implements core JMX sampling, processing and dissemination features. Maven dependency:

   <dependency>
       <groupId>com.jkoolcloud.tnt4j.stream</groupId>
       <artifactId>tnt4j-stream-jmx-core</artifactId>
       <version>2.1.1</version>
   </dependency>

J2EE (O) - J2EE API addition to handle J2EE API defined attributes and implements abstract Stream-JMX servlet API. Maven dependency:

   <dependency>
       <groupId>com.jkoolcloud.tnt4j.stream</groupId>
       <artifactId>tnt4j-stream-jmx-j2ee</artifactId>
       <version>2.1.1</version>
   </dependency>

WAS (O, requires J2EE) - IBM WebSphere Application Server (WAS) root module.

Api (O) - builds specific API used to sample WAS JMX (jar). Maven dependency:

   <dependency>
       <groupId>com.jkoolcloud.tnt4j.stream</groupId>
       <artifactId>tnt4j-stream-jmx-was-api</artifactId>
       <version>2.1.1</version>
    </dependency>

War (O) - builds WAS compliant web application package (war). Maven dependency:

   <dependency>
       <groupId>com.jkoolcloud.tnt4j.stream</groupId>
       <artifactId>tnt4j-stream-jmx-was-war</artifactId>
       <version>2.1.1</version>
    </dependency>

Ear (O) - builds WAS compliant enterprise application package (ear). Maven dependency:

   <dependency>
       <groupId>com.jkoolcloud.tnt4j.stream</groupId>
       <artifactId>tnt4j-stream-jmx-was-ear</artifactId>
       <version>2.1.1</version>
    </dependency>

Liberty (O, requires J2EE) - IBM WebSphere Liberty root module.

Api (O) - builds specific API used to sample Liberty JMX (jar). Maven dependency:

   <dependency>
       <groupId>com.jkoolcloud.tnt4j.stream</groupId>
       <artifactId>tnt4j-stream-jmx-liberty-api</artifactId>
       <version>2.1.1</version>
    </dependency>

War (O) - builds Liberty compliant web application package (war). Maven dependency:

   <dependency>
       <groupId>com.jkoolcloud.tnt4j.stream</groupId>
       <artifactId>tnt4j-stream-jmx-liberty-war</artifactId>
       <version>2.1.1</version>
    </dependency>

ZK (O) - JMX connections resolution for VMs orchestrated by ZooKeeper. Maven dependency:

   <dependency>
       <groupId>com.jkoolcloud.tnt4j.stream</groupId>
       <artifactId>tnt4j-stream-jmx-zk</artifactId>
       <version>2.1.1</version>
    </dependency>

Distribution (OU) - build distribution packages. After running, see build/ directory.

All optional modules (extensions) depends on core module and can't be build and run without it.

NOTE: Distribution module performs Maven post-build release assemblies delivery to build/ directory.

Requirements

All other required dependencies are defined in project modules pom.xml files. If Maven is running online mode it should download these defined dependencies automatically.

Manually installed dependencies

Some of required and optional dependencies may be not available in public Maven Repository. In this case we would recommend to download those dependencies manually into module's lib directory and install into local Maven repository by running Maven script lib/pom.xml with install goal. For example see tnt4j-stream-jmx/tnt4j-stream-jmx-was/tnt4j-stream-jmx-was-api/lib/pom.xml how to do this.

`Core` module

This module does not require manually downloaded dependencies, but depends on JDK contained attach instrumentation library.

For Java versions prior 9, dependency is defined in:

Maven POM script by

  <dependency>
      <groupId>com.sun</groupId>
      <artifactId>tools</artifactId>
      <version>${project.java.version}</version>
      <scope>system</scope>
      <systemPath>${java.home}/../lib/tools.jar</systemPath>
  </dependency>

System executables bin/stream-jmx*.bat or bin/stream-jmx*.sh by variable TOOLS_PATH
- .bat/cmd
```
set TOOLS_PATH="%JAVA_HOME%\lib\tools.jar"
```
- .sh
```
TOOLS_PATH="$JAVA_HOME/lib/tools.jar"
```

For Java versions 9+, jdk.attach module is enabled in:

Maven POM script by adding maven-compiler-plugin plugin configuration

  <compilerArgs>
      <arg>--add-exports=jdk.attach/sun.tools.attach=ALL-UNNAMED</arg>
      <arg>--add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED</arg>
  </compilerArgs>

System executables bin/stream-jmx*.bat or bin/stream-jmx*.sh by variable TNT4JOPTS

.bat/cmd

TNT4JOPTS="--add-exports=jdk.attach/sun.tools.attach=ALL-UNNAMED" "--add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED"

.sh

TNT4JOPTS="--add-exports=jdk.attach/sun.tools.attach=ALL-UNNAMED --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED"

NOTE: you may need to change paths if these do not match your environment.

`WAS` module

NOTE: Because this module requires manually downloaded libraries, it is commented out in main project pom file tnt4j-stream-jmx/pom.xml by default. If you want to use it uncomment this line of pom.xml file. But WAS module will be ready to build only when manually downloaded libraries will be installed to local Maven repository.

What to download manually or copy from your existing IBM MQ installation:

IBM WAS (e.g., version 8.5) client libs
j2ee API library

Download the above libraries and place into the tnt4j-stream-jmx/tnt4j-stream-jmx-was/tnt4j-stream-jmx-was-api/lib directory like this:

    lib
     + was
         |- com.ibm.ws.admin.client_8.5.0.jar
         |- com.ibm.ws.ejb.thinclient_8.5.0.jar
         |- com.ibm.ws.orb_8.5.0.jar

(O) marked libraries are optional

Additionally alter system executables bin/stream-jmx*.bat or bin/stream-jmx*.sh by extending environment variable LIBPATH value:

MS Windows

set WAS_HOME=C:\IBM\WebSphere\AppServer
set WAS_PATH=%WAS_HOME%\runtimes\*;%WAS_HOME%\lib\webadmin\management.jar;%WAS_HOME%\plugins\com.ibm.ws.runtime.jar
set LIBPATH=%RUNDIR%..\*;%RUNDIR%..\lib\*;%WAS_PATH%

*nix

WAS_HOME="/opt/IBM/WebSphere/AppServer"
WAS_PATH="$WAS_HOME/runtimes/*:$WAS_HOME/lib/webadmin/management.jar:$WAS_HOME/plugins/com.ibm.ws.runtime.jar"
LIBPATH="$SCRIPTPATH/../*:$SCRIPTPATH/../lib/*:$WAS_PATH"

If you don't have actual WAS installation, WAS_PATH may also refer to jars located in tnt4j-stream-jmx/tnt4j-stream-jmx-was/lib/* directory.

Building

To build the project, run Maven goals clean package
To build the project and install to local repo, run Maven goals clean install
To make distributable release assemblies use one of profiles: pack-bin or pack-all:
- containing only binary (including test package) distribution: run mvn -P pack-bin
- containing binary (including test package), source and javadoc distribution: run mvn -P pack-all
To make Maven required source and javadoc packages, use profile pack-maven
To make Maven central compliant release having source, javadoc and all signed packages, use maven-release profile

By default, Maven will build all modules defined in tnt4j-stream-jmx/pom.xml file.

If you do not want to build some of optional modules, comment those out like WAS module is. Or you can define Maven to build your preferred set of modules using -pl, --projects argument (comma separated modules list) together with -am, --also-make argument, e.g.:

mvn -pl tnt4j-stream-jmx-core,tnt4j-stream-jmx--distribution -am clean install -P pack-bin

mvn --projects tnt4j-stream-jmx-core,tnt4j-stream-jmx--distribution --also-make clean install -P pack-bin

NOTE: modules list should be without spaces after comma!

Issuing these commands, Maven will build only tnt4j-stream-jmx-core and tnt4j-stream-jmx--distribution modules.

Release assemblies are built to build/ directory.

NOTE: sometimes Maven fails to correctly handle dependencies. If dependency configuration looks fine, but Maven still complains about missing dependencies try to delete local Maven repository by hand: e.g., on MS Windows delete contents of c:\Users\[username]\.m2\repository directory.

Running samples

See Using Stream-JMX section.

tnt4j-stream-jmx tnt4j-stream-jmx copied to clipboard

Metadata

Stream-JMX

Major Log4j12 to Log4j2 migration notes to consider

Why Stream-JMX

Using Stream-JMX

Common command line arguments

Running Stream-JMX as -javaagent

Command line to run

Attaching Stream-JMX to running JVM

Command line to run

Coding into API

Connecting Stream-JMX to local or remote JMX service

Command line to run

To connect to local JVM process

To connect to JMX service over URL

Multiple VMs monitoring over single stream run

Multiple local JVM processes monitoring

Multiple VM's monitoring configuration using external file

Connections configuration using stanza divided properties file

Multiple VM's monitoring notes for Source

ZooKeeper orchestrated VMs access

How to migrate Source definitions from single to multi VM sampling

Connecting remote WebSphere Application Server (WAS)

Connecting remote WebLogic (version 12c) server instance

Coding into API

Sampling local process runner JVM

Command line to run

Coding into API

Running Stream-JMX as servlet

Generic servlet page features

WebSphere Application Server (WAS)

WebSphere Liberty Server

Embed Stream-JMX code into your application

Coding

Command line to run

Stream-JMX configuration

System properties used

Program arguments used

JMX Sampling Agent sampler options

TNT4J Source fields configuration

Stream-JMX event data formatters

MBeans collected data aggregations

Where do the streams go?

To meshIQ: (Requires JESL libraries)

To socket

To file

And even more

Streaming to multiple endpoints

Auto-generating application state dump

Overriding default SamplerFactory

Managing Sample Behavior

Conditions and Actions

How to Build TNT4J-Stream-JMX

Modules

Requirements

Manually installed dependencies

Core module

WAS module

Building

Running samples

Related Projects

Available Integrations

← Metadata

Owner

Metadata

tnt4j-stream-jmx
tnt4j-stream-jmx copied to clipboard

Major `Log4j12` to `Log4j2` migration notes to consider

Running Stream-JMX as `-javaagent`

Overriding default `SamplerFactory`

`Core` module

`WAS` module