Monitoring and graphing Weblogic performance using Graphite and metrics-sampler

My current project is to take our Weblogic monitoring setup from parsing gc logs in Splunk up to the next level. For other things metrics we do use Graphite already. Graphite is an awesome app for graphing any sort of metrics. You just need to get them in there somehow. Some days ago I stumbled over an outstanding piece of software written by Dimo Velev: metrics-sampler

It's able to read metrics from various inputs, like e.g. the JMX tree of the Weblogic Runtime MBean Server. Exactly this usecase is well documented in the configuration examples. The syntax is very well thought. It's not worth documenting it here or posting examples. Yust read the docs and the examples included, it's really straight forward.

Setup prerequisites:

For talking to Weblogic via JMX, you need to generate a wlfullclient.jar and put it in metrics-sampler/lib directory. See http://docs.oracle.com/cd/E15051_01/wls/docs103/client/jarbuilder.html for details on how to generate it.

By default, the Weblogic Runtime MBean Server does not expose the domains from the Plattform Mbean Server of the JVM, so any java.lang/* stuff can't be queried there. You either need to enable the JVM's remote management (which doesn't allow to specify a listenaddress for the MBean server and instead listens on all interfaces by default), or you configure the Weblogic Runtime MBean server to act as the Plattform Mbean server. The second option allows to query both, the com.bea and the JVM's java.lang domains - so every metric you can get - all from one source.

In our case it also was very impractical to install metrics sampler directly on the production weblogic cluster machines. Instead, I confiured a dedicated box with a SSH tunnel to the IP of every Weblogic node. As JMX requires the remote and "local" port to be the same, the tunnels were created using a loopback IP for every node on the local metrics-sampler box. So to speak, server01 was tunneled to 127.0.0.2, server02 to 127.0.0.3, server03 to 127.0.0.4, and so on. 

Once this was setup, it just required to edit the provided config.xml.template in metrics-sampler to match our DataSource namings, add some nodes to get samples from - and it was ready to go. This is really some well done JUST WORKS[tm] sort of software. I love it. Next step: mashing up dashboards in Graphite.

Clap clap clap!