Getting metrics from Graphite into Nagios and Centreon

Getting metrics from logs and various other sources into Graphite is quite simple. The most interesting metrics do represent critical performance data, and the pro-active-monitoring approach, means a person sitting there and waching the dashboard, isn’t suited to our needs. We use Nagios with Centreon as our monitoring plattform, and we want to alert on some of the metrics collected in Graphite. Also since version 2.4 Centreon supports custom dashboard views and, although this might sound like doublemobble, we wanted to get the metrics graphically integrated into the Centreon interface, as RRD graphs that is.

Looking around I found the check_graphite plugin by obfuscurity, and greatly enhanced it to support multiple metrics in one call, performance data with customizable metric shortnames and retry calls in case there were no datapoints in the given duration. It’s called check_graphite_multi, available from my nagios-scripts perfdata branch on github, and is especially usefull if you’d like to get multiple metrics of the same type into one RRD graph in Centreon or PNP4Nagios or thelike. Our usecase is a graph with JVM heap generation usage and garbage collector statistics. We alert on full old generation and high GC durations.

Here are some short usage notes:

–metrics|-m accepts a string of metrics, seperated by a pipe |

--metrics "scale(weblogic.server01.jvm.gc.oldgen_used_before_gc)|scale(weblogic.server01.jvm.gc.oldgen_used_after_gc)"

–shortname|-s accepts a comma separated list of aliases for the output of status and performance data

--shortname "jvm_eden_before_gc,jvm_eden_after_gc"

If no –shortname is specified for the given metric, it defaults to the full metric name.

–warn|-w also accepts a comma separated list

--warn "100,150"

At least one value is required, if only one value is given for multiple metrics, the given value counts for all

–critical|-c works the same as –warn

Also note:

When specifying multiple metrics, make sure to keep the order for all parameters, like

-m "metric1|metric2" -s "alias1,alias2" -w "warn1,warn2" -c "crit1,crit2"

If at least one of the metrics returns a CRITICAL state, the plugin exits with CRITICAL return code. Dito for WARNING.

By default, if the metric has no datapoints in the given –duration timeframe, the plugin retries with 10times the given duration. This is mostly cosmetic to prevent holes in RRD graphs, and I might make that configurable in the future. Unfortunately Graphite has no option via the render API to just return the last datapoint, so this is a hack to work around that.

Monitoring and graphing Weblogic performance using Graphite and metrics-sampler

My current project is to take our Weblogic monitoring setup from parsing gc logs in Splunk up to the next level. For other things metrics we do use Graphite already. Graphite is an awesome app for graphing any sort of metrics. You just need to get them in there somehow. Some days ago I stumbled over an outstanding piece of software written by Dimo Velev: metrics-sampler

It's able to read metrics from various inputs, like e.g. the JMX tree of the Weblogic Runtime MBean Server. Exactly this usecase is well documented in the configuration examples. The syntax is very well thought. It's not worth documenting it here or posting examples. Yust read the docs and the examples included, it's really straight forward.

Setup prerequisites:

For talking to Weblogic via JMX, you need to generate a wlfullclient.jar and put it in metrics-sampler/lib directory. See http://docs.oracle.com/cd/E15051_01/wls/docs103/client/jarbuilder.html for details on how to generate it.

By default, the Weblogic Runtime MBean Server does not expose the domains from the Plattform Mbean Server of the JVM, so any java.lang/* stuff can't be queried there. You either need to enable the JVM's remote management (which doesn't allow to specify a listenaddress for the MBean server and instead listens on all interfaces by default), or you configure the Weblogic Runtime MBean server to act as the Plattform Mbean server. The second option allows to query both, the com.bea and the JVM's java.lang domains – so every metric you can get – all from one source.

In our case it also was very impractical to install metrics sampler directly on the production weblogic cluster machines. Instead, I confiured a dedicated box with a SSH tunnel to the IP of every Weblogic node. As JMX requires the remote and "local" port to be the same, the tunnels were created using a loopback IP for every node on the local metrics-sampler box. So to speak, server01 was tunneled to 127.0.0.2, server02 to 127.0.0.3, server03 to 127.0.0.4, and so on. 

Once this was setup, it just required to edit the provided config.xml.template in metrics-sampler to match our DataSource namings, add some nodes to get samples from – and it was ready to go. This is really some well done JUST WORKS[tm] sort of software. I love it. Next step: mashing up dashboards in Graphite.

Clap clap clap!

Indexing and searching Weblogic logs using Logstash and Graylog2

Update 2013/10: we decided to replace Graylog2 with Kibana3 completely. The article below is just for reference, the logstash config is outdated since logstash 1.2 and the setup as described below is suboptimal anyway. I’ll post a new article shortly.

Update 2014/02: Finally, the new guide is here: Indexing and searching Weblogic logs using Logstash, Elasticsearch and Kibana.

 

Recently we decided to get rid of our Splunk “Free” log indexing solution as the 500MB limit is too limited for our 20+ Weblogic environments (we couldn’t even index all production apps with that) and $boss said “he won’t pay over 20000€ for a 2GB enterprise license, that’s just rediculous”. So I went out on the interwebs to watch out for alternatives, and stumbled over Graylog2 and Logstash. This stuff seemed to have potential, so I started playing around with it.

Logstash is a log pipeline that features various input methods, filters and output plugins. The basic process is to throw logs at it, parse the message for the correct date, split the message into fields if desired, and forward the result to some indexer and search it using some frontend. Logstash scales vertically, and for larger volumes it’s recommended to split the tasks of logshipping and log parsing to dedicated logstash instances. To avoid loosing logs when something goes down and to keep maintenance downtimes low, it’s also recommended to put some message queue between the shipper(s) and the parser(s).

Redis fits exactly in that pictures, and acts as a key-value message queue in the pipeline. Logstash has a hard coded queue size of 20 events per configured input. If the queue fills up, the input gets blocked. Using a dedicated message queue instead is a good thing to have.

Graylog2 consits of a server and a webinterface. The server stores the logs in Elasticsearch, the frontend lets you search the indexes.

So, our whole pipeline looks like this:

logfiles logstash shipper redis logstash indexer cluster gelf graylog2-server elasticsearch cluster graylog2-web-interface

Logstash is able to output to Elasticsearch directly, and there is the great Kibana frontend for it which is in many ways superior to graylog2-web-interface, but for reasons I explain at the end of the post we chose Graylog2 for weblogic logs.

Installation

The first step was to get graylog2, elasticsearch and mongodb up and running. We use RHEL 6, so this howto worked almost out of the box. I changed following:

  • latest stable elasticsearch-0.19.10
  • latest stable mongodb 2.2.0
  • default RHEL6 ruby 1.8.7 (so I left out any rvm stuff in that howto, and edited the provided scripts removing any rvm commands)

Prepare access to the logfiles for logstash

Next was to get logstash to index the logfiles correctly.

We decided to use SSHFS for mounting the logfile folders of all Weblogic instances onto a single box, and run the logstash shipper on that one using file input and output to redis. The reason for using SSHFS instead of installating logstash directly on the Weblogic machines and using for example a log4j appenders to logstash log4j inputs was mainly that our Weblogics are managed by a bank’s data centre, so getting new software installed requires a lot work. The SSH access was already in place.

We have weblogic server logs (usually the weblogic.log), and each application generates a log4j-style logfile.

/data/logfiles/prod/server01/app1.log
/data/logfiles/prod/server01/app2.log
/data/logfiles/prod/server01/weblogic.log
/data/logfiles/prod/server02/app1.log
...
/data/logfiles/qsu/server01/app1.log
... and so on

This is the configuration file for the file-to-redis shipper. The only filter in place is the multiline filter, so that multiline messages get stored in redis as a single event already.

input {
  # server logs
  file {
    type => "weblogic-log"
    path => [ "/data/logfiles/*/*/weblogic.log" ]
  }
  # application logs
  file {
    type => "application"
    path => [ "/data/logfiles/*/*/planethome.log",
              "/data/logfiles/*/*/marktplatz*.log",
              "/data/logfiles/*/*/immoplanet*.log",
              "/data/logfiles/*/*/planetphone.log" ]
  }
}
filter {
  # weblogic server log events always start with ####
  multiline {
    type => "weblogic"
    pattern => "^####"
    negate => true
    what => "previous"
  }
  # application logs use log4j syntax and start with the year. So below will work until 31.12.2099
  multiline {
    type => "application"
    pattern => "^20"
    negate => true
    what => "previous"
  }
}
output {
  redis {
    host => "phewu01"
    data_type => "list"
    key => "logstash-%{@type}"
  }
}

And this is the config for the logstash parsers. Here the main work happens, logs get parsed, fileds get extracted. This is CPU intensive, so depending on the amount of messages, you can simply add more instances with the same config.

input {
  redis {
    type => "weblogic"
    host => "phewu01"
    data_type => "list"
    key => "logstash-weblogic"
    message_format => "json_event"
  }
  redis {
    type => "application"
    host => "phewu01"
    data_type => "list"
    key => "logstash-application"
    message_format => "json_event"
  }
}

filter {
  ###################
  # weblogic server logs
  ###################
  grok {
     # extract server environment (prod, uat, dev etc..) from logfile path
     type => "weblogic"
     patterns_dir => "./patterns"
     match => ["@source_path", "%{PH_ENV:environment}"]
  }
  grok {
    type => "weblogic"
    pattern => ["####<%{DATA:wls_timestamp}> <%{WORD:severity}> <%{DATA:wls_topic}> <%{HOST:hostname}> <(%{WORD:server})?> %{GREEDYDATA:logmessage}"]
    add_field => ["application", "server"]
  }
  date {
    type => "weblogic"
    # joda-time doesn't know about localized CEST/CET (MESZ in German), 
    # so use 2 patterns to match the date
    wls_timestamp => ["dd.MM.yyyy HH:mm 'Uhr' 'MESZ'", "dd.MM.yyyy HH:mm 'Uhr' 'MEZ'"]
  }
  mutate {
    type => "weblogic"
    # set the "Host" in graylog to the environment the logs come from (prod, uat, etc..)
    replace => ["@source_host", "%{environment}"]
  }

  ######################
  # application logs
  ######################
  # match and pattern inside one single grok{} doesn't work
  # also using a hash in match didn't work as expected if the field is the same,
  # so split this into single grok{} directives
  grok {
    type => "application"
    patterns_dir => "./patterns"
    match => ["@source_path", "%{PH_ENV:environment}"]
  }
  grok {
    # extract app name from logfile name
    type => "application"
    patterns_dir => "./patterns"
    match => ["@source_path", "%{PH_APPS:application}"]
  }
  grok {
    # extract node name from logfile path
    type => "application"
    patterns_dir => "./patterns"
    match => ["@source_path", "%{PH_SERVERS:server}"]
  }
  grok {
    type => "application"
    pattern => "%{DATESTAMP:timestamp} %{DATA:some_id} %{WORD:severity} %{GREEDYDATA:logmessage}"
  }
  date {
    type => "application"
    timestamp => ["yyyy-MM-dd HH:mm:ss,SSS"]
  }
  mutate {
    type => "application"
    replace => ["@source_host", "%{environment}"]
  }
}

output {
  gelf {
    host => "localhost"
    facility => "%{@type}"
  }
}

In ./patterns there is a file containing 3 lines for the PH_ENV etc grok patterns to match.

I had several issues initially:

  • rsync to jumpbox and mounting to logserver via ssfhs would cause changes to go missing on some, but not all files
  • chaining rsyncs would cause some messages to be indexed with partial @message, as some files are huge and slow to transfer.
    • This was solved by mounting all logfile folders on the different environments directly with SSHFS, where possible.
    • The remaining rsync’ed files are rsync’ed with “–append –inplace” rsync parameters
  • indexing files would always start at position 0 of the file, over and over again
    • Only happened for rsync, using “–append –inplace” fixes this

Meanwhile I also took a look at Kibana, another great frontend to ElasticSearch. For the weblogic logs I’ll keep Graylog2, as it allows saving predefined streams and provides that easy-to-use quickfilter, which eases up log crawling for our developers (they only want to search for a string in a timeframe – they also never made use of the power of Splunk). Also Kibana doesn’t provide a way to view long stack traces in an acceptable fashion yet (cut message a n characters and provide a more link, something like that). But I added Apache logs in logstash, and those are routed to a logstash elasticsearch output and Kibana as WebUI. I’d really like to see some sort of merge between Kibana and Graylog2 and add saved searches to that mix – that would make a realy competitive Splunk alternative.