Indexing and searching Weblogic logs using Logstash and Graylog2

Update 2013/10: we decided to replace Graylog2 with Kibana3 completely. The article below is just for reference, the logstash config is outdated since logstash 1.2 and the setup as described below is suboptimal anyway. I'll post a new article shortly.

Update 2014/02: Finally, the new guide is here: Indexing and searching Weblogic logs using Logstash, Elasticsearch and Kibana.

 

Recently we decided to get rid of our Splunk "Free" log indexing solution as the 500MB limit is too limited for our 20+ Weblogic environments (we couldn't even index all production apps with that) and $boss said "he won't pay over 20000€ for a 2GB enterprise license, that's just rediculous". So I went out on the interwebs to watch out for alternatives, and stumbled over Graylog2 and Logstash. This stuff seemed to have potential, so I started playing around with it.

Logstash is a log pipeline that features various input methods, filters and output plugins. The basic process is to throw logs at it, parse the message for the correct date, split the message into fields if desired, and forward the result to some indexer and search it using some frontend. Logstash scales vertically, and for larger volumes it's recommended to split the tasks of logshipping and log parsing to dedicated logstash instances. To avoid loosing logs when something goes down and to keep maintenance downtimes low, it's also recommended to put some message queue between the shipper(s) and the parser(s).

Redis fits exactly in that pictures, and acts as a key-value message queue in the pipeline. Logstash has a hard coded queue size of 20 events per configured input. If the queue fills up, the input gets blocked. Using a dedicated message queue instead is a good thing to have.

Graylog2 consits of a server and a webinterface. The server stores the logs in Elasticsearch, the frontend lets you search the indexes.

So, our whole pipeline looks like this:

logfiles logstash shipper redis logstash indexer cluster gelf graylog2-server elasticsearch cluster graylog2-web-interface

Logstash is able to output to Elasticsearch directly, and there is the great Kibana frontend for it which is in many ways superior to graylog2-web-interface, but for reasons I explain at the end of the post we chose Graylog2 for weblogic logs.

Installation

The first step was to get graylog2, elasticsearch and mongodb up and running. We use RHEL 6, so this howto worked almost out of the box. I changed following:

  • latest stable elasticsearch-0.19.10
  • latest stable mongodb 2.2.0
  • default RHEL6 ruby 1.8.7 (so I left out any rvm stuff in that howto, and edited the provided scripts removing any rvm commands)

Prepare access to the logfiles for logstash

Next was to get logstash to index the logfiles correctly.

We decided to use SSHFS for mounting the logfile folders of all Weblogic instances onto a single box, and run the logstash shipper on that one using file input and output to redis. The reason for using SSHFS instead of installating logstash directly on the Weblogic machines and using for example a log4j appenders to logstash log4j inputs was mainly that our Weblogics are managed by a bank's data centre, so getting new software installed requires a lot work. The SSH access was already in place.

We have weblogic server logs (usually the weblogic.log), and each application generates a log4j-style logfile.

/data/logfiles/prod/server01/app1.log
/data/logfiles/prod/server01/app2.log
/data/logfiles/prod/server01/weblogic.log
/data/logfiles/prod/server02/app1.log
...
/data/logfiles/qsu/server01/app1.log
... and so on

This is the configuration file for the file-to-redis shipper. The only filter in place is the multiline filter, so that multiline messages get stored in redis as a single event already.

input {
  # server logs
  file {
    type => "weblogic-log"
    path => [ "/data/logfiles/*/*/weblogic.log" ]
  }
  # application logs
  file {
    type => "application"
    path => [ "/data/logfiles/*/*/planethome.log",
              "/data/logfiles/*/*/marktplatz*.log",
              "/data/logfiles/*/*/immoplanet*.log",
              "/data/logfiles/*/*/planetphone.log" ]
  }
}
filter {
  # weblogic server log events always start with ####
  multiline {
    type => "weblogic"
    pattern => "^####"
    negate => true
    what => "previous"
  }
  # application logs use log4j syntax and start with the year. So below will work until 31.12.2099
  multiline {
    type => "application"
    pattern => "^20"
    negate => true
    what => "previous"
  }
}
output {
  redis {
    host => "phewu01"
    data_type => "list"
    key => "logstash-%{@type}"
  }
}

And this is the config for the logstash parsers. Here the main work happens, logs get parsed, fileds get extracted. This is CPU intensive, so depending on the amount of messages, you can simply add more instances with the same config.

input {
  redis {
    type => "weblogic"
    host => "phewu01"
    data_type => "list"
    key => "logstash-weblogic"
    message_format => "json_event"
  }
  redis {
    type => "application"
    host => "phewu01"
    data_type => "list"
    key => "logstash-application"
    message_format => "json_event"
  }
}

filter {
  ###################
  # weblogic server logs
  ###################
  grok {
     # extract server environment (prod, uat, dev etc..) from logfile path
     type => "weblogic"
     patterns_dir => "./patterns"
     match => ["@source_path", "%{PH_ENV:environment}"]
  }
  grok {
    type => "weblogic"
    pattern => ["####<%{DATA:wls_timestamp}> <%{WORD:severity}> <%{DATA:wls_topic}> <%{HOST:hostname}> <(%{WORD:server})?> %{GREEDYDATA:logmessage}"]
    add_field => ["application", "server"]
  }
  date {
    type => "weblogic"
    # joda-time doesn't know about localized CEST/CET (MESZ in German), 
    # so use 2 patterns to match the date
    wls_timestamp => ["dd.MM.yyyy HH:mm 'Uhr' 'MESZ'", "dd.MM.yyyy HH:mm 'Uhr' 'MEZ'"]
  }
  mutate {
    type => "weblogic"
    # set the "Host" in graylog to the environment the logs come from (prod, uat, etc..)
    replace => ["@source_host", "%{environment}"]
  }

  ######################
  # application logs
  ######################
  # match and pattern inside one single grok{} doesn't work
  # also using a hash in match didn't work as expected if the field is the same,
  # so split this into single grok{} directives
  grok {
    type => "application"
    patterns_dir => "./patterns"
    match => ["@source_path", "%{PH_ENV:environment}"]
  }
  grok {
    # extract app name from logfile name
    type => "application"
    patterns_dir => "./patterns"
    match => ["@source_path", "%{PH_APPS:application}"]
  }
  grok {
    # extract node name from logfile path
    type => "application"
    patterns_dir => "./patterns"
    match => ["@source_path", "%{PH_SERVERS:server}"]
  }
  grok {
    type => "application"
    pattern => "%{DATESTAMP:timestamp} %{DATA:some_id} %{WORD:severity} %{GREEDYDATA:logmessage}"
  }
  date {
    type => "application"
    timestamp => ["yyyy-MM-dd HH:mm:ss,SSS"]
  }
  mutate {
    type => "application"
    replace => ["@source_host", "%{environment}"]
  }
}

output {
  gelf {
    host => "localhost"
    facility => "%{@type}"
  }
}

In ./patterns there is a file containing 3 lines for the PH_ENV etc grok patterns to match.

I had several issues initially:

  • rsync to jumpbox and mounting to logserver via ssfhs would cause changes to go missing on some, but not all files
  • chaining rsyncs would cause some messages to be indexed with partial @message, as some files are huge and slow to transfer.
    • This was solved by mounting all logfile folders on the different environments directly with SSHFS, where possible.
    • The remaining rsync'ed files are rsync'ed with "--append --inplace" rsync parameters
  • indexing files would always start at position 0 of the file, over and over again
    • Only happened for rsync, using "--append --inplace" fixes this

Meanwhile I also took a look at Kibana, another great frontend to ElasticSearch. For the weblogic logs I'll keep Graylog2, as it allows saving predefined streams and provides that easy-to-use quickfilter, which eases up log crawling for our developers (they only want to search for a string in a timeframe - they also never made use of the power of Splunk). Also Kibana doesn't provide a way to view long stack traces in an acceptable fashion yet (cut message a n characters and provide a more link, something like that). But I added Apache logs in logstash, and those are routed to a logstash elasticsearch output and Kibana as WebUI. I'd really like to see some sort of merge between Kibana and Graylog2 and add saved searches to that mix - that would make a realy competitive Splunk alternative.

Category: 

Comments

Have you looked at kibana 3. It has a logstash dashboard where you can drill down data by histogram, apply facets, save custom dashboard and so on. The first thing that turned me away from graylog2 was its UI.

To be honest, we meanwhile threw out Graylog2 completely, and use only Kibana.

For Weblogic we still use the ruby version as Kibana3 has a bug regarding multiline message display in detailed view. For everything else (Apache, Syslog) we use Kibana3 - it's great. Though I really miss the splunk-like pipe commands (mostly score).

Thanks for your blog on Splunk alternative. We are in process of going the route of using Logstash and Graylog2. Would like to know your reviews and experience on the same. Are you using both Graylog and Kibana. Any alerting you have implemented with the same on streams. Is there any other open source alternative that you have come across.
Looking forward to hear from you.

As said in my reply above, our developers found Kibana more usable than Graylog2 in the end. So we ditched Graylog2 completely in favor of Kibana. The saved streams feature is great and I'm sure Graylog2 is getting better, but for our needs Kibana is just fine.Last time I tried, alerts didn't work in Graylog2. We do it directly in logstash now. Since version 1.2 logstash supports conditionals which makes it quite easy to implement basic alerts. In combination with statsd, Graphite and a Nagios plugin to read stuff from Graphite we can also alert on message counts throughput (like in Graylog2, something like number of error messages in production greater than 10 per minute -> alert).I don't want to recommend against Graylog2, it's great software, but in our case it ended up being a big piece of software sitting between logstash and elasticsearch just causing maintenance overhead.