Prometheus: run queries against Elasticsearch and turn it into metrics and alerts

Long time no read. I'm in the middle of re-implementing our monitoring solution, this time using Prometheus.

We have tons of web- and application-server logs in Elasticsearch and need to query it for http 5xx error rates, application error rates and similar things to get that data into Prometheus for alerting fun.

Enter braedon's prometheus-es-exporter. There's a ready-to-go docker image on docker hub, so all it takes to get things working in kubernetes is this:

The way to Elasticsearch 2.0, or how to reindex your dot fields with logstash and ruby filters

The Elasticsearch 2.0 release intruced a major annoyance by removing support for dots in field names. We use ES for our apache logs, with retention policy of 365 days, and of course _all_ of the indices contained fields with a dot in the name.

What's even worse, at some point in time i had the idea to filter out the request parameters from the uri and run a kv filter on it. As we never used the resulting mess of request_params.* fields, those could just be dropped.

First step was to update our logstash configuration so no dots are used for field names.

Indexing and searching Weblogic logs using Logstash, Elasticsearch and Kibana

This is a re-edit of my previous post "Indexing and searching Weblogic logs using Logstash and Graylog2". Meanwhile our setup has settled to use Kibana instead of the Graylog2 frontend. This Howto is meant to be a complete installation guide for "The Elasticsearch ELK stack" and using it to index tons of Weblogic server and application logs, from DEV over UA to the Production environment.

Indexing and searching Weblogic logs using Logstash and Graylog2

Update 2013/10: we decided to replace Graylog2 with Kibana3 completely. The article below is just for reference, the logstash config is outdated since logstash 1.2 and the setup as described below is suboptimal anyway. I'll post a new article shortly.

Update 2014/02: Finally, the new guide is here: Indexing and searching Weblogic logs using Logstash, Elasticsearch and Kibana.