Prometheus: run queries against Elasticsearch and turn it into metrics and alerts

Long time no read. I'm in the middle of re-implementing our monitoring solution, this time using Prometheus.

We have tons of web- and application-server logs in Elasticsearch and need to query it for http 5xx error rates, application error rates and similar things to get that data into Prometheus for alerting fun.

Enter braedon's prometheus-es-exporter. There's a ready-to-go docker image on docker hub, so all it takes to get things working in kubernetes is this:

The way to Elasticsearch 2.0, or how to reindex your dot fields with logstash and ruby filters

The Elasticsearch 2.0 release intruced a major annoyance by removing support for dots in field names. We use ES for our apache logs, with retention policy of 365 days, and of course _all_ of the indices contained fields with a dot in the name.

What's even worse, at some point in time i had the idea to filter out the request parameters from the uri and run a kv filter on it. As we never used the resulting mess of request_params.* fields, those could just be dropped.

First step was to update our logstash configuration so no dots are used for field names.

Indexing and searching Weblogic logs using Logstash, Elasticsearch and Kibana

This is a re-edit of my previous post "Indexing and searching Weblogic logs using Logstash and Graylog2". Meanwhile our setup has settled to use Kibana instead of the Graylog2 frontend. This Howto is meant to be a complete installation guide for "The Elasticsearch ELK stack" and using it to index tons of Weblogic server and application logs, from DEV over UA to the Production environment.

Getting metrics from Graphite into Nagios and Centreon

Getting metrics from logs and various other sources into Graphite is quite simple. The most interesting metrics do represent critical performance data, and the pro-active-monitoring approach, means a person sitting there and waching the dashboard, isn't suited to our needs. We use Nagios with Centreon as our monitoring plattform, and we want to alert on some of the metrics collected in Graphite.

Indexing and searching Weblogic logs using Logstash and Graylog2

Update 2013/10: we decided to replace Graylog2 with Kibana3 completely. The article below is just for reference, the logstash config is outdated since logstash 1.2 and the setup as described below is suboptimal anyway. I'll post a new article shortly.

Update 2014/02: Finally, the new guide is here: Indexing and searching Weblogic logs using Logstash, Elasticsearch and Kibana.


SOHO Mailserver with Postfix + Postgresql + Dovecot + SpamAssassin + Roundcube

This HowTo describes my Home-Mailserver Setup. Basically this is a sum-it-all-up article from various resources on the net. 

Used Software:

  • Arch Linux OS
  • Postfix MTA
  • PostgreSQL database backend
  • Dovecot IMAP Server
  • Roundcube Webmail + Apache Webserver
  • Spamassassin junk filter
  • Server-side filtering with Sieve
  • fetchmail (for pulling all spread accounts in this one place)

Preconditions in my setup:

Howto create a youtube video from mp3/ogg audio using a picture

If you want to create a youtube video from an audio file, here is how to do this.
All you need is the audio file, a single picture, and ffmpeg.

First find out the lenght of the audio file in seconds, you'll need it. Here is an example with a 420 seconds file:

ffmpeg -loop_input -i picture.jpg -vcodec mpeg4 -r 25.00 -qscale 2 -s 480x360 \
-i audiofile.mp3 -acodec libmp3lame -ab 128k -t 420 video.avi

This will create a Hi-Res MPEG-4 video with 128k audio. The trick here is to use that one picture and loop it for -t seconds.

Howto print text between tags or characters with awk or sed

Took me a while to figure out this one. All I wanted to do is to print the text between two tags/characters/strings.

The IMHO nicer Awk way:

$ echo "bla(foo)"|awk -F'[(|)]' '{print $2}'
$ echo "bla=@@foo@@"|awk -F'[@@|@@]' '{print $3}'

 and with sed:

$ echo "blah(foo)"|sed -n 's/.*(\([^ ]*\))/\1/p'
echo "aaafoobbb"|sed -n 's/.*aaa\([^ ]*\)bbb/\1/p'