Prometheus: run queries against Elasticsearch and turn it into metrics and alerts

Long time no read. I'm in the middle of re-implementing our monitoring solution, this time using Prometheus.

We have tons of web- and application-server logs in Elasticsearch and need to query it for http 5xx error rates, application error rates and similar things to get that data into Prometheus for alerting fun.

Enter braedon's prometheus-es-exporter. There's a ready-to-go docker image on docker hub, so all it takes to get things working in kubernetes is this:

The way to Elasticsearch 2.0, or how to reindex your dot fields with logstash and ruby filters

The Elasticsearch 2.0 release intruced a major annoyance by removing support for dots in field names. We use ES for our apache logs, with retention policy of 365 days, and of course _all_ of the indices contained fields with a dot in the name.

What's even worse, at some point in time i had the idea to filter out the request parameters from the uri and run a kv filter on it. As we never used the resulting mess of request_params.* fields, those could just be dropped.

First step was to update our logstash configuration so no dots are used for field names.

Indexing and searching Weblogic logs using Logstash, Elasticsearch and Kibana

This is a re-edit of my previous post "Indexing and searching Weblogic logs using Logstash and Graylog2". Meanwhile our setup has settled to use Kibana instead of the Graylog2 frontend. This Howto is meant to be a complete installation guide for "The Elasticsearch ELK stack" and using it to index tons of Weblogic server and application logs, from DEV over UA to the Production environment.

Analyzing OpenWrt firewall logs with Splunk

This article explains how to analyze dropped and rejected traffic from OpenWrt (or any other Iptables based) firewall logs using Splunk and the Netfilter Iptables App.

What you will need is:

  • a remote syslog server (I use syslog-ng)
  • a machine that runs Splunk and can access the logfiles

Preparing OpenWrt

On OpenWrt we need to enable remote logging, as well as firewall logging

Howto migrate SonarQube from MySQL to Oracle

Recently we had the need to move Sonar off our small virtualized MySQL server due to the fact that the Sonar database has begun to grow huge. Really HUGE. We'd like to keep data for about 3 months, and 1 month is already worth several GB of data, and our MySQL server isn't setup for this amount of data.

So we decided to move it to our Oracle database. Thanks to SQL Developer, this was a quite easy process.

Pre-Setup: Create a tablespace and user for Sonar

Howto easily update GPS-A data on a Sony Alpha 65/77/99 and others on Linux/Mac

In order to speed up GPS locking on a Sony Alpha 65 (or similar) SLT camera, it's possible to update the GPS-A data (also called almanac data). Like on any other modern GPS device, the almanac data is used to give the device a hint where the satellites are located. The data usually is valid for only some weeks,then it needs to updated again.

Integrate DSPAM into postfix + dovecot + any mail client

Recently I figured that my spamasassin setup stopped working correctly for some reason. At first I didn't see that, then I didn't care immediately as Thunderbird ist still 99% right, but when using Roundcube while "Thunderbird at home" is shut down, it became more and more annoying. I rechecked my setup twice, started all over, re-trained it for almost a week, to no dice.

So I watched out for alternatives. DSPAM. There is nothing else, really. To say one thing upfront: it works from the start, even while being in training phase still.

The benefits:

Getting metrics from Graphite into Nagios and Centreon

Getting metrics from logs and various other sources into Graphite is quite simple. The most interesting metrics do represent critical performance data, and the pro-active-monitoring approach, means a person sitting there and waching the dashboard, isn't suited to our needs. We use Nagios with Centreon as our monitoring plattform, and we want to alert on some of the metrics collected in Graphite.