I wrote a Fluentd plugin which sends checks to Sensu monitoring system. You can utilize the plugin to detect anomaly of logs and send alerts using Sensu.
This plugin is functionally analogous to fluent-plugin-nsca, which is for Nagios/NSCA.
Installation
Install fluent-plugin-sensu gem.
Use case: “too many server errors” alert
Assume you have a web server which runs:
- Apache HTTP server
- Fluentd
- sensu-client
and a monitoring server which runs:
- Redis
- RabbitMQ
- sensu-server
You want to be notified when Apache responds too many server errors, for example 5 errors per minute as WARNING, and 50 errors per minute as CRITICAL.
This can be implemented as the following figure shows.
Fluentd configuration
This setting utilizes fluent-plugin-datacounter, fluent-plugin-record-reformer, and of course fluent-plugin-sensu. So, first of all, install the gems of those plugins.
Next, add these lines to the Fluentd configuration file.
# Parse Apache access log <source> type tail tag access format apache2 # The paths vary by setup path /var/log/httpd/access_log pos_file /var/lib/fluentd/pos/httpd-access_log.pos </source> # Count 5xx errors per minute <match access> type datacounter tag count.access unit minute aggregate all count_key code pattern1 error ^5\d\d$ </match> # Calculate the severity level <match count.access> type record_reformer tag server_errors enable_ruby true <record> level ${error_count < 5 ? 'OK' : error_count < 50 ? 'WARNING' : 'CRITICAL'} </record> </match> # Send checks to sensu-client <match server_errors> type sensu server localhost port 3030 check_name server_errors check_type standard check_status_field level check_ttl 100 </match>
The next figure shows the data flow.
Alternatives
You can use record_transformer filter instead of fluent-plugin-record-reformer on Fluentd 0.12.0 and above.
If you are concerned with scalability, fluent-plugin-norikra may be a better option than datacounter and record_reformer.
Another alternative configuration for the use case is sending the error count to Graphite using fluent-plugin-graphite, and making Sensu monitor the value on Graphite with check-data.rb.
Contributing
Please submit an issue or a pull request on the Github repository.
Feedback to @miyakawa_taku on Twitter is also welcome.
