Handling Platform Service Events from PowerKVM servers using OpenSource Tools

ELK stack which stands for Elasticsearch Logstash and Kibana is a very common opensource log management solution, providing real time log analysis. Elasticsearch provides the storage and search engine, Logstash is the data collection and parsing component whereas Kibana is the visualization component.

In this post let us see how this ubiquitous opensource log management solution can be used to handle service events in addition to regular events from the scale-out Power servers running PowerKVM.

This post assumes that you are already using an ELK based solution for log management and doesn’t go into the details of ELK installation and configuration.

The deployment topology for a centralized log management solution looks similar to the diagram shown below. Logs from the PowerKVM servers are sent to the centralized log management server for analysis and processing. The individual components of the log management server can be either on the same machine or different machines as required. PowerKVM servers are configured to send the logs to remote syslog (rsyslog) server which then pushes the logs to logstash server for further processing like alert generation, indexing and viewing in dashboard.

service

For PowerKVM servers to send the log messages to remote syslog (rsyslog) server, modify the rsyslog.conf file appropriately. As an example edit /etc/rsyslog.conf to add the following line to forward logs to remote syslog server.

*.* @rsyslog_server:port

Platform Service Events in PowerKVM server

A platform event that is logged in the syslog looks like the following:
“May 20 10:44:16 llmjuno03b ELOG[34914]: LID[5034a000]::SRC[11007201]::External Environment::Predictive Error::Service action is required”

The events logged in syslog can be one of the following three types:

  • Service action and call home are required
  • Service action is required
  • No service action is required

If you are interested in more details, please refer to the link on platform diagnostics for Power servers here.

In order to handle the platform service events that are logged in syslog, we’ll be required to write an input grok filter for logstash to process the service events. A very simple input grok filter can be something like the following:

filter {
 if [type] == "syslog" {
   grok {
     match =>  [ "message" , "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} ELOG(?:\[%{NUMBER}\])?: %{GREEDYDATA:syslog_message} ]
   }
   if 'No Service action is required' not in [syslog_message] {
     mutate { add_tag => "Service Event" }
   }
  }
}

Parsing of the service log entry by logstash results in the following:

{
             "message" => "Apr  3 19:32:14 llmjuno03b ELOG[21743]: LID[5094ddc3]::SRC[11007203]::External Environment::Unrecoverable Error::Service action is required",
            "@version" => "1",
          "@timestamp" => "2015-04-04T10:26:14.459Z",
                "type" => "syslog",
                "host" => "logstash_server",
                "path" => "/var/log/messages",
    "syslog_timestamp" => "Apr  3 19:32:14",
     "syslog_hostname" => "llmjuno03b",
      "syslog_message" => "LID[5094ddc3]::SRC[11007203]::External Environment::Unrecoverable Error::Service action is required",
                "tags" => [
        [0] "Service Event"
    ]
}

The output could be fed to Nagios/Icinga or Riemann for displaying the alert or for executing any additional actions like email, sms etc.
Hope this gives you an idea on how to integrate the new scale-out Power servers running PowerKVM into your existing data center managed by opensource tools.

Pradipta Kumar Banerjee

I'm a Cloud and Linux/ OpenSource enthusiast, with 16 years of industry experience at IBM. You can find more details about me here - Linkedin

You may also like...