If you are like me you tend to ignore the logs from your Ruby on Rails applications until something goes wrong, then once I find the error message I want to answer questions like: how frequently has this error happened?, were any customers affected?, what actions were being done just before the error? Does the error only happen at a certain time of day? These questions and more are exactly what searching your log files is good for. Searching a few thousand lines of log files with grep is fast enough, but as soon as you want to look at more than a few hours of logs from a production server, using grep can get pretty tedious. And grep can help you analyze trends.

Paglo uses inverted index technology (the same technology Google uses to search the Internet) to search through logs files blazingly fast. The really nice part is that Paglo is web based/SaaS so there is no complex software to install.

These instructions are designed for Unix and OS X hosts. They will not work under Windows. I am considering writing a pure-ruby interface to Paglo log search which would work under Windows. Let me know if this would be useful to you.

sudo gem install SyslogLogger

Configure the syslog logger to be used when the application runs in production mode by adding the following lines to RAILS_ROOT/config/environments/production.rb

require 'syslog_logger'
logger = SyslogLogger.new("application_name")
logger.level = Logger::INFO
config.logger = logger

You can customize the application_name to the name of your application. This name will be attached to each log message and will be used by syslog-ng and Paglo to separate the messages from other applications.

Now you are ready to configure syslog-ng. Note you can do a similar thing with RSyslog, instructions are available for configuring RSyslog within Paglo at https://app.paglo.com/log_manage/configure. I far prefer working with syslog-ng because its configuration files are logical and easy to understand. RSyslog tries to maintain backwards compatibility with the original syslog and its configuration files are cryptic.

You need to be running syslog-ng version 3.0 or newer. You can check what version of syslog-ng you are running using the command:

syslog-ng --version

Syslog-ng is completely controlled by the contents of the syslog-ng.conf file. On my system it is in /etc/syslog-ng/syslog-ng, but I have also seen it in /usr/local/etc/syslog-ng/syslog-ng.conf. We need to add some configuration to the file so it knows what to do with the log messages received from the SyslogLogger we just added to our rails application. SyslogLogger is sending its messages to the default syslog destination, on linux this is /dev/log. In my existing configuration file this source shows up near the top of the file and is called s_all:

# all known message sources
source s_all {
        # message generated by Syslog-NG
        internal();
        # standard Linux log source (this is the default place for the syslog()
        # function to send logs to)
        unix-stream("/dev/log");
        # messages from the kernel
        file("/proc/kmsg" log_prefix("kernel: "));
        # use the following line if you want to receive remote UDP logging messages
        # (this is equivalent to the "-r" syslogd flag)
        # udp();
};

We want to separate out the log messages from our Rails application so we define a filter based on the program name.

filter rails_app { 
  program("application_name "); 
};

Now we can configure the destination which will stream the data to Paglo.

destination log_to_paglo {
        syslog("crawler.paglo.com"
                port(6514)
                transport("tls")
                tls( ca_dir("/etc/syslog-ng/ca.d")
                        key_file("/etc/syslog-ng/cert.d/paglo.key")
                        cert_file("/etc/syslog-ng/cert.d/paglo.crt")
                        peer_verify(required-trusted))
        );
};

This configures syslog-ng to stream the logs to crawler.paglo.com using the syslog over TLS protocol. The files specified in the tls section are the private key and certification that authenticates this machine with Paglo, and also allows this machine to ensure that it is talking to the ‘real’ Paglo server and not an imposter. The certificate files are generated by Paglo at https://app.paglo.com/log_manage/configure.

Finally we can link all of these configurations together with a log directive which specifies a logging action:

log {
        source(s_all);
        filter(rails_app);
        destination(log_to_paglo);
};

Save the configuration and tell syslog-ng to reread the configuration file by restarting it or sending a HUP signal. I have found it useful to troubleshoot configuration problems by running syslog-ng from the command line in the foreground like:

syslog-ng --foreground

When run like this it print an error messages directly to the console, If the configuration is good then stop it with Ctrl-C.

If everything is working then the next time the Rails application generates a log message it will be sent to Paglo and automatically indexed. Log messages show up in the Paglo search index about 30 seconds after they were generated. Now I can log into Paglo and search for those pesky error messages.

Since we added the SyslogLogger to the Rails application no logs will be recorded in the standard log/production.log file. This can sometimes be a bit frustrating, but we can use syslog-ng to log the same information to multiple destinations. So we can add a file destination like:

destination rails_production_log {
        file("/apps/my_application/current/log/production.log");
};

And add this destination to the same log directive so the Rails logs go to both places:

log {
    source(s_all);
    filter(rails_app);
    destination(log_to_paglo);
    destination(rails_production_log);
};

Then restart syslog-ng again and you are ready to go.