Here at Paglo we like to monitor all sorts of things about IT infrastructure and (just about any other data that can be captured and monitored over time). Being able to grok how systems and applications are being used, where the hot spots are, really gives us a buzz.

You can see this in our recent Paglo Crawler plugin for VMware virtualization monitoring

You can now also use this plugin to see a VMware guest’s usage not from just inside the device, but from outside, as the VMware host sees it.

When Amazon recently announced their CloudWatch service — we were excited. In essence it was the same sort of view of the EC2 instances that the VMware API gave us.

Being no stranger to Amazon’s APIs we decided to whip up a Paglo Crawler plugin that would gather all the statistics that it could from the Amazon CloudWatch API for monitoring EC2 instances.

With our new release of the Crawler the VMware and Amazon plugins are automatically included. When you configure the Amazon plugin in the “Crawlers” application with your EC2 credentials it will query Amazon’s CloudWatch API every 5 minutes gathering the last 5 minutes of every statistic available.

And you don’t have to worry about doing anything in your Paglo account to handle the new data. This is because the your account’s Paglo index is semi-structured which lets us gather data from any source and plug it in.

All the statistics gathered from Amazon’s CloudWatch are stored under:

/apps/com/paglo/aws/cloud_watch/

Under here there will be a ‘statistic’ subnode for every stat we gather from CloudWatch.

For every statistic captured, we index all of the possible aggregations provided by CloudWatch. For example, the query:

select * from /apps/com/paglo/aws/cloud_watch

will provide something like:

statistic namespace AWS/EC2 measure_name networkout samples 15.0 sum 2.2491354E7 average 1499423.6 unit Bytes minimum 868.0 maximum 7221048.0 total true statistic namespace AWS/EC2 measure_name cpuutilization samples 5.0 sum 6.14 average 1.228 unit Percent minimum 0.0 maximum 3.46 instanceid i-c11759a8 statistic namespace AWS/EC2 measure_name diskwritebytes samples 15.0 sum 3.3189888E7 average 2212659.2 unit Bytes minimum 32768.0 maximum 8671232.0 instancetype m1.small

This query shows you (for the last 5 minutes) the number of samples, sum, average, min and max for NetworkOut in bytes across all instances, CPUUtilization for InstanceID ‘i-c11759a8’, and DiskWriteBytes for all instances with the InstanceType ‘m1.small’

Because of Paglo’s historical value and data analytics capabilities this lets you immediately begin to graph these various statistics over time.

For example here is some data that we have been tracking. Here is CPUUtialization for our three Amazon EC2 instances over the past six hours:

select instanceid||':'||measure_name||' '||unit as caption,average from /apps/com/paglo/aws/cloud_watch/statistic where instanceid is not null and measure_name in ('cpuutilization') history from '6 hours ago' to 'now'

Network in and out for each of our three instances over the last six hours:

select instanceid||':'||measure_name||' '||unit as caption,average from /apps/com/paglo/aws/cloud_watch/statisticwhere instanceid is not null and measure_name in ('networkin', 'networkout') history from '6 hours ago' to 'now'

In addition to charts you can also generate some useful tables, like: when was the last time we gathered statistics for an instance:

select instanceid, measure_name, format(observed_at(instanceid),'timestamp') from /apps/com/paglo/aws/cloud_watch/statistic where instanceid is not null

If you want to know the names of all the measures collected:

select distinct measure_name from /apps/com/paglo/aws/cloud_watch/statistic

As you can see, the data provided by Amazon’s CloudWatch API coupled with the data gathering, quering, and analytics ability of Paglo makes it easy to monitor your Amazon EC2 instances.

Let us know what you think and what additional data you would like to capture, search, and store via Paglo.

And remember that the Paglo Crawler is released under the GPL which makes it easy for you to add your own plugins too.