Perl throw-away: finding mtime of cron jobs commands from cron log (AIX)

We had two AIX boxes use all of their memory (real + paging) today. Looking back over performance logs this happened suddenly and exactly on the hour.

The precise time at the top of the hour points to a possible cron job problem. You know how humans are: always scheduling things at nice round numbers.

I wanted to parse the cron logs for all the commands there were run. The first thing I wanted to do was to check the mtime and see if anything changed recently.

Here is a sample of the cron logs AIX spits out.

bob1    : CMD ( [[ -f /opt/bin/Logging.ksh ]] && /opt/bin/Logging.ksh start >/dev/null 2>&1 & ) : PID ( 512208 ) : Wed Mar 16 20:15:00 2011
bob1    : CMD ( /home/bob1/scripts/check.pl ) : PID ( 843892 ) : Wed Mar 16 20:15:00 2011
root      : CMD ( /etc/init.d/agent status > /dev/null 2>&1 || /etc/init.d/agent start > /dev/null 2>&1
) : PID ( 258288 ) : Wed Mar 16 20:15:00 2011
root      : CMD ( /opt/bin/get_down >/dev/null 2>&1 ) : PID ( 548972 ) : Wed Mar 16 20:15:00 2011
Cron Job with pid: 512208 Successful
Cron Job with pid: 548972 Successful
Cron Job with pid: 258288 Successful
Cron Job with pid: 843892 Successful
bob1    : CMD ( [[ -f /etc/init.d/xyz_3.6 ]] && /etc/init.d/xyz_3.6 restart >/dev/null 2>&1 ) : PID ( 950448 ) :
 Wed Mar 16 20:16:00 2011
root      : CMD ( /opt/bin/get_down >/dev/null 2>&1 ) : PID ( 258052 ) : Wed Mar 16 20:16:00 2011
Cron Job with pid: 258052 Successful

Here is my perl script to grab the command and report on their mtimes.

#!/usr/bin/perl -wan

use strict;
use warnings;
use POSIX qw(strftime);

# Hash key = command, value = mtime.
our %cmds;

if ( /CMD \(/ ) {
    if ($F[4] =~ /^\//) {
        $cmds{$F[4]} = strftime('%Y-%m-%dT%H:%M:%S', localtime((stat($F[4]))[9]));
    }
}

END {
    for my $cmd (sort(keys %cmds)) {
        print "$cmds{$cmd}\t$cmd\n";
    }
}

Here is the example output.

# ./1.pl log
2009-11-25T16:03:42     /etc/init.d/agent
2008-04-04T14:33:25     /home/archie/bin/clearWebLogs.sh
2010-09-24T16:49:29     /home/bob1/scripts/check.pl
2003-09-10T13:32:04     /opt/bin/get_webdat
2004-01-06T16:28:08     /opt/bin/nim_prep
2004-01-06T16:28:08     /opt/bin/clr_tmp
2003-09-11T08:12:17     /opt/bin/get_down
2004-04-12T14:36:29     /opt/bin/xy_clr_tmp
2010-03-01T19:18:20     /usr/bin/errclear
2006-10-24T23:34:02     /usr/lib/ras/dumpcheck
2010-10-04T01:55:44     /var/perf/pm/bin/pmcfg

If you're thinking, "Hey, stupid, just look in the crontab files for that info," ummm, yeah, I had a reason for doing it this way, I'm sure of it. I just can't think of it right now.

You'll also see that my script does not actually get all the commands. I guess I need to fix that.