Perl throw-away: finding mtime of cron jobs commands from cron log (AIX)

We had two AIX boxes use all of their memory (real + paging) today. Looking back over performance logs this happened suddenly and exactly on the hour.

The precise time at the top of the hour points to a possible cron job problem. You know how humans are: always scheduling things at nice round numbers.

I wanted to parse the cron logs for all the commands there were run. The first thing I wanted to do was to check the mtime and see if anything changed recently.

Here is a sample of the cron logs AIX spits out.

bob1    : CMD ( [[ -f /opt/bin/Logging.ksh ]] && /opt/bin/Logging.ksh start >/dev/null 2>&1 & ) : PID ( 512208 ) : Wed Mar 16 20:15:00 2011
bob1    : CMD ( /home/bob1/scripts/check.pl ) : PID ( 843892 ) : Wed Mar 16 20:15:00 2011
root      : CMD ( /etc/init.d/agent status > /dev/null 2>&1 || /etc/init.d/agent start > /dev/null 2>&1
) : PID ( 258288 ) : Wed Mar 16 20:15:00 2011
root      : CMD ( /opt/bin/get_down >/dev/null 2>&1 ) : PID ( 548972 ) : Wed Mar 16 20:15:00 2011
Cron Job with pid: 512208 Successful
Cron Job with pid: 548972 Successful
Cron Job with pid: 258288 Successful
Cron Job with pid: 843892 Successful
bob1    : CMD ( [[ -f /etc/init.d/xyz_3.6 ]] && /etc/init.d/xyz_3.6 restart >/dev/null 2>&1 ) : PID ( 950448 ) :
 Wed Mar 16 20:16:00 2011
root      : CMD ( /opt/bin/get_down >/dev/null 2>&1 ) : PID ( 258052 ) : Wed Mar 16 20:16:00 2011
Cron Job with pid: 258052 Successful

Here is my perl script to grab the command and report on their mtimes.

#!/usr/bin/perl -wan

use strict;
use warnings;
use POSIX qw(strftime);

# Hash key = command, value = mtime.
our %cmds;

if ( /CMD \(/ ) {
    if ($F[4] =~ /^\//) {
        $cmds{$F[4]} = strftime('%Y-%m-%dT%H:%M:%S', localtime((stat($F[4]))[9]));
    }
}

END {
    for my $cmd (sort(keys %cmds)) {
        print "$cmds{$cmd}\t$cmd\n";
    }
}

Here is the example output.

# ./1.pl log
2009-11-25T16:03:42     /etc/init.d/agent
2008-04-04T14:33:25     /home/archie/bin/clearWebLogs.sh
2010-09-24T16:49:29     /home/bob1/scripts/check.pl
2003-09-10T13:32:04     /opt/bin/get_webdat
2004-01-06T16:28:08     /opt/bin/nim_prep
2004-01-06T16:28:08     /opt/bin/clr_tmp
2003-09-11T08:12:17     /opt/bin/get_down
2004-04-12T14:36:29     /opt/bin/xy_clr_tmp
2010-03-01T19:18:20     /usr/bin/errclear
2006-10-24T23:34:02     /usr/lib/ras/dumpcheck
2010-10-04T01:55:44     /var/perf/pm/bin/pmcfg

If you're thinking, "Hey, stupid, just look in the crontab files for that info," ummm, yeah, I had a reason for doing it this way, I'm sure of it. I just can't think of it right now.

You'll also see that my script does not actually get all the commands. I guess I need to fix that.

Perl one-liner: count Apache connections to local app servers.

I had this one-liner lying around that I meant to drop on here. Looking at it I'm fairly certain the scenario was as follows.

This one box has a single Apache (IBM HTTP Server, really) instance talking to a lot of different application servers (WebSphere), which also run locally. Ignore the wisdom or lack thereof for this configuration. For whatever reason (verify load balance settings?), I had to count the number of connections Apache was making to each application server.

root@somebox:/tmp
# lsof -i TCP -P | perl -ane 'next unless /^httpd/ && /->somebox/; $p{(split(":", $F[-2]))[2]}++; END { for $i (keys %p) {print  "$i = $p{$i}\n"}}'
8056 = 305
8054 = 30
8053 = 27
8057 = 31
8050 = 353

So as we can see, at the time I ran this script, the balance of connections was really lopsided. The application server listening on port 8056 had 305 connections and the one listening at port 8054 had 30 connections, and so on.

Perl throw-away: extracting info from giant list of file names.

I had a giant list of files that followed a pattern similar to this:

me@mybox:~/sandbox
$ ls -rlt
total 0
-rw-r--r-- 1 root root 0 Mar  4 18:10 bob.1234.x
-rw-r--r-- 1 root root 0 Mar  4 18:10 bob.9870.x
-rw-r--r-- 1 root root 0 Mar  4 18:10 bob.3245.x

Except there were thousands of these files, not just three.

I need to get a sorted list of the numbers in each file name. I did this:

me@mybox:~/sandbox
$ perl -e '@f = glob("bob*"); @s = map {substr($_,4,4)} @f; for $n (sort(@s)) {print "$n\n";}'
1234
3245
9870

Yes, I could have done this in the shell.

me@mybox:~/sandbox
$ ls -1 bob* | cut -c5-8 | sort
1234
3245
9870

Using join instead of for now looks pretty because of Anonymous comment below. Thanks, buddy.

jbm@Foucault:~/tmp
$ ls bob*
bob.0012.x bob.7123.x bob.9865.x
jbm@Foucault:~/tmp
$ perl -le '@f = glob("bob*"); @s = map {substr($_,4,4)} @f; print join("\n",sort(@s));'
0012
7123
9865

UPDATE

poisonbit and Yanick have more flexible solutions in the comments below because they don't rely on my brain dead method of counting the exact columns in which the digits appear in the file name.

Of course, poisonbit's solution requires cool bash features, and Yanick's requires newer Perl builds. Using their better answer in crustier Perl or Korn shell is left as an exercise for the reader.