Critique my code: using Lisp to parse lots of "ps -elf" files in AIX

Greetings, all,

I seek critiques and recommendations on the linked code (see below
under "Requests"). It's my first use of Lisp at work, and maybe my
third use "in anger", as the saying goes.

The situation: at work we had a machine crash due to running out of
memory and swap. Another project (under the same manager) had a ksh
script you could run as a cronjob to gather "ps -elf" snapshots and
write them to files, and another ksh script to analyze the output. My
team lead suggested that our manager might not trust anything but the
original scripts, since the other project had apparently had a lot of
success using them to diagnose some of their own memory problems, so I
installed the first script in a cronjob that ran every minute, and
then took a look at the second, analytic script.

What a disaster. I don't remember exactly, but I think it took about
an hour to process about an hour's worth of data (i.e. 60 data files).
Nothing like using <filename><nnn> to simulate arrays to slow down
your script. People that don't know that awk(1) has more features
than "{ print $2 }" should not write shell scripts.

So I threw my team lead's advice out the window and decided to rewrite
it in Lisp, using clisp (clisp.cons.org).

The basic idea is to read each file (a snapshot of "ps -elf", as
mentioned), keep track of the memory used by each process, and then
print out the memory deltas per process once it's read all the files.

Example output stanza for a particular process:

680002, ran for 111 units, -ksh
First appeared in reading_3May13:10:00
(524 (stable for 11 units) 8 (stable for 98 units))

It lists the PID, how long it ran, the command line, what data file it
first appeared in, and then a list of numbers, starting with the
initial memory usage, followed by a list of changes in memory usage.
This example shows that a ksh process started circa 5/3 @ 1:10pm, with
524 kb in use, stayed the same for 11 minutes, grew by 8 kb, stayed
the same for 98 more minutes, and then exited.

Design goals:

- correctness (obviously :)
- No use of external libraries -- not enough time to download them
and get them working (or so it seemed at the time). E.g. the
script might benefit from a regular expression library, but at the
time I didn't want to bother.
- Reasonable speed. My current version can process an hour's data
in 3-10 seconds (depending on the size of the data readings, i.e.
the number of processes running). I consider this reasonable and
"fast enough", actually, but I wouldn't mind making it even
faster.

Non-goals:

Portability. I don't care if it'll run or not under other Lisp
implementations. E.g. I don't know or care if my use of DIRECTORY
will work correctly under other Lisps; if I need to run it under
something else I'll download Peter's portable pathname library.
(Actually, I've already done this, but haven't integrated it into
this code.)

Design:

Have an EQUAL hash table keyed on (pid . command-line), with a list
of memory readings. Read all data files. Post-process the hash
table to output information formatted as above.

Possible changes not yet implemented:

Instead of storing each memory reading, just store the difference
between the current reading and the previous reading.

Requests:

Please see http://www.theclapp.org/process-mem-readings.lisp and/or
http://www.theclapp.org/process-mem-readings.lisp.html.

How can I make it
- faster?
- more readable?
- more useful / functional?
- Lispier? (most of my background is in C and Perl)

Commentary:

- I developed this script using neither SLIME nor my own VILisp
tool. You can get a great deal done with just a few utility
functions, plain ol' cut-and-paste, and clisp's command history
function. (I have since installed xemacs, but not SLIME yet.)
- I really liked the ability to load a bunch of data into memory,
and then step-wise refine the code to post-process it, without
having to spend three minutes reloading the data each time I
wanted to generate output. I could not have developed this code
as quickly in any other language. (On further thought, this last
comment probably isn't true. If I'd run my tests against a subset
of the data files, I probably could have done it as quickly in
Perl. Oh well. :) Also, someone else might have done it as fast
in something else, using e.g. Python or Scheme or any other
language with a REPL, but I don't know any of them and didn't have
them already installed on our development box anyway.)
- I wrote a short ksh script to do a similar analysis, and comparing
it to the output of the Lisp script revealed a subtle bug in the
shell version that I don't think I'd have noticed otherwise.

Thank you for any assistance or advice you can provide.

-- Larry Clapp

.

Relevant Pages

  • How to tell what is using memory
    ... My script uses up a lot of memory because it is reading large data files. ... Perl doesn't normally release any memory back to the operating system until ...
    (comp.lang.perl.misc)
  • Re: Problem with ASP on Webserver
    ... No, I know the memory model, I have made my own Virtual memory dump program ... function in the script ... (Accept-Language:) ...
    (microsoft.public.windowsce.platbuilder)
  • Re: Problem with ASP on Webserver
    ... Entrek's TOOLBOX is a lot better for memory stuff. ... function in the script ... (Accept-Language:) ...
    (microsoft.public.windowsce.platbuilder)
  • Re: Hello World
    ... This is my nth attempt to learn Lisp. ... documentation I managed to write my first script in sbcl. ... number for our primality check. ... eg. a `print-primes' function that does the heavy-lifting work ...
    (comp.lang.lisp)
  • Re: arc tangent
    ... lisp until a more algebraic notation that was good enough could be found. ... I dare you to cleanly write a Unix ... languages to be invoked to process Unix shell commands with -parms? ... So to emulate nested parameters to a script, ...
    (comp.lang.lisp)
Loading