Re: Why I never got into Lisp

Larry Clapp <larry@xxxxxxxxxxxx> writes:

On 2006-12-08, Tim X <timx@xxxxxxxxxxxxxxx> wrote:
Zach Beane <xach@xxxxxxxx> writes:
Chris Barts <puonegf+hfrarg@xxxxxxxxx> writes:
I bet an ultra-competent macro head could have shrunk code
considerably over the course of the whole program. But in small
blocks, Lisp tends to look flabby compared to Huffman-coded
languages like Perl.

I think this has a detrimental effect in how you think about
solutions in languages like Perl. Since certain solution styles
have optimized syntax baked right into the language, going outside
those styles suddenly feels like a lot of hassle and not worth the
trouble. I think that's why a lot of solutions converge tweaking
strings with regular expressions and sticking them into hash
tables.

If you can accept Lisp's notion that most constructs (including
method calls, control structures, array references, etc) look
somewhat like a function call, it doesn't feel like you're really
going outside the preferred realm of solutions by simply adding
more features that look somewhat like function calls. Code that
uses a specialized forms of table storage or arrays doesn't look
unnatural simply for not being able to take advantage of a
compressed syntax using {} or [].


I think I agree....

I recently had to write some fairly simple scripts to analyse some
data files.
[ cool story snipped ]

I enjoyed reading that and would be interested in hearing more about
the specifics of the problem, e.g. what concepts you tried in Perl,
and what you tried in Lisp, and how Lisp mapped better, for you. I'd
like to put a pointer to your story in the FAQ
(http://www.lispniks.com/faq/faq.html) in the "Toto -- I've a feeling
we're not in Kansas anymore" section (or somewhere in there, anyway).


I'll see if I can expand a bit.

The Problem
===========

- The application generates "event logs" which are similar in style to
syslog records, but its not done through syslog and unfortunatley
there is some inconsistency in record formats, which muddy the
waters a bit (need quite a lot of error checking an alternative
parsing formats)
- There is no documentation. The application is in binary and was
written in the early 90s. We have source code - actually, we have
lots of source code, but it has not been managed well and we can't
determine which source code was used to create the binary (some of
it won't compile anymore without considerable changes - its early
C++).
- The system is Tru64 on a server with approx 30,000 user accounts.
However, only approx 10k - 15k of the users use the application.
Usage frequency varies greatly. Log records average around 150Mb per
month. Initially, we want to process 1 years data. We have data
going back around 7 years.
- Records have a "session" relationship, but the only information
linking a session is the process id, which is not unique due to both
normal pid rotation and the way the application's server is
implemented. There is no documentation regarding record formats,
possible values or even meaning of some values etc.
- We want to get a "picture" of user interaction with the system, so
we need to try and tie together the records making up individual
sessions. Although there is a record which indicates the end of the
session, it seems that the existance of this record depends on how
the user exits the app, plus there seems to be frequent 'drop outs'
or possibly records being lost (i.e. race condition writing to log
files or something similar).

I think one of the main things that made lisp so useful in addressing
this problem was the level of data exploration that was needed to get
to understand the event logs. When developing the first version in
perl, it was a slightly annoying process of add a bit of code, try
parsing a section of a data file, tweak the code, try again, tweak ,
try, tweak, try - each time, starting from scratch. With lisp, the
exploration was a joy. I just entered code into the repl and refined
it in the repl while being able to keep variables and other data
structures. Essentially it was faster and less intrusive with respect
to concentration.

This problem was really well suited to an iterative style
of development where you incrementally refined what the program did.
While this can be done to some extent in any language, lisp seems to
suit this approach more than perl. The lisp code ended up looking a
lot clearer, while the perl code got less elegant and more muddy.
Adjusting data structures seemed easier with lisp than perl as well,
but that might be a reflection of how quickly I forget how to do
nested structures in perl - for example, if I wanted to create a hash
in perl with elements that were lists of lists, I always have trouble
trying to remember the syntax unless I've been working fairly
constantly in the language. With lisp, I don't seem to have this
problem.

As the code developed, in lisp, I was able to simplify some bits with
some really simple macros. To simplify the code in perl, I had to
write a new subroutine and either stick with global variables or deal
with yet another different referencing/dereferencing syntax and make
decisions on whether to use prototypes or not etc.

When I decided to try Ruby (mainly because I had been looking for a
real problem to try it out on), I had a better understanding of the
event records and the level of variation. I took a different approach
and created record objects and session objects to represent the main
data structures. I was actually quite shocked at how fast and
productive I was with this langauge given that it was my first ever
attempt to program with it. I found it had many of the features I
liked in both smaltalk and CLOS. Above all, I liked the fact it did
not have the overhead and verbosity of Java or clumsiness of C++. I
liked the fact that Ruby, like lisp, seems to have sensible defaults
for most things. Unlike lisp, I found the naming convention a lot more
familiar, which made finding built-in objects, methods etc easier to
find (for me, the hardest part of CL has been identifying the
appropriate functions and getting familiar with nameing conventions).
I didn't bother trying an OO perl based solution. although I've done
OO perl some time ago, I do remember there were a couple of "gotchas"
to watch out for (which I've forgotten the details of) and you pretty
much have to define everything in your object - there is no sensible
default accessors like there is in CL or Ruby (though from memory, I
didn't mind OO perl when I did it, which was a reflection of my
distaste for Java and C++).

The results I ended up with from Ruby look quite good. The code seems
easy to follow and logical, but this could be a reflection of my
better understadning of the problem than in any particular stremgth of
the language. Using objects certaily made refinement simpler and kept
the code cleaner than in perl - for example, in Ruby, defining to_s
(to string) methods in my objects made report generation very easy and
when I refined the object, a simple tweak of the to_s method updated
everything. In perl, I defined a subroutine to print out the data
structures to make the report. frequently, I would update the data
structure and forget to update the printing subroutine - running the
file would either crash or not report correctly and I'd have to switch
back to the editor and fix that before trying again - not a big issue,
but it added to my frustration.

I got similar positive results with CLOS and in some ways, found
things did what I expected more often. There were a couple of gotchas
with Ruby, but again, these could be just due to my lack of experience
with the language. For example, I was surprised that a line like

puts "My object is "+object

didn't work, but

puts "My object is "+object.to_s

did. I would have thought it would have been able to determine the need to
call the to_s method due to the context of being referenced in a puts
statement. I actually still think this is probably just something I've
overlooked rather than alimitation of the language.

As a big part of this problem involved parsing records and breaking
them up into tokens and interpreting some action from the set of
toekns, I also think it is the sort of problem which either suits lisp
or at least is the type of problem I think of in terms whih suit lisp.
Given that the main "selling point" of perl is its text processing
abilities, I had expected this sort of problem to fall out quite
naturally with perl. However, either the specifics of this problem
didn't map well onto perl, or more likely, the way I thought about the
problem and solution resulted in a conceptual model which didn't map
to perl as cleanly as it mapped to lisp and/or Ruby.

For me, the main thing I think I learned was that while the languages
may be functionally equivalent in a turing sense and while they all
possessed the necessary built-in features (though I did use cl-pcre),
one language may suit an individual better than another because it has
a better mapping from how the programmer thinks to how this is
translated into code. It is also likely that the way you think about a
problem could be significantly affected by what language was used to
introduce you to programming. This would explain why some people love
C++ and java. On the other hand, my first language was basic on a
TRS-80 in c1977, followed later by logo, pascal and C (in that order)
and yet it was lisp which re-inspired me wrt programming.

I'm now going to continue to try nd refine my program some more, but
have given up the perl version completely. The main version is likely
to be ruby (because its something new), but my exploration of the
problem domain will continue to be done in lisp (because thats easy
and allows me to concentrate on the problem rather than the language).

In conclusion, I really like lisp, but suspect Ruby is going to
continue to grow in popularity. It is interesting that although I
spent more time looking at python than I have with Ruby, I have
already achieved more with Ruby than I ever did in Python. It wouldn't
be a shock to find in five years time, Ruby is more popular than
Python and people will be saying things like "Oh Python, yeah, I
remember - that was that language with the weird whitespace thing that
couldn't decide exactly what type of language it was."

Of course if it was a really fair and balanced world, in five years
time people would also be saying "PHP? What's that?"

Tim

--
tcross (at) rapttech dot com dot au
.

Relevant Pages

  • Re: Ruby vs. PHP
    ... done is checked out which versions of Perl, Python, YARV, Ruby, jRuby ... dynamic language to the seconds for the benchmark with gcc. ... YARV, Python, Perl and PHP have fairly close medians and the boxes are ...
    (comp.lang.ruby)
  • Re: For performance, write it in C - Part 2, comparing C, Ruby and Java
    ... knowing Perl is helping in my learning of Ruby. ... let's just say I think Ruby is a vast improvement on Perl. ... Frankly, I think one of the biggest reasons Java and, to a slightly ... Language" sense of entitlement -- not universally, ...
    (comp.lang.ruby)
  • Re: Is Lisp ready to become popular?
    ... the lack of "standard" networking libraries and the like, the fact that free Lisps depend on Emacs as an editor and Emacs is _not_ the default editor of many developers, the unfortunate fact that Lisp syntax is so obviously different. ... It's generating a fair amount of conversation in the Java community. ... Tate is clearly impressed by Ruby, but he dismisses Lisp's chances based on its "history and reputation to overcome". ... or in a programming language ...
    (comp.lang.lisp)
  • Re: Best unix programming language or tool for dealing with plaintext DBs?
    ... >> I plan to learn both perl, php, and ruby, and also lisp, scheme, guile ... A perl solution won't be much longer here's a ruby ... It would be nearly as short with SCSH (which has an awk makro for such ...
    (comp.unix.programmer)
  • Re: Why I never got into Lisp
    ... look flabby compared to Huffman-coded languages like Perl. ... language issues - its one of those places where they waste far too ... rewarding for this type of explorational programming. ... I found the solution I developed in Ruby also mapped more readily wrt ...
    (comp.lang.lisp)