Re: Programming By The Seat Of Your Pants

From: Scott McIntire (mcintire_charlestown_at_comcast.net)
Date: 10/22/03 

Date: Wed, 22 Oct 2003 14:50:56 GMT

"David Steuber" <david.steuber@verizon.net> wrote in message
news:m2he21a30w.fsf@verizon.net...
> This isn't really quite on topic for c.l.l, but it is strongly
> related.
>
> I am in a situation with myself where I need to basically code by the
> seat of my pants. Why? I have no design, no solid requirements, and
> generally have only a vague idea of what I want. Lisp* seems like
> the tool for the job. Besides, the only way I'm going to learn Lisp
> is to program in it.
>
> Programming by the seat of my pants is something that I do more often
> than not. Sure, I don't have a solid idea for my project, just a
> vague notion. It is never really that different in the business
> world that I have had experience with. The business wants X but they
> don't seem to want to tell me what X really is.
>
> When pressed, the business will come up with what they think is a
> precise definition of X. To me, it is still hand waving, only with
> pictures. Then they like to say, "so how long will this take?" Um.
> Yeah. If I knew how long it would take, then I would know what the
> problem was and its solution. If I know the solution, I have already
> solved the problem.
>

Buisiness people and managers are playing the power game. They don't want
craftsman, they want interchangeable parts. With that midset comes
necessarily the belief that what you do is factory work. They give you w
vague idea what they want and you should just tell them how long it will
take.

> I can't seem to convince people that time estimates are crap. There
> are even programmers out there who think that meaningful time
> estimates are possible. If you are one of those people, please tell
> me what the hell you smoke so I can get some.
>

I have had similar experiences. There is always the belief that if one
writes a careful spec then one can make precise time estimates. The problem,
as Paul Graham points out is that specs and code are made of the same thing.
If a spec were that precise it would be the code! And, its true that the
customer may say that want X but you have to look deeper to see that they
need Y. In the best case, when you're done they'll say "Yeah, but we also
need Z, that's easy enough to do right?" And if you have a GUI and there are
n people involved (and you have no bugs with your system) you can expect to
have n+1 opinions about how the GUI should be changed.

What I found was that if you are developing an product that is a "me too"
application, you find that even with poor specifications that everyone has
pretty nuch internalized a spec for the product. And times estimates in the
large can be more accurate if you'be done similar things. It's when you are
working on something that hasn't been built before that time estimaties are
very hard to do; that specifications written or otherwise are bad. You can
expect to have someone make you give an estimate of the time and then you
can expect to have the product change directions many times. During this
process, there is much more experimentation going on; of course, if you ever
mention to someone that you are experimenting you might be shown the door. I
found at the time that I could use scripting languages, Tcl/Tk (along with
various extentions like [Incr Tcl] that gave you an object system. Later it
was Python, a better scripting language. You had to fight against the rest
of the company that used "real" langauges like C or C++. Most people didn't
get that (especially new) product development could be done much faster with
dynamic interactive langauges. The problem I had was that for speed I had to
drop down to a "real" language. Thus, ultimately I ended up with
applications that had some scriptability, but suffered from
1). Having to interface 2 languages more than I wanted.
2). Working with lame scripting languages.

I wondered if I could find a language that was this interactive, yet could
compile to native code when I was done. I also wondered if I could find a
language that didn't have a dicked-up single dispatch object system. The
answer of course is Lisp. The problem is that once you start using Lisp it
is going to be really hard to go back to any other language. The problems I
see with using Lisp are the following:

1). Not much in the way of libraries.
     a). Well, if you pick a hard problem, there most likely won't be much
library support in other langages either.
    b). You can still write the core in Lisp, get scriptability and use a
foreign function interface to other libraries. Yes, I know the 2 langauges
thing. At least you won't be doing it for performance reasons.

2). Lisp is different. There is a lot to learn and the concepts don't map
easily to other langauges. We need better books or examples of how to use
the various features.
   a). This is changing, I think you'll see at least one, maybe two books
next year that address this.

3). Standardization: This really impacts point (1). This is not so much of a
problem for an application as it is for people building libraries. We need
something akin to Perls CPAN. Without libraries that you can pull off the
internet and install easily, Lisp is going to find it hard to compete with
the scripting langagues in a lot of areas. I personally don't want to go
back to those languages.
   a). I would like to see the ALU bless "defacto" standards and get the
vendors to work on standards for multi-processing, sockets, etc.
     By defacto standards I mean the following:
      defsystem --> ASDF
      test harness --> PTESTER
     Web server/HTML generation --> ASERVE, LML2
     Regular Expressions --> CL-PPCRE
     Foreign Function Interface --> UFFI
     Emacs interafction --> ELI (A good interactive environment shows the
power of the interactive nature of Lisp.)

    This is sure to spark some debate. Let me just say, if we can't agree on
one item, bless 2. So, if we can't determine whether ASDF or mkdefsystem is
better bless both. But lets try to agree to focus on writting ANSI compliant
libraries in a way that people with a minimal number of standard packages
can pull down libraries off the internet and just have them work. We should
also try and pull together some of the libraries we do have and put them in
one place.

 4). GUI's:
      Maybe the answer is to use a web interace. Or maybe there should be
something that the vendors can standardize (this is asking a lot and may be
unreasonable). Or, maybe the answer is CLIM. We'll need a lot more examples
of how to use this puppy if that is the case.

> As for specifications that are not vague, I guess UML would be the
> king. The problem with UML is that you model a solution or process
> to such exacting detail that you have basically just programmed the
> bloody thing. Just run the UML through a code generator and you are
> done. UML and design patterns that go with it are popular because
> C++ and, later, Java are really the suck for programming anything
> that is not well defined ahead of time.
>

Amen to that.

> I don't work well that way. As a C++ programmer the text editor has
> always been a doodle pad for me. I have always had a very tight code
> compile test loop. When programming in Perl, the loop just got
> tighter. The editor was still a scratch pad. I would program top
> down. I need to do something like X. Well I can do that if I do A,
> B, and C. And so on until the details were filled in.
>

Lisp is your friend here.

> I keep hearing about Lisp as a bottom up programming language. Is
> that really the case? Yeah, you use macros to build the language up
> to the problem level. If you do that unguided though, what are you
> building up to? Top down seems more natural. You start out with
> (DWIM X) and start to define DWIM.
>
> Or do you just do stuff with macros and feel clever when you can do
> something useful? That doesn't seem like a goal oriented approach to
> me.
>

There are much more eperienced people here than I. I have found that it is
best to think more closely at how defun is defined. It is a macro that
associates code with a symbol. You can do the same with whatever system you
want to define. I would first build the raw functions you need for your app,
then add a macro layer that allows you to write things the way you want to
write them. For instance, I have a fuzzy logic package. I have a macro that
defines a fuzzy system, its called deffuzzy. With this macro I can write a
fuzzy system the way that a customer (or myself) would like to see it
specified. The macro then takes a look at the arguements and does a lot of
checking. The macro checks the structure of the arguements, ranges,
consistency between various bits of information. Something, that a static
type checker can't do by the way. Performance - don't care here, I do as
much checking as I want. After checking, the fuzzy rules of the fuzzy
variables are COMPILED. Performance, - don't care here. By the way, the
compiler is built in, I didn't have to make one. When one runs the rules, I
have compiled code that is fast. Performance - I care now.

In other general purpose languages, the language designer made the tradoffs
for performance and error checking for a general purpose language. In Lisp,
I can, for a given domain, make the appropriate decisions myself. Did I
mention that I can call the compiler too?