Re: Program compression

Haskell, SML, OCaml, Mathematica, F# and Scala all allow real
problems to be solved much more concisely than with Lisp. Indeed, I
think it is difficult to imagine even a single example where Lisp
is competitively concise.
What does "solved much more concisely" mean??
From: Jon Harrop <j...@xxxxxxxxxxxxxxxxx>
The solutions are shorter in modern FPLs.

OK, let's say you have a text string which contains the notation
for a nested list starting from the beginning of the string. How
many lines of code, in various modern FPLs, would it take to parse
that nested-list notation to produce an actual nested list
structure, and also report back in another value the position in
the string where the parse left off at the end of the nested-list
notation? In Common Lisp it's just one line of code:
(multiple-value-setq (ls ei) (read-from-string ts))
where ts has the given text string,
ls gets the resultant list structure,
and ei gets the resultant end index.

Now let's say you want to pick up where you left off, because after
that first nested list notation there's another within the same
text string you were given. How many lines of code in your favorite
FPLs? In Common Lisp, it's another one line of code:
(multiple-value-setq (ls2 ei2) (read-from-string ts :start ei))

My point is that different languages are based on different things
being important enough to put into the core language, but with
add-on libraries it's possible to make a lot of things be
essentially the same length in a whole bunch of languages. If you
don't mind Unix-style ultra-short names for commonly use utilities,
and you do a lot of this kind of stuff, if you really want to make
your code as short as possibly, you can use macros to abbreviate
the special operator and function used above, so the two lines of
code will easily fit into just one run-on line of code:
(mvs (ls ei) (rfs ts)) (mvs (ls2 ei2) (rfs ts :start ei))

So I propose when comparing languages that we allow liberal use of
macros (in languages such as Lisp which support them) to tailor the
function names to the same length in all languages, so that when we
look at the code we are comparing the size of the parse tree and
*not* the relative lengths of names of functions. And furthermore
we allow add-on libraries to compress the size of the parse tree in
the application itself down to just the part that's different from
one application to another. So the only real difference is whether
the syntax of the language (with all available help from macros and
libraries) supports the most natural expression of the application
of the tool(s).

So let's do a check-list. Which languages allow natural expression of:
- case expresssions (not case statements!)
- loops with a typical wide variety of ways to map various indices/pointers
across lists/ranges
- anonymous functions which lexically close variables from the
surrounding lexical scope
- a mix of symbols which are in a master table so that they can be
canonicalized upon input and symbols that are *not* in such a
master table so that multiple instances by the same name can be
created at runtime without them interferring with each other, and
where each symbol of either type can have an unlimited number of
tagged properties hanging off it, including the traditional
variable (value) and function-definition
- formal specification of the parse tree of source code as data
that can be manipulated at runtime just the same as ordinary data
can
- choice or mix of two ways to specialize functions and datatypes,
either by having a single function defined in one place which
explicitly dispatches on the type(s) of its parameters, and/or
having multiple definitions of the same-name function distributed
into multiple places where keeping track of all the variants of the
same-name function is automatic and dispatching from the same
generic function per parameter datatypes is set up automatically
- virtually identical semantics between code typed directly into a
read-eval-print loop one line at a time and lines of code embedded
within function definitions which are either entered from the REP
as a unit or loaded from a source file en masse or compiled to an
object file and then that file loaded, with easy intermix of all
four modes of lines-of-codes
Common Lisp can do all of those. How many of your favorite FPLs can?

Macros are very rare in modern languages.

Why? Because their parse tree isn't well defined and documented, so
it would be impossible for the author of a macro (in the Lisp
sense) to know what to do? Or because the syntax of the source code
is so different from the parse tree that it would be just too
difficult to mentally keep switching back and forth between
thinking about the parse tree (when designing the function of the
macro, what the macro is supposed to do) and writing code
(including the code of the macro-expander itself)? Or because
authors of most modern languages are unaware how valuable macros
can be in improving the "out of the box" syntax to be better fit a
problem domain or to capture the essense of common design patterns
into a single syntax?

Let's think about design patterns more. The whole idea of a design
pattern is that the language itself doesn't support direct
expression of a design, so the programmer must constantly think of
the pattern and write code in that form to put the pieces of a
pattern together into a coherent hole that is self-consistent. But
with a macro, all the programmer need do is call the name of the
design pattern and the parameters to it, and the macro-expader
automatically generates the whole self-consistent design pattern.
So from the view of the programmer, it's no longer a "pattern",
it's a utility.

Here's an example of what was originally a design pattern:
(unwind-protect
(let ((chan (open "foo.txt")))
(process-file chan))
(close chan))
The idea is that if something goes wrong while the file is open,
and PROCESS-FILE throws control out of the scope of its toplevel,
the CLOSE of the channel will nevertheless be executed, so the file
**will** end up closed before control reaches the main level of
program above this code.

So with a macro, it's no longer necessary to go to that pain of
writing all the stuff in just the right relationship. Now it's just:
(with-open-file (chan "foo.txt") (process-file chan))

Now that particular macro wasn't in older versions of Lisp before
Common Lisp, but the design pattern was so commonly useful that the
macro was incorporated into Common Lisp. But back in those earlier
versions of Lisp, programmers didn't have to write the
UNWIND-PROTECT pattern over and over. Instead they just wrote a
macro once and then called it. They didn't have to wait for Common
Lisp to do it. They could realize all by themselves that the design
pattern was common, and that the macro would condense the pattern
into a foolproof utility, so they just did it without waiting for
Common Lisp to provide it built-in. That's the value of
user-definable macros.

Now there are probably a *lot* of design patterns used in Common
Lisp and other languages which are not yet covered by pre-defined
macros or other mechanisms, because at the time Common Lisp or the
other laguages were standardized those patterns weren't recognized
as common enough in use with well-established macro that it seemed
reasonable to incorporate them into the standard language. After
time passes, in some problem domains it becomes common knowledge
that certain design patterns are common enough that it *really*
would be nice if the language supported them as a compact
expression *instead* of requiring the program to repeat the design
pattern over and over and over. Here is where languages such as
Lisp which have parse-tree macros, and other languages that don't
have them, differ: In Lisp, the appliation programmer has the power
to define a macro and thereby convert a design pattern into a
simple call. In languages without macros, the programmer does *not*
have that option, and must forever write out the entire design
pattern over and over and over until and unless the people defining
the standard and implement it decide to make that simple call part
of the language.

OCaml has a full macro system but it is rarely used.

Is it a parse-tree macro system, like Lisp has, or is it a
string-syntax macro system, like C has? If the former, then why
isn't it used? Perhaps as I guessed above, because the parse tree
is so different from the source syntax that it's confusing to
switch back and forth between the two? If it's a string-syntax
macro system like C, then it's total crap and I can understand why
it's rarely used.

F# has no macro system.

Then F# is crap compared to Lisp.

Why do you think statically typed languages completely dominate
general purpose programming?

Because a lot of people don't know any better and are stuck
with installed code base (legacy code) which must be maintained.

(The exception is in tight loops
where some machine type happens to work efficiently and where that
particular loop eats up most of the CPU time, which is precisely
where Lisp allows type declarations and consequent CPU-efficient
code as an *option* where it would actually be worth the cost.)
Lisp is unable to regain the performance cost of its dynamic design:

As long ago as around the late 1970's, MacLisp achieved a
reputation that tight code loops with proper declarations ran
faster than the equivalent loop in Fortran, because the code within
each function was identical but Lisp had more efficient
function-call linkage. Do you dispute that, or do you claim that
Common Lisp has taken a step backward and no longer is as fast in
tight loops with proper declarations as it should be?

By the way, there *is* one apparent problem with Common Lisp:
Getting those declarations correct is a rather tricky skill, and
different implementations of Common Lisp present different levels
of difficulty for optimizing. See the two threads where I had
trouble optimizing a benchmark of random access within large array,
and where I had trouble optimizing a brute-force trial
factorization, in both cases it was doing CONSing when it shouldn't
have. (SBCL was able to avoid CONSing, but with the same source
code the very old version of CMUCL that my ISP has was doing lots
of CONSing.)

finalizers,
You mean like to automatically close a system resource whenever the
Lisp hook for it is GC'd? Normally system resources like that are
lexically bound, whereby UNWIND-PROTECT is sufficient to make sure
they get closed when that block of code is exited. Is there
*really* a good reason to have such resources kept around per data
object instead of per block-entry/exit?
Yes. Finalizers are useful for the same reason that GC itself is
good (and Ada is bad): you don't want to have to shoehorn
everything you do into scopes just to ensure that things get freed
correctly.

On a philosophical level, I totally agree with you on this point.
I'm convinced you're not a troll!
Although you frequently go overboard on exaggerations in favor of
other languages and against Lisp, you *are* bringing up good issues
and occasionally (as here) actually showing some good insight.
So I'm not sorry I engaged you in debate.

Now let's discuss the practical matter, on a case by case basis:

If you are reading a file, you probably want only one agent reading
it at a time, and you probably want to process the contents of the
file straight through (or bulk copy the whole contents of the file
to a text-string and immediately close the file and process from
the string rather than from the file itself). Accordingly lexical
scope and UNWIND-PROTECT (such as in the WITH-OPEN-FILE macro) is
totally sufficient.

If you have a connection to a remote database, you may have
multiple agents sharing a single connection. But you want to close
the connection whenever the connection goes idle too long, and
automatically re-open it whenever somebody starts to use it again,
rather than keep it open yet idle the whole time anybody has a
handle on it. Accordingly a clock-timeout mechanism makes more
sense than a finalizer tied into the GC.

I'm hard pressed to think of any case where the best mechanism is a
GC-finalizer rather than lexical scope or clock-timeout and
auto-reopen. Please enlighten me with a few examples (no details,
just the essence of the problem to be solved) where GC-finalizer is
the *best* solution to resource management, OK?

Another reason is that garbage collectors often free sooner than that.

I don't really believe that. Either the pointer to the handle to
the resource is lexically scoped, in which case GC won't reclaim it
until that pointer goes out of scope, in fact won't reclaim it
until sometime *after* the pointer goes out of scope; Or you have
multiple pointers being passed around in a non-lexical way and GC
won't reclaim it until the very last of those pointers is
inaccessible, whereas clock-timeout will close the resource as soon
as those pointers are no longer actively being used, hence again GC
will close the resource *after* the other method. Sure, once in a
while all the multiple pointers will suddenly go away at once, and
GC will by chance happen sooner than the idle-timeout. But other
times GC won't have yet happened when idle-timeout happens. So it's
a race that can go either way with no consistent winner. Please
present a scenerio where GC gets there *first* more often than not.
Or an example where idle-timeout is *not* an option, where the
resource absolutely must be kept active until the very last of the
references to it is gone and GC then can reclaim it, even if none
of those references are actively using it.

But if you can think of a really good use for random closure, I may
reconsider my opinion.
I used finalizers in OCaml to collect unused OpenGL display lists and
textures. The results were excellent: the code was greatly
simplified at no cost.

Sorry, I don't know what those are, and they don't seem to be a
central issue that is worth my spending a half hour doing a Google
search to find a tutorial to teach me what they are. Would you
please define their important characteristics in regard to this
discussion, i.e. where they are physically located, what reason
they have for being kept around (until all references are gone),
why they can't be easily re-created if idle-timeout causes their
deletion but re-activity forces their re-existance, why they can't
be kept around nearly *forever* (until they have been idle a week
or two and it really would be surely a waste to keep them around
longer), etc.? On Unix/Linux there's a concept of various processes
in a tree, with each process dependent on the next parent process.
When a parent dies, all children automatically die. It would make
sense for a display list or texture to be a property of a
sub-process, which stays around so long as the main process is
around, then when the main process dies the sub-processes are
auto-killed and the display list or texture then disappears. Why
doesn't that model fit what's actually happening with display lists
and textures? The only reason I can think of is when the main
process is some persistent daemon, such as the Apache server. But I
don't think the Apache server or any of the other persistent
daemons are written in OCaml, so there must be some other reason
why that sub-process model wouldn't solve the problem
(hypothetically if OCaml didn't have finalizers, or if you decided
to translate your application to some other language that doesn't
have finalizers). So if you can succinctly enlighten me, please do.

The vast majority of computers do not have virtual memory because
they are embedded devices.

Yeah, that's a different environment from anything I've ever
programmed for. I would suspect that in an embedded device you
wouldn't want GC in the first place, you'd do better with mostly
static allocation with perhaps some non-cyclic dynamic structures
where reference count would work just fine. But it's late and maybe
I've lost track of the context.

I think you should aspire to earn money directly from customers
rather than asking people to give you money to develop your
software.
I like that idea very much. Will you find me a potential customer
who wants some serverside application that doesn't already exist?
You need to write a compelling sales page and make it indexable
by search engines and make it as easy as possible for users to give
you their money before plastering all of the news websites that you
can find with descriptions of your new service and links to it.

That's a strawman comparison. In fact before doing *either*, I need
to find some customers who desire some new server-side application
that is within my capability, and who are willing to work with me
to define use cases that I might offer to implement, and then once
that's all done *then* we start talking about a fixed-price
contract to implement those use cases, and alpha testing, etc. Then
after I have one set of satisied customers, only then would it be
reasonble to advertise to see if there might be lots of other
customers for an existing customer-pleasing application.

So can you find me a first brainstorm-use-cases customer to get
this whole process started? I've mentionned lots of new software
technology I've implemented to the pointer where I feel they might
possibly be useful for a variety of applications that might be of
value to others. But until I find a potential customer who is
actually interested in what I have done and how I can work it into
a salable product, I don't know if any of the wondrous things I've
programmed really could lead to a salable product.

By the way, Tuesday of last week I gave a partial demo of some of
my computer-assisted instruction software, and the person seeing
the demo seemed to think it would be of interest to two of his
technical acquaintances, one very far away (about 1000 miles) and
one closer (about 40 miles), so he'll be contacting them to try to
get them interested.
.

Relevant Pages

  • Re: Better is better: improving productivity through programming
    ... Nothing in Common Lisp prevents us from doing that. ... some other popular languages takes a *lot* of work. ... The idea of using a language with better abstraction facility doesn't ...
    (comp.lang.lisp)
  • Re: newLISP is simple, terse, and well documented
    ... Err..., no, MACLISP chose speed and Common Lisp followed. ... compilers and macro packages.'' ... ``It is widely held among members of the MIT Lisp community that FEXPR, ...
    (comp.lang.lisp)
  • Re: A "killer" macro
    ... Many contemporary languages boast their sharing most ... of the functional features of Lisp. ... thing they lack is uniform syntax, and hence the service of a powerful ... I have earlier used an implementation of a lexer macro as an example, ...
    (comp.lang.lisp)
  • Re: a LISP raytracer
    ... My languages professor right now is a huge fan ... and not so much with LISP. ... Brevity is another major advantage of MLs, e.g. SML and OCaml, over Lisp. ... > and his "def" macro is a great example of this. ...
    (comp.graphics.rendering.raytracing)
  • Re: Three questions
    ... of the functionality you are doing here already exists in lisp. ... Another way to implement this macro, ... really understand all of the binding forms in Common Lisp and also make ... often KEY for properly implementing control structures. ...
    (comp.lang.lisp)