Re: DBMS and lisp, etc.

From: Will Hartung (willh_at_msoft.com)
Date: 05/18/04 

Date: Tue, 18 May 2004 11:29:35 -0700

"Chris C apel" <chris@ibanktech.net> wrote in message
news:69d0f981.0405180947.8f32f4d@posting.google.com...
> So here's my question to the group. What's a good way to start out
> using normal lisp structures in global variables as a psuedo-database
> during initial development and then switch to a real RDBMS backend? I
> can see two issues that need to be considered right off.
>
> 1) Data access migration. I think this is simple, though, as I can
> write a data access layer specific to my application and implement it
> for a RDBMS after I have a clear idea of my data structure. Are there
> packages to generate such a layer based on CLOS classes, perhaps?

Well, obviously you can do whatever you want to do, I'm not aware of any
tools to help you here though.

The other problem is essentially enforing the Relational view of your data
in a decidedly non-relational system. The reason this is important is that
if you don't take the discipline now to view your data relationally for
eventual storage in a DBMS, then you'll have more headache later when you
only want to change the query and persistence layer.

Also, depending on your goals, you may want to pencil in any transactional
boundaries in your system even if your crude backing store does not actually
handle them. That way you can think through the basic issues on the
assumption that you will inevitably deal with a transactional store later
on, and not have to retrofit that in later.

Finally, you'll need some way to filter and order your data. As you can
imagine, it's fairly easy to simply use hash tables to get basic references,
but one of the features of the relational model is being able to query and
order on arbitrary data elements, including those that are not indexed at
all. Now, you may not feel you'll need to do that at all, but it's a shame
to toss out a real benefit of the DBMS and design around it at this point.
Kind of depends on your data model.

Also, try and think of "big" queries in your system, getting as much as
practical in a single query rather than grabbing the nodes of your tree, and
then lazily filling them in with queries to the child bits. That's fast on
in memory systems, but will destroy performance on a DBMS. Round trips to
the DBMS are Evil. SQL Joins are your friend.

You don't need a "generic OO -> Relational" mapping system. Just a layer
that seperates your data from the back end, and unless you have 1000 tables,
you can write this yourself, by hand. By hand you can specialize your
queries with a lot less complexity than using an OR mapper.

> 2) Persistence. This was the main issue I had in mind writing this
> post. When my SLIME backend goes down and I need to restart emacs (not
> necessary, I'm sure, but I'm no SLIME guru) I don't want to forever
> lose all of my test data. Perhaps the solution is to dump a core file
> with my test data in it periodically (what are the functions for that,
> now?). But I'd rather persist just those objects related to my test
> data, and not everything else as well.

As you say, you can store everything is one, large, single rooted data
structure and the simply "print" to save it, and then "read" to load it.
That will work peachy for smaller data sets (with the definition of "small"
being related to your machine performance and your patience for waiting for
it to save and load).

When that seems to be too slow, and you're not ready to layer in a DBMS yet,
you may want to look into some implementation specific bits of your Lisp, as
many have the ability to dump, essentially, FASL files, which are normally
used to store compiled code, but some Lisps can also save and load FASLs of
any data structure. These load and save MUCH faster than "print" and "read",
but of course you have an enormous binary blob of data that can't be changed
outside of your image.

Regards,

Will Hartung
(willh@msoft.com)

Relevant Pages

  • Re: fasl and lisp image
    ... fundamental difference between the lisp image and the set of .fasl ... need anything more than a remembered set of fasl to load (both ... When I work with a Lisp machine, ... When I later restart the Lisp, ...
    (comp.lang.lisp)
  • Re: Poll : Are you interested seeing Qi as embedded language in lisp?
    ... LOAD the following into Qi. ... why it wouldn't work on SBCL and other lisp implementations. ... Now, Qi doesn't have an asdf system definition, but it's a somewhat ... currently in a single file anyway, you just compile and load it and ...
    (comp.lang.lisp)
  • Re: How to load and clear files?
    ... You can just load new definitions to replace the old ones. ... by the old name in the lisp image, ... CLISP is written in portable C code: you can compile it on almost any ... or:EXECUTE to precise your idea. ...
    (comp.lang.lisp)
  • [Lit.] Buffer Overruns - II
    ... I suppose that if I were familiar enough with Lisp in 1989 that I might have ... > relative to actual load address' is unfortunately not clear ... just use the appropriate load library system call. ... typedef struct {... ...
    (sci.crypt)
  • Re: ILC2005: McCarthy denounces Common Lisp, "Lisp", XML, and Rahul
    ... > However, is Common Lisp source code, especially with reader macros, ... Reader macros have to return objects, ... If we are going to store structured data in files, ... let's pretend XML is useful. ...
    (comp.lang.lisp)