Re: DBMS and lisp, etc.
From: Will Hartung (willh_at_msoft.com)
Date: Tue, 18 May 2004 11:29:35 -0700
"Chris C apel" <firstname.lastname@example.org> wrote in message
> So here's my question to the group. What's a good way to start out
> using normal lisp structures in global variables as a psuedo-database
> during initial development and then switch to a real RDBMS backend? I
> can see two issues that need to be considered right off.
> 1) Data access migration. I think this is simple, though, as I can
> write a data access layer specific to my application and implement it
> for a RDBMS after I have a clear idea of my data structure. Are there
> packages to generate such a layer based on CLOS classes, perhaps?
Well, obviously you can do whatever you want to do, I'm not aware of any
tools to help you here though.
The other problem is essentially enforing the Relational view of your data
in a decidedly non-relational system. The reason this is important is that
if you don't take the discipline now to view your data relationally for
eventual storage in a DBMS, then you'll have more headache later when you
only want to change the query and persistence layer.
Also, depending on your goals, you may want to pencil in any transactional
boundaries in your system even if your crude backing store does not actually
handle them. That way you can think through the basic issues on the
assumption that you will inevitably deal with a transactional store later
on, and not have to retrofit that in later.
Finally, you'll need some way to filter and order your data. As you can
imagine, it's fairly easy to simply use hash tables to get basic references,
but one of the features of the relational model is being able to query and
order on arbitrary data elements, including those that are not indexed at
all. Now, you may not feel you'll need to do that at all, but it's a shame
to toss out a real benefit of the DBMS and design around it at this point.
Kind of depends on your data model.
Also, try and think of "big" queries in your system, getting as much as
practical in a single query rather than grabbing the nodes of your tree, and
then lazily filling them in with queries to the child bits. That's fast on
in memory systems, but will destroy performance on a DBMS. Round trips to
the DBMS are Evil. SQL Joins are your friend.
You don't need a "generic OO -> Relational" mapping system. Just a layer
that seperates your data from the back end, and unless you have 1000 tables,
you can write this yourself, by hand. By hand you can specialize your
queries with a lot less complexity than using an OR mapper.
> 2) Persistence. This was the main issue I had in mind writing this
> post. When my SLIME backend goes down and I need to restart emacs (not
> necessary, I'm sure, but I'm no SLIME guru) I don't want to forever
> lose all of my test data. Perhaps the solution is to dump a core file
> with my test data in it periodically (what are the functions for that,
> now?). But I'd rather persist just those objects related to my test
> data, and not everything else as well.
As you say, you can store everything is one, large, single rooted data
structure and the simply "print" to save it, and then "read" to load it.
That will work peachy for smaller data sets (with the definition of "small"
being related to your machine performance and your patience for waiting for
it to save and load).
When that seems to be too slow, and you're not ready to layer in a DBMS yet,
you may want to look into some implementation specific bits of your Lisp, as
many have the ability to dump, essentially, FASL files, which are normally
used to store compiled code, but some Lisps can also save and load FASLs of
any data structure. These load and save MUCH faster than "print" and "read",
but of course you have an enormous binary blob of data that can't be changed
outside of your image.