Re: teaching a child - console or GUI

From: Bjørge Sæther (bjorge_at_hahaha_itte.no)
Date: 07/28/04 

Date: Wed, 28 Jul 2004 12:26:46 +0200

Marco van de Voort wrote:
> On 2004-07-27, J French <erewhon@nowhere.com> wrote:
>> - or rather what is it that makes it slow if you keep most of the
>> data on disk ?
>
> Filters/queries over 6 million objects (in 8 entities/tables) must be
> in an
> acceptable time. Multiple users might run queries at the same time,
> but not
> many (say 1-4 users)
>
> Also note that 6 million objects are a lot more lines in a RDBMS,
> because you
> have all kind of coupling tables for 1 to many relations.
>
> The use of indexes is limited, or you really need a lot of indexes.
>
>>>> Try it and kick the data set up to 8Gb and see what happens
>>>
>>> That happens in 2020. 300MB/year
>>
>> Will they be interested in 2001 data in 2020 ?
>
> No, 5 years max. And even that is unlikely. They probably could do
> with 2,3
> years real data, and global stats from the other 2,3 years. But it is
> not
> worthwhile to code that.
>
> However that 300MB/year figure and the five year figure is the
> current situation.
>
> We didn't know the exact sizes and date ranges yet when we made the
> decision. At the time we were afraid of hitting the Delphi limit of
> 2-3 GB.
> It was more 780MB/year in the initial version, and application memory
> on top of that. Improved indexing, and packing some data decreased the
> size of the data.

Did you have a strategy on what to do if you couldn't make it all in RAM ?
The application I'm working on (and concidering loading all data in memory
sometime in the near future) hit the wall the day they had a customer whose
data was suddenly some 50x larger.
The requirement was to be able to complete calculations withing 48 hours
(1999), and I was hired as an "expert" to help them make it. It showed that
what works with 100,000 rows doesn't necessarily work with 6,000,000 rows,
and that some database operation loads are typically evolving in an n^2
manner. It took me 4 months and a total rewrite of both the application and
the database to get there.
Now there are even bigger databases in use, and the undergoing rework is
about to enable using data for a longer period. In one installation we will
possibly reach 30 million rows within short....

I believe this story is somewhat more typical, an application growing far
beyond it's original limits.

>> Realistically I have things that should really be re-written, but I
>> defend my not doing so on the grounds of
>> - why upset it
>> - its working so the users will not benefit
>> - I'm idle
>> - I'm not interested
>>
>> I can seldom point to anything and say that it cannot be improved, or
>> rather 'future proofed'.
>
> I would not try to retrofit such system to a RDBMS based solution,
> unless
> you have severe (as in magnitudes) performance problems. The way one
> works
> is too different. It is more meant for new development.

An important point when concidering using RDBMSes is that mastering an RDBMS
is not simpler than writing a Delphi application. Its just different. If you
take Oracle, there is at least a handful of different job descriptions:
Architect, Tuner, Installation maintainer, Security etc., etc..
I've been working on 4 Oracle projects, and I still feel I have only
scratched the surface. The 4 months of trying to make Interbase fly, OTOH,
resulted in quite another feel for what's going on under the hood. Then,
some 3 years later, I had to spend some 2 months to refactor the application
to use a correct transactions scheme (much due to bugs and peculiarities of
IBObjects).
I've seen some terrible examples on Delphi programmers abusing database
engines. In such cases flat files would probably be much better. But one
should not underestimate the power of Delphi on top of an RDBMS that one
knows well...especially because knowledge on RDBMS and SQL is widely spread. 

-- 
Regards,
Bjørge Sæther
bjorge@haha_itte.no
-------------------------------------
I'll not spend any money on American Software products
until armed forces are out of Iraq.

Relevant Pages

  • Re: teaching a child - console or GUI
    ... At the time we were afraid of hitting the Delphi limit of ... > the database to get there. ... > An important point when concidering using RDBMSes is that mastering an RDBMS ... > is not simpler than writing a Delphi application. ...
    (comp.lang.pascal.delphi.misc)
  • Re: Delphi Jobs
    ... I am a Delphi database programmer. ... Source Safe and Source Off Site on the server side and client side. ... design several pieces for the interactive website. ...
    (comp.lang.pascal.delphi.misc)
  • Re: Good Books on MultiValue Databases
    ... new info there, right?), the database structure when converting from ... an MV environment to an RDBMS actually does require re-architecting. ... Codd wrote his rules specifically ... management system, rather asking the management system to manage ...
    (comp.databases.pick)
  • Re: Good Books on MultiValue Databases
    ... new info there, right?), the database structure when converting from ... an MV environment to an RDBMS actually does require re-architecting. ... Codd wrote his rules specifically ... management system, rather asking the management system to manage ...
    (comp.databases.pick)
  • Re: When to use sub-values
    ... from the outset as if each discrete problem is a database problem -- ... "optimizations" are merely optimizations. ... within LOCATE (depending on the controlling-nesting levels you want to ... reimplemented our own simulated RDBMS to be able to store/retrieve/sort ...
    (comp.databases.pick)