Re: NoSQL Movement?

On Sat, 13 Mar 2010 23:42:31 -0800
Jonathan Gardner <jgardner@xxxxxxxxxxxxxxxxxxx> wrote:
On Fri, Mar 12, 2010 at 11:23 AM, Paul Rubin <no.email@xxxxxxxxxxxxxx> wrote:
"D'Arcy J.M. Cain" <darcy@xxxxxxxxx> writes:
Just curious, what database were you using that wouldn't keep up with
you?  I use PostgreSQL and would never consider going back to flat
files.

Try making a file with a billion or so names and addresses, then
compare the speed of inserting that many rows into a postgres table
against the speed of copying the file.

That's a straw man argument. Copying an already built database to
another copy of the database won't be significantly longer than copying
an already built file. In fact, it's the same operation.

Also consider how much work it is to partition data from flat files
versus PostgreSQL tables.

Another straw man. I'm sure you can come up with many contrived
examples to show one particular operation faster than another.
Benchmark writers (bad ones) do it all the time. I'm saying that in
normal, real world situations where you are collecting billions of data
points and need to actually use the data that a properly designed
database running on a good database engine will generally be better than
using flat files.

The only thing I can think of that might make flat files faster is
that flat files are buffered whereas PG guarantees that your
information is written to disk before returning

Don't forget all the shadow page operations and the index operations,
and that a lot of these operations require reading as well as writing
remote parts of the disk, so buffering doesn't help avoid every disk
seek.

Not sure what a "shadow page operation" is but index operations are
only needed if you have to have fast access to read back the data. If
it doesn't matter how long it takes to read the data back then don't
index it. I have a hard time believing that anyone would want to save
billions of data points and not care how fast they can read selected
parts back or organize the data though.

Plus the fact that your other DB operations slow down under the load.

Not with the database engines that I use. Sure, speed and load are
connected whether you use databases or flat files but a proper database
will scale up quite well.

--
D'Arcy J.M. Cain <darcy@xxxxxxxxx> | Democracy is three wolves
http://www.druid.net/darcy/ | and a sheep voting on
+1 416 425 1212 (DoD#0082) (eNTP) | what's for dinner.
.

Relevant Pages

  • Re: NoSQL Movement?
    ... I use PostgreSQL and would never consider going back to flat ... Copying an already built database to ... Don't forget all the shadow page operations and the index operations, ...
    (comp.lang.python)
  • Re: NoSQL Movement?
    ... another copy of the database won't be significantly longer than copying ... versus PostgreSQL tables. ... using flat files. ... RDBMs do wonderful things in the ...
    (comp.lang.python)
  • Re: Writing data to disk?!
    ... your customer what is needed. ... Maybe flat files would be better for this. ... standard format (database, xml, etc). ... - Database servers ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Windows Ada database support.
    ... >>for example) promote the use of Object IDs (basically Row IDs). ... >>Still others don't support the concept at all (MySQL). ... >>within certain database design restrictions). ... PostgreSQL uses an API that opens/creates etc. and returns ...
    (comp.lang.ada)
  • Running queries on large data structure
    ... the money) that parses multiple large text files containing nested data ... data from the PostgreSQL database by different criteria. ... Networks ...
    (comp.lang.python)