Re: BerkeleyDB Queue Database Array Size

From: kallestad@xxxxxxxxx [mailto:kallestad@xxxxxxxxx]

I've been playing around with BerkeleyDB Queue database type, and I've
found something curious.

I'm tying an array variable to the queue database:
$db = tie @samplearray, 'BerkeleyDB::Queue',
-Flags => DB_CREATE,
-Filename => $config->get('db_queue_filename'),
-Env => $env,
-Len => 300
or print "$BerkeleyDB::Error\n" ;

I'm adding records to the database by pushing values into the array:
push @samplearray, $u or die "can't push $u $!\n";

And I'm pulling records from the beginning of the database using
shift:
shift @samplearray;

Everything is working brilliantly - although I haven't tried
multiprocess access to the database which is a desired future step.
The curiosity that I have found is this:

$k = $#samplearray;
print "$k \n";

When the array is not "tied" to the BerkeleyDB, this number will
shrink
every time I shift a record out of the array - If I pull out 5 of 5
items, it would report 3,2,1,0,-1. However, when it is tied to the
database, it will always report the maximum queue size until there is
nothing left in the queue, at which point it will return -1 like it
normally would if it was just an array. In the above example - it's
4,4,4,4,-1

I tried calling $db->sync after pulling items out, but the issue still
exists. Being a bit of a newbie, I don't know if this is a problem
with the way I am implementing things, my technical understanding, or
if this is an actual bug. Any guidance is appreciated.

This is a limitation in the current perl interface to Berkeley DB.

The problem with providing an interface to return the number of records
in the queue (or any of the other database types that Berkeley DB
provides) is that the underlying Berkeley DB library has to traverse the
complete database to get an accurate count. If you have a large queue,
this is not desirable.

Have a look at this page. In particular, look at the DB_FAST_STAT entry.

http://www.sleepycat.com/docs/api_c/db_stat.html

If the limitations of DB_FAST_STAT are good enough for your purposes,
you can use it like this


$count = $db->db_stat("qs_nrecs", DB_FAST_STAT);

If you want to take the hit of traversing the database to count the
number of records, this should do the trick

$count = $db->db_stat("qs_nrecs");

Paul

Relevant Pages

  • Re: BerkeleyDB Queue Database Array Size
    ... I'm tying an array variable to the queue database: ... I'm adding records to the database by pushing values into the array: ... This is a limitation in the current perl interface to Berkeley DB. ...
    (comp.lang.perl.modules)
  • Re: BerkeleyDB Queue Database Array Size
    ... I'm tying an array variable to the queue database: ... I'm adding records to the database by pushing values into the array: ... This is a limitation in the current perl interface to Berkeley DB. ...
    (comp.lang.perl.modules)
  • Re: BerkeleyDB Queue Database Array Size
    ... I'm tying an array variable to the queue database: ... I'm adding records to the database by pushing values into the array: ... This is a limitation in the current perl interface to Berkeley DB. ...
    (comp.lang.perl.modules)
  • Re: CLOS and databases
    ... Updating any slot ought to add that object to a queue of objects to be ... the queue gets flushed to the database. ... Supporting this for the add/drop of a slot in one class is trivial. ...
    (comp.lang.lisp)
  • Re: Parallel Programming
    ... database and pass them in as a parameter (looking a the Parallel.For ... I run my worker objects in a seperate thread als code is encapsulated in a ... worker class that implements my queue interface it can comunicate with my ... I need to grab the data from the database and pass it to a web service ...
    (microsoft.public.dotnet.languages.vb)