Re: Stale data in DB_File?

From: Bob Walton (invalid-email_at_rochester.rr.com)
Date: 10/22/03 

Date: Wed, 22 Oct 2003 03:14:19 GMT

Mark wrote:

> Hello,
>
> I was wondering whether someone could explain to me a weird problem I am
> having with DB_File (Perl 5.8.0).
>
> Okay. A daemon uses this:
>
> use DB_File;
> use Fcntl;
>
> tie %drac, DB_File, '/usr/local/etc/dracd.db', O_RDONLY, 0440, $DB_BTREE;
>
> Within the daemon I then check like this:
>
> if ($drac{'192.168.0.1'}) {
> ...
>
> Works fine. Except, that when I delete an entry (with an external program,
> also using DB_File), like so,
>
> delete $drac{'192.168.0.1'};
>
> Then, within the daemon the entry still exists! And vice versa: if I add an
> entry to the dabase, then, within the daemon, this addition is not seen.
> Within the daemon I do this, for test purposes:
>
> foreach (sort keys %drac) {
> write_syslog ("$_, $drac{$_}");
> }
>
> The only way I was able to avoid stale data, is to evoke the tie call, each
> time, within the subroutine that tests $drac{'entry'}. Surely I am not
> supposed to do that, right? I thought you had to do the tie only once, after
> which %drac stays tied throughout the process.
>
> The drac database, of course, is updated from the outside (like my test
> programs did).
>
> Please, tell me what I'm missing and/or doing wrong.
>
> Thanks!
>
> - Mark
>
>
>

When you are reading and writing DBM-type files (or any other type of
file, for that matter) from more than one process or thread
simultaneously, you need to treat all the operations between the tie and
the untie as atomic from a file locking perspective. See:

    perldoc -f flock
    perldoc -q lock
    perldoc DB_File

So the basic operations are: In a given process, establish a lock, then
tie the DBM-type file, do whatever to it (read, write), untie the
DBM-type file, and remove the lock. For best results, use a separate
empty file (or a file whose contents don't matter) for locking purposes
and only touch the locked DBM-type file when your process has a lock on
the lock file. Be sure to very carefully read all your system's
documentation about file locking -- that topic probably varies more than
any other amongst the various operating systems. Note that some OS's
require the lock file be opened for write to establish an exclusive lock
and that it be opened for read to establish a shared lock. And some
infamous wanna-be OS's don't support file locking at all, in which case
you cannot safely use file I/O in multiple processes simultaneously (in
that case, get an OS). Be sure and test your locking scheme to be
certain it is functional.

Note also that you risk a lot more than just loss of data synchronicity
("stale data") if you don't lock -- you can easily corrupt the file. 

-- 
Bob Walton
Email: http://bwalton.com/cgi-bin/emailbob.pl

Relevant Pages

  • [NT] File Locking and Security (Group Policy DoS on Windows 2000 Domains)
    ... The following security advisory is sent to the securiteam mailing list, and can be found at the SecuriTeam web site: http://www.securiteam.com ... EXCLUSIVE lock on a file. ... file locking is only checked ... Windows, things are different. ...
    (Securiteam)
  • Re: Mandatory File Locking
    ... I need to put a mandatory lock on /dev/fb0 in Linux ... Based on what you posted the file locking should work. ...
    (Ubuntu)
  • Re: UWASH IMAP and flock over iscsi
    ... inetd initiated daemon, instead of an autonomous daemon, and therefore in a modern environment where a user can potentially connect to the IMAP server in several ways the probability of stray inetd initiated IMAP daemons staying around is higher. ... Either it is a bug in the underlying system lock infrastructure or there is a bug in imapd. ...
    (comp.mail.imap)
  • Re: Adding and deleting files in an atomic way
    ... Reader, and your Adder and Merger are both writers. ... I think NIO gives you access to the system file locking mechanisms. ... For example, if someone comes along after you and starts opening and modifying these files in a fourth process (say, a Perl script somewhere) you need to make sure you can lock their script out when you need to. ...
    (comp.lang.java.programmer)
  • Re: Multithreading and syncronization
    ... > I have written a dll that performs file IO that will be called frequently. ... > have been running into problems where file locking is occuring when a thread ... How do I lock this IO ... I suggest you use a semaphore for each resource - there's no semaphore ...
    (microsoft.public.dotnet.languages.csharp)