Re: flock seems very unsafe, python fcntl bug?



Jens Henrik Leonhard Jensen <jhlj@xxxxxxxxxxxxxxxxxxx> wrote:
Your problem is that open(...,'w') is not locked.

Use something like:
lockf = open('aaa', 'a')
fnctl.flock(lockf,fnctl.LOCK_EX)
file = open('aaa', 'w')
file.write('asdf')
file.close()
lockf.close()

I've not seen that trick before - it is a good one. It wouldn't work
for mandatory locking though...

The basic problem is getting the file open for read/write without
destroying the contents before you have the lock. None of the
arguments to open() do the right thing....

Using an append open is a nice solution to that, apart from the fact
that you can only append data onto the file hence the other open.

Here are some other possible solutions and a test harness! The low
level open is probably the most unixy solution. If you could specify
os.O_RDWR|os.O_CREAT to the python open() function then it would be
the best, but none of the documented open strings do that

$ strace python -c 'open("aaa", "r+")' 2>&1 | grep 'open("aaa"'
open("aaa", O_RDWR|O_LARGEFILE) = 3
$ strace python -c 'open("aaa", "w+")' 2>&1 | grep 'open("aaa"'
open("aaa", O_RDWR|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
$ strace python -c 'open("aaa", "wr+")' 2>&1 | grep 'open("aaa"'
open("aaa", O_RDWR|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
$ strace python -c 'open("aaa", "a+")' 2>&1 | grep 'open("aaa"'
open("aaa", O_RDWR|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 3


------------------------------------------------------------
"""
Testing writing stuff to a file with locking
"""

import os
import sys
import fcntl

def method_0(data):
"""
Use an normal open

This doesn't work because it truncates the data before it has the
lock
"""
fd = open('aaa', 'w')
fcntl.flock(fd,fcntl.LOCK_EX)
fd.write(data)
fd.close()

def method_1(data):
"""
Use an additional append open to lock
"""
lock_fd = open('aaa', 'a')
fcntl.flock(lock_fd,fcntl.LOCK_EX)
fd = open('aaa', 'w')
fd.write(data)
fd.close()
lock_fd.close()

def method_2(data):
"""
Use a low level open
"""
fd = os.fdopen(os.open('aaa', os.O_CREAT|os.O_RDWR), "r+")
fcntl.flock(fd, fcntl.LOCK_EX)
fd.truncate()
fd.write(data)
fd.close()

def method_3(data):
"""
Using high level functions only

Possibly racy on open when file doesn't exist
"""
if not os.path.exists('aaa'):
fd = open('aaa', 'w+')
else:
fd = open('aaa', 'r+')
fcntl.flock(fd,fcntl.LOCK_EX)
fd.truncate()
fd.write(data)
fd.close()

def check():
fd = open('aaa', 'r')
fcntl.flock(fd,fcntl.LOCK_EX)
data = fd.read()
fd.close()
if data not in ("sausage", "potato"):
raise AssertionError("Wrong data %r" % data)

if os.path.exists("aaa"):
os.unlink("aaa")

if len(sys.argv) < 2:
print "Syntax: %s <method number 0..3>" % sys.argv[0]
raise SystemExit(1)
method = globals()["method_"+sys.argv[1]]
if os.fork():
while 1:
method("sausage")
check()
else:
while 1:
method("potato")
check()
------------------------------------------------------------


--
Nick Craig-Wood <nick@xxxxxxxxxxxxxx> -- http://www.craig-wood.com/nick
.



Relevant Pages

  • Re: is there enough information?
    ... waiting for the prior index to complete before starting the next. ... from threading import Lock ... def iexit(self, index, *a): ... Sequence numbers are looked up in a hash; ...
    (comp.lang.python)
  • Re: Sharing objects between processes
    ... one updating thread takes user input, ... I can use a single lock over the entire object; ... a write is something I also want to avoid. ... def write_random: ...
    (comp.lang.python)
  • Re: Trying to wrap my head around futures and coroutines
    ... Each corresponds to executing a particular activity, A or B, which take ... def do_A: ... from threading import Lock ... Regarding cancellation, I presume your code polls some cancellation ...
    (comp.lang.python)
  • Re: yield, curry, mix-in, new.function, global, closure, .... what will work?
    ... Should I lock global variables i, j during the execution of run? ... So I don't even mind to copy changer() every ... if spam is meal: ... def change_i: ...
    (comp.lang.python)
  • Re: Sharing objects between processes
    ... def write_random: ... I've attempted to modify it to use multiprocessing without success. ... This works when the threading import is used, ... #from multiprocessing import Process as ControlType, Lock ...
    (comp.lang.python)