Re: "RuntimeError: dictionary changed size during iteration" ; Good atomic copy operations?



EleSSaR^ wrote:
robert si è profuso/a a scrivere su comp.lang.python tutte queste
elucubrazioni:


own deepcopy: thus, do you already know if the existing deepcopy has the same problem as cPickle.dump ? (as the problem araises rarely, it is difficult for me to test it out)

I don't know the exact specs of your object, and I don't know what
operations are you performing on that object, nor the way they're atomic.

There is not much to know. Python object trees consist only of dicts and lists as far as variable non-atomic datastructures are concerned. (unless you use advanced extension libs like NumPy)

Thus the RuntimeError problem is only about modified dicts/lists during Iteration in pickly/copy.


It seems like you're trying to save periodically the state of such object
while it is being modified (a sort of backup?), and Python complains about
that. A self-implemented deepcopy might raise anomalies (i.e. your dumped
object may be partly a 'before' object and partly an 'after' object ) as
well.

Yes, a "backup" / autosave while all threads are running. It doesn't matter if 'before' of 'after' another item has been added/deleted atomically.


By the way, you could try employing locks from other threads to dump the
object as well... this would prevent additional locking.

Don't understand.
The threads work all simulatniously on the object tree, add and detach atomically only valid sub-trees.

Regarding what AM said, I would have to lock _each_ dict/list operation on the tree, thus almost each change, because even a single attribute change "subobj.x='y'" is a dictionary operation. That would make threaded programming very arduous.

AFAIK about the current Python implementation: This RuntimeError is only thrown "during true Iteration over dict/list, when the number of items changes". (and not when e.g. a single item is changed). Thus a

def rt_save_dict_copy()
tod={}
for k in fromd.keys():
try: tod[k]=fromd[k]
except: pass
return tod

without true iteration over the original dict whould copy without RuntimeError.

(or maybe equivalent: "dict(fromd.items())" ? )

I don't know if dict.copy() works so - but I think so, as dict.keys() and dict.items() have the same footprint.

The algorithm in cPickle.dump does not work so. Guess it does something like "for k in fromd: ..."(!) internally. This might be a "90%-bug"?

Will have to see what copy/deepcopy does ...

Robert

.