Re: Can someone explain this weakref behavior?

From: David MacQuigg (dmq_at_gain.com)
Date: 06/11/04 

Date: Fri, 11 Jun 2004 14:15:50 -0700

On Fri, 11 Jun 2004 15:38:29 -0400, "Tim Peters" <tim.one@comcast.net>
wrote:

>[Tim Peters]
>>> That will pass under CPython today, but there's no general guarantee
>>> about exactly when a weak dict will notice that keys (or values) have
>>> become unreachable by strong references.
>
>[David MacQuigg]
>> OUCH!! We just built a module that uses weak references to keep a
>> "robust count" of instances in various classes. The claim is that this
>> is more robust than simply incrementing and decrementing class variables
>> using __init__ and __del__. The module seems to be working OK,
>> immediately deleting the weak reference as soon as all references to the
>> corresponding instance are deleted.
>
>Then it must be the case that you're running CPython, and that these
>instances aren't involved in cycles. Because CPython primarily uses
>reference-counting to recycle garbage, its behavior is predictable in the
>absence of cycles.

I'm not worried about cyclic references, but it is something to keep
in mind.

>CPython's use of reference counting is an implementation detail, and that
>internal weakref lists are traversed "immediately" upon an object's refcount
>reaching 0 is also an implementation detail. Nothing in the language
>definition guarantees these behaviors.
>
>> If I understand you correctly, there is some chance that a future
>> implementation of Python may have the weak references "out-of-sync" with
>> the actual count of live instances. Is that a remote possibility, or
>> something quite likely to occur?
>
>Well, it's been the case for a long time in JPython. I judge the odds of it
>changing in CPython as slim. I personally wouldn't worry about it ever
>changing in CPython. If a PyPy- or Parrot-based implementation of Python
>takes off, behavior will depend on its implementation details.
>
>> I have to decide now whether to rip out some risky code.
>>
>> Is there a good way to track the count of instances?
>
>If you want it enough, you can build Python in a mode that tracks this
>automatically (see the discussion of COUNT_ALLOCS in Misc/SpecialBuilds.txt
>-- for each distinct type object, the total # of allocations, # of
>deallocations, and highwater mark (max(#alloc - #dealloc) over time) are
>maintained in a COUNT_ALLOCS build).
>
>> If not, would it make sense to request a guarantee on the current
>> behavior of weak references? Maybe it could be an option, assuming there
>> is some performance penalty, an option to be used when accuracy is more
>> important than speed.
>
>You can request anything you can dream up <wink>. If it's something your
>business needs, the only way to guarantee it is to get involved in Python
>development deeply enough so that, if worse comes to worse, you can maintain
>your own Python implementation. That's unreasonably paranoid in my
>estimation, but it's a judgment call.

Seems like we could do this more easily with a function that lists
instances, like __subclasses__() does with subclasses. This doesn't
have to be efficient, just reliable. So when I call
cls.__instances__(), I get a current list of all instances in the
class.

Maybe we could implement this function using weak references. If I
understand the problem with weak references, we could have a
WeakValueDictionary with references to objects that actually have a
refcount of zero. There may be too many entries in the dictionary,
but never too few. In that case, maybe I could just loop over every
item in my WeakValueDictionary, and ignore any with a refcount of
zero.

    def _getInstances(cls):
        d1 = cls.__dict__.get('_instances' , {})
        d2 = {}
        for key in d1:
            if sys.getrefcount(d1[key]) > 0:
                d2[key] = d1[key]
        return d2
    _getInstances = staticmethod(_getInstances)

I'm making some assumptions here that may not be valid, like
sys.getrefcount() for a particular object really will be zero
immediately after all normal references to it are gone. i.e. we don't
have any temporary "out-of-sync" problems like with the weak
references themselves.

Does this seem like a safe strategy?

-- Dave

Relevant Pages

  • A re-announce on GCs defects
    ... The need for weak reference makes the destruction delay logically incorrect. ... Weak references refer to references who do ... When all strong references to a target go out of their lifetime, ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: nss_ldap broken
    ... I had no idea that libpthread would be loaded then unloaded. ... I'm unsure how nss_ldap was built to depend on libpthread (or ... weak references (not definitions, like you see in our libc ... By making the pthread_* references weak, you don't need to be linked ...
    (freebsd-current)
  • Re: INTERN hygiene in Web applications
    ... Interned chunks of rubbish will eventually be garbage collected when the only reference to the rubbish is within, say, a weak hash table of symbols. ... ANSI Common Lisp does not provide weak references nor weak hash tables. ...
    (comp.lang.lisp)
  • Re: Resolve object pointer any use for this?
    ... So, all in all a very interesting technique, but unfortunately, no good ... By the way, weak references in .NET are not needed nearly as much, because ... ObjPtr for the object as the key. ...
    (microsoft.public.excel.programming)
  • Re: i am looking for a couple ALMOST impossible math problems-Alan B Fabian
    ... and came up with the four references below. ... dense set, but I thought it might be of interest anyway. ... HISTORICAL ESSAY ON CONTINUITY OF DERIVATIVES ... Frank S. Cater, "A derivative often zero and discontinuous", ...
    (sci.math)
Loading