Re: import in threads: crashes & strange exceptions on dual core machines
- From: robert <no-spam@xxxxxxxxxxxxxxxxxxx>
- Date: Tue, 31 Oct 2006 12:05:08 +0100
Klaas wrote:
It seems clear that the import lock does not include fully-executing
the module contents. To fix this, just import cookielib before the
What is the exact meaning of "not include fully-executing" - regarding the examples "import cookielib" ?
Do you really mean the import statement can return without having executed the cookielib module code fully?
(As said, a simple deadlock is not at all my problem)
threads are spawned. Better yet, use your own locks around the
acquisition of the opener instance (this code seems fraughtfully
thread-unsafe--fix that and you solve other problems besides this one).
thanks. I will probably have to do the costly pre-import of things in main thread and spread locks as I have also no other real idea so far.
Yet this costs the smoothness of app startup and corrupts my believe in Python capabs of "lazy execution on demand". I'd like to get a more fundamental understanding of the real problems than just a general "stay away and lock and lock everything without real understanding".
* I have no real explanation why the import of a module like cookielib is not thread-safe. And in no way I can really explain the real OS-level crashes on dual cores/fast CPU's. Python may throw this and that, Python variable states maybe wrong, but how can it crash on OS-level when no extension libs are (hopefully) responsible?
* The Import Lock should be a very hard lock: As soon as any thread imports something, all other threads are guaranteed to be out of any imports. A dead lock is not the problem here.
* cookielib module execution code consists only of definitions and of re.compile's. re.compile's should be thread safe?
* the things in my code patter are function local code except "opener = urlcookie_openers.get(user)" and "urlcookie_openers[user] = opener" : Simple dictionary accesses which are atomic from all my knowledge and experience. I think, I have thought about enough, what could be not thread safe. The only questionable things have to do with rare change of some globals, but this has not at all to do with the severe problems here and could only affect e.g wrong url2_proxy or double/unecessary re-creation of an opener, which is uncritical in my app.
I'm still puzzled and suspect there is a major problem in Python, maybe in win32ui or - no idea ... ?
-robert
==================================================================================
def f():
...
opener = urlcookie_openers.get(user)
if not opener:
import cookielib #<----1
cj=cookielib.CookieJar() #<----2 build_opener = urllib2.build_opener
httpCookieProcessor = urllib2.HTTPCookieProcessor(cj)
if url2_proxy:
opener = build_opener(url2_proxy,httpCookieProcessor)
else:
opener = build_opener(httpCookieProcessor)
opener.addheaders #$pycheck_no
opener.addheaders= app_addheaders
urlcookie_openers[user] = opener
ufile = opener.open(urllib2.Request(url,data,dict(headers)))
...
thread.start_new(f,()) .
- References:
- Prev by Date: Re: create global variables?
- Next by Date: Re: Where do nested functions live?
- Previous by thread: Re: import in threads: crashes & strange exceptions on dual core machines
- Next by thread: How can I import a script with an arbitrary name ?
- Index(es):
Relevant Pages
|
