Re: creating garbage collectable objects (caching objects)
- From: Simon Forman <sajmikins@xxxxxxxxx>
- Date: Sun, 28 Jun 2009 11:19:26 -0700 (PDT)
On Jun 28, 11:03 am, News123 <news...@xxxxxxx> wrote:
Hi.
I started playing with PIL.
I'm performing operations on multiple images and would like compromise
between speed and memory requirement.
The fast approach would load all images upfront and create then multiple
result files. The problem is, that I do not have enough memory to load
all files.
The slow approach is to load each potential source file only when it is
needed and to release it immediately after (leaving it up to the gc to
free memory when needed)
The question, that I have is whether there is any way to tell python,
that certain objects could be garbage collected if needed and ask python
at a later time whether the object has been collected so far (image has
to be reloaded) or not (image would not have to be reloaded)
# Fastest approach:
imgs = {}
for fname in all_image_files:
imgs[fname] = Image.open(fname)
for creation_rule in all_creation_rules():
img = Image.new(...)
for img_file in creation_rule.input_files():
img = do_somethingwith(img,imgs[img_file])
img.save()
# Slowest approach:
for creation_rule in all_creation_rules():
img = Image.new(...)
for img_file in creation_rule.input_files():
src_img = Image.open(img_file)
img = do_somethingwith(img,src_img)
img.save()
# What I'd like to do is something like:
imgs = GarbageCollectable_dict()
for creation_rule in all_creation_rules():
img = Image.new(...)
for img_file in creation_rule.input_files():
if src_img in imgs: # if 'm lucke the object is still there
src_img = imgs[img_file]
else:
src_img = Image.open(img_file)
img = do_somethingwith(img,src_img)
img.save()
Is this possible?
Thaks in advance for an answer or any other ideas of
how I could do smart caching without hogging all the system's
memory
Maybe I'm just being thick today, but why would the "slow" approach be
slow? The same amount of I/O and processing would be done either way,
no?
Have you timed both methods?
That said, take a look at the weakref module Terry Reedy already
mentioned, and maybe the gc (garbage collector) module too (although
that might just lead to wasting a lot of time fiddling with stuff that
the gc is supposed to handle transparently for you in the first place.)
.
- References:
- creating garbage collectable objects (caching objects)
- From: News123
- creating garbage collectable objects (caching objects)
- Prev by Date: Re: creating garbage collectable objects (caching objects)
- Next by Date: Re: Running a script from a windows right click menu..?
- Previous by thread: Re: creating garbage collectable objects (caching objects)
- Next by thread: Re: creating garbage collectable objects (caching objects)
- Index(es):
Relevant Pages
|
