Re: Information About A BlockingQueue

On 26.04.2007 14:57, Jason Cavett wrote:
On Apr 25, 9:57 am, Robert Klemme <shortcut...@xxxxxxxxxxxxxx> wrote:
On 25.04.2007 15:47, Jason Cavett wrote:

I am currently in the process of using a BlockingQueue
(LinkedBlockingQueue, to be exact) to help with a producer/consumer
aspect of an application I'm developing. The user can continuously
add items to the queue for processing (and this does work). However,
I'm a little confused about how a BlockingQueue works.
When I add an item to the BlockingQueue, is the items copied into the
queue, or is it just a reference to the actual object?
The blocking queue holds on to objects in the queue like any other
collection does.

The reason I ask is because I notice that after adding 9 to 11 items
into the queue, the application slows considerably (despite the fact
that the "queue" stuff is running in a separate thread).
What is an "item"?

After
watching the output from the garbage collector, it appears as though I
have a memory leak. I attempted to use hprof to profile my
application to see where the memory leak is taking place, but I'm not
hprof is unlikely to help you with detecting memory leaks other than
finding excessive creation of instances. But to get a leak you must
also be holding on to those instances longer than intended which is
something you do not get from hprof IIRC. You might want to look at one
of the various profiler / memory analyzer tools around (JProbe,
OptimizeIt, Eclipse TPTP).

fully understanding the output of hprof. When my application lags
like this, it somewaht defeats the purpose of the use of the
BlockingQueue since I want users to be able to queue up files to
process.
Can anybody clear up any possible issues or provide some insight with
the way BlockingQueue works and any possible memory issues I should be
keeping an eye out for (there isn't much information that I could find
from my own Google searches).
Here are some potential reasons for the slowdown:

- your objects are large and you are tight on JVM memory so GC has to
kick in more often or your JVM starts swapping to disk

- your processing is the slow bit (too much concurrency for example,
or it uses too much mem per processing instance)

There are probably more but without more detail we can't really tell.

Thanks for the tips. I checked into the "objects are large...so GC
hast to kick in more often" Turns out this is the problem. I forgot
I had done it, but I was making deep copies of objects to put onto the
BlockingQueue so the user could continue to edit the file but the file
(at that point in time) could also be processed.

Of course, creating deep copies quickly fills up memory, so I quickly
ran out of memory. I can easily fix this by putting the (reference
to) the object on the queue directly instead of a copy of the object,
but, unfortunately that raises the problem where users can continue
editing objects and what they originally put in the queue will not be
what's processed if they have continued to make changes to the file
(the processing is done in a separate thread).

Any suggestions on how to handle this problem?

Increase heap size. Or, since you are using a blocking queue anyway, use a lower max size for the queue. Deal with your "files" (whatever that means) more efficiently.

Anyway, even if there isn't (I've come up with a temporary solution
for now), thank you for your helpful suggestions on the memory problem.

I'm glad you figured out.

Kind regards

robert
.

Relevant Pages

  • Re: Information About A BlockingQueue
    ... add items to the queue for processing. ... I'm a little confused about how a BlockingQueue works. ... application to see where the memory leak is taking place, ... hprof is unlikely to help you with detecting memory leaks other than finding excessive creation of instances. ...
    (comp.lang.java.programmer)
  • Re: Cpp Considered Harmful
    ... was received would push it on the queue. ... It's main advantages are the multiple use ... research and experience into concurrency and system design in order to ... Threads share memory, the problem is they share all memory (well not ...
    (comp.lang.cpp)
  • Re: Wait free queue
    ... dealloc, lock-free remote memory dealloc among producers, wait-free ... memory allocator blows the msvcrt away--why bother with it, ... Writing a Generalized Concurrent Queue ... struct queue_node* next; ...
    (comp.programming.threads)
  • Re: P2P Events - growing memory
    ... The recommended practice is that the event callback keeps calling ... PeerGroupGetData until it get the PEER_S_NO_EVENT_DATA error code, ... signifying that you've handled all events in the queue. ... > is where all the memory allocations for all the structures are done. ...
    (microsoft.public.win32.programmer.networks)
  • Re: Best way to design multithreading application
    ... The data acquisition is more important than UI updates, ... adding it to a Queue 1 as it acquires more. ... B, which is medium-to-high priority, loops over and over, and when it ... (memory in this case, though having the UI lag behind the actual ...
    (microsoft.public.dotnet.framework)