Re: Problem with awaitTermination in ThreadpoolExecutor.



On 8/23/2010 2:25 PM, TomInDenver wrote:
On Aug 23, 11:52 am, Daniel Pitts
<newsgroup.spamfil...@xxxxxxxxxxxxxxxxxxx> wrote:
On 8/23/2010 7:20 AM, TomInDenver wrote:



Hi,

The javadoc for awaitTermination in ExecutorService and
ThreadPoolExecutor includes the following:

Description
Blocks until all tasks have completed execution after a shutdown
request, or the timeout occurs, or the current thread is interrupted,
whichever happens first.

Returns:
true if this executor terminated and false if the timeout elapsed
before termination

We have occasionally noticed that awaitTermination returns true when
tasks submitted to the executor are still running, a timeout has not
occurred, and the submitting thread was not interrupted. This has been
an infrequent occurrence, but when it happens it severely impacts our
application. Our log clearly shows the condition (log messages from
Runnables exist after the awaitTermination returned true), and the
application behavior reflects the result of this condition (failures
due to threads still running when it is expected that the threads
have completed).

Below is the relevant code. (In this instance the tasks are
downloading files from an FTP site using a 3rd party FTP library, one
file per thread.)

Can anyone point out anything in this code that might cause the
problem, or suggest how we might refactor the code so the chances of
the problem occurring are reduced, or let us know if you recall a
bugfix for a problem like this ? We are using java build 1.6.0_11-
b03.

// Create thread pool
ExecutorService downloadThreadPool = new ThreadPoolExecutor(
3, // corePoolSz
5, // maxPoolSz,
7, // keepalive (7 days)
TimeUnit.DAYS,
new LinkedBlockingQueue<Runnable>(),
new ThreadFactory() {
public Thread newThread(Runnable r) {
Thread t = new Thread(r);
t.setDaemon(false);
t.setName("XF-Download-Thread-Pool");
return t;
}
});

//The application creates many Runnables and then executes the
following line in a loop for each:
// (code to create Runnables not shown here)
downloadThreadPool.execute(aRunnable);

// Make threadpool wait up to 7 days for Runnables to end, after
which a threadpool timeout will occur.
downloadThreadPool.shutdown();
try {
if (!downloadThreadPool.awaitTermination(7, TimeUnit.DAYS)) {
Log.log(SPLogger.LogLevel.WARNING, "Threadpool timeout occurred",
SPLogger.LogPhase.UNKNOWN);
}
} catch (InterruptedException ie) {
Log.log(SPLogger.LogLevel.WARNING, "Threadpool prematurely
terminated due to interruption in thread that created pool",
SPLogger.LogPhase.UNKNOWN);
}

Thank you,

Tom Vicker

The problem is in the code you didn't post.

Please create an SSCCE and post it here.
<http://sscce.org/>

I suspicion is that perhaps it *is* terminated., but you're log might be
buffered and the buffer isn't flushed in the order you expect.

--
Daniel Pitts' Tech Blog:<http://virtualinfinity.net/wordpress/>

Daniel,

Thanks for response. It seems you're interested in seeing the code of
the thread class. There is quite a bit of code I would need to post
and I am not sure it would be productive. The run() method code is
within a try/catch and the catch is for "Exception", which is handled
without rethrowing.

I am curious what code could be in the Runnable that would cause the
ThreadPoolExecutor to think the thread has terminated when it actually
has not terminated. Do you know of anything that would cause this
condition ? Some of the code in the Runnable are calls to objects in a
3rd party lib, so I cannot see what that code is doing. If I knew what
could cause the problem condition, I can check our code and also
request the 3rd party vendor to check their code. So please, if you
know what would cause it, please respond.

The buffer flush scenario you described isn't happening because we see
evidence of the thread running before the timeout expiration and well
after the awaitTermination unblocked. Furthermore, the behavior of the
application is such that if the threads did not terminate, subsequent
processing would fail, which is exactly what happens.

Tom Vicker
Is it possible then that your runnables are spinner off yet more threads, and *those* threads are the ones you observe after awaitTermination completes?

I'm not asking you to provide your entire code-base. And SSCCE is specifically the smallest running program which exhibits your problem. It is definitely worth the exercise of creating a test harness and attempting to recreate the errant conditions. You may find that you needn't post *any* code here, as you could discover the problem simply by trying to recreate it.

--
Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>
.



Relevant Pages

  • Re: Problem with awaitTermination in ThreadpoolExecutor.
    ... request, or the timeout occurs, or the current thread is interrupted, ... before termination ... tasks submitted to the executor are still running, ... request the 3rd party vendor to check their code. ...
    (comp.lang.java.programmer)
  • Re: Problem with awaitTermination in ThreadpoolExecutor.
    ... request, or the timeout occurs, or the current thread is interrupted, ... tasks submitted to the executor are still running, ... Runnables exist after the awaitTermination returned true), ... confused where the termination of a distant Runnable that was spun off ...
    (comp.lang.java.programmer)
  • Re: Problem with awaitTermination in ThreadpoolExecutor.
    ... request, or the timeout occurs, or the current thread is interrupted, ... tasks submitted to the executor are still running, ... Runnables exist after the awaitTermination returned true), ... confused where the termination of a distant Runnable that was spun off ...
    (comp.lang.java.programmer)
  • Re: Problem with awaitTermination in ThreadpoolExecutor.
    ... request, or the timeout occurs, or the current thread is interrupted, ... tasks submitted to the executor are still running, ... Runnables exist after the awaitTermination returned true), ... confused where the termination of a distant Runnable that was spun off ...
    (comp.lang.java.programmer)
  • monitoring a process with proc_get_status()
    ... has terminated within the timeout, then my function has to return the ... process exit code and this is always -1. ... So my question is - how do I monitor if the process has terminated ... *AND* obtain its exit code after the termination? ...
    (php.general)