Re: JDBC Threads Locking up

From: "joeNOSPAM@xxxxxxx" <joe.weinstein@xxxxxxxxx>
Date: Fri, 15 Jun 2007 23:20:16 -0000

On Jun 15, 11:02 am, samee...@xxxxxxxxx wrote:

Hi,

We are using Apache,Tomcat 5,ojdbc14.jar,Oracle 10g based system and
observing that all 100 tomcat threads are getting stuck in following
situation.

87 threads are stuck in "waiting to lock <0x718fc270> (a
oracle.jdbc.pool.OracleConnectionEventListener)"

TP-Processor100 daemon prio=1 tid=0x5c430080 nid=0x7964 waiting for
monitor entry [0x546c2000..0x546c3868]
at
oracle.jdbc.pool.OracleConnectionEventListener.connectionClosed(OracleConnectionEventListener.java:
129)
- waiting to lock <0x718fc270> (a
oracle.jdbc.pool.OracleConnectionEventListener)
at
oracle.jdbc.pool.OraclePooledConnection.callListener(OraclePooledConnection.java:
482)
at
oracle.jdbc.pool.OraclePooledConnection.logicalClose(OraclePooledConnection.java:
445)
- locked <0x6254f030> (a oracle.jdbc.pool.OraclePooledConnection)
at
oracle.jdbc.driver.OracleConnection.logicalClose(OracleConnection.java:
2936)
- locked <0x6254f0f0> (a oracle.jdbc.driver.OracleConnection)
at oracle.jdbc.driver.OracleConnection.close(OracleConnection.java:
1454)
- locked <0x6254f0f0> (a oracle.jdbc.driver.OracleConnection)

One thread has locked "<0x718fc270> (a
oracle.jdbc.pool.OracleConnectionEventListener)" and it is waiting to
lock <0x718fc0e0> (a oracle.jdbc.pool.OracleConnectionCacheImpl).

TP-Processor45 daemon prio=1 tid=0x081cf9b8 nid=0x7d0 waiting for
monitor entry [0x562fa000..0x562fb868]
at
oracle.jdbc.pool.OracleConnectionCacheImpl.closeSingleConnection(OracleConnectionCacheImpl.java:
677)
- waiting to lock <0x718fc0e0> (a
oracle.jdbc.pool.OracleConnectionCacheImpl)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.reusePooledConnection(OracleConnectionCacheImpl.java:
592)
at
oracle.jdbc.pool.OracleConnectionEventListener.connectionClosed(OracleConnectionEventListener.java:
130)
- locked <0x718fc270> (a
oracle.jdbc.pool.OracleConnectionEventListener)
at
oracle.jdbc.pool.OraclePooledConnection.callListener(OraclePooledConnection.java:
482)
at
oracle.jdbc.pool.OraclePooledConnection.logicalClose(OraclePooledConnection.java:
445)
- locked <0x62354878> (a oracle.jdbc.pool.OraclePooledConnection)
at
oracle.jdbc.driver.OracleConnection.logicalClose(OracleConnection.java:
2936)
- locked <0x62354938> (a oracle.jdbc.driver.OracleConnection)
at oracle.jdbc.driver.OracleConnection.close(OracleConnection.java:
1454)
- locked <0x62354938> (a oracle.jdbc.driver.OracleConnection)

One thread has locked <0x718fc0e0>, but it is waiting in
socketRead0(), so the lock can not released.
This stack trace is originating from while obtaining a new database
connection.

TP-Processor10 daemon prio=1 tid=0x084d4f50 nid=0x70d5 runnable
[0x5cd7c000..0x5cd7e868]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Unknown Source)
at oracle.net.ns.DataPacket.receive(Unknown Source)
at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
t oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:931)
at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:893)
at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:375)
at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1894)
at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:
1094)
- locked <0x62bb7a10> (a oracle.jdbc.ttc7.TTC7Protocol)
at
oracle.jdbc.driver.OracleStatement.executeNonQuery(OracleStatement.java:
2132)
at
oracle.jdbc.driver.OracleStatement.doExecuteOther(OracleStatement.java:
2015)
at
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:
2877)
at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:
925)
- locked <0x62bc4228> (a oracle.jdbc.driver.OracleStatement)
- locked <0x62bb7a80> (a oracle.jdbc.driver.OracleConnection)
at oracle.jdbc.dbaccess.DBAccess.setNlsParamsClient(DBAccess.java:
1680)
at oracle.jdbc.ttc7.TTC7Protocol.initNls(TTC7Protocol.java:1592)
- locked <0x62bb7a10> (a oracle.jdbc.ttc7.TTC7Protocol)
at oracle.jdbc.driver.OracleConnection.<init>(OracleConnection.java:
406)
at
oracle.jdbc.driver.OracleDriver.getConnectionInstance(OracleDriver.java:
521)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:325)
at java.sql.DriverManager.getConnection(DriverManager.java:512)
- locked <0xa1efd890> (a java.lang.Class)
at java.sql.DriverManager.getConnection(DriverManager.java:140)
- locked <0xa1efd890> (a java.lang.Class)
at
oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:
171)
- locked <0x718fc1e0> (a
oracle.jdbc.pool.OracleConnectionPoolDataSource)
at
oracle.jdbc.pool.OracleConnectionPoolDataSource.getPhysicalConnection(OracleConnectionPoolDataSource.java:
149)
at
oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:
95)
- locked <0x718fc1e0> (a
oracle.jdbc.pool.OracleConnectionPoolDataSource)
at
oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:
63)
- locked <0x718fc1e0> (a
oracle.jdbc.pool.OracleConnectionPoolDataSource)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.getNewPoolOrXAConnection(OracleConnectionCacheImpl.java:
547)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.getPooledConnection(OracleConnectionCacheImpl.java:
404)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.getConnection(OracleConnectionCacheImpl.java:
298)
- locked <0x718fc0e0> (a oracle.jdbc.pool.OracleConnectionCacheImpl)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.getConnection(OracleConnectionCacheImpl.java:
268)
- locked <0x718fc0e0> (a oracle.jdbc.pool.OracleConnectionCacheImpl)

Did somebody ran into a similar situation? Please Help.

Thanks,
Sameer

Show how your pool is defined. This points to a serious lack of
concurrency
consideration in the Oracle driver. One terrible thing we can see in
the
driver is that internally it is calling DriverManager.getConnection()!
This
is a huge gaff on the part of the driver because all DriverManager
calls
are class-synchronized, and many other of these calls are made by
drivers
and even SQLException constructors, so one long-running DriverManager
call
can block all other JDBC in the entire JVM.
I suggest you stop using the Oracle driver-based pooling if Tomcat
offers
it's own pooling.
Joe Weinstein at BEA Systems

.

Follow-Ups:
- Re: JDBC Threads Locking up
  - From: sameergn
- Re: JDBC Threads Locking up
  - From: Tom Hawtin

References:
- JDBC Threads Locking up
  - From: sameergn

Prev by Date: JDBC Threads Locking up
Next by Date: Re: JDBC Threads Locking up
Previous by thread: JDBC Threads Locking up
Next by thread: Re: JDBC Threads Locking up
Index(es):
- Date
- Thread

Relevant Pages

Re: JDBC Threads Locking up
... oracle.jdbc.pool.OracleConnectionEventListener)" and it is waiting to ... lock. ... driver is that internally it is calling DriverManager.getConnection! ... That's not the whole pool definition. ...
(comp.lang.java.databases)
Re: JDBC Threads Locking up
... monitor entry ... oracle.jdbc.pool.OracleConnectionEventListener)" and it is waiting to ... lock. ... driver is that internally it is calling DriverManager.getConnection! ...
(comp.lang.java.databases)
Re: JDBC Threads Locking up
... observing that all 100 tomcat threads are getting stuck in following ... monitor entry ... lock. ... driver is that internally it is calling DriverManager.getConnection! ...
(comp.lang.java.databases)
Re: OS Question
... driver, should not distress you. ... I've made the unwarranted assumption that having multiple threads is ... To protect data in user mode ... Permanence (if apps all crash the lock state is retained because ...
(microsoft.public.development.device.drivers)
Re: LOR with netisr changes
... ::> It's just a bug in the driver. ... the softc and is intended to be covered by the driver's softc lock. ... this is a hack to work around a recursion in net80211. ...
(freebsd-current)