Re: JDBC Threads Locking up

On Jun 19, 10:59 am, samee...@xxxxxxxxx wrote:
On Jun 15, 4:20 pm, "joeNOS...@xxxxxxx" <joe.weinst...@xxxxxxxxx>
wrote:



On Jun 15, 11:02 am, samee...@xxxxxxxxx wrote:

Hi,

We are using Apache,Tomcat 5,ojdbc14.jar,Oracle 10g based system and
observing that all 100 tomcat threads are getting stuck in following
situation.

87 threads are stuck in "waiting to lock <0x718fc270> (a
oracle.jdbc.pool.OracleConnectionEventListener)"

TP-Processor100 daemon prio=1 tid=0x5c430080 nid=0x7964 waiting for
monitor entry [0x546c2000..0x546c3868]
at
oracle.jdbc.pool.OracleConnectionEventListener.connectionClosed(OracleConnectionEventListener.java:
129)
- waiting to lock <0x718fc270> (a
oracle.jdbc.pool.OracleConnectionEventListener)
at
oracle.jdbc.pool.OraclePooledConnection.callListener(OraclePooledConnection.java:
482)
at
oracle.jdbc.pool.OraclePooledConnection.logicalClose(OraclePooledConnection.java:
445)
- locked <0x6254f030> (a oracle.jdbc.pool.OraclePooledConnection)
at
oracle.jdbc.driver.OracleConnection.logicalClose(OracleConnection.java:
2936)
- locked <0x6254f0f0> (a oracle.jdbc.driver.OracleConnection)
at oracle.jdbc.driver.OracleConnection.close(OracleConnection.java:
1454)
- locked <0x6254f0f0> (a oracle.jdbc.driver.OracleConnection)

One thread has locked "<0x718fc270> (a
oracle.jdbc.pool.OracleConnectionEventListener)" and it is waiting to
lock <0x718fc0e0> (a oracle.jdbc.pool.OracleConnectionCacheImpl).

TP-Processor45 daemon prio=1 tid=0x081cf9b8 nid=0x7d0 waiting for
monitor entry [0x562fa000..0x562fb868]
at
oracle.jdbc.pool.OracleConnectionCacheImpl.closeSingleConnection(OracleConnectionCacheImpl.java:
677)
- waiting to lock <0x718fc0e0> (a
oracle.jdbc.pool.OracleConnectionCacheImpl)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.reusePooledConnection(OracleConnectionCacheImpl.java:
592)
at
oracle.jdbc.pool.OracleConnectionEventListener.connectionClosed(OracleConnectionEventListener.java:
130)
- locked <0x718fc270> (a
oracle.jdbc.pool.OracleConnectionEventListener)
at
oracle.jdbc.pool.OraclePooledConnection.callListener(OraclePooledConnection.java:
482)
at
oracle.jdbc.pool.OraclePooledConnection.logicalClose(OraclePooledConnection.java:
445)
- locked <0x62354878> (a oracle.jdbc.pool.OraclePooledConnection)
at
oracle.jdbc.driver.OracleConnection.logicalClose(OracleConnection.java:
2936)
- locked <0x62354938> (a oracle.jdbc.driver.OracleConnection)
at oracle.jdbc.driver.OracleConnection.close(OracleConnection.java:
1454)
- locked <0x62354938> (a oracle.jdbc.driver.OracleConnection)

One thread has locked <0x718fc0e0>, but it is waiting in
socketRead0(), so the lock can not released.
This stack trace is originating from while obtaining a new database
connection.

TP-Processor10 daemon prio=1 tid=0x084d4f50 nid=0x70d5 runnable
[0x5cd7c000..0x5cd7e868]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Unknown Source)
at oracle.net.ns.DataPacket.receive(Unknown Source)
at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
t oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:931)
at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:893)
at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:375)
at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1894)
at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:
1094)
- locked <0x62bb7a10> (a oracle.jdbc.ttc7.TTC7Protocol)
at
oracle.jdbc.driver.OracleStatement.executeNonQuery(OracleStatement.java:
2132)
at
oracle.jdbc.driver.OracleStatement.doExecuteOther(OracleStatement.java:
2015)
at
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:
2877)
at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:
925)
- locked <0x62bc4228> (a oracle.jdbc.driver.OracleStatement)
- locked <0x62bb7a80> (a oracle.jdbc.driver.OracleConnection)
at oracle.jdbc.dbaccess.DBAccess.setNlsParamsClient(DBAccess.java:
1680)
at oracle.jdbc.ttc7.TTC7Protocol.initNls(TTC7Protocol.java:1592)
- locked <0x62bb7a10> (a oracle.jdbc.ttc7.TTC7Protocol)
at oracle.jdbc.driver.OracleConnection.<init>(OracleConnection.java:
406)
at
oracle.jdbc.driver.OracleDriver.getConnectionInstance(OracleDriver.java:
521)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:325)
at java.sql.DriverManager.getConnection(DriverManager.java:512)
- locked <0xa1efd890> (a java.lang.Class)
at java.sql.DriverManager.getConnection(DriverManager.java:140)
- locked <0xa1efd890> (a java.lang.Class)
at
oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:
171)
- locked <0x718fc1e0> (a
oracle.jdbc.pool.OracleConnectionPoolDataSource)
at
oracle.jdbc.pool.OracleConnectionPoolDataSource.getPhysicalConnection(OracleConnectionPoolDataSource.java:
149)
at
oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:
95)
- locked <0x718fc1e0> (a
oracle.jdbc.pool.OracleConnectionPoolDataSource)
at
oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:
63)
- locked <0x718fc1e0> (a
oracle.jdbc.pool.OracleConnectionPoolDataSource)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.getNewPoolOrXAConnection(OracleConnectionCacheImpl.java:
547)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.getPooledConnection(OracleConnectionCacheImpl.java:
404)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.getConnection(OracleConnectionCacheImpl.java:
298)
- locked <0x718fc0e0> (a oracle.jdbc.pool.OracleConnectionCacheImpl)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.getConnection(OracleConnectionCacheImpl.java:
268)
- locked <0x718fc0e0> (a oracle.jdbc.pool.OracleConnectionCacheImpl)

Did somebody ran into a similar situation? Please Help.

Thanks,
Sameer

Show how your pool is defined. This points to a serious lack of
concurrency
consideration in the Oracle driver. One terrible thing we can see in
the
driver is that internally it is calling DriverManager.getConnection()!
This
is a huge gaff on the part of the driver because all DriverManager
calls
are class-synchronized, and many other of these calls are made by
drivers
and even SQLException constructors, so one long-running DriverManager
call
can block all other JDBC in the entire JVM.
I suggest you stop using the Oracle driver-based pooling if Tomcat
offers
it's own pooling.
Joe Weinstein at BEA Systems

Here are the connection pool parameters
CacheFixedWaitIdleTime = 30
CacheFixedWaitTimeout = 30
CacheFixedWaitTimeout = 30
CacheScheme = 2
CacheSize = 30
CacheTimeToLiveTimeout = 180
MaxLimit = 100
MaxStatements = 0
MinLimit = 30
NetworkProtocol = tcp
StmtCacheSize = 0
ThreadWakeUpInterval = 300
LoginTimeout = 0
ExplicitCachingEnabled = false
ImplicitCachingEnabled = false

(I posted this 2 days ago, but looks like it did not get posted)

Download the latest (10.2.0.3) version of the ojdbc14.jar,
and try it. Duplicate the problem and post the entire thread dump.
That's not the whole pool definition. You should be specifying
the oracle.Driver class or an Oracle XADataSource. I suspect you're
defining the pool to use the wrong driver class and/or with
the wrong properties.
Joe Weinstein at BEA Systems

.

Relevant Pages

  • Re: OS Question
    ... driver, should not distress you. ... I've made the unwarranted assumption that having multiple threads is ... To protect data in user mode ... Permanence (if apps all crash the lock state is retained because ...
    (microsoft.public.development.device.drivers)
  • Re: JDBC Threads Locking up
    ... monitor entry ... oracle.jdbc.pool.OracleConnectionEventListener)" and it is waiting to ... lock. ... driver is that internally it is calling DriverManager.getConnection! ...
    (comp.lang.java.databases)
  • Re: LOR with netisr changes
    ... ::> It's just a bug in the driver. ... the softc and is intended to be covered by the driver's softc lock. ... this is a hack to work around a recursion in net80211. ...
    (freebsd-current)
  • Re: [PATCH] Remove process freezer from suspend to RAM pathway
    ... My question referred to drivers trying to bind or unbind a device ... The thread trying to bind is holding a lock which is ... Spinning in the driver with the lock not held is impossible, ... would be easiest to jump directly into the freezer! ...
    (Linux-Kernel)
  • Re: [patch 00/15] clocksource / timekeeping rework V4 (resend V3 + bug fix)
    ... which lock already depends on the new lock. ... now at 1915353302473 nsecs ... # You can enable one or both FireWire driver stacks. ...
    (Linux-Kernel)