Deadlock in JMS while ApplicationContext shutdown [SPR-4124]

[spring-projects-issues]

Eberhard Wolff opened SPR-4124 and commented

While Spring shuts down a DefaultMessageListenerContainer the JMS Connection is closed. This in turn closes the JMS Sessions as can be seen in the thread dump for Thread-2. The JMS specification says that a close() on a JMS Connection means that you don't need to close the sessions, producers and consumers of a closed connection. However, the AsyncMessageListenerInvoker notices that it is shut down and tries to close its JMS Session (messageListenerContainer-1 thread). For this the JMS Connection is needed.

This leads to a deadlock: Thread-2 has the JMS Connection and tries to acquire the lock on the JMS Session to close it. messageListenerContainer-1 has the JMS Session but tries to acquire the lock on the JMS Connection. Voila - deadlock.

This happens using Oracle Advanced Queueing 9i but it is suspected to happen on IBM WebSphere MQ 6.0 and 5.3. This is outside an Application Server i.e. oracle.jms.AQjmsFactory is used and the Destinations are resolved by name.

Here is the thread dump:

Found one Java-level deadlock:

---

"Thread-2":
waiting to lock monitor 002f1578 (object 69aa96f0, a oracle.jms.AQjmsSession),
which is held by "messageListenerContainer-1"
"messageListenerContainer-1":
waiting to lock monitor 002f15f8 (object 69a9c488, a oracle.jms.AQjmsConnection),
which is held by "Thread-2"

Java stack information for the threads listed above:

---

"Thread-2":
at oracle.jms.AQjmsSession.closeSessionResources(AQjmsSession.java)

• waiting to lock <69aa96f0> (a oracle.jms.AQjmsSession)
  at oracle.jms.AQjmsSession.close(AQjmsSession.java)
  at oracle.jms.AQjmsConnection.closeConnectionResources(AQjmsConnection.java)
• locked <69a9c488> (a oracle.jms.AQjmsConnection)
  at oracle.jms.AQjmsConnection.close(AQjmsConnection.java)
• locked <69a9c488> (a oracle.jms.AQjmsConnection)
  at org.springframework.jms.connection.ConnectionFactoryUtils.releaseConnection(ConnectionFactoryUtils.java:80)
  at org.springframework.jms.listener.AbstractJmsListeningContainer.shutdown(AbstractJmsListeningContainer.java:302)
• locked <69a9c538> (a java.lang.Object)
  at org.springframework.jms.listener.AbstractJmsListeningContainer.destroy(AbstractJmsListeningContainer.java:264)
  at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:146)
  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:344)
  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:317)
  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:290)
• locked <6985c0b0> (a java.util.LinkedHashMap)
  at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:706)
  at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:684)
  at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:621)
  "messageListenerContainer-1":
  at oracle.jms.AQjmsConnection.removeJmsSession(AQjmsConnection.java)
• waiting to lock <69a9c488> (a oracle.jms.AQjmsConnection)
  at oracle.jms.AQjmsSession.closeSessionResources(AQjmsSession.java)
• locked <69aa96f0> (a oracle.jms.AQjmsSession)
  at oracle.jms.AQjmsSession.close(AQjmsSession.java)
  at org.springframework.jms.support.JmsUtils.closeSession(JmsUtils.java:104)
  at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:899)
  at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:864)
  at java.lang.Thread.run(Thread.java:595)

Found 1 deadlock.

---

Affects: 2.0.7

Backported to: 2.0.8

[spring-projects-issues]

Juergen Hoeller commented

Ouch, Oracle seems to have some really bad locking between its Connection and Session there... This is arguably an Oracle bug.

Nevertheless, I've revised DefaultMessageListenerContainer to avoid a deadlock between its Connection and Session through the use of a shared monitor.

Juergen