Wednesday, November 7, 2018

Application Hung Caused by Corrupted Connection in DBCP Connection Pool

There was an incident where an unplanned network refresh causing the active connection object in the DBCP Connection Pool to lose the 'connection' to the corresponding database session and this resulted the application to hang upon getting connection from the pool.

This case was unique and was not reproducible due to no privilege to perform a network refresh intentionally. However I managed to reproduce this by intentionally killing the database sessions (Sql Server) while having SoapUI to fire the application continuously.

Solution
- via DBCP configuration. There are 2 solutions:
  1. For Sql Server, it's fixed by setting "maxIdle" of the Apache DBCP as 0, the application always establishes a new connection upon new request.
  2. For Oracle, what I observed from setting the "maxIdle=0" solution at local connecting to remote Oracle database was that the application was experiencing slowness upon getting connection from the DBCP pool. I did not want to compromise the application performance, therefore I tried another method which is to set the "maxWait" property, I set it at 500 milliseconds and at the same time had SOAPUI firing the application continuously. After killing the corresponding Oracle Connection Session, I noticed the application was able to create a new connection very soon, as opposed to previously the application would just be hung.