Sunday, December 16, 2012

java.sql.SQLRecoverableException: IO Error: Connection reset and /dev/urandom

Following stack trace could be seen when the application server was starting.
java.sql.SQLRecoverableException: IO Error: Connection reset
        at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:886)
        at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1167)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289)
        at oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1491)
        at oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:406)
        
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at oracle.net.ns.Packet.receive(Packet.java:300)
        at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
        at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
        at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
        at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
        at oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
        at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
        at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
        at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
        at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:193)
        at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:873)
        ... 6 more
The application servers (6 of them) were running on RHEL 5 (x86_64) and this is a new deployment so there's less activity on the servers. The JDBC drivers were 11.2.0.3.
It seems the JDBC thin clients running on Linux system uses /dev/random for security purposes and the amount of activity (entropy - disorder in the system to be precise) determine how fast /dev/random can deliver random numbers. A good explanation of this is available here. By default JDBC uses /dev/random and could cause JDBC connection creation to hang when the /dev/random cannot generate random numbers fast enough.


One of the solutions (more available on the above mentioned link) is to make JDBC use /dev/urandom which can deliver random numbers with less activity than /dev/random. (According link above these values are "theoretically vulnerable to a cryptographic attack").
JDBC could be made to use /dev/urandom by specifying it as a system property (triple "/" is needed)
-Djava.security.egd=file:///dev/urandom
For some systems alternate formats of the URL may be needed and these are available in the metalink note given below.

Useful metalink notes
ODI Receives Timeout Messages On Linux x86 With Java 1.5 Onwards [ID 1277360.1]
Oracle Service Registry 11.1 Fails To Install with 'java.sql.SQLRecoverableException: IO Error: Connection reset' Error [ID 1343776.1]
JDBC 11.2.0.3 Application Fails With java.sql.SQLRecoverableException: IO Error: Connection reset [ID 1575238.1]
Oracle Service Registry 11.1 Fails To Install with 'java.sql.SQLRecoverableException: IO Error: Connection reset' Error [ID 1343776.1]