Why does the following simple piece of Java code using pooled connections from commons.dbcp block?

Question

I have written a simple Java code, which attempts to store files in an hsqldb database. All it does is reads the files from a certain directory and puts them in the DB. It is single threaded, but I am using pooled connections from the apache commons.dbcp in order to be able to cope with the multithreaded access later.

The problem is that the code blocks after reading a few files.

Please, find below the entire source code.

Program.java

import java.io.File;
import java.io.IOException;
import java.sql.SQLException;

import javax.sql.DataSource;

import org.apache.commons.dbcp.ConnectionFactory;
import org.apache.commons.dbcp.DriverManagerConnectionFactory;
import org.apache.commons.dbcp.PoolableConnectionFactory;
import org.apache.commons.dbcp.PoolingDataSource;
import org.apache.commons.pool.KeyedObjectPoolFactory;
import org.apache.commons.pool.impl.GenericKeyedObjectPoolFactory;
import org.apache.commons.pool.impl.GenericObjectPool;

public class Program {
  public static DataSource getPoolingDataSource(String driverClass, String url, String user, String password) throws ClassNotFoundException {
    Class.forName(driverClass);
    ConnectionFactory connectionFactory = new DriverManagerConnectionFactory(url, user, password);
    GenericObjectPool connectionPool = new GenericObjectPool();
    KeyedObjectPoolFactory stmtPool = new GenericKeyedObjectPoolFactory(null);
    new PoolableConnectionFactory(connectionFactory, connectionPool, stmtPool, null, false, true);
    return new PoolingDataSource(connectionPool);
  }

  public static void main(String[] args) throws ClassNotFoundException, SQLException, IOException, InterruptedException {
    String root = args.length == 0 ? "c:/Work/java/ntxdb" : args[0];

    Runtime run = Runtime.getRuntime();
    Process pr = run.exec("cmd /c del /s/q c:\\tmp\\file.db*");
    pr.waitFor();
    DataSource ds = getPoolingDataSource("org.hsqldb.jdbcDriver", "jdbc:hsqldb:file:c:/tmp/file.db", "sa", "");
    HsqldbFileStorage fs = new HsqldbFileStorage(ds);
    putFiles(fs, new File(root));
  }

  private static void putFiles(HsqldbFileStorage fs, File parent) throws IOException, SQLException {
    for (File child : parent.listFiles()) {
      if (child.isDirectory()) {
        putFiles(fs, child);
      } else {
        System.out.println(child.getCanonicalPath());
        fs.put(child);
      }
    }
  }
}

HsqldbFileStorage.java

import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;

import javax.sql.DataSource;

public class HsqldbFileStorage {
  private static final String SET_SQL = "MERGE INTO test" +
    "  USING (VALUES ?, CAST(? AS BLOB)) I (name, data)" +
    "  ON (test.name=I.name)" +
    "  WHEN MATCHED THEN UPDATE SET test.data = I.data" +
    "  WHEN NOT MATCHED THEN INSERT (name, data) VALUES (I.name, I.data)";
  private DataSource m_dataSource;

  public HsqldbFileStorage(DataSource dataSource) throws SQLException {
    super();
    m_dataSource = dataSource;
    Connection c = dataSource.getConnection();
    c.createStatement().execute("Create Cached Table IF NOT EXISTS test (name VARCHAR(256), data BLOB(10M));");
  }

  public void put(File file) throws IOException, SQLException {
    put(file.getCanonicalPath(), file);
  }

  public void put(String name, File file) throws IOException, SQLException {
    InputStream is = new BufferedInputStream(new FileInputStream(file));
    try {
      put(name, is);
    } finally {
      is.close();
    }
  }

  public void put(String name, InputStream data) throws SQLException, IOException {
    PreparedStatement set = m_dataSource.getConnection().prepareStatement(SET_SQL);
    try {
      set.setString(1, name);
      set.setBinaryStream(2, data);
      set.executeUpdate();
    } finally {
      set.close();
    }
  }
}

The code depends on commons-dbcp 1.4, commons-pool 1.6 and hsqldb 2.2.9

Running it on the project directory itself should put 62 files in the DB (I have much more files there than the two aforementioned source files), printing a line for each file.

Unfortunately, it blocks on the eighth file with the following stack trace:

  at java.lang.Object.wait(Object.java:-1)
  at java.lang.Object.wait(Object.java:485)
  at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1118)
  at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
  at HsqldbFileStorage.put(HsqldbFileStorage.java:41)
  at HsqldbFileStorage.put(HsqldbFileStorage.java:34)
  at HsqldbFileStorage.put(HsqldbFileStorage.java:28)
  at Program.putFiles(Program.java:42)
  at Program.putFiles(Program.java:39)
  at Program.putFiles(Program.java:39)
  at Program.main(Program.java:33)

What am I doing wrong?

Answer 1

You are calling:

m_dataSource.getConnection()

in your put(String, InputStream) method. This creates a new connection, which you never close.

So when you are hitting putting 62 files in at some stage it will hit the maximum number of connections in your pool, after which the pool will wait for a connection to be returned to the pool.

If you modify your method as such:

  public void put(String name, InputStream data) throws SQLException, IOException {
    Connection con = null;
    PreparedStatement set = null;
    try {
      con = m_dataSource.getConnection();
      set.prepareStatement(SET_SQL);
      set.setString(1, name);
      set.setBinaryStream(2, data);
      set.executeUpdate();
    } finally {
      if (set != null) {
        set.close();
      }
      if (con != null) {
        con.close();
      }
    }
  }

Note, when you access connections through a pool, you still need to call close. This won't actually close the connection but return it to the pool.