You want DB->compact(). From the C API reference:
The DB->compact() method compacts Btree, Hash, and Recno access method databases, and optionally returns unused Btree, Hash or Recno database pages to the underlying filesystem.
http://docs.oracle.com/cd/E17076_02/html/api_reference/C/frame_main.html