Golang - pgbouncer and transaction usage

Question

Tech Detail

• go version 1.2
• postrgres library for go bmizerany/pq

This issue is driving me mad and I'm hoping someone will be able to help.

I have developed an application in golang to read data from a postgres database and for each record make an http request and then update the database.

This is all simple enough. However, we have pgbouncer in place. The configuration we have for pgbouncer is such that it does not support prepared statements. Go silently wraps all queries in a prepared statement. The way around this for pgbouncer is to set up a transaction. That is all well and good for things like insert/update/delete.

In the case of the select statement I am wrapping it in transaction:

func TransactionQuery(db *sql.DB, baseQuery string) (rows *sql.Rows, code int, err error) {
        tx, txErr := db.Begin()
        if txErr != nil {
            return nil, -1, txErr
        }

        selectStmt, prepErr := tx.Prepare(baseQuery)
        if prepErr != nil {
            return nil, -1, fmt.Errorf("Failed to prepare statment: %s Error: %v", baseQuery, prepErr)
        }

        defer func() {
            if stmtErr := selectStmt.Close(); stmtErr != nil {
                rows = nil
                code = -2
                err = fmt.Errorf("Failed to close statement: %v.", stmtErr)
            }
        }()

        rows, err = selectStmt.Query()
        if err != nil {
            fmt.Errorf("Failed to retrieve data: %v", err)
            return nil, -1, err
        }
        return rows, 0, nil
    }

(hhmm, that seems to have throw the indenting off a little) AsS you can see I am starting bnut not closing the transaction. This causes a problem in the pg side of things where every select is left in a 'idle in transaction" state.

I have tried tx.Commit() and tx.Rollback() and in both cases I get errors:

"unknown response for simple query '3'"

or

"unknown response for simple query 'D'"

Ho do I successfully close the transaction in Go? I am hoping to get our pgbouncer.ini updated to allow me to switch to lib/pq for the driver library but I'm not sure if that will directly help this issue.

So, how do I close the tx object correctly or is there a way to force Go to not use prepared statements under the hood?

Thanks Nathan

I've tried to change things up a bit:

func TransactionQuery(db *sql.DB, baseQuery string) (rows *sql.Rows, code int, err error) {
    tx, txErr := db.Begin()
    if txErr != nil {
        return nil, -1, txErr
    }

    /*selectStmt, prepErr := tx.Prepare(baseQuery)
      if prepErr != nil {
          return nil, -1, fmt.Errorf("Failed to prepare statment: %s Error: %v", baseQuery, prepErr)
      }
    */
    rows, err = tx.Query(baseQuery)
    if err != nil {
        fmt.Errorf("Failed to retrieve data: %v", err)
        return nil, -1, err
    }

    /*    if stmtErr := selectStmt.Close(); stmtErr != nil {
          rows = nil
          code = -2
          err = fmt.Errorf("Failed to close statement: %v.", stmtErr)
      }*/

    if txCloseErr := tx.Commit(); txErr != nil {
        rows = rows
        code = -3
        err = txCloseErr
    }
    return rows, 0, nil
}

What I see in the logs with this code:

pq: unexpected describe rows response: '3'

However, I should point out that this is when trying a select statement for the second time. This application selects a batch, deals with it and then selects a subsequent batch. This error happens on the second select. There are no issues with the very first select.

Answer 1

For anyone else who hits this I have solved this.

The code I had was returning the rows object to the calling code (not shown) above. Closing the transaction before the rows were read seemed to be the cause of the problem. I don't fully understand why so please clarify if you do.

Now I return both the rows and the transaction object. Then when I have read all the rows I rollback the transaction. this keeps everything working.

Thanks Nathan

Answer 2

Tx.Query returns an instance of sql.Rows, which is a cursor to the result set. The data hasn't necessarily been retrieved while you are iterating this cursor. The transaction must remain open while you are iterating the cursor.

This is why you could no longer iterate the cursor once the transaction was closed.