Question

I am working on a terminal app in C and SQL for tagging files. The file/tag relationships are saved in an SQLite 3 database. Up until now I've been using statically allocated prepared statements, which are reset and bound on every call to the wrapper functions. Recently I was testing for memory leaks etc with valgrind, and it was complaining a lot because obviously the static sqlite3_stmt in the wrappers are not freed upon program termination. This also causes the database to not be closed properly (because of non-finalized statements).

The original reason for doing this is performance. I read in the C interface intro that reusing prepared statements is very good for performance.

Using sqlite3_reset() on an existing prepared statement rather than creating a new prepared statement avoids unnecessary calls to sqlite3_prepare(). In many SQL statements, the time needed to run sqlite3_prepare() equals or exceeds the time needed by sqlite3_step(). So avoiding calls to sqlite3_prepare() can result in a significant performance improvement.

Here's some code to show what I'm doing:

int tag_file(const char *file, const char *tag)
{
    static sqlite3_stmt *sql_prep = NULL;
    static const char *sql_str = "INSERT OR IGNORE INTO Tag VALUES (?, ?);";

    // Prepare if null, else reset
    prepare_or_reset(&sql_prep, sql_str);

    if (sqlite3_bind_text(sql_prep, 1, file, -1, SQLITE_STATIC) != SQLITE_OK ||
        sqlite3_bind_text(sql_prep, 2, tag, -1, SQLITE_STATIC) != SQLITE_OK)
        return ERROR;

    if (sqlite3_step(sql_prep) != SQLITE_DONE)
        return ERROR;

    return SUCCESS;
}

What I tried, which will satisfy valgrind, but completely misses the performance gain of reusing the prepared statement is to make sql_prep non-static and calling sqlite3_finalize(sql_prep) at the end of the function, but I guess this is worse performance wise. Can I prepare the statement just once without a memory leak in an elegant way?

Another thing I started worrying about was memory consumption. In the future I am planning to create a GUI for this app, which would keep all this in memory much longer. The performance gain could potentially be bigger, but that statement would just be sitting there on the heap from the first call to exit. Is this a fair space-time tradoff?

Edit: Would it be ugly/weird to keep all "static" prepared statements in a global array, which I can free atexit?.

Was it helpful?

Solution

  1. You must finalize all statements before closing the database.

    This is not possible if the statements are not known outside of functions such as tag_file. You should have a global list of statements, and let the prepare_or_reset function add the statement to that list dynamically.

  2. Prepared statements are tiny; there is no comparison to all the stuff allocated by GUI libraries. It is very likely that you will not even notice their memory usage.

    However, prepared statements do not give much of a speed improvement if you do not execute them so often that the preparing time is noticeable. So it probably does not matter either way. ☺

OTHER TIPS

You are mentioning two issues.

  1. The first one is essentially how to manage your prepared statement.

    Here it seem appropriate to re-design your int tag_file(const char *file, const char *tag) function a little bit.

    Either you maintain the stmt one level higher and pass it to the function each time. Then you can clean up at the very end of the program.

    Another way could be to define your function's semantics in a way that you give it special sentinel functions (maybe just (NULL, NULL)) which make the function free the statement.

  2. The second issue is the question of heap size. As long as you don't have hundreds or thousands of prepared stmts, I think you can have it all the time as it will probably only cost about some few bytes.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top