How to change the dart-sqlite code from synchronous style to asynchronous?

Question

I'm trying to use Dart with sqlite, with this project dart-sqlite.

But I found a problem: the API it provides is synchronous style. The code will be looked like:

// Iterating over a result set
var count = c.execute("SELECT * FROM posts LIMIT 10", callback: (row) {
    print("${row.title}: ${row.body}");
});
print("Showing ${count} posts.");

With such code, I can't use Dart's future support, and the code will be blocking at sql operations.

I wonder how to change the code to asynchronous style? You can see it defines some native functions here: https://github.com/sam-mccall/dart-sqlite/blob/master/lib/sqlite.dart#L238

_prepare(db, query, statementObject) native 'PrepareStatement';
_reset(statement) native 'Reset';
_bind(statement, params) native 'Bind';
_column_info(statement) native 'ColumnInfo';
_step(statement) native 'Step';
_closeStatement(statement) native 'CloseStatement';
_new(path) native 'New';
_close(handle) native 'Close';
_version() native 'Version';

The native functions are mapped to some c++ functions here: https://github.com/sam-mccall/dart-sqlite/blob/master/src/dart_sqlite.cc

Is it possible to change to asynchronous? If possible, what shall I do?

If not possible, that I have to rewrite it, do I have to rewrite all of:

1. The dart file
2. The c++ wrapper file
3. The actual sqlite driver

---

UPDATE:

Thanks for @GregLowe's comment, Dart's Completer can convert callback style to future style, which can let me to use Dart's doSomething().then(...) instead of passing a callback function.

But after reading the source of dart-sqlite, I realized that, in the implementation of dart-sqlite, the callback is not event-based:

int execute([params = const [], bool callback(Row)]) {
  _checkOpen();
  _reset(_statement);
  if (params.length > 0) _bind(_statement, params);
  var result;
  int count = 0;
  var info = null;
  while ((result = _step(_statement)) is! int) {
    count++;
    if (info == null) info = new _ResultInfo(_column_info(_statement));
    if (callback != null && callback(new Row._internal(count - 1, info, result)) == true) {
      result = count;
      break;
    }
  }
  // If update affected no rows, count == result == 0
  return (count == 0) ? result : count;
}

Even if I use Completer, it won't increase the performance. I think I may have to rewrite the c++ code to make it event-based first.

Answer 1

You should be able to write a wrapper without touching the C++. Have a look at how to use the Completer class in dart:async. Basically you need to create a Completer, return Completer.future immediately, and then call Completer.complete(row) from the existing callback.

Re: update. Have you seen this article, specifically the bit about asynchronous extensions? i.e. If the C++ API is synchronous you can run it in a separate thread, and use messaging to communicate with it. This could be a way to do it.

Answer 2

The big problem you've got is that SQLite is an embedded database; in order to process your query and provide your results, it must do computation (and I/O) in your process. What's more, in order for its transaction handling system to work, it either needs its connection to be in the thread that created it, or for you to run in serialized mode (with a performance hit).

Because these are fairly hard constraints, your plan of switching things to an asynchronous operation mode is unlikely to go well except by using multiple threads. Since using multiple connections complicates things a lot (as you can't share some things between them, such as TEMP TABLEs) let's consider going for a single serialized connection; all activity will be serialized at the DB level, but for an application that doesn't use the DB a lot it will be OK. At the C++ level, you'd be talking about calling that execute from another thread and then sending messages back to the caller thread to indicate each row and the completion. But you'll take a real hit when you do this; in particular, you're committing to only doing one query at a time, as the technique runs into significant problems with semantic effects when you start using two connections at once and the DB forces serialization on you with one connection.

It might be simpler to do the above by putting the synchronous-asynchronous coupling at the Dart level by managing the worker thread and inter-thread communication there. That would let you avoid having to change the C++ code significantly. I don't know Dart well enough to be able to give much advice there.

Myself, I'd just stick with synchronous connection processing so that I can make my application use multi-threaded mode more usefully. I'd be taking the hit with the semantics and giving each thread its own connection (possibly allocated lazily) so that overall speed was better, but I do come from a programming community that regards threads as relatively heavyweight resources, so make of that what you will. (Heavy threads can do things that reduce the number of locks they need that it makes no sense to try to do with light threads; it's about overhead management.)