Designing a sqlite component in C++

Question

I work on a small component in an embedded device (sensor). This component :

• Every 5 seconds, sends requests to other components using sockets (get health check status, operating status etc) and stores the response in memory.
• Creates a socket, listens for incoming requests from UI. And sends the stored stats.

To keep the design simple, component is single threaded and asynchronous. I use boost asio, timers. Some components take a while to respond but the async calls (socket create/listen/connect/send/receive) make sure none of the operations block the main thread. This has been working really well.

I am working on a new task:

• Talk to a new component(every 5 secs) and get a list of active events. There could be thousands of active events at any point of time.
• Store this info in a database. Hundreds of thousands of records in a month.

component_id integer
event_starttime integer // timestamp
event_endtime integer // timestamp
event_info text

• Send this information to the UI.

There will be a single writer and multiple readers. I chose sqlite because it fits my needs: simple, fast, suitable for embedded system. Here are some questions that I have:

1. Indices: UI will query things like list of active events (endtime is null), amount of events per component in a data range. I could make component_id an index. Would it make sense to make a text column an index if it's used in searching?

2. UI can query historical stats and close the connection. It can also get continuous feed of active events in a single http connection. These db read operations could block the main thread from talking to other components. Would it make sense to:

   a. Use a separate thread and boost io_context to listen for incoming db requests from UI? io_context will handle the listening/accepting connections asynchronously. The new thread will handle db reads synchronously. Even if it takes time, it won't block the main thread. Cons: Short queries might be delayed by longer ones since they run in the same thread, share db connection.

   b. Spawn a new thread per connection? Also, share/create a new db connection?

3. In async_receive(), need to store the event info into database. I can't find any sqlite3 async methods to write this info in the database. The events are stored in a vector and it will insert/update each item in a foreach loop. This will block the main thread which I want to avoid. Any suggestions?

4. I have seen many recommending WAL mode. Will it be suitable for my use case?

Edit: Slightly unrelated, I was reading about nodejs sqlite libraries and come across a discussion on node-sqlite3 vs better-sqlite3. Apparently, better-sqlite3 is much faster than the former even though it is synchronous while node-sqlite3 is async. They say sql operations are CPU intensive, doesn't make sense to use async. Thoughts?

Answer 1

1. Indices: UI will query things like list of active events (endtime is null), amount of events per component in a data range. I could make component_id an index. Would it make sense to make a text column an index if it's used in searching?

Indices in a database are mostly useful when searching for an exact match. If that is also how you look for entries in your text column, then it can certainly be useful to make an index for it. Otherwise, it is probably not useful, but you could run an experiment with it to see if an index improves performance without increasing the storage requirements too much.

2. UI can query historical stats and close the connection. It can also get continuous feed of active events in a single http connection. These db read operations could block the main thread from talking to other components. Would it make sense to:

   a. Use a separate thread and boost io_context to listen for incoming db requests from UI? io_context will handle the listening/accepting connections asynchronously. The new thread will handle db reads synchronously. Even if it takes time, it won't block the main thread. Cons: Short queries might be delayed by longer ones since they run in the same thread, share db connection.

   b. Spawn a new thread per connection? Also, share/create a new db connection?

This depends entirely on if handling the database requests is dominated by processing time on your CPU or time that your CPU is waiting for external events. In an embedded device, this depends a lot on the choices made for the microprocessor and persistent storage, so we can't give a general advise on it.

If the handling of database requests is dominated by CPU processing time, then your CPU will not be able to do anything else without slowing down the processing of those database requests. It is up to you to determine when the cost of multiple threads (in both processing time due to context switching and more complicated development) outweighs the potential delay in handling other requests because the main thread is busy.

4. I have seen many recommending WAL mode. Will it be suitable for my use case?

Yes, WAL mode should work fine in your use case.