in-memory database in Python
-
27-09-2019 - |
Question
I'm doing some queries in Python on a large database to get some stats out of the database. I want these stats to be in-memory so other programs can use them without going to a database.
I was thinking of how to structure them, and after trying to set up some complicated nested dictionaries, I realized that a good representation would be an SQL table. I don't want to store the data back into the persistent database, though. Are there any in-memory implementations of an SQL database that supports querying the data with SQL syntax?
Solution
SQLite3 might work. The Python interface does support the in-memory implementation that the SQLite3 C API offers.
From the spec:
You can also supply the special name :memory: to create a database in RAM.
It's also relatively cheap with transactions, depending on what you are doing. To get going, just:
import sqlite3
conn = sqlite3.connect(':memory:')
You can then proceed like you were using a regular database.
Depending on your data - if you can get by with key/value (strings, hashes, lists, sets, sorted sets, etc) - Redis might be another option to explore (as you mentioned that you wanted to share with other programs).
OTHER TIPS
You could possibly use a database like SQLite. It's not strictly speaking in memory, but it is fairly light and would be completely separate from your main database.