Why would someone need an in-memory database?

Question

I read that a few databases can be used in-memory but can't think of reason why someone would want to use this feature. I always use a database to persist data and memory caches for fast access.

Answer 1

Cache is also a kind of database, like a file system is. 'Memory cache' is just a specific application of an in-memory database and some in-memory databases are specialized as memory caches.

Other uses of in-memory databases have already been included in other answers, but let me enumerate the uses too:

1. Memory cache. Usually a database system specialized for that use (and probably known as 'a memory cache' rather than 'a database') will be used.
2. Testing database-related code. In this case often an 'in-memory' mode of some generic database system will be used, but also a dedicated 'in-memory' database may be used to replace other 'on-disk' database for faster testing.
3. Sophisticated data manipulation. In-memory SQL databases are often used this way. SQL is a great tool for data manipulation and sometimes there is no need to write the data on disk while computing the final result.
4. Storing of transient runtime state. There are application that need to store their state in some kind of database but do not need to persist that over application restart. Think of some kind of process manager – it needs to keep track of sub-processes running, but that data is only valid as long as the application and the sub-processes run.

Answer 2

A common use case is to run unit/integration tests.

You don't really care about persisting data between each test run and you want tests to run as quickly as possible (to encourage people to do them often). Hosting a database in process gives you very quick access to the data.

Answer 3

Does your memory cache have SQL support?

How about you consider the in-memory database as a really clever cache?

That does leave questions of how the in-memory database gets populated and how updated are managed and consistency is preserved across multiple instances.

Answer 4

Searching for something among 100000 elements is slow if you don't use tricks like indexes. Those tricks are already implemented in a database engine (be it persistent or in-memory).

A in-memory database might offer a more efficient search feature than what you might be able to implement yourself quickly over self-written structures.

Answer 5

In-memory databases are roughly at least an order of magnitude faster than traditional RDBMS for general purpose (read side) queries. Most are disk backed providing the very same consistency as a normal RDBMS - only catch the entire dataset must fit into RAM.

The core idea is disk backed storage has huge random access penalties which does not apply to DRAM. Data can be index/organized in a random access optimized way not feasible using traditional RDBMS data caching schemes.

Answer 6

Applications, which require real time responses would like to use an in memory database, perhaps application to control aircraft, plants where the response time is critical

Answer 7

An in memory database is also useful in game programming. You can store data in an in memory database which is much faster than permanent databases.

Answer 8

They are used as an advanced data structure to store, query and modify runtime data.

Answer 9

You may need a database if several different applications are going to access the dataset. A database has a consistent interface for accessing / modifying data, which your hash table (or whatever else you use) won't have.

If a single program is dealing with the data, then it's reasonable to just use a data structure in whatever language you are using though.

Answer 10

In-memory database is better than performing database caching. Database caching works similar to in-memory databases when it comes to READ operations.

On the other hand, when it comes to WRITE operations, in-memory databases are faster when compared to database caches, where the data is persisted onto disk (which leads to IO overhead).

Also, with database caching you can end with cache misses but you will never end up with cache misses when using in-memory databases.

Answer 11

Given their speed and the declining price of RAM, it’s likely that in-memory databases will become the dominant technology in the future. There are already some that have developed sophisticated features like SQL queries, secondary indexes, and engines for processing datasets larger than RAM.