Mécanisme de stockage de base de données le plus performant

Question

J'ai besoin d’idées pour mettre en place un mécanisme de stockage / base de données en mémoire extrêmement performant. Dans la plage de stockage de plus de 20 000 objets, chaque objet étant mis à jour toutes les 5 secondes environ. Je souhaite une solution FOSS .

Quelle est ma meilleure option? Quelles sont vos expériences?

Je travaille principalement en Java, mais le magasin de données doit être performant pour que la solution de magasin de données ne soit pas nécessairement java centric.

J'ai également besoin de pouvoir interroger ces objets et de pouvoir restaurer tous les objets au démarrage du programme.

Answer 1

SQLite is an open-source self-contained database that supports in-memory databases (just connect to :memory:). It has bindings for many popular programming languages. It's a traditional SQL-based relational database, but you don't run a separate server – just use it as a library in your program. It's pretty quick. Whether it's quick enough, I don't know, but it may be worth an experiment.

Java driver.

Answer 2

are you updating 20K objects every 5 seconds or updating one of the 20K every 5 seconds?

What kind of objects? Why is a traditional RDBMS not sufficient?

Answer 3

Check out HSQLDB and Prevayler. Prevayler is a paradigm shift from traditional RDBMS - one which I have used (the paradigm, that is, not specifically Prevayler) in a number of projects and found it to have real merit.

Answer 4

Depends exactly how you need to query it, but have you looked into memcached?

http://www.danga.com/memcached/

Other options could include MySQL MEMORY Tables, the APC Cache if you're using PHP.

Some more detail about the project/requirements would be helpful.

Answer 5

An in-memory storage ?

1) a simple C 'malloc' array where all your structures would be indexed.

2) berkeleyDB: http://www.oracle.com/technology/products/berkeley-db/index.html. It is fast because you build your own indexes (secondary database) and there is no SQL expression to be evaluated.

Answer 6

Look at some of the products listed here: http://en.wikipedia.org/wiki/In-memory_database

Answer 7

What level of durability do you need? 20,000 updates every 5 seconds will probably be difficult for most IO hardware in terms of number of transactions if you write the data back to disc for every one.

If you can afford to lose some updates, you could probably flush it to disc every 100ms with no problem with fairly cheap hardware if your database and OS support doing that.

If it's really an in-memory database that you don't want to flush to disc often, that sounds pretty trivial. I've heard that H2 is pretty good, but SQLite may work as well. A properly tuned MySQL instance could also do it (But may be more convoluted)

Answer 8

Oracle TimesTen In-Memory Database. See: http://www.informationweek.com/whitepaper/Business-Intelligence/Datamarts-Data-Warehouses/oracle-timesten-in-memory-databas-wp1228511232361

Answer 9

Chronicle Map is an pure Java key-value store

• it has really high performance, sustaining 1 million writes/second from a single thread. It's a myth that a fast database couldn't be written in Java.
• Seamlessly stores and loads any serializable Java objects, provides a simple Map interface
• LGPLv3

Since you don't have many "tables" a full-blown SQL database could be an overkill solution, indexes & queries could be implemented with a handful of distinct key-value stores which are updated manually by vanilla Java code. Chronicle Map provides mechanisms to make such updates concurrently isolated from each other, if you need it.