Mecanismo de almacenamiento de bases de datos de más alto rendimiento

Question

Necesito ideas para implementar un Mecanismo de almacenamiento / base de datos en memoria (realmente) de alto rendimiento. En el rango de almacenamiento de más de 20,000 objetos, con cada objeto actualizado cada 5 segundos más o menos. Me gustaría una solución FOSS .

¿Cuál es mi mejor opción? ¿Cuáles son tus experiencias?

Estoy trabajando principalmente en Java, pero necesito que el almacén de datos tenga un buen rendimiento, por lo que la solución del almacén de datos no necesita estar centrada en Java.

También necesito poder consultar estos objetos y necesito poder restaurar todos los objetos al iniciar el programa.

Answer 1

SQLite is an open-source self-contained database that supports in-memory databases (just connect to :memory:). It has bindings for many popular programming languages. It's a traditional SQL-based relational database, but you don't run a separate server – just use it as a library in your program. It's pretty quick. Whether it's quick enough, I don't know, but it may be worth an experiment.

Java driver.

Answer 2

are you updating 20K objects every 5 seconds or updating one of the 20K every 5 seconds?

What kind of objects? Why is a traditional RDBMS not sufficient?

Answer 3

Check out HSQLDB and Prevayler. Prevayler is a paradigm shift from traditional RDBMS - one which I have used (the paradigm, that is, not specifically Prevayler) in a number of projects and found it to have real merit.

Answer 4

Depends exactly how you need to query it, but have you looked into memcached?

http://www.danga.com/memcached/

Other options could include MySQL MEMORY Tables, the APC Cache if you're using PHP.

Some more detail about the project/requirements would be helpful.

Answer 5

An in-memory storage ?

1) a simple C 'malloc' array where all your structures would be indexed.

2) berkeleyDB: http://www.oracle.com/technology/products/berkeley-db/index.html. It is fast because you build your own indexes (secondary database) and there is no SQL expression to be evaluated.

Answer 6

Look at some of the products listed here: http://en.wikipedia.org/wiki/In-memory_database

Answer 7

What level of durability do you need? 20,000 updates every 5 seconds will probably be difficult for most IO hardware in terms of number of transactions if you write the data back to disc for every one.

If you can afford to lose some updates, you could probably flush it to disc every 100ms with no problem with fairly cheap hardware if your database and OS support doing that.

If it's really an in-memory database that you don't want to flush to disc often, that sounds pretty trivial. I've heard that H2 is pretty good, but SQLite may work as well. A properly tuned MySQL instance could also do it (But may be more convoluted)

Answer 8

Oracle TimesTen In-Memory Database. See: http://www.informationweek.com/whitepaper/Business-Intelligence/Datamarts-Data-Warehouses/oracle-timesten-in-memory-databas-wp1228511232361

Answer 9

Chronicle Map is an pure Java key-value store

• it has really high performance, sustaining 1 million writes/second from a single thread. It's a myth that a fast database couldn't be written in Java.
• Seamlessly stores and loads any serializable Java objects, provides a simple Map interface
• LGPLv3

Since you don't have many "tables" a full-blown SQL database could be an overkill solution, indexes & queries could be implemented with a handful of distinct key-value stores which are updated manually by vanilla Java code. Chronicle Map provides mechanisms to make such updates concurrently isolated from each other, if you need it.