Meccanismo di archiviazione del database con le massime prestazioni

Question

Ho bisogno di idee per implementare un meccanismo di archiviazione / database in-memory (davvero) ad alte prestazioni. Nella gamma di memorizzazione di oltre 20.000 oggetti, con ogni oggetto aggiornato ogni 5 secondi circa. Vorrei una soluzione FOSS .

Qual è la mia migliore opzione? Quali sono le tue esperienze?

Sto lavorando principalmente in Java, ma ho bisogno che il datastore abbia buone prestazioni, quindi la soluzione del datastore non deve essere java centric.

Devo anche essere in grado di eseguire query su questi oggetti e devo essere in grado di ripristinare tutti gli oggetti all'avvio del programma.

Answer 1

SQLite is an open-source self-contained database that supports in-memory databases (just connect to :memory:). It has bindings for many popular programming languages. It's a traditional SQL-based relational database, but you don't run a separate server – just use it as a library in your program. It's pretty quick. Whether it's quick enough, I don't know, but it may be worth an experiment.

Java driver.

Answer 2

are you updating 20K objects every 5 seconds or updating one of the 20K every 5 seconds?

What kind of objects? Why is a traditional RDBMS not sufficient?

Answer 3

Check out HSQLDB and Prevayler. Prevayler is a paradigm shift from traditional RDBMS - one which I have used (the paradigm, that is, not specifically Prevayler) in a number of projects and found it to have real merit.

Answer 4

Depends exactly how you need to query it, but have you looked into memcached?

http://www.danga.com/memcached/

Other options could include MySQL MEMORY Tables, the APC Cache if you're using PHP.

Some more detail about the project/requirements would be helpful.

Answer 5

An in-memory storage ?

1) a simple C 'malloc' array where all your structures would be indexed.

2) berkeleyDB: http://www.oracle.com/technology/products/berkeley-db/index.html. It is fast because you build your own indexes (secondary database) and there is no SQL expression to be evaluated.

Answer 6

Look at some of the products listed here: http://en.wikipedia.org/wiki/In-memory_database

Answer 7

What level of durability do you need? 20,000 updates every 5 seconds will probably be difficult for most IO hardware in terms of number of transactions if you write the data back to disc for every one.

If you can afford to lose some updates, you could probably flush it to disc every 100ms with no problem with fairly cheap hardware if your database and OS support doing that.

If it's really an in-memory database that you don't want to flush to disc often, that sounds pretty trivial. I've heard that H2 is pretty good, but SQLite may work as well. A properly tuned MySQL instance could also do it (But may be more convoluted)

Answer 8

Oracle TimesTen In-Memory Database. See: http://www.informationweek.com/whitepaper/Business-Intelligence/Datamarts-Data-Warehouses/oracle-timesten-in-memory-databas-wp1228511232361

Answer 9

Chronicle Map is an pure Java key-value store

• it has really high performance, sustaining 1 million writes/second from a single thread. It's a myth that a fast database couldn't be written in Java.
• Seamlessly stores and loads any serializable Java objects, provides a simple Map interface
• LGPLv3

Since you don't have many "tables" a full-blown SQL database could be an overkill solution, indexes & queries could be implemented with a handful of distinct key-value stores which are updated manually by vanilla Java code. Chronicle Map provides mechanisms to make such updates concurrently isolated from each other, if you need it.