Multiuser Embedded Database

Question

I have been tasked with finding a good fit for a database to use with an upcoming project. I am having a hard time finding a good match. Our application is a .Net Framework 4.0 C# application.

First I looked at SQLite but I am running into issue because SQLite locks it's database file during writes.

I have 2 separate applications running at the same time using the same database and I cannot be certain when one is writing. I have attempted to just wait for the locks to release but this has caused some data loss. So I thought that i would invent a queue system, but both applications would need to use the same queue...So I thought that I would just make a separate Window's service. All of this started to sound like a traditional RDBMS to me.

I decided to backtrack and ask myself "Am I using the right database?".

Does anybody know of a database that might meet the requirements that I have been given?

Here are my requirements: 1. Easy to install. This includes not having a separate "installer" that users have to run separate from our application's installation. This is what first led me to believe that I need an embedded database but I think there maybe some clever ways of including a Database installation inside of our installer.

1. Hard to break. The biggest want here is that it should be able to work over named pipes instead of TCP/IP for security concerns (or anything that reduces the ability for people's system configurations to break our application). Our applications will always be running on the same machine. They need to share the same database, therefore using TCP/IP could potentially cause issues if firewalls/security/"smart users" get involved.

2. Long running operations on the database should not prevent other users from reading/writing to the database. I thought about writing some code to handle this requirement and still use Sqlite but decided to do more homework.

3. Licensing: We want something that we can redistribute royalty free. We dont want our end user to need to purchase their own separate license for a database that we depend on.

4. Size limitations: we plan on storing quite a bit of data in these database and arbitrary limits are not acceptable. We are willing to accept limits imposed by the operating system for file size. SQLite's size limit (which is the size limit of any windows machine's file) was greater than 2 terabytes.

5. Nice to have: Able to use Entity Framework.

6. Nice to have: Relational database. We may allow 3rd party applications to read our database and would not mind if it was relational as opposed to a document or object database. However, we would be willing to budge on this one for something that meets all our requirements.

Databases I've looked at so far: RavenDB SQLite CubeSQLite SQL CE

Oracle and Microsoft SQL Server are licensed and there for difficult to redistribute.

I understand that these requirements are VERY restrictive. I am not even certain that there is an out of the box solution available. I appreciate any help anybody can provide.

Answer 1

You need to use SQL Server CE 4.0:

• xcopy deployment
• support encryption
• freely distributable
• support multi-theading (in your case, multi-process) Only since version 4.0
• support 4 Gb space
• support EF (even CodeFirst)
• it's relational, and supports a subset of SQL Server T-SQL

You have the wrong misconception that it requires licensing:

Microsoft SQL Server Compact 4.0 is a free, embedded database that software developers can use for building ASP.NET websites and Windows desktop applications. SQL Server Compact 4.0 has a small footprint and supports private deployment of its binaries within the application folder, easy application development in Visual Studio and WebMatrix, and seamless migration of schema and data to SQL Server.

From Microsoft SQL Server Compact 4.0 (MS download)

Unless the 4Gb is not enough for you, you can use it.

The SQL Server Express is also free, and has less limitations, but needs installation.

The main difference betqween SQL CE and SQL Server is that CE doesn't support Sotred Procedures. But If you use EF that isn't a problem at all.

EDIT Additional info on licensing:

SQL Server Compact 4.0 is freely redistributable under a redistribution license agreement and application developers redistributing SQL Server Compact 4.0 can optionally register at the SQL Server Compact redistribution site. Registering will help the developers in getting information about SQL Server Compact critical security patches and hot fixes that can be further applied to the client installations.

At the bottom of the donwload page

Answer 2

I realise this is an old question, but I'm currently dealing with almost the same problems and was wondering if the situation had changed so here goes!

From the SQLite documentation it would seem like this is definitely possible with SQLite.

They clearly state that is supports multi-process access on the same computer. It is only when you try to share over a network that SQLite has problems. http://www.sqlite.org/faq.html#q5

This part is of specific interest: "When any process wants to write, it must lock the entire database file for the duration of its update. But that normally only takes a few milliseconds. Other processes just wait on the writer to finish then continue about their business. Other embedded SQL database engines typically only allow a single process to connect to the database at once."

Would this not have been a solution to the problem? It seems that if a write only takes a few milliseconds you could basically just loop the call until it worked (With an appropriate timeout and delay).

I would go with CE but I just love open source too much ;)