Good ISAM library or other simple file manager for large files on Windows x64

Question

We have some very large data files (5 gig to 1TB) where we need quick read/write access. Since we have a fixed record size it seems like some form of ISAM would be the way to go. But would be happy to hear other suggestions.

Ideally the solution would have an Apache or LGPL style license but we will pay if we have to.

Must haves:

Scalable - over at least 1 TB files
Stable - either doesn't corrupt data or has fast recovery process
Runs well on X64 Windows

Nice to have:

Can participate in 2 phase commits
Intrinsic compression facilities
Portable to *nix platforms
C# API or Java API

Thanks,

Terence

Answer 1

Give Berkeley DB a try. Opinions vary, but it's scalable, stable (if you use all necessary layers) and AFAIK runs well on x64 windows. Also portable to *nix and has C and Java API. Don't know about C# API.

Answer 2

You can also use the ESENT database engine which is built into Windows. As far as your requirements go:

• Scalable: the maximum database size is 16TB. Multi-TB datbases have been used in production.
• Stable: crash recovery with write-ahead logging.
• X64 Windows: ESENT is part of Windows, so it is present on your 64-bit machine.

Nice to have:

• 2 phase commits: No.
• Compression: No.
• Portable to *nix: No.
• C# API or Java API: Not really (there is a C# interop layer on Codeplex but it isn't complete).

The documentation is here: http://msdn.microsoft.com/en-us/library/ms684493(VS.85).aspx

You can get the header file and lib by downloading the Windows SDK.