How to distribute a Python program with embedded Firebird SQL for Linux and Windows

Question

Summary:

What is the best (easiest, most flexible, simplest) way to redistribute a Firebird SQL database with Python code in a way that end users can use it without going to the trouble of installing and maintaining Firebird?

Background (somewhat long-winded):

I've been trying to write a program that sifts through stock fundamentals and appraises different companies' stock based on those fundamentals and randomized weights. I noticed that after some length of time, the program seemed to just stall. I did use multi-threading here and there, and I considered dead/livelocks, but apart from just combing through the code and seeing if it makes sense, I can't debug that. I noticed I was also eating up a lot of RAM, since all this data was held in big Python dicts in memory. So I figured putting it in SQL databases would fix that.

After a few weeks, I got the code working again with SQLAlchemy and SQLite. Now the problem is that the appraisal function takes ten minutes (!) per stock. Multiplied by a total of twelve "genomes" competing initially, this would add up to around 200 hours. I started thinking maybe this had to do with SQLite's concurrency locks, or something along those lines, so I started trying to use Firebird, since it is the only other one I know that stores a database in a file.

Question elaboration:

Ideally, I would be able to just throw my code on a disk or a server, take it to another computer with Python on it, and run everything out of the box. That's doable with SQLite. Is it possible with Firebird? I know that there is a separate embedded package for Windows, but that Linux only has the libfbembed library that ships with the Classic server. The docs said that Linux always requires some version of the Firebird server proper to be installed.

Will end users need to do any database administration to make that work; that is, will they need to set up users and such manually, as if I had just given them an fdb file and told them to figure out the rest? Or is it enough to install the basic packages for Firebird? Will I ever be able to get something close to the simplicity of SQLite in redistributing Firebird databases? Is there any special syntax I need to pass to SQLalchemy/FDB/Kinterbasedb to use an embedded server? (I could not find anything about this on either SQLalchemy or FDB's websites). Can my program run seamlessly on Linux and Windows, or will there need to be slightly different setups for each case?

Thanks in advance, anyone who can answer some of these questions.

Answer 1

Well, i only can give partial answers. But i think that'll be enough to start with.

Let's start with the Firebird embedded thing: As you have written - using linux as os you have to provide a full install. There is no other way.

HINT: Use the native tgz provided from firebird, not any package delivered from distribution - to avoid dependency hell.

Installing Firebird on Windows : The Windows Firebird Installer is mostly a 'click-through' thing. Luckily you can customize the installer: Install Firebird and look into doc\scripted-install.txt.

HINT: on Win7/8 don't install into %PROGRAM FILES% or %PROGRAM FILES (x86)%

Talking to firebird: AFAIK you have two options, but for both i don't know if and how they will work with SQLAlchemy:

• The fdb module which comes from firebird. When installing the fdb package make shure the appropriate fbclient.dll is in the search path.
• the pyfirebirdsql module: https://github.com/nakagami/pyfirebirdsql/ which needs no dll or that. Partial drawback - it is not as fast as the fdb module as there is no real database engine. Personally i only use it for short lookups.

Using the fdb module you also can talk to the firebird services api - from creating over dropping databases to querying header statistics, ending up with backup/restore actions.

That should at least answer the question if the end user needs to perform any database administration.