How can I merge many SQLite databases?

StackOverflow https://stackoverflow.com/questions/80801

  •  09-06-2019
  •  | 
  •  

Question

If I have a large number of SQLite databases, all with the same schema, what is the best way to merge them together in order to perform a query on all databases?

I know it is possible to use ATTACH to do this but it has a limit of 32 and 64 databases depending on the memory system on the machine.

Was it helpful?

Solution 2

This would be done on demand, possible several times a day. The way I would see it working is as in http://sqlite.1065341.n5.nabble.com/Attempting-to-merge-large-databases-td39548.html where the databases are merged into a large DB, the query performed and then the large database deleted.

OTHER TIPS

To summarize from the Nabble post in DavidM's answer:

attach 'c:\test\b.db3' as toMerge;           
BEGIN; 
insert into AuditRecords select * from toMerge.AuditRecords; 
COMMIT; 
detach toMerge;

Repeat as needed.

Note: added detach toMerge; as per mike's comment.

If you only need to do this merge operation once (to create a new bigger database), you could create a script/program that will loop all your sqlite databases and then insert the data into your main (big) database.

Although a very old thread, this is still a relevant question in today's programming needs. I am posting this here because none of the answers provided yet is concise, easy, and straight-to-point. This is for sake of Googlers that end up on this page. GUI we go:

  1. Download Sqlitestudio
  2. Add all your database files by using the Ctrl + O keyboard shortcut
  3. Double-click each now-loaded db file to open/activate/expand them all
  4. Fun part: simply right-click on each of the tables and click on Copy, and then go to the target database in the list of the loaded database files (or create new one if required) and right-click on the target db and click on Paste

I was wowed to realize that such a daunting task can be solved using the ancient programming skill called: copy-and-paste :)

With no offense, just as one developer to another, I'm afraid that your idea seems terribly inefficient. It seems to me that instead of uniting SQLite databases you should probably be storing several tables within the same Database file.

However if I'm mistaken I guess you could ATTACH the databases and then use a VIEW to simplify your queries. Or make an in-memory table and copy over all the data (but that's even worse performance wise, especially if you have large databases)

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top