Options for storing large text blobs in/with an SQL database?

Question

I have some large volumes of text (log files) which may be very large (up to gigabytes). They are associated with entities which I'm storing in a database, and I'm trying to figure out whether I should store them within the SQL database, or in external files.

It seems like in-database storage may be limited to 4GB for LONGTEXT fields in MySQL, and presumably other DBs have similar limits. Also, storing in the database presumably precludes any kind of seeking when viewing this data -- I'd have to load the full length of the data to render any part of it, right?

So it seems like I'm leaning towards storing this data out-of-DB: are my misgivings about storing large blobs in the database valid, and if I'm going to store them out of the database then are there any frameworks/libraries to help with that?

(I'm working in python but am interested in technologies in other languages too)

Answer 1

Your misgivings are valid.

DB's gained the ability to handle large binary and text fields some years ago, and after everybody tried we gave up.

The problem stems from the fact that your operations on large objects tend to be very different from your operations on the atomic values. So the code gets difficult and inconsistent.

So most veterans just go with storing them on the filesystem with a pointer in the db.

Answer 2

I know php/mysql/oracle/prob more lets you work with large database objects as if you have a file pointer, which gets around memory issues.