Delphi support for SQLite manifest typing

Question

SQLite has the interesting "feature" of letting you store anything in any field regardless of its data type.

http://www.sqlite.org/different.html#typing

I'm having to read some externally created SQLite files that were created by (ab)using this "feature". They have a field defined as VARCHAR(30) but are using it to store strings up to 100 characters or more. SQLite happily does this without any trimming taking place if you make direct calls to the SQLite DLL to store your data.

I'm currently using DevArt UniDAC 3.70.0.19 which has SQLite support to read these files, however it quite reasonably respects the size of the field as it is defined and so creates a TStringField object with a length of 30 characters. All characters stored beyond this 30 character limit are inaccessible to me.

I'm aware of all the available Delphi solutions for SQLite however can someone tell me which of them if any can handle this "feature"?

Answer 1

You can do this with any direct access layer to sqlite engine, without any TDataset layer.

For instance our open source wrapper http://blog.synopse.info/post/2011/07/22/SynDBSQLite3%3A-SQLite3-direct-access

In fact, unlike most SQL databases, SQLite does not restrict the type of data that may be inserted into a column based on the column's declared type. Instead, SQLite uses dynamic typing. The declared type of a column is used to determine the affinity of the column only. Direct access without the Db.pas layer allows use of this unique feature.

Answer 2

Your solution can lay in sqlite itself. The schema is by default not writable (you can not UPDATE sqlite_master table) but with sane knowledge and a little help of PRAGMA writable_schema=ON; you can do this. So some changes are safe, for example changing VARCHAR(N) to VARCHAR(M), sqlite doesn't care about the number in the brackets.

Step 1, your schema is limited to 30 chars

CREATE TABLE [TestTable] (
[Id] INTEGER PRIMARY KEY AUTOINCREMENT,
[Txt] VARCHAR(30)
)

The line below allows sqlite_table changes

PRAGMA writable_schema=ON;

And the following statement will change the limits to 100

Update sqlite_master set sql='CREATE TABLE [TestTable] (
[Id] INTEGER PRIMARY KEY AUTOINCREMENT,
[Txt] VARCHAR(100)
)' where tbl_name='TestTable' and type='table'

But you should be aware of what you're doing since some changes are no welcome because sqlite expects some storage format based on the information in the schema. Varchar to varchar conversion does not change the storage format

Answer 3

Because I wanted a really thin wrapper around the sqlite3.dll, I wrote my own, heavily based on OleVariants:

https://github.com/stijnsanders/TSQLite

It should also 'disregard' column type like you discribe, since I've added no other type checking than what the query results provide.

Answer 4

Need change the output type to TEXT, use CAST

SELECT cast(column as TEXT) FROM `tablename`