Sqlite database optimization

Question

I have sqlite3 database with three tables:

CREATE TABLE document (
  id Int PRIMARY KEY NOT NULL,
  root_id Int,
  name Varchar(100),
  active Tinyint
);
CREATE INDEX IDX_documentId ON document (id);
CREATE INDEX IDX_documentName ON document (name);

CREATE TABLE dictionary (
  id Int PRIMARY KEY NOT NULL,
  word Varchar(100) NOT NULL
);
CREATE INDEX IDX_dictionaryId ON dictionary (id);
CREATE UNIQUE INDEX IDX_dictionaryWord ON dictionary (word ASC);

CREATE TABLE document_index (
  id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
  document_id Int NOT NULL,
  word_id Int NOT NULL,
  FOREIGN KEY(document_id) REFERENCES document(id),
  FOREIGN KEY(word_id) REFERENCES dictionary(id)
);
CREATE INDEX IDX_documentIndexId ON document_index (id);
CREATE INDEX IDX_documentIndexDocId ON document_index (document_id);
CREATE INDEX IDX_documentIndexWordId ON document_index (word_id);

And I have sql script to select all documents wich contains word from the dictionary:

SELECT document.id, document.name
FROM document
     INNER JOIN document_index on document_index.document_id=document.id
     INNER JOIN dictionary on dictionary.id=document_index.word_id
WHERE dictionary.word LIKE @pQuery
   AND document.active = 1
   AND document.root_id in (@pRoot1, @pRoot2, @pRoot3, @pRoot4, @pRoot5, @pRoot6, @pRoot7)

When the dictionary contains ~= 400,000 records, document ~= 1000 records and document_index ~= 500,000 records, the query is executing about 30 seconds on my iPad 2.

How to optimize query or change the structure of the database (add indexes for example) to reduce the query time?

Answer 1

I found the solution. This solution increased the speed of the query execution in 60! times. I found it here and more detailed - here. It's simple, I replaced LIKE expression to compare >= and <:

old:

dictionary.word LIKE 'prezident%'

new:

dictionary.word >= 'prezident' AND dictionary.word < 'prezidentz' /* Added z to the second string*/ 

This solution has one limitation, I can look for by a part of the string, but at the end of the string ony, i.e. 'expr%'.

Thanks all for the help!

Answer 2

I don't think there is any practical way to make your queries work faster other than using SQLite Full Text Search extension.

FTS allows to use fast version of MATCH clause instead of inherently slow LIKE.

Unfortunately, FTS is not enabled by default on iOS, but apparently you can still do that if you build your own app and include your own copy of SQLite with FTS enabled.

Answer 3

The bottleneck is very likely the part WHERE dictionary.word LIKE @pQuery.

1. you have no index on dictionary.word, so SQLite needs to scan the complete table
2. You are using the LIKE operator, which can't use indices in most cases.

Is it really necessary for your use-case to use a LIKE query instead of just checking for string equality?

Answer 4

Try using dictionary.word = @pQuery instead of dictionary.word LIKE @pQuery

Run "analyze"