Fastest way to query repeatedly big table

Question

I have a table with 50 million records, but a compact structure (id, int1, int1, int2, int3). All relevant indexes are implemented.

I need to query it about 50 times for each user interaction. This is taking about 5 seconds, using regular prepared queries against the db server. All the queries are simple selects.

My question is: what can I do to speed up this, even spending significantly more memory? The Locate method for queries is not flexible enough and using filters directly into the query is slow.

The main query I am running is

select knowledge_id, knowledge_family_id, tag_level, tag_order, 
  total_title_direct_words, total_title_parenthesis_words from knowledge_tags 
  where dictionary_word_id = XX order by tag_level, tag_order

Can anyone suggest a strategy? Would TVirtualTable increase speed?

Answer 1

I my opinion it would be much faster load entire data in a TClientDataSet and use FINDKEY to Seek those records.

To use FindKey() you must define Indexes like this:

Cds.IndexDefs.ADD('IDX1', 'FieldA;FieldB',[]);
Cds.IndexName := 'IDX1';
if Cds.FindKey([A,B]) then begin
  //Do something
end;

You can also create multiple indexes and use it whenever you want:

Cds.IndexDefs.ADD('IDX1', 'FieldA;FieldB',[]);
Cds.IndexDefs.ADD('IDX2', 'FieldD;FieldB',[]);
Cds.IndexDefs.ADD('IDX3', 'FieldA;FieldC',[]);  

if SeekAB then 
  Cds.IndexName := 'IDX1'
else if SeekDB then 
  Cds.IndexName := 'IDX2'
else 
  Cds.IndexName := 'IDX3';

Answer 2

Considering that to load the entire table into the memory is not a problem I suggest:

1. Execute a query to select all the records
2. Walk on the dataset one single time to copy the records into an instance of TObjectList<TMyRecord>, where TMyRecord is class that can contain all the fields of a dataset record; making it so what you have now is a copy of the dataset in an object list
3. If possible, sort the list (or select it correctly ordered) so you can run a binary search when you need
4. Free the dataset and work only on the list.

In this scenario you will avoid all the dataset overhead and each search will be much faster.

An example of the TMyRecord is:

interface

type
  TMyRecord = class
  private
    FId: Integer;
    FCol1: Integer;
    FCol2: Integer;
    FCol3: Integer;
    FCol4: Integer;
  public
    constructor Create(aId, aCol1, aCol2, aCol3, aCol4: Integer);
    property Id: Integer read FId write FId;
    property Col1: Integer read FCol1 write FCol1;
    property Col2: Integer read FCol2 write FCol2;
    property Col3: Integer read FCol3 write FCol3;
    property Col4: Integer read FCol4 write FCol4;
  end;

implementation

constructor TMyRecord.Create(aId, aCol1, aCol2, aCol3, aCol4: Integer);
begin
  FId := aId;
  FCol1 := aCol1;
  FCol2 := aCol2;
  FCol3 := aCol3;
  FCol4 := aCol4;
end;