Get big result set from mysql

Question

I've a big table with about 20 millions of rows and every day it grows up and I've a form which get a query from this table. Unfortunately query returns hundreds of thousands of rows. Query is based on Time, and I need all records to classify them by 'clid' base on some rules.So I need all records to do some process on them to make a result table. This is my table :

    CREATE TABLE IF NOT EXISTS `cdr` (
  `gid` bigint(20) NOT NULL AUTO_INCREMENT,
  `prefix` varchar(20) NOT NULL DEFAULT '',
  `id` bigint(20) NOT NULL,
  `start` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
  `clid` varchar(80) NOT NULL DEFAULT '',
  `duration` int(11) NOT NULL DEFAULT '0',
  `service` varchar(20) NOT NULL DEFAULT '',
  PRIMARY KEY (`gid`),
  UNIQUE KEY `id` (`id`,`prefix`),
  KEY `start` (`start`),
  KEY `clid` (`clid`),
  KEY `service` (`service`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf-8 ;

and this is my query :

SELECT * FROM `cdr` 
    WHERE 
        service = 'test' AND 
        `start` >= '2014-02-09 00:00:00' AND 
        `start` < '2014-02-10 00:00:00' AND 
        `duration` >= 10

Date period could be various from 1 hour to maybe 60 day or even more.(like :

DATE(start) BETWEEN '2013-02-02 00:00:00' AND '2014-02-03 00:00:00'

)

The result set has about 150,000 rows for every day. When i try to get result for bigger period or even one day database crashes. Does anybody have any idea ?

Answer 1

I don't know how to prevent it from crashing, but one thing that I did with my large tables was partition them by date.

Here, I partition the rows by date, twice a month. As long as your query uses the partitioned column, it will only search the partitions containing the key. It will not do a full table scan.

CREATE TABLE `identity` (
  `Reference` int(9) unsigned NOT NULL AUTO_INCREMENT,
  ...
  `Reg_Date` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
  PRIMARY KEY (`Reference`),
  KEY `Reg_Date` (`Reg_Date`)
) ENGINE=InnoDB AUTO_INCREMENT=28424336 DEFAULT CHARSET=latin1
PARTITION BY RANGE COLUMNS (Reg_Date) (

  PARTITION p20140201 VALUES LESS THAN ('2014-02-01'),
  PARTITION p20140214 VALUES LESS THAN ('2014-02-14'),
  PARTITION p20140301 VALUES LESS THAN ('2014-03-01'),
  PARTITION p20140315 VALUES LESS THAN ('2014-03-15'),
  PARTITION p20140715 VALUES LESS THAN (MAXVALUE)

);

So basically, you just do a dump of the table, create it with partitions and then import the data into it.