La mejor manera de eliminar muy grande de registros en Oracle

Question

Me gestionar una aplicación que tiene una muy grande (cerca de 1 TB de datos con más de 500 millones de filas en una tabla) back-end de base de datos Oracle. La base de datos en realidad no hace nada (no hay sprocs, no hay factores desencadenantes o nada) es sólo un almacén de datos.

Cada mes se requiere que los registros de purga de los dos de los cuadros principales. Los criterios para la purga varía y es una combinación de la edad fila y un par de campos de estado. Por lo general terminamos purga entre 10 y 50 millones de filas al mes (agregamos aproximadamente 3-5 millones de filas a la semana a través de las importaciones).

En la actualidad tenemos que hacer esto de eliminación en lotes de alrededor de 50.000 filas (es decir. Borrar 50000, Comit, de eliminación 50000, comprometerse, repetición). El intento de eliminar el lote completo de una sola vez hace que la base de datos no responde durante una hora aproximadamente (dependiendo del # de filas). La eliminación de las filas en lotes como éste es muy áspero en el sistema y que por lo general tienen que hacerlo "como el tiempo lo permite" a lo largo de una semana; permitiendo que la secuencia de comandos para ejecutar de forma continua puede dar lugar a la una degradación del rendimiento que es inaceptable para el usuario.

Creo que este tipo de lotes eliminando también degrada el rendimiento del índice y tiene otros impactos que eventualmente causa el rendimiento de la base de datos para degradar. Hay 34 índices en una sola mesa, y el tamaño de los datos de índice es en realidad más grande que los propios datos.

Aquí está el script que uno de nuestra gente utiliza para hacer esta purga:

BEGIN
LOOP

delete FROM tbl_raw 
  where dist_event_date < to_date('[date]','mm/dd/yyyy') and rownum < 50000;

  exit when SQL%rowcount < 49999;

  commit;

END LOOP;

commit;

END;

Esta base de datos debe tener hasta 99,99999% y sólo tenemos una ventana de mantenimiento día 2 una vez al año.

Estoy buscando un mejor método para eliminar estos registros, pero todavía tengo que encontrar ninguna. ¿Alguna sugerencia?

Answer 1

The logic with 'A' and 'B' might be "hidden" behind a virtual column on which you could do the partitioning:

alter session set nls_date_format = 'yyyy-mm-dd';
drop   table tq84_partitioned_table;

create table tq84_partitioned_table (
  status varchar2(1)          not null check (status in ('A', 'B')),
  date_a          date        not null,
  date_b          date        not null,
  date_too_old    date as
                       (  case status
                                 when 'A' then add_months(date_a, -7*12)
                                 when 'B' then            date_b
                                 end
                        ) virtual,
  data            varchar2(100) 
)
partition   by range  (date_too_old) 
( 
  partition p_before_2000_10 values less than (date '2000-10-01'),
  partition p_before_2000_11 values less than (date '2000-11-01'),
  partition p_before_2000_12 values less than (date '2000-12-01'),
  --
  partition p_before_2001_01 values less than (date '2001-01-01'),
  partition p_before_2001_02 values less than (date '2001-02-01'),
  partition p_before_2001_03 values less than (date '2001-03-01'),
  partition p_before_2001_04 values less than (date '2001-04-01'),
  partition p_before_2001_05 values less than (date '2001-05-01'),
  partition p_before_2001_06 values less than (date '2001-06-01'),
  -- and so on and so forth..
  partition p_ values less than (maxvalue)
);

insert into tq84_partitioned_table (status, date_a, date_b, data) values 
('B', date '2008-04-14', date '2000-05-17', 
 'B and 2000-05-17 is older than 10 yrs, must be deleted');


insert into tq84_partitioned_table (status, date_a, date_b, data) values 
('B', date '1999-09-19', date '2004-02-12', 
 'B and 2004-02-12 is younger than 10 yrs, must be kept');


insert into tq84_partitioned_table (status, date_a, date_b, data) values 
('A', date '2000-06-16', date '2010-01-01', 
 'A and 2000-06-16 is older than 3 yrs, must be deleted');


insert into tq84_partitioned_table (status, date_a, date_b, data) values 
('A', date '2009-06-09', date '1999-08-28', 
 'A and 2009-06-09 is younger than 3 yrs, must be kept');

select * from tq84_partitioned_table order by date_too_old;

-- drop partitions older than 10 or 3 years, respectively:

alter table tq84_partitioned_table drop partition p_before_2000_10;
alter table tq84_partitioned_table drop partition p_before_2000_11;
alter table tq84_partitioned_table drop partition p2000_12;

select * from tq84_partitioned_table order by date_too_old;

Answer 2

The classic solution to this is to partition your tables, e.g. by month or by week. If you have not come across them before, a partitioned table is like several identically structured tables with an implicit UNION when selecting, and Oracle will automatically store a row in the appropriate partition when inserting it based on the partitioning criteria. You mention indexes - well each partition gets its own partitioned indexes too. It is a very cheap operation in Oracle to drop a partition (it is analogous to a TRUNCATE in terms of load because that is what you are really doing - truncating or dropping one of these invisible sub-tables). It will be a significant amount of processing to partition "after the fact", but there's no sense crying over spilt milk - the advantages to doing so far outweigh the costs. Every month you would split the top partition to create a new partition for the next month's data (you can easily automate ths with a DBMS_JOB).

And with partitions you can also exploit parallel query and partition elimination, which should make your users very happy...

Answer 3

One aspect to consider is how much of the delete performance result from indexes and how much from the raw table. Every record deleted from the table requires the same deletion of the row from every btree index. If you've got 30+ btree indexes, I suspect most of your time is spent in index maintenance.

This has an impact on the usefulness of partitioning. Say you have an index on name. A standard Btree index, all in one segment, might have to do four jumps to get from the root block to the leaf block and a fifth read to get the row. If that index is partitioned into 50 segments and you don't have the partition key as part of the query, then each of those 50 segments will need to be checked. Each segment will be smaller, so you may only have to do 2 jumps but you may still end up doing 100 reads rather than the previous 5.

If they are bitmap indexes, the equations are different. You probably aren't using indexes to identify individual rows, but rather sets of them. So rather than a query using 5 IOs to return a single record, it was using 10,000 IOs. As such the extra overhead in extra partitions for the index won't matter.

Answer 4

deletion of 50 million records per month in batches of 50,000 is only 1000 iterations. if you do 1 delete every 30 minutes it should meet your requirement. a scheduled task to run the query you posted but remove the loop so it only executes once should not cause a noticeable degredation to users. We do about the same volume of records in our manufacturing plant that runs pretty much 24/7 and it meets our needs. We actually spread it out a little more 10,000 records every 10 minutes, which executes in about 1 or 2 second running on our Oracle unix servers.

Answer 5

If disk space is not at a premium, you could be able to create a "work" copy of the table, say my_table_new, using CTAS (Create Table As Select) with criteria that would omit the records to be dropped. You can do the create statement in parallel, and with the append hint to make it fast, and then build all your indexes. Then, once it it finished, (and tested), rename the existing table to my_table_old and rename the "work" table to my_table. Once you are comfortable with everything drop my_table_old purge to get rid of the old table. If there are a bunch of foreign key restraints, take a look at the dbms_redefinition PL/SQL package. It will clone your indexes, contraints, etc. when using the appropriate options. This is a summation of a suggestion by Tom Kyte of AskTom fame. After the first run, you can automate everything, and the create table should go much quicker, and can be done while the system is up, and application downtime would be limited to less than a minute to doing the renaming of the tables. Using CTAS will be much faster than doing several batch deletes. This approach can be particularly useful if you don't have partitioning licensed.

Sample CTAS, keeping rows with data from the last 365 days and flag_inactive = 'N':

create /*+ append */ table my_table_new 
   tablespace data as
   select /*+ parallel */ * from my_table 
       where some_date >= sysdate -365 
       and flag_inactive = 'N';

-- test out my_table_new. then if all is well:

alter table my_table rename to my_table_old;
alter table my_table_new rename to my_table;
-- test some more
drop table my_table_old purge;

Answer 6

when dropping a partition, you leave global indexes unusable, that need to rebuild, the rebuild of global indexes would be a big issue, as if you do it online, it will be quite slow, otherwise you need downtime. in either case, can't fit for the requirement.

"We typically end up purging between 10 and 50 million rows per month"

i would recommended using PL/SQL batch delete, several hours is ok i think.