How to delete large amounts of data from Foxpro

Question

The problem

We have a legacy Visual FoxPro reservation system with many tables. I have been asked to do some housekeeping on the tables to reduce their size.

• The tables are badly designed with no auto incrementing primary key.
• The largest table is 3 million rows.
• I am attempting to delete 380,000 rows.
• Due to the volume of data in the tables, I am trying to develop a solution which batch deletes.

What I've got so far

I have created a C# application which accesses the database files via the vfpoledb.1 driver. This application uses recno() to batch the deletion. This is an example of the query I'm using:

delete from TableA 
where TableA.Key in (
    select Key from TableB 
    where Departure < date(2010,01,01) and Key <> ""
) and recno() between 1 and 10000

Executing this via vfpoledb.1 does not delete anything. Executing a select statement with the same where clause does not return anything.

It seems that the combination of the recno() function and an in() function is causing the issue. Testing the query with each clause in turn returns results.

Questions

1. Is there another way of batch deleting data from Visual FoxPro?
2. Why are recno() and in() not compatible?
3. Is there anything else I'm missing?

Additional information

• ANSI is set to TRUE
• DELETED is set to TRUE
• EXCLUSIVE is set to TRUE

Answer 1

Instead of doing in batch of so many record numbers, why not a simpler approach. You are looking to kill off everything prior to some date (2010-01-01).

Why not try based on starting with 2009-12-31 and keep working backwards to the earliest date on file you are trying to purge off. Also note, I don't know if Departure is a date vs datetime, so I changed it to

TTOD( Departure ) (meaning convert time to just the date component)

DateTime purgeDate = new DateTime(2009, 12, 31);

// the "?" is a parameter place-holder in the query
string SQLtxt = "delete from TableA "
              + " where TableA.Key in ( "
              + "     select Key from TableB "
              + "        where TTOD( Departure ) <  ? and Key <> \"\" )";


OleDbCommand oSQL = new OleDbCommand( SQLtxt, YourOleDbConnectionHandle );
// default the "?" parameter place-holder
oSQL.Parameters.AddWithValue( "parmDate", purgeDate );

int RecordsDeleted = 0;

while( purgeDate > new DateTime(2000,1,1) )
{
   // always re-apply the updated purge date for deletion
   oSQL.Parameters[0].Value = purgeDate;
   RecordsDeleted += oSQL.ExecuteNonQuery();

   // keep going back one day at a time...
   purgeDate = purgeDate.AddDays(-1);
}

This way, it does not matter what RECNO() you are dealing with, it will only do whatever keys are for that particular day. If you have more than 10,000 entries for a single day, then I might approach differently, but since this is more of a one-time cleanup, I would not be too concerned with doing 1000+ iterations ( 365 days per year for however many years) through the data... Or, you could do it with a date range and do maybe weekly, just change the WHERE clause and adjust the parameters... something like... (The date of 1/1/2000 is just a guess for how far back the data goes). Also, since this is doing entire date range, no need to convert possible TTOD() of the departure field.

DateTime purgeDate = new DateTime(2009, 12, 31);
DateTime lessThanDate = new DateTime( 2010, 1, 1 );

// the "?" is a parameter place-holder in the query
string SQLtxt = "delete from TableA "
              + " where TableA.Key in ( "
              + "     select Key from TableB "
              + "        where Departure >=  ? "
              + "          and Departure < ? "
              + "          and Key <> \"\" )";

OleDbCommand oSQL = new OleDbCommand( SQLtxt, YourOleDbConnectionHandle );
// default the "?" parameter place-holder
oSQL.Parameters.AddWithValue( "parmDate", purgeDate );
oSQL.Parameters.AddWithValue( "parmLessThanDate", LessThanDate );

int RecordsDeleted = 0;

while( purgeDate > new DateTime(2000,1,1) )
{
   // always re-apply the updated purge date for deletion
   oSQL.Parameters[0].Value = purgeDate;
   oSQL.Parameters[1].Value = lessThanDate;
   RecordsDeleted += oSQL.ExecuteNonQuery();

   // keep going back one WEEK at a time for both the starting and less than end date of each pass
   purgeDate = purgeDate.AddDays(-7);
   lessThanDate = lessThanDate.AddDays( -7);
}

Answer 2

I'm interested in the best way to accomplish this too. We use a lot of poorly designed dBaseIII files that are sometimes quite large.

We do this a lot but it's a nasty, manual process:

1. Import dbf files into a temp database using DTS (management studio import/export wizard for version 2005 + )

2. Run the cleanup scripts using SSMS

3. Export the dbf files and replace the original ones (backing them up) with the newly modified files.

It works for us.

Answer 3

It looks like your condition with date isn't working. Try to execute SELECT statement with using of CTOD() function instead of DATE() you've used.

When your condition will work, then you'll be able to run DELETE statement. But remember that as a result of the DELETE execution the rows will only be marked as deleted. To remove them completely you should run PACK statement after DELETE.

As an another way, you can also try our DBF editor - DBF Commander Professional. It allows to execute SQL queries, including command-line (batch) mode. E.g.:

dbfcommander. exe -q "DELETE FROM 'D:\table_name.dbf' WHERE RECNO()<10000"
dbfcommander. exe -q "PACK 'D:\table_name.dbf'"

You can use it for free within 20 days full-featured trial period.