How to Import a Lot of Data into a Production Database? (We are using Java)

Question

We have to import a lot of data into a production database from Web services, flat files and other external sources. We are using spring batch to do this.

One of the major problem is that some of those data are related to each other but won't be imported at the same time. The other major problem is that there is a lot of data, so I can't really make a huge transaction and roll back if a problem occurs.

How could I do that?

Answer 1

Your best best is to load the data into a "holding" table that is not used by your running application. Then look at using SELECT INTO to copy the data into the application tables when the application is least busy.

The advantages to this approach are

1. Loading the data into the holding table has no locking implications on your application tables
2. Depending on your database configuration SELECT INTO can be done with minimal (or no) writing to the transaction log making it very efficient

Answer 2

Assuming that the database is not in serevice while you are doing this: back up the database, turn off all constraint checks, import your data, turn constraints back on. If all fails then at least you have the backup to fall back on.