Bulk DB Operation Strategy

Question

I've got a SQLite DB running within an Android app that transfers user data in one month batches.

Upon first run of the app, with no data yet downloaded, storing is a trivial matter with INSERTS.

However, when the user re-downloads a months worth of data (to get any potential updates from the master DB (I want to push small, one record updates, but that's a 2.0 feature)) I need to find an efficient way to store the records.

Currently, I use brute force and simply DELETE all records for the month being refreshed, followed by INSERTS. I didn't want to have a whole bunch of logic and DB accesses, or store meta data, to keep track of which records are new and which are existing, and then perform INSERTS and UPDATES accordingly.

My boss said that DELETE can be an expensive operation, is this true ? Should I put in some logic to figure out what records should be updated and which should be inserted, rather than DELETING and then INSERTING ? This strategy would also need to figure out which records were deleted on the master DB side as well (another reason I opted out).

If there is a completely better/efficient way to do this that I haven't mentioned by all means please let me know. Thanks.

Answer 1

My boss said that DELETE can be an expensive operation, is this true ?

That depends on one's definition of "expensive". I am not aware that a DELETE is significantly more expensive than an UPDATE for the same affected records (in fact, it may be cheaper).

Should I put in some logic to figure out what records should be updated and which should be inserted, rather than DELETING and then INSERTING ?

We have no good way of answering that question, because it is largely dependent upon your environment, such as your database schema.

Make sure that you are wrapping these operations in a transaction, as that has a major impact on the speed of bulk operations.

Beyond that, you would need to run some performance benchmarks of your own, to determine how much more expensive DELETE+INSERT is over UPDATE, and therefore whether it is worth the development headache or not to try to switch to an UPDATE-centric strategy.