Best way to perform insert/delete batch spring-jdbc/mysql

Question

First, I will try to describe what I am willing to do and then, I will ask my questions. I need to do the following:

• List all rows corresponding to some conditions
• Do some tests (e.g: check if it wasn't already inserted), if test passes then insert row into another database
• Delete row (whether it passed tests or not)

The following is my implementation

List<MyObject> toAdd = new ArrayList<MyObject>();
for(MyObject obj:list){
    if(!notYetInserted(obj){
        toAdd.add(obj);
    }
}
myObjectDAO.insertList(toAdd);
myObjectDAO.deleteList(list);

The service method is marked transactional.

In my DAO methods for deleteList and insertList are pretty similar so I will just put here method for insert.

public void insertList(final List<MyObject> list){
String sql = "INSERT INTO table_test " +
    "(col_id, col2, col3,col4) VALUES (?, ?, ?,?)";

List<Object[]> paramList = new ArrayList<Object[]>();

for (MyObject myObject : list) {
    paramList.add(new Object[] {myObject.getColId(), 
        myObject.getCol2(), myObject .getCol3(), myObject.getCol4()}
    );
}
simpleJdbcTemplate.batchUpdate(sql, paramList);        

}

I am not sure about the best way to perform such operations, I read here that calling for update inside a loop may slow down the system (especially in my case, I will have about 100K insert/delete at a time). I wonder if these additional loops inside DAO won't slow down my system even more and what would happen if problem happened repeatedly while processing that batch (I thought also about moving test from service to DAO to have only one loop and an additional test, I don't really know if it's a good idea). So, I would like your advices. Thanks a lot.

PS: if you need more details feel free to ask!

Answer 1

This is not necessarily a bad approach, but you are right, it might be really slow. If I were to do a process like this that inserted or deleted this many rows I would probably do it all in a stored procedure. My code would just execute the proc and the proc would handle the list and the enumeration through it (as well as the inserts and deletes).