Google App-Engine Java Batch Update

Question

I need to upload a .csv file and save the records in bigtable. My application successfully parse 200 records in the csv files and save to table.

Here is my code to save the data.

for (int i=0;i<lines.length -1;i++) //lines hold total records in csv file
{
   String line = lines[i]; 

   //The record have 3 columns integer,integer,Text 

   if(line.length() > 15)
   {
 int n = line.indexOf(",");

 if (n>0)
 {
  int ID = lInteger.parseInt(ine.substring(0,n));
  int n1 = line.indexOf(",", n + 2);

  if(n1 > n)
  {
     int Col1 = Integer.parseInt(line.substring(n + 1, n1));
     String Col2 = line.substring(n1 + 1);

     myTable uu = new myTable();

     uu.setId(ID);
     uu.setCol1(MobNo);

     Text t = new Text(Col2);         
     uu.setCol2(t);

       PersistenceManager pm = PMF.get().getPersistenceManager();
            pm.makePersistent(uu);             
     pm.close();
  }
    }
    }
}

But when no of records grow it gives timeout error.

The csv file may have upto 800 records. Is it possible to do that in App-Engine?

(something like batch update)

Answer 1

GAE limit you app request to 30 sec, and you can't run long task.

Best approach is to split this CSV into smaller chunks, and process them individually, one after one. In the case when you can upload it only as one big file, you can store it as binary data, and then process (split and parse) using Task Queue (note that it's also limited to 10 minutes per request, but you can always make a chain of tasks). Or you can user backend to process.

Answer 2

You could store your CSV file in the Blobstore (gzipped or not) and use a MapReduce job to read and persist each line in the Datastore.