Inserting many rows in SQLite from a large CSV file - Window is full error

Question

I'm trying to read a CSV file delimited by tabulations ('\t') and insert each cell into my local database (SQLite) in Android. I could read the file, and insert each row in my database but I have an error :

04-17 14:52:00.529: W/CursorWindow(27577): Window is full: requested allocation 204 bytes, free space 198 bytes, window size 2097152 bytes

04-17 14:52:00.545: E/Prod Count(27577): Prod count: 7009

All the rows seem to have been inserted (I have 7009 rows) but every time I get a certain number of rows in my CSV file I get the Window is full error. I have seen that this error happens when there is BLOB in the database, but I don't have.

The application seems to work but I would like to understand a bit more what I used to do that (I searched a lot and used what I found to improve performances, but didn't understand it very well).

Code :

            // Find the directory for the SD Card using the API
    // *Don't* hardcode "/sdcard"
    File sdcard = Environment.getExternalStorageDirectory();

    // Get the text file
    File file = new File(sdcard, "/my_application/update/testCSV7.txt");
    // It's a CSV file

    BufferedReader buffer = new BufferedReader(new InputStreamReader(
            new FileInputStream(file), "Cp1252")); // Cp1252 = ANSI
    String line = "";
    line = buffer.readLine();
    int cpt = 0;
    int cpt2 = 0;
    SQLiteDatabase db = this.getWritableDatabase();
    Cursor myCursor = db.rawQuery("PRAGMA synchronous=OFF", null);  // Is this usefull?
    myCursor.close();
    try {
        db.setLockingEnabled(false); // Is it usefull?
        String sql = "INSERT INTO " + TABLE_PROD
                + " VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?);";
        SQLiteStatement statement = db.compileStatement(sql);
        db.beginTransaction();
        while ((line = buffer.readLine()) != null) {
            try {
                String[] cells = line.split("\t", -1);
                if (cells.length != 186) {
                    cpt2++;
                    Log.d("ERROR", "Strange size " + line);
                } else {
                    statement.clearBindings();
                    int c = 1;                      
                    statement.bindString(c++, cells[20] + "");
                    statement.bindString(c++, cells[22] + "");
                    statement.bindString(c++, cells[23] + "");
                    statement.bindString(c++, cells[3] + "");
                    statement.bindString(c++, cells[6] + "");
                    statement.bindString(c++, cells[8] + "");
                    statement.bindString(c++, cells[10] + "");
                    statement.bindString(c++, cells[13]);
                    statement.bindString(c++, cells[31] + "");
                    statement.bindString(c++, cells[104] + "");
                    statement.bindString(c++, cells[105] + "");
                    statement.bindString(c++, cells[106] + "");
                    statement.bindString(c++, cells[112] + 0);
                    statement.bindString(c++, cells[114] + 0);
                    statement.bindString(c++, cells[140] + cells[141] + "");
                    statement.bindString(c++, cells[184] + "");
                    statement.bindString(c++, cells[185] + "");
                    statement.execute();
                    Log.d("COMPTEUR : ", cpt + " CPT 2 : " + cpt2);
                }
            } catch (Exception e) {
                e.printStackTrace();
                Log.d("Exception : ", cpt + " Error");
            }
            cpt++;
        }
        db.setTransactionSuccessful();
        db.endTransaction();
    } finally {
        db.setLockingEnabled(true); // Is it usefull?
    }
    db.close();
    buffer.close();
}

My CSV file contains 186 columns and 7010 rows (with the first one). I only want few of them.

I would appreciate if anyone can explain me why I get this message, what I am doing wrong and if some things are useless in my code.

My local database :

    CREATE TABLE GERS_prod(prod_num TEXT PRIMARY KEY,
    prod_design TEXT,prod_design_complete TEXT,prod_marque TEXT,prod_rayon TEXT,
    prod_famille TEXT,prod_ss_famille TEXT,prod_code_vie TEXT,prod_quantite REAL,
    prod_gencod_foire TEXT,prod_gencod_ref TEXT,prod_emballage TEXT,
    prod_colisage REAL,prod_ss_colisage REAL,prod_desc TEXT,prod_couleur TEXT,
    prod_matiere TEXT)

EDIT : I also made the transactions by 200 thinking it was the problem, but I have the same error. I don't understand why I get a CursorWindow, is it used in the transaction?

Code, if I'm doing something wrong :

    File sdcard = Environment.getExternalStorageDirectory();
    File file = new File(sdcard, "/my_application/update/testCSV7.txt");
    BufferedReader buffer = new BufferedReader(new InputStreamReader(
            new FileInputStream(file), "Cp1252")); // Cp1252 = ANSI
    String line = "";
    line = buffer.readLine();
    int cpt = 0;
    int cpt2 = 0;
    SQLiteDatabase db = this.getWritableDatabase();
    Cursor myCursor = db.rawQuery("PRAGMA synchronous=OFF", null);
    myCursor.close();
    String sql = "INSERT INTO " + TABLE_PROD
            + " VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?);";
    while (line != null) {
        SQLiteStatement statement = db.compileStatement(sql);
        db.beginTransaction();
        cpt = 0;
        while ((line = buffer.readLine()) != null && cpt < 200) {
            try {
                String[] cells = line.split("\t", -1);
                if (cells.length != 186) {
                    cpt2++;
                    Log.d("ERROR", "Strange size : " + line.length() + " Line : " + line);
                } else {
                    statement.clearBindings();
                    int c = 1;
                    statement.bindString(c++, cells[20] + ""); // Numprod
                    statement.bindString(c++, cells[22] + ""); // Désignation
                    statement.bindString(c++, cells[23] + ""); // Désignation
                                                                // complète
                    statement.bindString(c++, cells[3] + ""); // marque
                    statement.bindString(c++, cells[6] + ""); // rayon
                    statement.bindString(c++, cells[8] + ""); // famille
                    statement.bindString(c++, cells[10] + ""); // sous-famille
                    statement.bindString(c++, cells[13]); // Code vie
                    statement.bindString(c++, cells[31] + ""); // Quantité
                    statement.bindString(c++, cells[104] + ""); // gencod
                                                                // Foire
                    statement.bindString(c++, cells[105] + ""); // gencod
                                                                // ref
                    statement.bindString(c++, cells[106] + ""); // emballage
                    statement.bindString(c++, cells[112] + 0); // colisage
                    statement.bindString(c++, cells[114] + 0); // sous-colisage
                    statement.bindString(c++, cells[140] + cells[141] + ""); // Concat
                                                                                // des
                                                                                // descriptions
                    statement.bindString(c++, cells[184] + ""); // couleur
                    statement.bindString(c++, cells[185] + ""); // matière
                    statement.execute();
                    Log.d("COMPTEUR : ", cpt + " CPT 2 : " + cpt2);
                }
            } catch (Exception e) {
                e.printStackTrace();
                Log.d("COMPTEUR : ", cpt + " CPT 2 : " + cpt2);
            }
            cpt++;
        }
        db.setTransactionSuccessful();
        db.endTransaction();

    }

    db.close();
    buffer.close();

EDIT 2 : My problem was in my getProdCount() function that I call in my MainActivity. I finally resolved it using LIMIT and OFFSET.

Thank you in advance for any help you can give me.

Answer 1

You can try to use multiple transactions for the insert. You can play around with, say, 200 inserts per transaction and work from there. Playing around with the amount of inserts will also have dramatic impact on performance.