Memory management of large data from oracle database

Question

I am pulling out large data from oracle database using cx_oracle using below sample script:

from cx_Oracle import connect

TABLEDATA = []

con = connect("user/password@host")
curs = con.cursor()
curs.execute("select * from TABLE where rownum < 100000")

for row in curs:
    TABLEDATA.append([str(col) for col in list(row)])

curs.close()                        
con.close()

Problem with storing in list is that it ends up to about 800-900 mb of RAM usages. I know I can instead save this in file and not store in list but I am using this list to display table using QTABLEVIEW and QABSTRACTTABLE MODEL.

Is there any alternate or more effient way where I can minimise memory usage of storing this data and also use it to display my table?

Answer 1

I have tried multiple possobilities, I don't think qsqltablemodel works for me. Though it load data directly from database, as you keep scrolling down it loads more and more data in table and hence the memory usage keep on increasing.

What I think will ideally work is being able to load set number of rows in model. As you scroll down it loads new rows but also at the same time unloads what's already there. So at any point of time we only have set number of rows loaded in model.

Answer 2

If you don't want to store all the data in RAM, then you need to use a model for you tableview that get's information from the database as needed. Fortunately, Qt natively supports this, and can connect to oracle databases.

You will want to look into:

• http://qt-project.org/doc/qt-4.8/sql-driver.html
• http://qt-project.org/doc/qt-4.8/sql-model.html
• http://qt-project.org/doc/qt-4.8/qsqltablemodel.html
• http://qt-project.org/doc/qt-4.8/qsqldatabase.html

Note this is c++ documentation, but it is fairly easy to translate to PyQt (I always use the c++ documentation despite never coding in c++). You may also want to subclass QSqlTableModel to provide slightly different behaviour to the standard interface!