Multi-threading SQLAlchemy and return results to ObjectListView

Question

I have run into another issue with a program I am working on. Basically what my program does is it takes up to 4 input files, processes them and stores the information I collect from them in a SQLite3 database on my computer. This has allowed me to view the data any time I want without having to run the input files again. The program uses a main script that is essentially just an AUI Notebook that imports an input script, and output scripts to use as panels.

To add the data to the database I am able to use threading since I am not returning the results directly to my output screen(s). However, when I need to view the entire contents from my main table I end up with 25,000 records that are being loaded. While these are loading my GUI is locked and almost always displays: "Program not responding".

I would like to use threading/multiprocessing to grab the 25k records from the database and load them into my ObjectListView widget(s) so that my GUI is still usable during this process. When I attempted to use a similar threading class that is used to add the data to the database I get nothing returned. When I say I get nothing I am not exaggerating.

So here is my big question, is there a way to thread the query and return the results without using global variables? I have not been able to find a solution with an example that I could understand, but I may be using the wrong search terms.

Here are the snippets of code pertaining to the issue at hand:

This is what I use to make sure the data is ready for my ObjectListView widget.

class OlvMainDisplay(object):
    def __init__(self, id, name, col01, col02, col03, col04, col05,
                 col06, col07, col08, col09, col10, col11,
                 col12, col13, col14, col15):
        self.id    = id
        self.name  = name
        self.col01 = col01
        self.col02 = col02
        self.col03 = col03
        self.col04 = col04
        self.col05 = col05
        self.col06 = col06
        self.col07 = col07
        self.col08 = col08
        self.col09 = col09
        self.col10 = col10
        self.col11 = col11
        self.col12 = col12
        self.col13 = col13
        self.col14 = col14
        self.col15 = col15

The 2 tables I am pulling data from:

class TableMeta(base):
    __tablename__ = 'meta_extra'
    id = Column(String(20), ForeignKey('main_data.id'), primary_key=True)
    col06 = Column(String)
    col08 = Column(String)
    col02 = Column(String)
    col03 = Column(String)
    col04 = Column(String)
    col09 = Column(String)
    col10 = Column(String)
    col11 = Column(String)
    col12 = Column(String)
    col13 = Column(String)
    col14 = Column(String)
    col15 = Column(String)


class TableMain(base):
    __tablename__ = 'main_data'
    id    = Column(String(20), primary_key=True)
    name  = Column(String)
    col01 = Column(String)
    col05 = Column(String)
    col07 = Column(String)

    extra_data = relation(
        TableMeta, uselist=False, backref=backref('main_data', order_by=id))

I use 2 queries to collect from these 2 tables, one grabs all records while the other one is part of a function definition that takes multiple dictionaries and applies filters based on the dictionary contents. Both queries are part of my main "worker" script that is imported by each of my notebook panels.

Here is the function that applies the filter(s):

def multiFilter(theFilters, table, anOutput, qType):
    session = Session()
    anOutput = session.query(table)
    try:
        for x in theFilters:
            for attr, value in x.items():
                anOutput = anOutput.filter(getattr(table, attr).in_(value))
    except AttributeError:
        for attr, value in theFilters.items():
            anOutput = anOutput.filter(getattr(table, attr).in_(value))

    anOutput = convertResults(anOutput.all())
    return anOutput
    session.close()

The theFilters can either be a single dictionary or a list of dictionaries, hence the "Try:" statement. Once the function has applied the filters it then runs the returned results through another function that puts each result returned through the OlvMainDisplay class and adds them to a list to be passed on to the OLV Widget.

Again the big question, is there a way to thread the query (or queries) and return the results without using global variables? Or possibly grab around 200 records at a time and add the data "in chunks" to the OLV widget?

Thank you in advance.
-MikeS

--UPDATE--
I have reviewed "how to get the return value from a thread in python" and the accepted answer does not return anything or still locked the GUI (not sure what is causing the variance). I would like to limit the number of threads created to about 5 at the most.

--New Update-- I made some corrections to the filter function.

Answer 1

You probably don't want to load the entire database into memory at once. That is usually a bad idea. Because ObjectListView is a wrapper of the ListCtrl, I would recommend using the Virtual version of the the underlying widget. The flag is wx.LC_VIRTUAL. Take a look at the wxPython demo for an example, but basically you load data on demand via virtual methods OnGetItemText(), OnGetItemImage(), and OnGetItemAttr(). Note that that refers to the ListCtrl methods...that may be different in OLV land. Anyway, I know that the OLV version is called VirtualObjectListView and works in much the same way. I'm pretty sure there's an example in the source download.

Answer 2

Ok, I finally managed to get the query to run in a thread and be able to display the results in a standard ObjectListView. I used the answer HERE with some modifications.

I added the code to my main worker script which is imported into my output panel as EW.
Since I am not passing arguments to my query these lines were changed:

    def start(self, params):

    self.thread = threading.Thread(target=self.func, args=params)

to

    def start(self):

    self.thread = threading.Thread(target=self.func)

In my output panel I changed how I call upon my default query, the one that returns 25,000+ records. In my output panel's init I added self.worker = () as a placeholder and in my function that runs the default query:

def defaultView(self, evt):
    self.worker = EW.ThreadWorker(EW.defaultQuery)
    self.worker.start()
    pub.sendMessage('update.statusbar', msg='Full query started.')

I also added:

def threadUpdateOLV(self):
    time.sleep(10)
    anOutput = self.worker.get_results()
    self.dataOLV.SetObjects(anOutput)

pub.subscribe(self.threadUpdateOLV, 'thread.completed')

the time.sleep(10) was added after trial an error to get the full 25,000+ results, and I found a 10 seconds delay worked fine.

And finally, at the end of my default query I added the PubSub send right before my output return:

wx.CallAfter(pub.sendMessage, 'thread.completed')
return anOutput
session.close()

To be honest I am sure there is a better way to accomplish this, but as of right now it is serving the purpose needed. I will work on finding a better solution though.

Thanks
-Mike S