usage of generators as a progression notifier

Question

I am currently using generators as a quick way to get the progress of long processes and I'm wondering how is it done usually as I find it not very elegant...

Let me explain first, I have a engine.py module that do some video processing (segmentation, bg/fg subtraction, etc) which takes a lot of time (from seconds to several minutes).

I use this module from a GUI written in wxpython and a console script. When I looked at how to implement progress dialogs in wxpython, I saw that I must get somehow a progress value to update my dialog, which is pure logic you'll admit... So I decided to use the number of frame processed in my engine functions, yield the current frame number every 33 frames and yield None when the processing is finished.

by doing that here's what it looks like:

dlg = wx.ProgressDialog("Movie processing", "Movie is being written...",
                           maximum = self.engine.endProcessingFrame,self.engine.startProcessingFrame,
                           parent=self,
                           style = wx.PD_APP_MODAL | wx.PD_ELAPSED_TIME | wx.PD_SMOOTH | wx.PD_CAN_ABORT)
state = self.engine.processMovie()
f = state.next()
while f != None:
    c, s = dlg.Update(f, "Processing frame %d"%f)
    if not c:break
    f = state.next()
dlg.Destroy()

That works very well, there is absolutely no noticeable speed loss, but I would like to be able to call processMovie() function without having to deal with generators if I don't want to.

For instance my console script which uses the engine module doesn't care of the progress, I could use it but it is destined to be executed in an environment where there is no display so I really don't care about the progress...

Anyone with another design that the one I came up with? (using threads, globals, processes, etc)

There must be a design somewhere that does this job cleany I think :-)

Answer 1

Using a generator is fine for this, but the whole point of using generators is so you can builtin syntax:

for f in self.engine.processMovie():
    c, s = dlg.Update(f, "Processing frame %d"%f)
    if not c: break

If you don't care about that, then you can either say:

for f in self.engine.processMovie(): pass

or expose a function (eg. engine.processMovieFull) to do that for you.

You could also use a plain callback:

def update_state(f):
    c, s = dlg.Update(f, "Processing frame %d"%f)
    return c
self.engine.processMovie(progress=update_state)

... but that's not as nice if you want to process the data piecemeal; callback models prefer to do all the work at once--that's the real benefit of generators.

Answer 2

This sounds like a perfect case for events. The process sends a "status update event", and anyone who wants to know (in this case the dialog) listens to that event.

Answer 3

First of all, if you use a generator, you might as well use it as an iterator:

state = self.engine.processMovie()

for f in state:
    c, s = dlg.Update(f, "Processing frame %d"%f)
    if not c:
        break

dlg.Destroy()

And don't yield None; stop yielding when you're done and leave the function; alternatively raise StopIteration. This is the correct way of ending generation (and when using a for loop, it's necessary).

Other than that, I like the idea. In my opinion, this is a very valid use of generators.

You might want to make the 33 configurable (i.e. passable to processMovie as a parameter); 33 seems like an arbitrary choice, and if your process a two-hour movie, there's no need to update the progress bar every 33 frames I guess.