Few questions on generator expressions and speed efficient alternatives

Question

Consider the following code, integral to my questions below:

import functools

N = 3

class Struct:
    """Create an instance with argument=value slots.
    This is for making a lightweight object whose class doesn't matter."""
    def __init__(self, **entries):
        self.__dict__.update(entries)

    def __repr__(self):

        args = ['%s=%s' % (k, repr(v)) for (k, v) in vars(self).items()]
        return '\nStruct(%s)' % ', '.join(args)

def doit( move ):

    ( rowIn, colIn ) = move
    something = rowIn  + ( 10 * colIn ) # An involved computation here in real life

    return Struct( coord = ( rowIn, colIn ), something = something )

legalStates = [ ( row, col ) for row in xrange( N ) for col in xrange( N ) ] # A more complicated function that generates the list in real life. Call it 'complicatedFunction'

genExpFn = lambda : ( ( s.something, m, s ) for ( m, s ) in ( ( move, doit( move ) ) for move in legalStates ) )  #Q1

successorsSortedGenFn = lambda : ( p for p in sorted( genExpFn(), reverse = True ) )

def bFunc( s, a ):

    #print "a * s ->", a * s
    return a * s # An involved computation here in real life

def aFunc( ( v, m, s ) ): #Q2

    assert( s.something == v )
    return bFunc( s.something, 10 )

print "min( successorsSortedGen ) -> " + str( min( successorsSortedGenFn(), key=functools.partial( aFunc )) ) #Q3
print
print "max( successorsSortedGen ) -> " + str( max( successorsSortedGenFn(), key=functools.partial( aFunc )) ) #Q4

My questions are based on the statements marked as "#Q":

Q1 : It's apparent that the generator is completely instantiated ( all elements are executed ), as we call sorted() on it ( which generates all the elements and creates a temporary unsorted list which it sorts and returns as a new list? ).

Is there a space efficient way, that minimizes the creation of temporaries and yeilds a sorted list?

I tried to, but could not write a list comprehension that could sort in place using list.sort()

This was the kind of expression I was thinking about:

successorsSorted = [ ( s.something, m, s ) for ( m, s ) in ( ( move, doit( move ) ) for move in legalStates ) ].sort( reverse = True )

Q2 : Notice that 'aFunc' is just a wrapper around 'bFunc' because I was not able to write an equivalent representation in the functools.partial( aFunc ) call.

What is the expression 'aFunc' in functools.partial( aFunc ) that I am looking for that will allow me to call 'bFunc' directly?

---

EDIT : The answer to Q2 is lambda ( v, m, s ): bFunc(s.something, 10)

Thus, the statements become:

print "min( successorsSortedGen ) -> " + str( min( successorsSortedGenFn(), key=functools.partial( lambda ( v, m, s ): bFunc(s.something, 10)) ) )
print
print "max( successorsSortedGen ) -> " + str( max( successorsSortedGenFn(), key=functools.partial( lambda ( v, m, s ): bFunc(s.something, 10)) ) )

I know it kinda seems lame I did not think about this earlier, but oh well ( thanks to aaronasterling for the gentle prodding ).

---

Q3, Q4 : Note that the elements passed to min() and max() are already sorted.

Is it possible for me to make this hint to min() and max() so that it does not instantiate the whole list as a temporary and then iterate through the whole list to locate the min or max element?

If not, is there a module or custom function in existence that does not instantiate the whole list, but, given that the list passed to it is sorted, returns the min or max element while inspecting the least number of elements?

Answer 1

Q1. [x for x in somelist].sort() creates a list and calls the sort method. This returns None The assignment assigns None to successorSorted. If you want to do this, you'll have to implement it yourself and it will probably be way slower than the builtin sort creating the temporary list.

Q2. You could take apart the code object and rearrange the argument list so that a is the first argument and then rewrite all the bytecode to account for the new positions of the locals. (Yes this can actually be done). You could then use functools.partial on that. Or you could use a wrapper like you're doing now or in a few other ways. I'm +1 on the wrapper. (though let me know if you want a bytecode hack, I think they're fun and the cool thing about providing them as answers on Stack Overflow is that I get to write them but don't have to use them ;)

Q3, Q4. Not really. To get the tenth element of an iterator, you need to go through all of the earlier ones. If you know that you want the first element, you can just do

smallest = next(sorted_iterator)

and for the last

for item in iterable: pass
largest = item

The first will eat the first element of the iterator and the last will eat your whole iterator. Bye Bye iterator.