Python Optimize Grouper Function to avoid None elements

Question

Hello I am using something the Grouper function from python's itertools to cut large chunks of select where in(idlist) queries down for sqlite performance. The problem is grouper fills in the whole space of the chunksize even if the list is much smaller so I had to add a loop and comparison before which now I want to optimize.

# input list shorter than grouper chunk size
input = (1,2,3,4,5)

grouper(10,input)
# desired output = (1,2,3,4,5)
# actual output = (1,2,3,4,5,None,None,None,None,None)

# current fix for this issue
list_chunks = tuple(tuple(n for n in t if n) for t in grouper(10, input))

I think there has to be a way to do this without this loop and comparison.

Note: using python 2.5

Answer 1

Something like this?

>>> filter(bool, (1,2,3,4,5,None,None,None,None,None))
(1, 2, 3, 4, 5)

For more complicated case (eg, with 0 in list or you need to drop something differ from None) you can implement own lambda:

>>> filter(lambda n: n is not None, (0,1,2,3,4,5,None,None,None,None,None))
(0, 1, 2, 3, 4, 5)

Or even

>>> from functools import partial
>>> from operator import ne
>>> filter(partial(ne, None), (0,1,2,3,4,5,None,None,None,None,None))
(0, 1, 2, 3, 4, 5)

Answer 2

If instead of filtering out the None entries, rewriting grouper() to return what you want is an option, you can use the following solution using itertools.islice:

def grouper(n, iterable):
    it = iter(iterable)
    x = tuple(islice(it, n))
    while x:
        yield x
        x = tuple(islice(it, n))

Or a shorter equivalent (that is slightly more difficult to understand):

def grouper(n, iterable):
    it = iter(iterable)
    return iter(lambda: tuple(islice(it, n)), ())

Example:

>>> list(grouper(5, range(12)))
[(0, 1, 2, 3, 4), (5, 6, 7, 8, 9), (10, 11)]

Answer 3

You can make use of filter and map:

map(lambda x: filter(bool, x), grouper(10, my_input))

Here is an example:

>>> my_input = (1,2,3,4,5,6,7,8,9,1,2,3,3)
>>> map(lambda x: filter(bool, x), list(grouper(10, my_input)))
[(1, 2, 3, 4, 5, 6, 7, 8, 9, 1), (2, 3, 3)]

Lastly, you can wrap this in a tuple() call if you want it to be a tuple instead of a list.