Build a Basic Python Iterator
Question
How would one create an iterative function (or iterator object) in python?
Solution
Iterator objects in python conform to the iterator protocol, which basically means they provide two methods: __iter__()
and next()
. The __iter__
returns the iterator object and is implicitly called at the start of loops. The next()
method returns the next value and is implicitly called at each loop increment. next()
raises a StopIteration exception when there are no more value to return, which is implicitly captured by looping constructs to stop iterating.
Here's a simple example of a counter:
class Counter:
def __init__(self, low, high):
self.current = low
self.high = high
def __iter__(self):
return self
def next(self): # Python 3: def __next__(self)
if self.current > self.high:
raise StopIteration
else:
self.current += 1
return self.current - 1
for c in Counter(3, 8):
print c
This will print:
3
4
5
6
7
8
This is easier to write using a generator, as covered in a previous answer:
def counter(low, high):
current = low
while current <= high:
yield current
current += 1
for c in counter(3, 8):
print c
The printed output will be the same. Under the hood, the generator object supports the iterator protocol and does something roughly similar to the class Counter.
David Mertz's article, Iterators and Simple Generators, is a pretty good introduction.
OTHER TIPS
There are four ways to build an iterative function:
- create a generator (uses the yield keyword)
- use a generator expression (genexp)
- create an iterator (defines
__iter__
and__next__
(ornext
in Python 2.x)) - create a class that Python can iterate over on its own (defines
__getitem__
)
Examples:
# generator
def uc_gen(text):
for char in text:
yield char.upper()
# generator expression
def uc_genexp(text):
return (char.upper() for char in text)
# iterator protocol
class uc_iter():
def __init__(self, text):
self.text = text
self.index = 0
def __iter__(self):
return self
def __next__(self):
try:
result = self.text[self.index].upper()
except IndexError:
raise StopIteration
self.index += 1
return result
# getitem method
class uc_getitem():
def __init__(self, text):
self.text = text
def __getitem__(self, index):
result = self.text[index].upper()
return result
To see all four methods in action:
for iterator in uc_gen, uc_genexp, uc_iter, uc_getitem:
for ch in iterator('abcde'):
print ch,
print
Which results in:
A B C D E
A B C D E
A B C D E
A B C D E
Note:
The two generator types (uc_gen
and uc_genexp
) cannot be reversed()
; the plain iterator (uc_iter
) would need the __reversed__
magic method (which must return a new iterator that goes backwards); and the getitem iteratable (uc_getitem
) must have the __len__
magic method:
# for uc_iter
def __reversed__(self):
return reversed(self.text)
# for uc_getitem
def __len__(self)
return len(self.text)
To answer Colonel Panic's secondary question about an infinite lazily evaluated iterator, here are those examples, using each of the four methods above:
# generator
def even_gen():
result = 0
while True:
yield result
result += 2
# generator expression
def even_genexp():
return (num for num in even_gen()) # or even_iter or even_getitem
# not much value under these circumstances
# iterator protocol
class even_iter():
def __init__(self):
self.value = 0
def __iter__(self):
return self
def __next__(self):
next_value = self.value
self.value += 2
return next_value
# getitem method
class even_getitem():
def __getitem__(self, index):
return index * 2
import random
for iterator in even_gen, even_genexp, even_iter, even_getitem:
limit = random.randint(15, 30)
count = 0
for even in iterator():
print even,
count += 1
if count >= limit:
break
print
Which results in (at least for my sample run):
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32
First of all the itertools module is incredibly useful for all sorts of cases in which an iterator would be useful, but here is all you need to create an iterator in python:
yield
Isn't that cool? Yield can be used to replace a normal return in a function. It returns the object just the same, but instead of destroying state and exiting, it saves state for when you want to execute the next iteration. Here is an example of it in action pulled directly from the itertools function list:
def count(n=0):
while True:
yield n
n += 1
As stated in the functions description (it's the count() function from the itertools module...) , it produces an iterator that returns consecutive integers starting with n.
Generator expressions are a whole other can of worms (awesome worms!). They may be used in place of a List Comprehension to save memory (list comprehensions create a list in memory that is destroyed after use if not assigned to a variable, but generator expressions can create a Generator Object... which is a fancy way of saying Iterator). Here is an example of a generator expression definition:
gen = (n for n in xrange(0,11))
This is very similar to our iterator definition above except the full range is predetermined to be between 0 and 10.
I just found xrange() (suprised I hadn't seen it before...) and added it to the above example. xrange() is an iterable version of range() which has the advantage of not prebuilding the list. It would be very useful if you had a giant corpus of data to iterate over and only had so much memory to do it in.
I see some of you doing return self
in __iter__
. I just wanted to note that __iter__
itself can be a generator (thus removing the need for __next__
and raising StopIteration
exceptions)
class range:
def __init__(self,a,b):
self.a = a
self.b = b
def __iter__(self):
i = self.a
while i < self.b:
yield i
i+=1
Of course here one might as well directly make a generator, but for more complex classes it can be useful.
This question is about iterable objects, not about iterators. In Python, sequences are iterable too so one way to make an iterable class is to make it behave like a sequence, i.e. give it __getitem__
and __len__
methods. I have tested this on Python 2 and 3.
class CustomRange:
def __init__(self, low, high):
self.low = low
self.high = high
def __getitem__(self, item):
if item >= len(self):
raise IndexError("CustomRange index out of range")
return self.low + item
def __len__(self):
return self.high - self.low
cr = CustomRange(0, 10)
for i in cr:
print(i)
This is an iterable function without yield
. It make use of the iter
function and a closure which keeps it's state in a mutable (list
) in the enclosing scope for python 2.
def count(low, high):
counter = [0]
def tmp():
val = low + counter[0]
if val < high:
counter[0] += 1
return val
return None
return iter(tmp, None)
For Python 3, closure state is kept in an immutable in the enclosing scope and nonlocal
is used in local scope to update the state variable.
def count(low, high):
counter = 0
def tmp():
nonlocal counter
val = low + counter
if val < high:
counter += 1
return val
return None
return iter(tmp, None)
Test;
for i in count(1,10):
print(i)
1
2
3
4
5
6
7
8
9
All answers on this page are really great for a complex object. But for those containing builtin iterable types as attributes, like str
, list
, set
or dict
, or any implementation of collections.Iterable
, you can omit certain things in your class.
class Test(object):
def __init__(self, string):
self.string = string
def __iter__(self):
# since your string is already iterable
return (ch for ch in string)
It can be used like:
for x in Test("abcde"):
print(x)
# prints
# a
# b
# c
# d
# e
If you looking for something short and simple, maybe it will be enough for you:
class A(object):
def __init__(self, l):
self.data = l
def __iter__(self):
return iter(self.data)
example of usage:
In [3]: a = A([2,3,4])
In [4]: [i for i in a]
Out[4]: [2, 3, 4]
Inspired by Matt Gregory's answer here is a bit more complicated iterator that will return a,b,...,z,aa,ab,...,zz,aaa,aab,...,zzy,zzz
class AlphaCounter:
def __init__(self, low, high):
self.current = low
self.high = high
def __iter__(self):
return self
def __next__(self): # Python 3: def __next__(self)
alpha = ' abcdefghijklmnopqrstuvwxyz'
n_current = sum([(alpha.find(self.current[x])* 26**(len(self.current)-x-1)) for x in range(len(self.current))])
n_high = sum([(alpha.find(self.high[x])* 26**(len(self.high)-x-1)) for x in range(len(self.high))])
if n_current > n_high:
raise StopIteration
else:
increment = True
ret = ''
for x in self.current[::-1]:
if 'z' == x:
if increment:
ret += 'a'
else:
ret += 'z'
else:
if increment:
ret += alpha[alpha.find(x)+1]
increment = False
else:
ret += x
if increment:
ret += 'a'
tmp = self.current
self.current = ret[::-1]
return tmp
for c in AlphaCounter('a', 'zzz'):
print(c)