Python Iterator: Example Code and How it Works

To understand what a Python iterator is, you need to know two terms: iterator and iterable:

Iterator

An object that can be iterated, meaning we can keep asking it for a new element until there are no elements left. Elements are requested using a method called __next__.

Iterable

An object that implements another special method, called __iter__. This function returns an iterator.

Thanks to iterators, Python has very elegant for-loops. Iterators also make comprehension possible. Even though it requires some work to understand all the inner workings, they are very easy to use in practice!

How a Python iterator works

As stated above, a Python iterator object implements a function that needs to carry the exact name __next__. This special function keeps returning elements until it runs out of elements to return, in which case an exception is raised of type StopIteration. To get an iterator object, we need to first call the __iter__ method on an iterable object.

We can see this all at work using the built-in Python range function, which is a built-in Python iterable. Let’s do a little experiment:

>>> my_iterable = range(1, 3)
>>> my_iterator = my_iterable.__iter__()
>>> my_iterator.__next__()
1
>>> my_iterator.__next__()
2
>>> my_iterator.__next__()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>>

As you can see:

• range returns an iterable object, since it has the __iter__ method.
• We call the function and assign the iterator it returns to my_iterator. 
• Next, we start to repeatedly call the __next__ method, until the end of the range is reached and a StopIteration exception is raised. 

Of course, this is not how you use iterators in practice. I’m just demonstrating how they work internally. If you ever need to manually get an iterator, use the iter() function instead. And if you need to manually call the __next__ method, you can use Python’s next() function.

Since there’s only a next function, you can only go forward with an iterator. There is no way to reset an iterator (except for creating a new one) or get previous elements.

Why are iterators and iterables separate objects?

Iterators and iterables can be separate objects, but they don’t have to. Nothing is holding us back here. If you want, you can create a single object that is both an iterator and an iterable. You just need to implement both __iter__ and __next__.

So why did the wise men and women building the language decide to split these concepts? It has to do with keeping state. An iterator needs to maintain information on the position, e.g., the pointer into an internal data object like a list. In other words: it must keep track of which element to return next.

If the iterable itself maintains that state, you can only use it in one loop at a time. Otherwise, the other loop(s) would interfere with the state of the first loop. By returning a new iterator object with its own state, we don’t have this problem. This comes in handy, especially when you’re working with concurrency.

Built-in Python iterators

You’ll find that many Python types are iterable once you start looking for them. Here are some iterable types that are native to Python:

• Python lists
• Python sets
• Python dictionaries
• Python tuples
• Ranges
• Strings

There are also some special types of iterables called generators. The most prominent generator example is the range() function, which returns items in the specified range. This function generates these numbers instead of needing to materialize them in an actual list. The obvious advantage is that it can save a lot of memory since all it needs to do is keep a count of the last iteration number to calculate the next item.

How to use a Python iterator

Now that we understand how iterators work, let’s look at how to use Python iterators. You’ll quickly find out that it feels and looks natural and isn’t difficult at all!

Iterator in a for-loop

Unlike other programming languages, loops require an iterable. Here are two examples in which we iterate a list and a string:

>>> mystring = "ABC"
>>> for letter in mystring:
...     print(letter)
...
A
B
C
>>> mylist = ['A', 'B', 'C']
>>> for letter in mylist:
...     print(letter)
...
A
B
C

As you can see, a Python string behaves the same as a Python list in terms of iterability.

Iterators in comprehensions

Just like for-loops, comprehensions require an iterable object too:

>>> [x for x in 'ABC']
['A', 'B', 'C']
>>> [x for x in [1, 2, 3,4] if x > 2]
[3, 4]
>>>

Iterate Python dictionary keys

Python dictionaries are iterable so that we can loop over all a dictionary’s keys. A dictionary iterator only returns the keys, not the values. Here’s an example:

>>> d = {'name': 'Alice', 'age': 23, 'country': 'NL' }
>>> for k in d:
...     print(k)
...
name
age
country

Sometimes I see people use this instead: for k in d.keys(). Although the result is the same, it’s obviously less elegant.

Iterate dictionary values

To iterate Python dictionary values, you can use the values() method:

>>> for k in d.values():
...     print(k)
...
Alice
23
NL

Iterate dictionary keys and values

Use the items() method if you want both the keys and the values from a dictionary. You can use items() with a for-loop or with a Python list comprehension:

>>> for k,v in d.items():
...     print(k, v)
...
name Alice
age 23
country NL
>>>
>>> # With a list comprehension and f-string
>>> [f'{k}: {v}' for k, v in d.items()]
['name: Alice', 'age: 23', 'country: NL']

Convert Python Iterator to list, tuple, dict, or set

An iterator can be materialized into a list using the list() function. In the same way, you can materialize an iterator into a set using the set() function or to a tuple using the tuple() function:

>>> list(range(1, 4))
[1, 2, 3]
>>> set(range(1, 4))
{1, 2, 3}
>>> tuple(range(1, 4))
(1, 2, 3)

If you have an iterator that returns (key, value) tuples, you can materialize it with dict().

Read file line-by-line in Python

Reading a file line-by-line in Python is very easy, thanks to iterators:

with open('cities.txt') as cities:
    for line in cities:
        proccess_city(line)

The open() function returns an iterable object, that can be used in a for-loop. If you like, read my comprehensive article with all the details about handling files in Python.

Creating your own Python iterator

There’s no magic to creating your own iterator. I’ll demonstrate with a simple iterator class that returns even numbers.

As we’ve learned, we need to implement __iter__ and __next__. We’ll do this in one single class to keep things simple. We accomplish this with an __iter__ method that simply returns self. We could make this an endless iterator. But for the sake of demonstration, we’ll raise a StopIteration exception as soon as we pass the number 8.

Remember that if you build an iterator this way, you can not use it in a concurrent environment. In such cases, you should return a new object on each call to __iter__.

If you need a refresher, read our tutorial on classes and objects first.

class EvenNumbers:
    last = 0

    def __iter__(self):
        return self

    def __next__(self):
        self.last += 2

        if self.last > 8:
            raise StopIteration

        return self.last

even_numbers = EvenNumbers()

for num in even_numbers:
    print(num)

If you run this program, the output will be:

2
4
6
8

You can play with this example interactively if you like: