Python Iterator: Example Code and How it Works

To understand what a Python iterator is, you need to know two terms: iterator and iterable:

Iterator
An object that can be iterated, meaning we can keep asking it for a new element until there are no elements left. Elements are requested using a method called __next__.
Iterable
An object that implements another special method, called __iter__. This function returns an iterator.

Iterators make for very elegant for-loops in Python, and also make comprehension possible. Even though it requires some work to understand all the inner workings, they are actually very easy to use in practice!

How a Python iterator works

As stated above, a Python iterator object implements a function that needs to carry the exact name __next__. This special function keeps returning elements until it runs out of elements to return, in which case an exception is raised of type StopIteration. To get an iterator object, we need to first call the __iter__ method on an iterable object.

We can see this all at work using the built-in range function, which is a built-in Python iterable. Let’s do a little experiment:

>>> my_iterable = range(1, 3)
>>> my_iterator = my_iterable.__iter__()
>>> my_iterator.__next__()
1
>>> my_iterator.__next__()
2
>>> my_iterator.__next__()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>>

As you can see:

  • range returns an object that is iterable, since it has the __iter__ method.
  • We call the function an assign the iterator it returns to my_iterator
  • Next, we start to repeatedly call the __next__ method, until the end of the range is reached and a StopIteration exception is raised. 

Of course, this is not how you use iterators in practice. I’m just demonstrating how they work internally. If you ever need to manually get an iterator, use the iter() function instead. And if you need to manually call the __next__ method, you can use Python’s next() function.

Since there’s only a next function, you can only go forward with an iterator. There is no way to reset an iterator (except for creating a new one) or get previous elements.

Why are iterators and iterables separate objects?

Iterators and iterables can be separate objects, but they don’t have to. Nothing is holding us back here. If you want, you can create a single object that is both an iterator and an iterable. You just need to implement both __iter__ and __next__.

So why did the wise men and women building the language decide to split these concepts? It has to do with keeping state. An iterator needs to maintain information on the position, e.g. the pointer into an internal data object like a list. In other words: it must keep track of which element to return next.

If the iterable itself maintains that state, you can only use it in one loop at a time. Otherwise, the other loop(s) would interfere with the state of the first loop. By returning a new iterator object, with its own state, we don’t have this problem. This comes in handy especially when you’re working with concurrency.

Built-in Python iterators

You’ll find that many Python types are iterable once you start looking for them. Here are some iterable types that are native to Python:

There are also some special types of iterables, called generators. The most prominent example of a generator is the range() function, which returns items in the specified range. This function generates these numbers, instead of needing to materialize them in an actual list. The obvious advantage is that it can save a lot of memory since all it needs to do, is keep count of the last iteration number.

How to use a Python iterator

Now that we understand how iterators work exactly, let’s look at how to use Python iterators. You’ll quickly find out that it feels and looks natural and isn’t difficult at all!

Iterator in a for-loop

Unlike other programming languages, for loops in Python require an iterable. Here are two examples in which we iterate a list and a string:

>>> mystring = "ABC"
>>> for letter in mystring:
...     print(letter)
...
A
B
C
>>> mylist = ['A', 'B', 'C']
>>> for letter in mylist:
...     print(letter)
...
A
B
C

As you can see, a Python string behaves the same as a Python list in terms of iterability.

Iterators in comprehensions

Just like for-loops, comprehensions require an iterable object too:

>>> [x for x in 'ABC']
['A', 'B', 'C']
>>> [x for x in [1, 2, 3,4] if x > 2]
[3, 4]
>>>

Iterate Python dictionary keys

Python dictionaries are iterable, so we can loop over all the keys of a dictionary. The iterator a dictionary returns only contains the keys, not the values. Here’s an example:

>>> d = {'name': 'Alice', 'age': 23, 'country': 'NL' }
>>> for k in d:
...     print(k)
...
name
age
country

Sometimes I see people use this instead: for k in d.keys(). Although the result is the same, it’s obviously less elegant.

Iterate dictionary values

To iterate Python dictionary values, you can use the values() method:

>>> for k in d.values():
...     print(k)
...
Alice
23
NL

Iterate dictionary keys and values

If you want both the keys and the values from a dictionary, use the items() method. You can use items() with a for-loop or with a Python list comprehension:

>>> for k,v in d.items():
...     print(k, v)
...
name Alice
age 23
country The Netherlands
>>>
>>> # With a list comprehension and f-string
>>> [f'{k}: {v}' for k, v in d.items()]
['name: Alice', 'age: 23', 'country: NL']

Convert Python Iterator to list, tuple, dict, or set

An iterator can be materialized into a list using the list() function. In the same way, you can materialize and iterator into a set using the set() function or to a tuple using the tuple() function:

>>> list(range(1, 4))
[1, 2, 3]
>>> set(range(1, 4))
{1, 2, 3}
>>> tuple(range(1, 4))
(1, 2, 3)

If you have an iterator that returns (key, value) tuples, you can materialize it with dict().

Loop over a file in Python

Reading a file line-by-line in Python is very easy, thanks to iterators:

with open('cities.txt') as cities:
    for line in cities:
        proccess_city(line)

The open() function returns an iterable object, that can be used in a for-loop.

Creating your own Python iterator

There’s no magic to creating your own iterator. I’ll demonstrate with a simple iterator class that returns even numbers.

As we’ve learned, we need to implement __iter__ and __next__. We’ll do this in one single class, to keep things simple. We accomplish this with an __iter__ method that simply returns self. We could make this an endless iterator. But for the sake of demonstration, we’ll raise a StopIteration exception as soon as we go past the number 8.

Remember that if you build an iterator this way, you can not use it in a concurrent environment. In such cases, you should return a new object on each call to __iter__.

If you need a refreshment, read our tutorial on Python classes and object first.

class EvenNumbers:
    last = 0

    def __iter__(self):
        return self

    def __next__(self):
        self.last += 2

        if self.last > 8:
            raise StopIteration

        return self.last

en = EvenNumbers()

for num in en:
    print(num)   

If you run this program, the output will be:

2
4
6
8

About the author

Erik is the owner of Python Land and the author of many of the articles and tutorials on this website. He's been working as a professional software developer for 25 years, and he holds a Master of Science degree in computer science. His favorite language of choice: Python!