The Anatomy of a for Loop

What happens behind the scenes when you run a Python `for` loop? How complex can it be?

Learning to use the for loop is one of the first steps in everyone's Python journey. Early on, you learn how to loop using range() and how to loop through a list. A bit later, you discover you can loop through strings, tuples, and other iterable data types. You even learn the trick of looping through a dictionary using its items() method.

But what's really happening behind the scenes in a for loop?

Let's look inside the for loop and make sense of the "iter-soup" of terms: iterate, iterable, iterator, iter(), and __iter__().

---

I will not describe and explain how to use the for loop in this post. If you're looking for a "for loop primer", you can read Chapter 2 | Loops, Lists and More Fundamentals of The Python Coding Book.

I'll also use concepts from object-oriented programming in this article. If you need to read more about this topic, you can read Chapter 7 | Object-Oriented Programming in The Python Coding Book or A Magical Tour Through Object-Oriented Programming in Python • Hogwarts School of Codecraft and Algorithmancy, the series of articles about the topic on The Python Coding Stack.

---

Iterating Through Iterables

Let's jump straight to what you can loop through using a for loop. The data type at the end of the for statement must be iterable. We'll see what this means in a second. However, the cheap and cheerful definition of an iterable is "anything you can use in a for loop".

Let's define a class and add to it bit by bit to make it iterable. I'll explore different routes to create a custom class you can use in a for loop.

I'll create a class called RandomWalk which can hold several values. The twist is that each time you loop through a RandomWalk object, the loop will go through the items in a random order. However, the order in which the items are stored in the object doesn't change.

Let's start by defining the class and its __init__() special method:

copy code

You pass an iterable when you initialise a RandomWalk object and store it as a list in the data attribute .values. You also create an instance of this class with the names of several captains of starships from the Star Trek universe.

You use a list of strings as an argument for RandomWalk(). Does this mean you can use a RandomWalk instance in a for loop? You can run this code to find out:

 Traceback (most recent call last):
   File ... line 17, in <module>
     for captain in captains:
 TypeError: 'RandomWalk' object is not iterable

No, you cannot. The object captains is of type RandomWalk, which is not iterable. You can loop through captains.values, which is a list. But this is not the direction we'll take in this article. We'll make RandomWalk itself an iterable object.

A class needs to have the __iter__() special method defined to be iterable. Let's try adding one:

copy code

This code still raises an error, but it's a different error this time:

 Traceback (most recent call last):
   File ... line 21, in <module>
     for captain in captains:
 TypeError: iter() returned non-iterator of type 'NoneType'

This is progress. You're no longer told that the 'RandomWalk object is not iterable' as in the previous error message since __iter__() is defined now. However, the special method __iter__() must return an iterator. The method implicitly returns None, and that's why the error message refers to a NoneType object.

In the next section, we'll distinguish between iterable and iterator properly and see how they're connected through the __iter__() special method. For good measure, we'll throw in the built-in function iter(), too.

But for now, let's cheat a bit. First, let's try to return the list self.values in the __iter__() special method:

copy code

This raises a similar error message as before. The difference is that message refers to the list object now:

 Traceback (most recent call last):
   File ... line 21, in <module>
     for captain in captains:
 TypeError: iter() returned non-iterator of type 'list'

A list is iterable. But it's not an iterator. Let's preview the next section by stating that we can convert an iterable to an iterator using the built-in function iter(). Note the change in the return statement:

copy code

This works. The code prints out the list of captains:

 Pike
 Kirk
 Picard
 Sisko
 Janeway
 Riker

The RandomWalk class is now iterable, and you can use its instances, such as captains, in a for loop. But this class just mimics a list for now. In the next section, we'll explore iterators further and create a RandomWalkIterator class.

From Iterables to Iterators

An iterable is a data structure that can be used in a for loop. More generally, it's a data structure that can return its members one at a time. There are other processes in Python, besides the for loop, which rely on this feature of iterables. These include list comprehensions and unpacking, among others.

You can always create an iterator from an iterable. One way of doing this is by passing an iterable to the built-in function iter(), which calls the object's __iter__() special method.

An iterator is an object that doesn't contain its own data. Instead, it refers to data stored in another structure. Specifically, it keeps track of which item comes next. You can think of an iterator as a disposable data structure. It goes through each item in a structure, one at a time, returning the next value each time it's needed.

Each time you iterate through an iterable, such as in a for loop, a new iterator is created from the iterable. So, if you iterate through the same iterable several times, a different iterator is used for each one.

Let's create a RandomWalkIterator class which will act as the iterator for the RandomWalk iterable. You want the order of iteration to be random each time you iterate through the RandomWalk iterable:

copy code

Let's look at the change in RandomWalk.__iter__() first. This method now returns an object of type RandomWalkIterator. You pass the RandomWalk instance itself when you create the RandomWalkIterator instance.

In RandomWalkIterator.__init__(), you create a list of integers and shuffle it. Each time you create a new RandomWalkIterator, you generate a new random order of integers which you use as indices.

You also create self.idx and set it to 0. This index will keep track of the progress through the items in the iterable. You'll use this data attribute shortly.

Does this work? You can try running this as you did earlier:

copy code

But you find an error you've encountered several times today already:

 Traceback (most recent call last):
   File ... line 32, in <module>
     for captain in captains:
 TypeError: iter() returned non-iterator of type 'RandomWalkIterator'

RandomWalk.__iter__() returns an instance of RandomWalkIterator. But just having 'Iterator' in its name is not enough to make this object an iterator!

Let's see what comes next…

---