The Anatomy of a for Loop
What happens behind the scenes when you run a Python `for` loop? How complex can it be?
Learning to use the
for loop is one of the first steps in everyone's Python journey. Early on, you learn how to loop using
range() and how to loop through a list. A bit later, you discover you can loop through strings, tuples, and other iterable data types. You even learn the trick of looping through a dictionary using its
But what's really happening behind the scenes in a
Let's look inside the
for loop and make sense of the "iter-soup" of terms: iterate, iterable, iterator,
I will not describe and explain how to use the
for loop in this post. If you're looking for a "
for loop primer", you can read Chapter 2 | Loops, Lists and More Fundamentals of The Python Coding Book.
I'll also use concepts from object-oriented programming in this article. If you need to read more about this topic, you can read Chapter 7 | Object-Oriented Programming in The Python Coding Book or A Magical Tour Through Object-Oriented Programming in Python • Hogwarts School of Codecraft and Algorithmancy, the series of articles about the topic on The Python Coding Stack.
Iterating Through Iterables
Let's jump straight to what you can loop through using a
for loop. The data type at the end of the
for statement must be iterable. We'll see what this means in a second. However, the cheap and cheerful definition of an iterable is "anything you can use in a
Let's define a class and add to it bit by bit to make it iterable. I'll explore different routes to create a custom class you can use in a
I'll create a class called
RandomWalk which can hold several values. The twist is that each time you loop through a
RandomWalk object, the loop will go through the items in a random order. However, the order in which the items are stored in the object doesn't change.
Let's start by defining the class and its
__init__() special method:
You pass an iterable when you initialise a
RandomWalk object and store it as a list in the data attribute
.values. You also create an instance of this class with the names of several captains of starships from the Star Trek universe.
You use a list of strings as an argument for
RandomWalk(). Does this mean you can use a
RandomWalk instance in a
for loop? You can run this code to find out:
Traceback (most recent call last): File ... line 17, in <module> for captain in captains: TypeError: 'RandomWalk' object is not iterable
No, you cannot. The object
captains is of type
RandomWalk, which is not iterable. You can loop through
captains.values, which is a list. But this is not the direction we'll take in this article. We'll make
RandomWalk itself an iterable object.
A class needs to have the
__iter__() special method defined to be iterable. Let's try adding one:
This code still raises an error, but it's a different error this time:
Traceback (most recent call last): File ... line 21, in <module> for captain in captains: TypeError: iter() returned non-iterator of type 'NoneType'
This is progress. You're no longer told that the
'RandomWalk object is not iterable' as in the previous error message since
__iter__() is defined now. However, the special method
__iter__() must return an iterator. The method implicitly returns
None, and that's why the error message refers to a
In the next section, we'll distinguish between iterable and iterator properly and see how they're connected through the
__iter__() special method. For good measure, we'll throw in the built-in function
But for now, let's cheat a bit. First, let's try to return the list
self.values in the
__iter__() special method:
This raises a similar error message as before. The difference is that message refers to the
list object now:
Traceback (most recent call last): File ... line 21, in <module> for captain in captains: TypeError: iter() returned non-iterator of type 'list'
A list is iterable. But it's not an iterator. Let's preview the next section by stating that we can convert an iterable to an iterator using the built-in function
iter(). Note the change in the
This works. The code prints out the list of captains:
Pike Kirk Picard Sisko Janeway Riker
RandomWalk class is now iterable, and you can use its instances, such as
captains, in a
for loop. But this class just mimics a list for now. In the next section, we'll explore iterators further and create a
From Iterables to Iterators
An iterable is a data structure that can be used in a
for loop. More generally, it's a data structure that can return its members one at a time. There are other processes in Python, besides the
for loop, which rely on this feature of iterables. These include list comprehensions and unpacking, among others.
You can always create an iterator from an iterable. One way of doing this is by passing an iterable to the built-in function
iter(), which calls the object's
__iter__() special method.
An iterator is an object that doesn't contain its own data. Instead, it refers to data stored in another structure. Specifically, it keeps track of which item comes next. You can think of an iterator as a disposable data structure. It goes through each item in a structure, one at a time, returning the next value each time it's needed.
Each time you iterate through an iterable, such as in a
for loop, a new iterator is created from the iterable. So, if you iterate through the same iterable several times, a different iterator is used for each one.
Let's create a
RandomWalkIterator class which will act as the iterator for the
RandomWalk iterable. You want the order of iteration to be random each time you iterate through the
Let's look at the change in
RandomWalk.__iter__() first. This method now returns an object of type
RandomWalkIterator. You pass the
RandomWalk instance itself when you create the
RandomWalkIterator.__init__(), you create a list of integers and shuffle it. Each time you create a new
RandomWalkIterator, you generate a new random order of integers which you use as indices.
You also create
self.idx and set it to
0. This index will keep track of the progress through the items in the iterable. You'll use this data attribute shortly.
Does this work? You can try running this as you did earlier:
But you find an error you've encountered several times today already:
Traceback (most recent call last): File ... line 32, in <module> for captain in captains: TypeError: iter() returned non-iterator of type 'RandomWalkIterator'
RandomWalk.__iter__() returns an instance of
RandomWalkIterator. But just having 'Iterator' in its name is not enough to make this object an iterator!
Let's see what comes next…
Most articles on The Python Coding Stack are available in full for free. This is only the second paid-only article on The Stack. If you like reading this Substack and are in a position to support further, you can upgrade to a paid subscription