Python Generators - What are they?

Generators are a cool Python feature that was introduced back in Python 2.2 (as a __future__ import) and became standard in Python 2.3. They are a very neat abstraction that allows you to simplify control flow and in many instances the resulting elegance also leads to much lower memory consumption.

The key to generator functionality lies in the yield keyword. A very simple example of how it works is demonstrated below:

def abc():
  yield "a"
  yield "b"
  yield "c"

The first thing to note here is that abc() is neither the generator nor an ordinary function. Rather, the presence of the yield keyword within abc() makes it a generator factory. You define the body of the generator within abc(), but abc() itself is only ever invoked to acquire a separate generator object. You NEVER call abc() to execute the code written within it.

In the example below, we assign the generator object that abc() returns to a variable called genobj1 and call next() on it repeatedly. This is what yields the strings "a", "b" and "c" per successive call:

>>> genobj1=abc()

>>> genobj1.next()
'a'
>>> genobj1.next()
'b'
>>> genobj1.next()
'c'

The factory approach allows you to create as many generator objects as you like, and each will operate independently.

>>> genobj1=abc(); genobj2=abc()

>>> genobj1.next()
'a'
>>> genobj1.next()
'b'
>>> genobj2.next()
'a'
>>> genobj1.next()
'c'
>>> genobj2.next()
'b'
>>> genobj2.next()
'c'

Because a generator object supports the Iterator Protocol (also see http://www.python.org/dev/peps/pep-0234), you can use it in place of a list in a for statement (among other places)

>>> genobj1=abc()
>>> for x in genobj1: print x

a
b
c

The key idea that differentiates a generator from a function is that whereas a function finishes execution upon returning a value, a generator yields a value and at the same time remembers or bookmarks itself at the point of the yield. On the next invocation of the generator, execution will resume from this 'bookmarked' point. This is the reason we use the term yield [a value] rather than return. Because the generator never truly 'returns and terminates', all the local variables within that generator's scope remain intact. This completely eliminates the need to do bookkeeping of variables that need to persist between calls.

To illustrate with a counterexample in C, while Python does not explicitly support so-called 'static' variables that persist between function calls, generators allow you to achieve a similar effect in a much more elegant manner.

In C,

#include <stdio.h>

int main() {
  printf("0\n");
  printf("1\n");
  while (1)
    printf("%d\n",fib());
}

int fib() {
  static int first=0,second=1,next,retval;

  next=first+second;
  retval=next;
  first=second;
  second=next;

  return retval;
}

/* An even clumsier way to implement the above
   is to use global variables. */

In Python,

def fib():
  first=0; second=1
  yield first
  yield second

  while 1:
    next=first+second
    yield next
    first=second; second=next

And of course, whereas in C you have to create separate function declaration (with the same body) for each fibonacci-number generating function you might need ...

int fib1() {
  static int first=0,second=1,next,retval;

  next=first+second;
  retval=next;
  first=second;
  second=next;

  return retval;
}

int fib2() {
  static int first=0,second=1,next,retval;

  next=first+second;
  retval=next;
  first=second;
  second=next;

  return retval;
}

int fib3() {

.
.
.

... in Python, you just create as many of these fibonacci number generating objects as you like via a simple assignment statement as shown in the earlier examples.

So how do Python generators help you conserve memory? Basically you can use them anywhere you need a list of computed values, but will only need to access the values one at a time (e.g. iterate over them). Instead of pre-computing all the values at once and storing them in a real list, create a generator that computes the values on the fly.

Discussion:

Back to Top

© 2008 by Andy Sy
last updated: 2006-Apr-12