Generators

We've spent some time playing with lists -- they are ordered collections that are iterable, indexable -- and finite. But, what if we'd like to have an iterable collection -- that isn't necessarily finite. Imagine an ordered collection of the even numbers or the digits of pi? Although it isn't indexable, Python does give us a way of representing such an ordered sequence in an iterable way -- generators.

To do this, we write what looks a lot like a function that computes each item in the sequence, one at a time, stopping after each one. Then, each time we "iterate" upon the generator, it computes and yields the next item. The syntax of the generator looks a lot like the syntax of a function, but we "yield" a value instead of "return"ing the value. A "yield" is like a reusmable return -- the next time we poke the generator, it will pick up there.

In class we built up the example below. Notice that we used a loop to genrate the sequence; that if the loop isn't infinite, the sequence isn't either; that we can iterate using next(); and we can also iterate using a for-loop, as we can for any other iterable

---

def generateEvens(start):

  if ((start % 2) != 0): start += 1

  even = start
  while True:
    yield even
    even +=2

evens = generateEvens(3)

print next(evens)
print next(evens)
print next(evens)
print next(evens)


# Reset the generator: 
# Technically, create a new one and assign it to the same variable
evens = generateEvens(3)

# This would be infinite!
# for even in evens: print even

def generateEvensRangeInclusive(start, end):

  if ((start % 2) != 0): start += 1

  even = start
  while (even <= end):
    yield even
    even +=2

evens = generateEvensRangeInclusive(3, 10)
for even in evens: print even

# Reset 
evens = generateEvensRangeInclusive(3, 10)
print next(evens)
print next(evens)
print next(evens)
print next(evens)
print next(evens) # Notice the StopIterator exception
  

---

Generators "Comprehensions" Expressions

Just like we could use a closed form to initialize a list, we can do the same for generators. Instead of using []-brackets to define the comprehension, we use ()-parenthesis to define the generator "expression".

The example below shows a couple of genertor expressions and their use:

  

sentence = "The quick brown fox jumps over the lazy old dog."

# List comprehension
wordList = [word for word in sentence.split()]
print wordList # Prints the list as a list
for word in wordList: print word # Works as expected
print wordList[2] # Legal

# Similar generator expression
wordGenerator = (word for word in sentence.split())
print wordGenerator # Prints nothing useful
for word in wordList: print word # Works as expected
# print wordGenerator[2] # Syntax Error
  

  

Pipelining Generators

One of my favorite features of generators is the ability to use them in a pipeline for filtering. The example below does several steps in a pipeline: parses words, removes periods from them, upper cases them, numbers them:

---

sentence = "The quick brown fox jumps over the lazy old dog."

words = (sentence.split())
#for word in words: print word

wordsNoPeriods = (word.replace(".","") for word in words)
#for word in wordsNoPeriods: print word

wordsUpper = (word.upper() for word in wordsNoPeriods)
#for word in wordsUpper: print word

def numbersGenerator():
  number = 0
  while True:
    yield number
    number = number+1
numbers = numbersGenerator()

numberedWords = (str(numbers.next()) + ":" + word for word in wordsUpper)
for word in numberedWords: print word
  

---