Generator Details

When we implement an iterator, we need to record the current iterated state so that we can generate the next data based on the current state. In order to record the current state and use it iteratively with the next() function, we can use a simpler syntax, namely a generator. A generator is a special kind of iterator.

generators and yield s

If the function body contains the yield keyword, and the function is called again, the code of the function body will not be executed, and the return value obtained is the generator object

def my_range(start,stop,step=1):
     while start < stop:
         yield start

<generator object my_range at 0x104105678>

Generators have built-in __iter__ and __next__ methods, so the generator itself is an iterator

>>> g.__iter__
<method-wrapper '__iter__' of generator object at 0x1037d2af0>
>>> g.__next__
<method-wrapper '__next__' of generator object at 0x1037d2af0>

Therefore, we can use next (generator) to trigger the execution of the function corresponding to the generator,

>>> next(g) # Trigger function execution until it encounters yield, then stop, return the value after yield, and suspend the function at the current position
>>> next(g) # Call next(g) again, and the function continues execution from where it left off until yield is encountered again...
>>> next(g) # Week by week...
>>> next(g) # If the trigger function execution does not encounter yield, no value is returned, that is, an exception is thrown after the value is retrieved to end the iteration
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>

Since the generator object belongs to the iterator, it must be possible to use the for loop to iterate, as follows:

>>> for i in countdown(3):
...     print(i)
countdown start

With the yield keyword, we have a way to implement a custom iterator. Yield can be used to return a value, but unlike return, the function ends once it encounters return, while yield can save the running state of the function and suspend the function for returning multiple values.


yield expression application

A yield that can take the form of an expression inside a function

def eater():
     print('Ready to eat')
     while True:
         print('get the food: %s, and start to eat' %food)

The generator object that can get the function persists as the send value of the function body, as follows

>>> g=eater() # get the generator object
>>> g
<generator object eater at 0x101b6e2b0>
>>> next(e) # You need to "initialize" once in advance, let the function hang at food=yield, and wait for the g.send() method to be called to pass a value to it
Ready to eat
>>> g.send('steamed stuffed bun')
get the food: steamed stuffed bun, and start to eat
>>> g.send('drumstick')
get the food: drumstick, and start to eat

For yield in the form of an expression, the generator object must be initialized once in advance, so that the function hangs at the position of food=yield, waiting to call the g.send() method to pass a value to the function body, g.send(None) is equivalent to next (g).

We can write a decorator to complete the initialization of the corresponding generator for all expression forms yield, as follows

def init(func):
    def wrapper(*args,**kwargs):
        return g
    return wrapper

def eater():
    print('Ready to eat')
    while True:
        print('get the food: %s, and start to eat' %food)

Yield in expression form can also be used to return multiple values, that is, the form of variable name=yield value, as follows

>>> def eater():
...     print('Ready to eat')
...     food_list=[]
...     while True:
...         food=yield food_list
...         food_list.append(food)
>>> e=eater()
>>> next(e)
Ready to eat
>>> e.send('steamed lamb')
['steamed lamb']
>>> e.send('Steamed bear paws')
['steamed lamb', 'Steamed bear paws']
>>> e.send('steamed deer tail')
['steamed lamb', 'Steamed bear paws', 'steamed deer tail']


Ternary expressions, list comprehensions, generator expressions

Ternary expression

Ternary expression is a solution that python provides us to simplify the code, the syntax is as follows:

res = The value to return when the condition is true if condition else The value to return when the condition is not met


res = x if x > y else y

list comprehension

List comprehension is a simplified code solution provided by python for us to quickly generate lists. The syntax is as follows

[expression for item1 in iterable1 if condition1
for item2 in iterable2 if condition2
for itemN in iterableN if conditionN


egg_list=['egg%s' %i for i in range(10)]

generator expression

There are two ways to create a generator object, one is to call a function with the yield keyword, and the other is to use a generator expression, which has the same syntax format as the list comprehension, just replace [] with (), that is :

(expression for item in iterable if condition)


>>> g=(x*x for x in range(3))
>>> g
<generator object <genexpr> at 0x101be0ba0>

Compared with list comprehensions, the advantage of generator expressions is naturally that they save memory (only one value is produced in memory at a time)

If we want to read the number of bytes of a large file, it should be done based on the generator expression

with open('db.txt','rb') as f:
    nums=(len(line) for line in f)
    total_size=sum(nums) # Execute next(nums) in sequence, and then add them together to get the result


Tags: Python

Posted by Jezthomp on Tue, 31 May 2022 11:18:35 +0530