Fluent Python—Functions as First-Class Objects

Python

1. Callable Objects

  • Functions in Python are first-class objects. Integers, strings, and dictionaries are other examples of first-class objects in Python—nothing fancy here. Programming language researchers define a “first-class object” as a program entity that can be:
    • Created at runtime
    • Assigned to a variable or element in a data structure
    • Passed as an argument to a function
    • Returned as the result of a function
  • The call operator () may be applied to other objects besides functions. To determine whether an object is callable, use the callable() built-in function. As of Python 3.9, the data model documentation lists nine callable types:
    • User-defined functions: Created with def statements or lambda expressions
    • Built-in functions: A function implemented in C (for CPython), like len or time.strftime
    • Built-in methods: Methods implemented in C, like dict.get
    • Methods: Functions defined in the body of a class
    • Classes: When invoked, a class runs its __new__ method to create an instance, then __init__ to initialize it, and finally the instance is returned to the caller
    • Class instances: If a class defines a __call__ method, then its instances may be invoked as functions
    • Generator functions: Functions or methods that use the yield keyword in their body. When called, they return a generator object
    • Native coroutine functions: Functions or methods defined with async def. When called, they return a coroutine object. Added in Python 3.5
    • Asynchronous generator functions: Functions or methods defined with async def that have yield in their body. When called, they return an asynchronous generator for use with async for. Added in Python 3.6
  • Generators, native coroutines, and asynchronous generator functions are unlike other callables in that their return values are never application data, but objects that require further processing to yield application data or perform useful work. Generator functions return iterators. Native coroutine functions and asynchronous generator functions return objects that only work with the help of an asynchronous programming framework, such as asyncio
1
print([callable(obj) for obj in (abs, str, 'Ni!')])  # [True, True, False]

1.1. Higher-Order Functions

  • A function that takes a function as an argument or returns a function as the result is a higher-order function (e.g. map, filter, reduce, sorted)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
def factorial(n):
    """returns n!"""
    return 1 if n < 2 else n * factorial(n - 1)
print(factorial(42))  # 1405006117752879898543142606244511569936384000000000
# used to generate the help text of an object, can use help(factorial) in Python console
print(factorial.__doc__)  # returns n!
# function object is an instance of the function class
print(type(factorial))  # <class 'function'>

fact = factorial
print(fact)  # <function factorial at 0x102eec4a0>
print(fact(5))  # 120

# map and filter return generators
print(map(factorial, range(11)))  # <map object at 0x102fb3b80>
print(list(map(factorial, range(11))))  # [1, 1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800]
# listcomp or a genexp does the job of map and filter combined, but is more readable
print([factorial(n) for n in range(6)])  # [1, 1, 2, 6, 24, 120]
print(list(map(factorial, filter(lambda n: n % 2, range(6)))))  # [1, 6, 120]
print([factorial(n) for n in range(6) if n % 2])  # [1, 6, 120]

from functools import reduce
from operator import add
print(reduce(add, range(100)))  # 4950
print(sum(range(100)))  # 4950
#all(iterable)  # Returns True if there are no falsy elements in the iterable; all([]) returns True
#any(iterable)  # Returns True if any element of the iterable is truthy; any([]) returns False

1.2. Anonymous Functions

  • The lambda keyword creates an anonymous function within a Python expression. The best use of anonymous functions is in the context of an argument list for a higher-order function
  • However, the simple syntax of Python limits the body of lambda functions to be pure expressions. In other words, the body cannot contain other Python statements such as while, try, assignment with =, etc. The new assignment expression syntax using := can be used—but the lambda will be complicated and hard to read, and it should be refactored into a regular function
1
2
fruits = ['strawberry', 'fig', 'apple', 'cherry', 'raspberry', 'banana']
print(sorted(fruits, key=lambda word: word[::-1]))  # ['banana', 'apple', 'fig', 'raspberry', 'strawberry', 'cherry']

1.3. User-Defined Callable Types

  • A class implementing __call__ is an easy way to create function-like objects that have some internal state that must be kept across invocations. Another good use case for __call__ is implementing decorators. Decorators must be callable, and it is sometimes convenient to “remember” something between calls of the decorator (e.g., for memoization—caching the results of expensive computations for later use) or to split a complex implementation into separate methods. The functional approach to creating functions with internal state is to use closures
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import random
class BingoCage:
    def __init__(self, items):
        self._items = list(items)
        random.shuffle(self._items)
    def pick(self):
        try:
            return self._items.pop()
        except IndexError:
            raise LookupError('pick from empty BingoCage')
    def __call__(self):
        return self.pick()
bingo = BingoCage(range(3))
print(bingo.pick())  # 2
print(bingo())  # 0
print(callable(bingo))  # True

2. From Positional to Keyword-Only Parameters

  • One of the best features of Python functions is the extremely flexible parameter handling mechanism. Closely related are the use of * and ** to unpack iterables and mappings into separate arguments when we call a function
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# name keyword-only arguments after the argument prefixed with *
def tag(name, *content, class_=None, **attrs):
    """Generate one or more HTML tags"""
    if class_ is not None:
        attrs['class'] = class_
    attr_pairs = (f' {attr}="{value}"' for attr, value in sorted(attrs.items()))
    attr_str = ''.join(attr_pairs)
    if content:
        elements = (f'<{name}{attr_str}>{c}</{name}>' for c in content)
        return '\n'.join(elements)
    else:
        return f'<{name}{attr_str} />'
print(tag('br'))  # <br />
# any number of arguments after the first are captured by *content as a tuple
print(tag('p', 'hello'))  # <p>hello</p>
print(tag('p', 'hello', 'world'))  # <p>hello</p>\n<p>world</p>
# keyword arguments not explicitly named are captured by **attrs as a dict
print(tag('p', 'hello', id=33))  # <p id="33">hello</p>
# class_ can only be passed as a keyword argument
print(tag('p', 'hello', 'world', class_='sidebar'))  # <p class="sidebar">hello</p>\n<p class="sidebar">world</p>
print(tag(content='testing', name="img"))  # <img content="testing" />
my_tag = {'name': 'img', 'title': 'Sunset Boulevard', 'src': 'sunset.jpg', 'class': 'framed'}
print(tag(**my_tag))  # <img class="framed" src="sunset.jpg" title="Sunset Boulevard" />

# put a * by itself in the signature to specify keyword-only arguments
def f(a, *, b):
    return a, b
print(f(1, b=2))  # (1, 2)
#print(f(1, 2))  # TypeError: f() takes 1 positional argument but 2 were given

# all arguments to the left of the / are positional-only (Since Python 3.8)
def divmod(a, b, /):
    return (a // b, a % b)
print(divmod(10, 3))  # (3, 1)
#print(divmod(a=10, b=3))  # TypeError: divmod() got some positional-only arguments passed as keyword arguments: 'a, b'

3. Packages for Functional Programming

  • The operator module provides function equivalents for dozens of operators so you don’t have to code trivial functions
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
# functions defined in operator
import operator
print([name for name in dir(operator) if not name.startswith('_')])
# ['abs', 'add', 'and_', 'attrgetter', 'call', 'concat', 'contains', 'countOf', 'delitem', 'eq', 'floordiv', 
# 'ge', 'getitem', 'gt', 'iadd', 'iand', 'iconcat', 'ifloordiv', 'ilshift', 'imatmul', 'imod', 'imul', 
# 'index', 'indexOf', 'inv', 'invert', 'ior', 'ipow', 'irshift', 'is_', 'is_not', 'isub', 'itemgetter', 
# 'itruediv', 'ixor', 'le', 'length_hint', 'lshift', 'lt', 'matmul', 'methodcaller', 'mod', 'mul', 'ne', 
# 'neg', 'not_', 'or_', 'pos', 'pow', 'rshift', 'setitem', 'sub', 'truediv', 'truth', 'xor']

# arithmetic operator
import functools
def factorial(n):
    return functools.reduce(lambda a, b: a*b, range(1, n+1))
def factorial(n):
    return functools.reduce(operator.mul, range(1, n+1))

# itemgetter uses the [] operator, it supports sequences, mappings and any class that implements __getitem__
data = [
    ('A', 2, 1, (11, 111)),
    ('B', 1, 2, (33, 333)),
    ('C', 1, 1, (22, 222)),
]
for item in sorted(data, key=operator.itemgetter(2, 1)):
    print(item)  # CAB

# attrgetter extracts object attributes by name, support nested structure
from collections import namedtuple
Tup = namedtuple('Tup', 'v1 v2')
Data = namedtuple('Data', 'nm f1 f2 tup')
datas = [Data(nm, f1, f2, Tup(v1, v2)) for nm, f1, f2, (v1, v2) in data]
print(datas[0])  # Data(nm='A', f1=2, f2=1, tup=Tup(v1=11, v2=111))
print(datas[0].tup.v2)  # 111
nm_v2 = operator.attrgetter('nm', 'tup.v2')
for item in sorted(datas, key=operator.attrgetter('tup.v2')):
    print(nm_v2(item))  # ACB

# methodcaller creates a function by name on the object given as argument
s = 'The time has come'
upcase = operator.methodcaller('upper')
print(upcase(s))  # THE TIME HAS COME
hyphenate = operator.methodcaller('replace', ' ', '-')
print(hyphenate(s))  # The-time-has-come

# partial takes a callable as first argument, followed by an arbitrary number of positional and keyword arguments to bind
triple = functools.partial(operator.mul, 3)
print(triple(7))  # 21

import unicodedata
nfc = functools.partial(unicodedata.normalize, 'NFC')
s1 = 'café'
s2 = 'cafe\u0301'
print(s1, s2)  # café café
print(s1 == s2)  # False
print(nfc(s1) == nfc(s2))  # True

def tag(name, *content, class_=None, **attrs):
    return (name, content, class_, attrs)
picture = functools.partial(tag, 'img', class_='pic-frame')
print(picture(src='wumpus.jpeg'))  # ('img', (), 'pic-frame', {'src': 'wumpus.jpeg'})
print(picture)  # functools.partial(<function tag at 0x1092cede0>, 'img', class_='pic-frame')
print(picture.func)  # <function tag at 0x1092cede0>
print(picture.args)  # ('img',)
print(picture.keywords)  # {'class_': 'pic-frame'}

References

  • Fluent Python