Fluent Python—Special Methods for Sequences

Python

1. Protocols and Duck Typing

  • In the context of object-oriented programming, a protocol is an informal interface, defined only in documentation and not in code. For example, the sequence protocol in Python entails just the __len__ and __getitem__ methods. Any class Spam that implements those methods with the standard signature and semantics can be used anywhere a sequence is expected. Protocols are informal and unenforced, you can often get away with implementing just part of a protocol, if you know the specific context where a class will be used
  • This became known as duck typing: Don’t check whether it is-a duck: check whether it quacks-like-a duck, walks-like-a duck, etc., depending on exactly what subset of duck-like behavior you need to play your language-games with
  • Since Python 3.8, typing.Protocol supports static protocols, which formalize structural subtyping for static type checkers. One key difference is that static protocol implementations must provide all methods defined in the protocol class
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
from typing import Protocol
class SizedAndIndexable(Protocol):
    def __len__(self) -> int:
        ...
    def __getitem__(self, index: int) -> str:
        ...
class Words:
    def __init__(self, text):
        self._words = text.split()
    def __len__(self):
        return len(self._words)
    def __getitem__(self, index):
        return self._words[index]
class OnlyIndexable:
    def __init__(self, text):
        self._words = text.split()
    def __getitem__(self, index):
        return self._words[index]
def first_word(words: SizedAndIndexable) -> str:
    return words[0]

print(first_word(Words('hello python')))  # hello
# mypy error: Argument 1 to "first_word" has incompatible type "OnlyIndexable"; expected "SizedAndIndexable"  [arg-type]
print(first_word(OnlyIndexable('hello python')))  # hello

2. A Sliceable Sequence

  • The best practice for a sequence constructor is to take the data as an iterable argument in the constructor
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
from array import array
import reprlib
import math
class Vector:
    typecode = 'd'
    # take the data as an iterable argument
    def __init__(self, components):
        self._components = array(self.typecode, components)
    def __iter__(self):
        return iter(self._components)
    def __len__(self):
        return len(self._components)
    def __getitem__(self, index):
        return self._components[index]
    def __repr__(self):
        # safe representations of large or recursive structures by limiting the length
        components = reprlib.repr(self._components)  # array('d', [3.1, 4.2])
        components = components[components.find('['):-1]
        return f'Vector({components})'
    def __str__(self):
        return str(tuple(self))
    def __bytes__(self):
        return (bytes([ord(self.typecode)]) + bytes(self._components))
    def __eq__(self, other):
        return tuple(self) == tuple(other)
    def __abs__(self):
        # since Python 3.8, math.hypot accepts N-dimensional points
        #return math.sqrt(sum(x * x for x in self))
        return math.hypot(*self)
    def __bool__(self):
        return bool(abs(self))
    @classmethod
    def frombytes(cls, octets):
        typecode = chr(octets[0])
        memv = memoryview(octets[1:]).cast(typecode)
        # no need unpacking
        return cls(memv)

print(repr(Vector([3.1, 4.2])))  # Vector([3.1, 4.2])
print(repr(Vector((3, 4, 5))))  # Vector([3.0, 4.0, 5.0])
print(repr(Vector(range(10))))  # Vector([0.0, 1.0, 2.0, 3.0, 4.0, ...])

v1 = Vector([3, 4, 5])
print(len(v1))  # 3
print(v1[0], v1[-1])  # 3.0 5.0
print(repr(v1[:2]))  # array('d', [3.0, 4.0])

2.1. How Slicing Works

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
class MySeq:
    def __getitem__(self, index):
        return index

s = MySeq()
print(s[1])  # 1
print(s[1:4])  # slice(1, 4, None)
# start:stop:stride
print(s[1:4:2])  # slice(1, 4, 2)
print(s[1:4:2, 9])  # (slice(1, 4, 2), 9)
print(s[1:4:2, 7:9])  # (slice(1, 4, 2), slice(7, 9, None))
print(slice)  # <class 'slice'>
print(dir(slice))  # [..., 'indices', 'start', 'step', 'stop']

# S.indices(len) -> (start, stop, stride)
# this method produces "normalized" tuples of nonnegative start, stop, and stride integers tailored to a sequence of the given length
print(slice(None, 10, 2).indices(5))  # (0, 5, 2)
print(slice(-3, None, None).indices(5))  # (2, 5, 1)

2.2. A Slice-Aware __getitem__

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
class Vector:
    ...
    def __len__(self):
        return len(self._components)
    def __getitem__(self, key):
        if isinstance(key, slice):
            cls = type(self)
            # slicing a Vector should return a Vector
            return cls(self._components[key])
        # allow any of the numerous types of integers to be used as indexes, calls the __index__ special method
        index = operator.index(key)
        return self._components[index]

v7 = Vector(range(7))
print(v7[-1])  # 6.0
print(repr(v7[1:4]))  # Vector([1.0, 2.0, 3.0])
print(repr(v7[-1:]))  # Vector([6.0])
#print(v7[1, 2])  # TypeError: 'tuple' object cannot be interpreted as an integer
#print(v7[3.14])  # TypeError: 'float' object cannot be interpreted as an integer

3. Dynamic Attribute Access

  • We are now dealing with vectors that may have a large number of components, it is convenient to support shortcut names for the first few components: x, y, z, t. We could write four properties, but the __getattr__ special method provides a better way
  • The __getattr__ method is invoked by the interpreter when attribute lookup fails. In simple terms, given the expression my_obj.x, Python checks if the my_obj instance has an attribute named x; if not, the search goes to the class (my_obj.__class__), and then up the inheritance graph. If the x attribute is not found, then the __getattr__ method defined in the class of my_obj is called with self and the name of the attribute as a string (e.g., ‘x’)
  • Very often when you implement __getattr__, you need to code __setattr__ as well, to avoid inconsistent behavior in your objects
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
class Vector:
    ...
    # support positional pattern matching on the dynamic attributes supported by __getattr__
    __match_args__ = ('x', 'y', 'z', 't')
    def __getattr__(self, name):
        cls = type(self)
        try:
            pos = cls.__match_args__.index(name)
        except ValueError:
            pos = -1
        if 0 <= pos < len(self._components):
            return self._components[pos]
        raise AttributeError(f'{cls.__name__!r} object has no attribute {name!r}')
    def __setattr__(self, name, value):
        cls = type(self)
        if len(name) == 1:
            if name in cls.__match_args__:
                error = 'readonly attribute {attr_name!r}'
            elif name.islower():
                error = "can't set attributes 'a' to 'z' in {cls_name!r}"
            else:
                error = ''
            if error:
                msg = error.format(cls_name=cls.__name__, attr_name=name)
                raise AttributeError(msg)
        super().__setattr__(name, value)

v = Vector(range(5))
print(v.x, v.y, v.z, v.t)  # 0.0 1.0 2.0 3.0
print(v[0], v[1], v[2], v[3])  # 0.0 1.0 2.0 3.0
print(Vector.__match_args__)  # ('x', 'y', 'z', 't')
#print(v.k)  # AttributeError: 'Vector' object has no attribute 'k'

# __getattr__ is only a fallback. without __setattr__, v.x = 10 would create a real instance attribute and hide the dynamic v.x
#v.x = 10
#print(v.x)  # 10
#print(v)  # (0.0, 1.0, 2.0, 3.0, 4.0)

#v.x = 10  # AttributeError: readonly attribute 'x'
#v.a = 10  # AttributeError: can't set attributes 'a' to 'z' in 'Vector'
v.X = 10
print(v.X)  # 10
print(v)  # (0.0, 1.0, 2.0, 3.0, 4.0)

4. Hashing and a Faster ==

  • Vector may have thousands of components, so building a tuple only for hashing may be too costly
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
class Vector:
    ...
    def __eq__(self, other):
        # zip produces a generator of tuples made from the items in each iterable argument
        return len(self) == len(other) and all(a == b for a, b in zip(self, other))
    def __hash__(self):
        # generator expression lazily compute the hash of each component
        #hashes = (hash(x) for x in self._components)
        hashes = map(hash, self._components)
        # apply the xor operator to the hashes of every component in succession
        return functools.reduce(operator.xor, hashes, 0)

# reduce(function, iterable, initializer)
print(functools.reduce(lambda a, b: a ^ b, range(6)))  # 1
print(functools.reduce(operator.xor, range(6)))  # 1
# initializer is the value returned if the sequence is empty and is used as the first argument in the reducing loop
print(functools.reduce(operator.xor, [], 0))  # 0

# zip function is named after the zipper fastener, it stops producing values without warning as soon as one of the inputs is exhausted
print(zip(range(3), 'ABC'))  # <zip object at ...>
print(list(zip(range(3), 'ABC', [0.0, 1.1, 2.2, 3.3])))  # [(0, 'A', 0.0), (1, 'B', 1.1), (2, 'C', 2.2)]
#print(list(zip(range(3), 'ABC', [0.0, 1.1, 2.2, 3.3], strict=True)))  # ValueError: zip() argument 3 is longer than arguments 1-2
# optional fillvalue, None by default
print(list(zip_longest(range(3), 'ABC', [0.0, 1.1, 2.2, 3.3], fillvalue=-1)))  # [(0, 'A', 0.0), (1, 'B', 1.1), (2, 'C', 2.2), (-1, -1, 3.3)]
# transpose a matrix
a = [(1, 2, 3), (4, 5, 6)]
print(list(zip(*a)))  # [(1, 4), (2, 5), (3, 6)]

v1 = Vector([3, 4, 5])
v2 = Vector([3.0, 4.0, 5.0])
v3 = Vector([3, 4])
print(v1 == v2)  # True
print(v1 == v3)  # False
print(hash(v1) == hash(v2))  # True
print(hash(Vector([])))  # 0

5. Formatting

  • Instead of providing a custom display in polar coordinates, Vector will use spherical coordinates—also known as “hyperspherical” coordinates. Vector will be formatted as <r, angle1, angle2, ...>, where r is the magnitude (abs(v)), and the remaining numbers are the angular components
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
class Vector:
    ...
    # compute one of the angular coordinates
    def angle(self, n):
        r = math.hypot(*self[n:])
        a = math.atan2(r, self[n - 1])
        if (n == len(self) - 1) and (self[-1] < 0):
            return math.pi * 2 - a
        else:
            return a
    # iterable of all angular coordinates
    def angles(self):
        return (self.angle(n) for n in range(1, len(self)))
    def __format__(self, fmt_spec=''):
        if fmt_spec.endswith('h'):
            # hyperspherical coordinates
            fmt_spec = fmt_spec[:-1]
            coords = itertools.chain([abs(self)], self.angles())
            outer_fmt = '<{}>'
        else:
            # Cartesian coordinates
            coords = self
            outer_fmt = '({})'
        components = (format(c, fmt_spec) for c in coords)
        return outer_fmt.format(', '.join(components))

v1 = Vector([3, 4])
print(format(v1))  # (3.0, 4.0)
print(format(v1, '.2f'))  # (3.00, 4.00)
print(format(Vector(range(7))))  # (0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0)
print(format(Vector([1, 1]), 'h'))  # <1.4142135623730951, 0.7853981633974483>
print(format(Vector([1, 1]), '0.5fh'))  # <1.41421, 0.78540>
print(format(Vector([2, 2, 2]), '.3eh'))  # <3.464e+00, 9.553e-01, 7.854e-01>
print(format(Vector([0, 0, 0]), '0.5fh'))  # <0.00000, 0.00000, 0.00000>
print(format(Vector([2, 2, 2, 2]), '.3eh'))  # <4.000e+00, 1.047e+00, 9.553e-01, 7.854e-01>
print(format(Vector([0, 1, 0, 0]), '0.5fh'))  # <1.00000, 1.57080, 0.00000, 0.00000>

References

  • Fluent Python