Fluent Python—Data Class Builders

Python

1. Data Class Builders Overview

  • Python offers a few ways to build a simple class that is just a collection of fields, with little or no extra functionality. That pattern is known as a “data class”—and dataclasses is one of the packages that supports this pattern
  • None of the class builders discussed here depend on inheritance to do their work. Both collections.namedtuple and typing.NamedTuple build classes that are tuple subclasses. @dataclass is a class decorator that does not affect the class hierarchy in any way. Each of them uses different metaprogramming techniques to inject methods and data attributes into the class under construction
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
# 1. default class has no necessary __init__, __repr__, and __eq__ methods
class Coordinate:
    def __init__(self, lat, lon):
        self.lat = lat
        self.lon = lon

moscow = Coordinate(55.76, 37.62)
print(repr(moscow))  # <__main__.Coordinate object at 0x1057eb710>
location = Coordinate(55.76, 37.62)
print(location == moscow)  # False
print((location.lat, location.lon) == (moscow.lat, moscow.lon))  # True

# 2. collections.namedtuple class builder
from collections import namedtuple
Coordinate = namedtuple('Coordinate', 'lat lon')
print(issubclass(Coordinate, tuple))  # True
moscow = Coordinate(55.756, 37.617)
print(repr(moscow))  # Coordinate(lat=55.756, lon=37.617)
print(moscow == (55.756, 37.617))  # True

# 3. typing.NamedTuple class builder
import typing
Coordinate = typing.NamedTuple('Coordinate', [('lat', float), ('lon', float)])
#Coordinate = typing.NamedTuple('Coordinate', lat=float, lon=float)
#Coordinate = typing.NamedTuple('Coordinate', **{'lat': float, 'lon': float})
print(issubclass(Coordinate, tuple))  # True
print(typing.get_type_hints(Coordinate))  # {'lat': <class 'float'>, 'lon': <class 'float'>}

# Since Python 3.6, typing.NamedTuple can also be used in a class statement, with type annotations 
class Coordinate(typing.NamedTuple):
    # declare field with type annotation
    lat: float
    lon: float
    def __str__(self):
        ns = 'N' if self.lat >= 0 else 'S'
        we = 'E' if self.lon >= 0 else 'W'
        return f'{abs(self.lat):.1f}°{ns}, {abs(self.lon):.1f}°{we}'

# typing.NamedTuple uses the advanced functionality of a metaclass to customize the creation of the user's class
#print(issubclass(Coordinate, typing.NamedTuple))  # TypeError: issubclass() arg 2 must be a class, a tuple of classes, or a union
print(issubclass(Coordinate, tuple))  # True
print(Coordinate.__mro__)  # (<class '__main__.Coordinate'>, <class 'tuple'>, <class 'object'>)
print(Coordinate(55.756, 37.617))  # 55.8°N, 37.6°E

# 4. dataclass decorator class builder
from dataclasses import dataclass
@dataclass(frozen=True)
class Coordinate:
    lat: float
    lon: float
    def __str__(self):
        ns = 'N' if self.lat >= 0 else 'S'
        we = 'E' if self.lon >= 0 else 'W'
        return f'{abs(self.lat):.1f}°{ns}, {abs(self.lon):.1f}°{we}'

print(Coordinate.__mro__)  # (<class '__main__.Coordinate'>, <class 'object'>)
print(Coordinate(55.756, 37.617))  # 55.8°N, 37.6°E
namedtuple NamedTuple dataclass
mutable instances NO NO YES (by default, frozen=False)
class statement syntax NO YES YES
construct dict x._asdict() x._asdict() dataclasses.asdict(x)
get field names x._fields x._fields [f.name for f in dataclasses.fields(x)]
get defaults x._field_defaults x._field_defaults [f.default for f in dataclasses.fields(x)]
get field types N/A x.__annotations__ x.__annotations__
new instance with changes x._replace(…) x._replace(…) dataclasses.replace(x, …)
new class at runtime namedtuple(…) NamedTuple(…) dataclasses.make_dataclass(…)
  • The classes built by typing.NamedTuple and @dataclass have an __annotations__ attribute holding the type hints for the fields. However, reading from __annotations__ directly is not recommended. Instead, the recommended best practice to get that information is to call inspect.get_annotations(MyClass) (added in Python 3.10) or typing.get_type_hints(MyClass) (Python 3.5 to 3.9). That’s because those functions provide extra services, like resolving forward references in type hints

2. Classic Named Tuples

  • The collections.namedtuple function is a factory that builds subclasses of tuple enhanced with field names, a class name, and an informative __repr__. Classes built with namedtuple can be used anywhere where tuples are needed, and in fact many functions of the Python standard library that are used to return tuples now return named tuples for convenience, without affecting the user’s code at all
  • Each instance of a class built by namedtuple takes exactly the same amount of memory as a tuple because the field names are stored in the class
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
from collections import namedtuple
#City = namedtuple('City', ['name', 'country', 'population', 'coordinates'])
City = namedtuple('City', 'name country population coordinates')
# field values must be passed as separate positional arguments (while tuple constructor takes a single iterable)
tokyo = City('Tokyo', 'JP', 36.933, (35.689722, 139.691667))
print(tokyo)  # City(name='Tokyo', country='JP', population=36.933, coordinates=(35.689722, 139.691667))
print(tokyo.population)  # 36.933
print(tokyo.coordinates)  # (35.689722, 139.691667)
print(tokyo[1])  # JP
print(tokyo <= ('Tokyo', 'JP', 36.933, (35.689722, 139.691667)))  # True

# namedtuple offers a few attributes and methods in addition to those inherited from the tuple
print(City._fields)  # ('name', 'country', 'population', 'coordinates')
Coordinate = namedtuple('Coordinate', 'lat lon')
delhi_data = ('Delhi NCR', 'IN', 21.935, Coordinate(28.613889, 77.208889))
delhi = City._make(delhi_data)  # City(*delhi_data)
# return OrderedDict until Python 3.7. Since Python 3.8, it returns a simple dict
print(delhi._asdict())  # {'name': 'Delhi NCR', 'country': 'IN', 'population': 21.935, 'coordinates': Coordinate(lat=28.613889, lon=77.208889)}
import json
print(json.dumps(delhi._asdict()))  # {"name": "Delhi NCR", "country": "IN", "population": 21.935, "coordinates": [28.613889, 77.208889]}

# Since Python 3.7, namedtuple accepts defaults argument, providing an iterable of N default values for each of the N rightmost fields of the class
Coordinate = namedtuple('Coordinate', 'lat lon reference', defaults=[0, 'WGS84'])
print(Coordinate(0))  # Coordinate(lat=0, lon=0, reference='WGS84')
print(Coordinate._field_defaults)  # {'lon': 0, 'reference': 'WGS84'}
  • We can also add methods to a namedtuple, but it’s a hack. In Python, it’s not as common, because it doesn’t work with any built-in type—str, list, etc.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
ranks = [str(n) for n in range(2, 11)] + list('JQKA')
Card = namedtuple('Card', ['rank', 'suit'])
# add a class attribute to namedtuple
Card.suit_values = dict(spades=3, hearts=2, diamonds=1, clubs=0)
def spades_high(card):
    rank_value = ranks.index(card.rank)
    suit_value = card.suit_values[card.suit]
    return rank_value * len(card.suit_values) + suit_value
# add a function to namedtuple
Card.overall_rank = spades_high
lowest_card = Card('2', 'clubs')
highest_card = Card('A', 'spades')
print(lowest_card.overall_rank())  # 0
print(highest_card.overall_rank())  # 51

3. Typed Named Tuples

  • Classes built by typing.NamedTuple don’t have any methods beyond those that collections.namedtuple also generates—and those that are inherited from tuple. The only difference is the presence of the __annotations__ class attribute—which Python completely ignores at runtime

3.1. Type Hints Overview

  • Type hints—a.k.a. type annotations—are ways to declare the expected type of function arguments, return values, variables, and attributes, they are not enforced at all by the Python bytecode compiler and interpreter. Think about Python type hints as “documentation that can be verified by IDEs and type checkers”. That’s because type hints have no impact on the runtime behavior of Python programs
  • The basic syntax of variable annotation is var_name: some_type [= a_value]. In the context of defining a data class, these types are more likely to be useful:
    • A concrete class, for example, str or FrenchDeck
    • A parameterized collection type, like list[int], tuple[str, float], etc.
    • typing.Optional, for example, Optional[str]—to declare a field that can be a str or None
1
2
3
4
5
6
7
import typing
class Coordinate(typing.NamedTuple):
    lat: float
    lon: float = 0

trash = Coordinate('Ni!', None)
print(trash)  # Coordinate(lat='Ni!', lon=None)

3.2. Variable Annotations

  • Type hints have no effect at runtime. But at import time—when a module is loaded—Python does read them to build the __annotations__ dictionary that typing.NamedTuple and @dataclass then use to enhance the class
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
class DemoPlainClass:
    a: int
    b: float = 1.1
    c = 'spam'

# __annotations__ attribute is created by the interpreter to record the type hints that appear in the source code—even in a plain class
print(DemoPlainClass.__annotations__)  # {'a': <class 'int'>, 'b': <class 'float'>}
# a survives only as an annotation. It doesn't become a class attribute because no value is bound to it
#print(DemoPlainClass.a)  # AttributeError: type object 'DemoPlainClass' has no attribute 'a'
# b and c are stored as class attributes because they are bound to values
print(DemoPlainClass.b)  # 1.1
# c is just a plain old class attribute, not an annotation
print(DemoPlainClass.c)  # spam
# none of those three attributes will be in a new instance of DemoPlainClass
# o.a will raise AttributeError, while o.b and o.c will retrieve the class attributes
o = DemoPlainClass()
print(o.__dict__)  # {}
print(o.b, o.c)  # 1.1 spam

import typing
class DemoNTClass(typing.NamedTuple):
    a: int
    b: float = 1.1
    c = 'spam'

print(DemoNTClass.__annotations__)  # {'a': <class 'int'>, 'b': <class 'float'>}
# NamedTuple creates a and b class attributes, they are descriptors, similar to property getters, but don't require () to retrieve instance attribute
print(DemoNTClass.a)  # _tuplegetter(0, 'Alias for field number 0')
print(DemoNTClass.b)  # _tuplegetter(1, 'Alias for field number 1')
print(DemoNTClass.c)  # spam
print(DemoNTClass.__doc__)  # DemoNTClass(a, b)
# because the descriptors, a and b will work as read-only instance attributes
nt = DemoNTClass(8)
print(nt._fields)  # ('a', 'b')
print(nt.a)  # 8
print(nt.b)  # 1.1
print(nt.c)  # spam
print(nt[1])  # 1.1
#nt.a = 10  # AttributeError: can't set attribute
#nt.c = 10  # AttributeError: 'DemoNTClass' object attribute 'c' is read-only
#nt.d = 10  # AttributeError: 'DemoNTClass' object has no attribute 'd'

from dataclasses import dataclass
@dataclass
class DemoDataClass:
    a: int
    b: float = 1.1
    c = 'spam'

print(DemoDataClass.__annotations__)  # {'a': <class 'int'>, 'b': <class 'float'>}
print(DemoDataClass.__doc__)  # DemoDataClass(a: int, b: float = 1.1)
# a attribute will only exist in instances of DemoDataClass
#print(DemoDataClass.a)  # AttributeError: type object 'DemoDataClass' has no attribute 'a'
# b holding the default value for the b instance attribute
print(DemoDataClass.b)  # 1.1
print(DemoDataClass.c)  # spam
dc = DemoDataClass(9)
print(nt._fields)  # ('a', 'b')
print(dc.a)  # 9
print(dc.b)  # 1.1
print(dc.c)  # spam
print(nt[1])  # 1.1
# dataclass instances are mutable by default
dc.a = 10
dc.b = 'oops'
dc.c = 'whatever'
dc.z = 'secret stash'
print(dc.__dict__)  # {'a': 10, 'b': 'oops', 'c': 'whatever', 'z': 'secret stash'}
print(DemoDataClass.c)  # spam

4. More About @dataclass

  • The decorator accepts several keyword arguments. This is its signature: @dataclass(*, init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False)
    • The * in the first position means the remaining parameters are keyword-only
    • @dataclass emulates immutability by generating __setattr__ and __delattr__, which raise dataclass.FrozenInstanceError—a subclass of AttributeError—when the user attempts to set or delete a field
    • If the eq and frozen arguments are both True, @dataclass produces a suitable __hash__ method, so the instances will be hashable. The generated __hash__ will use data from all fields that are not individually excluded using a field option. If frozen=False, @dataclass will set __hash__ to None, signalling that the instances are unhashable, therefore overriding __hash__ from any superclass
Option Meaning Default Notes
init Generate __init__ True Ignored if __init__ is implemented by user
repr Generate __repr__ True Ignored if __repr__ is implemented by user
eq Generate __eq__ True Ignored if __eq__ is implemented by user
order Generate __lt__, __le__, __gt__, __ge__ False If True, raises exceptions if eq=False, or if any of the comparison methods that would be generated are defined or inherited
unsafe_hash Generate __hash__ False If True, force create a __hash__() method, even though it may not be safe. Otherwise, generate a __hash__() method according to how eq and frozen are set
frozen Generate __setattr__, __delattr__ False Instances will be reasonably safe from accidental change, but not really immutable

4.1. Field Options

  • The instance fields declared in a dataclass become parameters in the generated __init__. After declaring a field with a default value, all remaining fields must also have default values. Use dataclasses.field() when a field needs options beyond a simple default value. The most common option is default_factory, used to build a new mutable default value for each instance
Option Meaning Default
default Default value for field _MISSING_TYPE
default_factory 0-parameter callable used to produce a default _MISSING_TYPE
init Include field in parameters to __init__ True
repr Include field in __repr__ True
compare Use field in comparison methods True
hash Include field in __hash__ calculation None
metadata Mapping with user-defined data; ignored by @dataclass None 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
from dataclasses import dataclass, field

@dataclass
class ClubMember:
    name: str
    #guests: list = []  # ValueError: mutable default <class 'list'> for field guests is not allowed: use default_factory
    # always use a default factory to set mutable default values
    guests: list[str] = field(default_factory=list)
    athlete: bool = field(default=False, repr=False)

anna = ClubMember('Anna')
leo = ClubMember('Leo')
anna.guests.append('Bob')
print(anna)  # ClubMember(name='Anna', guests=['Bob'])
print(leo)  # ClubMember(name='Leo', guests=[])
print(anna.athlete)  # False

4.2. Post-init Processing

  • The __init__ method generated by @dataclass only takes the arguments passed and assigns them—or their default values, if missing—to the instance attributes that are instance fields. But you may need to do more than that to initialize the instance. If that’s the case, you can provide a __post_init__ method. When that method exists, @dataclass will add code to the generated __init__ to call __post_init__ as the last step. Common use cases for __post_init__ are validation and computing field values based on other fields
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
from dataclasses import dataclass, field, fields

@dataclass
class ClubMember:
    name: str
    guests: list[str] = field(default_factory=list)
@dataclass
class HackerClubMember(ClubMember):
    all_handles = set()
    handle: str = ''
    def __post_init__(self):
        # get the class of the instance
        cls = self.__class__
        if self.handle == '':
            self.handle = self.name.split()[0]
        if self.handle in cls.all_handles:
            raise ValueError(f'handle {self.handle!r} already exists.')
        cls.all_handles.add(self.handle)

print(HackerClubMember.__doc__)  # HackerClubMember(name: str, guests: list[str] = <factory>, handle: str = '')
anna = HackerClubMember('Anna Ravenscroft', handle='AnnaRaven')
print(anna)  # HackerClubMember(name='Anna Ravenscroft', guests=[], handle='AnnaRaven')
leo = HackerClubMember('Leo Rochael')
print(leo)  # HackerClubMember(name='Leo Rochael', guests=[], handle='Leo')
#HackerClubMember('Leo DaVinci')  # ValueError: handle 'Leo' already exists.

print(HackerClubMember.all_handles == {'AnnaRaven', 'Leo'})  # True
print([f.name for f in fields(HackerClubMember)])  # ['name', 'guests', 'handle']
print('all_handles' in HackerClubMember.__dict__)  # True

4.3. Typed Class Attributes

  • @dataclass decorator doesn’t care about the types in the annotations, except in two cases: if the type is ClassVar or InitVar, an instance field will not be generated for that attribute
1
2
3
4
5
6
7
from typing import ClassVar

@dataclass
class HackerClubMember(ClubMember):
    # class attribute of type set-of-str, with an empty set as its default value
    all_handles: ClassVar[set[str]] = set()
    ...

4.4. Initialization Variables

  • Sometimes you may need to pass arguments to __init__ that are not instance fields. Such arguments are called init-only variables, and will be arguments that the generated __init__ will accept, and it will be also passed to __post_init__
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
from dataclasses import dataclass, InitVar, fields

class Database:
    def lookup(self, key):
        return {'j': 20}[key]
@dataclass
class C:
    i: int
    j: int | None = None
    database: InitVar[Database | None] = None
    def __post_init__(self, database):
        if self.j is None and database is not None:
            self.j = database.lookup('j')

c = C(10, database=Database())
print(c)  # C(i=10, j=20)
print(c.__dict__)  # {'i': 10, 'j': 20}
print([f.name for f in fields(C)])  # ['i', 'j']

5. Pattern Matching Class Instances

  • Class patterns are designed to match class instances by type and—optionally—by attributes. The subject of a class pattern can be any class instance, not only instances of data classes. There are three variations of class patterns: simple, keyword, and positional
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# 1. Simple Class Patterns
# applies only to nine built-in types: bytes/dict/float/frozenset/int/list/set/str/tuple
def describe(value):
    match value:
        # matches instances of float without binding a new variable
        case float():
            return f'float: {value}'
        # binds the whole subject to name if the subject is an instance of str
        case str(name):
            return f'str: {name}'
        case [str(name), _, _, (float(lat), float(lon))]:
            return f'name: {name}, lat: {lat}, lon: {lon}'
        case _:
            return 'other'

print(describe(1.5))  # float: 1.5
print(describe('Tokyo'))  # str: Tokyo
print(describe(42))  # other
print(describe(('Tokyo', 'JP', 36.933, (35.689722, 139.691667))))  # name: Tokyo, lat: 35.689722, lon: 139.691667

# 2. Keyword and Positional Class Patterns
import typing
class City(typing.NamedTuple):
    continent: str
    name: str
    country: str
def match_cities(city):
    match city: 
        case City(name='Tokyo'):
            return city
        # cc variable is bound to the country attribute of the instance
        case City(continent='North America', country=cc):
            return cc
        # matches any City instance where the third attribute value is 'France'
        case City(_, _, 'France'):
            return city
        # combine keyword and positional arguments in a pattern
        case City('Africa', _, country=cc):
            return cc
        case _:
            return 'other'

print(match_cities(City('Asia', 'Tokyo', 'Japan')))  # City(continent='Asia', name='Tokyo', country='Japan')
print(match_cities(City('North America', 'New York', 'United States')))  # United States
print(match_cities(City('Europe', 'Paris', 'France')))  # City(continent='Europe', name='Paris', country='France')
print(match_cities(City('Africa', 'Nairobi', 'Kenya')))  # Kenya
print(match_cities(City('Oceania', 'Sydney', 'Australia')))  # other
# declares the names of the attributes in the order they will be used in positional patterns
print(City.__match_args__)  # ('continent', 'name', 'country')

References

  • Fluent Python