Fluent Python—Object References, Mutability, and Recycling

Python

1. Identity, Equality, and Aliases

  • Every Python object has an identity, a type, and a value. Only the value of an object may change over time. Python variables are like reference variables in Java, we can think of variables as labels with names attached to objects
  • The real meaning of an object’s ID is implementation dependent. In CPython, id() returns the memory address of the object, but it may be something else in another Python interpreter. The key point is that the ID is guaranteed to be a unique integer label, and it will never change during the life of the object
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
a = [1, 2, 3]
b = a  # attaches the label b to the object that already has the label a
a.append(4)
print(b)  # [1, 2, 3, 4]

charles = {'name': 'Charles L. Dodgson', 'born': 1832}
lewis = charles
print(lewis is charles)  # True (identity comparison)
print(id(charles), id(lewis))  # 4298217536 4298217536
lewis['balance'] = 950
print(charles)  # {'name': 'Charles L. Dodgson', 'born': 1832, 'balance': 950}
alex = {'name': 'Charles L. Dodgson', 'born': 1832, 'balance': 950}
print(alex == charles)  # True (__eq__)
print(alex is not charles)  # True

# checking for None is the only common use case for the is operator
x = None
print(x is None)  # True

# what can never change in a tuple is the identity of the items it contains
t1 = (1, 2, [30, 40])
t2 = (1, 2, [30, 40])
print(t1 == t2)  # True
print(id(t1[-1]))  # 4389904320
t1[-1].append(99)
print(t1)  # (1, 2, [30, 40, 99])
print(id(t1[-1]))  # 4389904320
print(t1 == t2)  # False

2. Deep and Shallow Copies

2.1. Copies Are Shallow by Default

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# using the constructor or [:] produces a shallow copy 
l1 = [3, [55, 44], (7, 8, 9)]
l2 = list(l1)
#l2 = l1[:]
print(l1 == l2)  # True
print(l1 is l2)  # False
l1.append(100)
l1[1].remove(55)
print(l1)  # [3, [44], (7, 8, 9), 100]
print(l2)  # [3, [44], (7, 8, 9)]
l2[1] += [33, 22]
l2[2] += (10, 11)
print(l1)  # [3, [44, 33, 22], (7, 8, 9), 100]
print(l2)  # [3, [44, 33, 22], (7, 8, 9, 10, 11)]

2.2. Copy Arbitrary Objects

  • The copy module provides the deepcopy and copy functions that return deep and shallow copies of arbitrary objects
  • A deep copy may be too deep in some cases. For example, objects may refer to external resources or singletons that should not be copied. You can control the behavior of both copy and deepcopy by implementing the __copy__() and __deepcopy__() special methods
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
class Driver:
    pass
ALEX = Driver()
class Bus:
    def __init__(self, passengers=None):
        self.passengers = list(passengers) if passengers else []
        self.driver = ALEX
    def pick(self, name):
        self.passengers.append(name)
    def drop(self, name):
        self.passengers.remove(name)
    def __deepcopy__(self, memo):
        # create uninitialized instance, avoids calling __init__
        new = Bus.__new__(Bus)
        memo[id(self)] = new
        # deepcopy mutable state
        new.passengers = copy.deepcopy(self.passengers, memo)
        # share the singleton
        new.driver = self.driver
        return new

import copy
bus1 = Bus(['Alice', 'Bill', 'Claire', 'David'])
bus2 = copy.copy(bus1)
bus3 = copy.deepcopy(bus1)
print(id(bus1), id(bus2), id(bus3))  # 4448273040 4448273106 4448273232
print(id(bus1.passengers), id(bus2.passengers), id(bus3.passengers))  # 4448272896 4448272896 4448171451
print(id(bus1.driver), id(bus2.driver), id(bus3.driver))  # 4382786704 4382786704 4382786704
bus1.drop('Bill')
print(bus2.passengers)  # ['Alice', 'Claire', 'David']
print(bus3.passengers)  # ['Alice', 'Bill', 'Claire', 'David']

# deepcopy can gracefully handle cyclic references
a = [10, 20]
b = [a, 30]
a.append(b)
print(a)  # [10, 20, [[...], 30]]
c = copy.deepcopy(a)
print(c)  # [10, 20, [[...], 30]]

2.3. Function Parameters as References

  • The only mode of parameter passing in Python is call by sharing, which means that each formal parameter of the function gets a copy of each reference in the arguments. In other words, the parameters inside the function become aliases of the actual arguments
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
def f(a, b):
    a += b
    return a
x = 1
y = 2
print(f(x, y))  # 3
print(x, y)  # 1 2
a = [1, 2]
b = [3, 4]
print(f(a, b))  # [1, 2, 3, 4]
print(a, b)  # [1, 2, 3, 4] [3, 4]
t = (10, 20)
u = (30, 40)
print(f(t, u))  # (10, 20, 30, 40)
print(t, u)  # (10, 20) (30, 40)

# avoid mutable objects as default values for parameters
class HauntedBus:
    def __init__(self, passengers=[]):
        self.passengers = passengers
    def pick(self, name):
        self.passengers.append(name)
    def drop(self, name):
        self.passengers.remove(name)
passengers = ['Alice', 'Bill']
bus1 = HauntedBus(passengers)
print(bus1.passengers)  # ['Alice', 'Bill']
bus1.pick('Charlie')
bus1.drop('Alice')
print(bus1.passengers)  # ['Bill', 'Charlie']
print(passengers)  # ['Bill', 'Charlie']
bus2 = HauntedBus()
bus2.pick('Carrie')
print(bus2.passengers)  # ['Carrie']
bus3 = HauntedBus()
print(bus3.passengers)  # ['Carrie']
bus3.pick('Dave')
print(bus2.passengers)  # ['Carrie', 'Dave']
print(bus2.passengers is bus3.passengers)  # True
print(bus1.passengers)  # ['Bill', 'Charlie']
print(HauntedBus.__init__.__defaults__)  # (['Carrie', 'Dave'],)
print(HauntedBus.__init__.__defaults__[0] is bus2.passengers)  # True

2.4. Tricks Python Plays with Immutables

  • For a tuple t, t[:] does not make a copy, but returns a reference to the same object. You also get a reference to the same tuple if you write tuple(t). The same behavior can be observed with instances of str, bytes, and frozenset. Note that fs[:] does not work if fs is a frozenset. But fs.copy() has the same effect
  • For immutable literals (like tuples, strings, and some integers), CPython often reuses objects as an optimization. For mutable objects (lists, dictionaries, sets), it must create a new object each time
1
2
3
4
5
6
7
8
9
10
t1 = (1, 2, 3)
t2 = tuple(t1)
print(t2 is t1)  # True
t3 = t1[:]
print(t3 is t1)  # True
t4 = (1, 2, 3)
print(t4 is t1)  # True
s1 = 'ABC'
s2 = 'ABC'
print(s2 is s1)  # True

3. del and Garbage Collection

  • del is not a function, it’s a statement. We write del x and not del(x)—although the latter also works, but only because the expressions x and (x) usually mean the same thing in Python
  • del deletes references, not objects. Python’s garbage collector may discard an object from memory as an indirect result of del, if the deleted variable was the last reference to the object. Rebinding a variable may also cause the number of references to an object to reach zero, causing its destruction
  • In CPython, the primary algorithm for garbage collection is reference counting. Essentially, each object keeps count of how many references point to it. As soon as that refcount reaches zero, the object is immediately destroyed: CPython calls the __del__ method on the object (if defined) and then frees the memory allocated to the object. In CPython 2.0, a generational garbage collection algorithm was added to detect groups of objects involved in reference cycles—which may be unreachable even with outstanding references to them, when all the mutual references are contained within the group. Other implementations of Python have more sophisticated garbage collectors that do not rely on reference counting, which means the __del__ method may not be called immediately when there are no more references to the object
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
a = [1, 2]
b = a
del a
print(b)  # [1, 2]
b = [3]

import weakref
s1 = {1, 2, 3}
s2 = s1
def bye():
    print('...like tears in the rain.')
# finalize holds a weak reference to {1, 2, 3}, do not increase its reference count
ender = weakref.finalize(s1, bye)
print(ender.alive)  # True
del s1
print(ender.alive)  # True
s2 = 'spam'  # ...like tears in the rain.
print(ender.alive)  # False

References

  • Fluent Python