Fluent Python—Unicode Text Versus Bytes

Python

1. Characters and Bytes

  • Python 3 introduced a sharp distinction between strings of human text and sequences of raw bytes. Implicit conversion of byte sequences to Unicode text is a thing of the past

1.1. Character Issues

  • A string is a sequence of characters. The best definition of “character” is Unicode character. Accordingly, the items we get out of a Python 3 str are Unicode characters, it is pretty much the Python 2 unicode type with a new name—not the raw bytes we got from a Python 2 str
  • The Unicode standard explicitly separates the identity of characters from specific byte representations:
    • The identity of a character—its code point—is a number from 0 to 1,114,111, shown in the Unicode standard as 4 to 6 hex digits with a “U+” prefix, from U+0000 to U+10FFFF. For example, the code point for the letter A is U+0041, the Euro sign is U+20AC. About 13% of the valid code points have characters assigned to them in Unicode 13.0.0, the standard used in Python 3.10.0b4
    • The actual bytes that represent a character depend on the encoding in use. An encoding is an algorithm that converts code points to byte sequences and vice versa. The code point for the letter A (U+0041) is encoded as the single byte \x41 in the UTF-8 encoding, or as the bytes \x41\x00 in UTF-16LE encoding. As another example, UTF-8 requires three bytes—\xe2\x82\xac—to encode the Euro sign (U+20AC), but in UTF-16LE the same code point is encoded as two bytes: \xac\x20
  • Converting from code points to bytes is encoding; converting from bytes to code points is decoding
1
2
3
4
5
6
7
8
9
s = 'café'
print(len(s))  # 4
# encode str to bytes using UTF-8 encoding
b = s.encode('utf8')
# the code point for "é" is encoded as two bytes in UTF-8
print(b)  # b'caf\xc3\xa9'
print(len(b))  # 5
# decode bytes to str using UTF-8 encoding
print(b.decode('utf8'))  # 'café'

1.2. Byte Essentials

  • The new binary sequence types are unlike the Python 2 str in many regards. There are two basic built-in types for binary sequences: the immutable bytes type introduced in Python 3 and the mutable bytearray, added way back in Python 2.6. Each item in bytes or bytearray is an integer from 0 to 255, and not a one-character string like in the Python 2 str. However, a slice of a binary sequence always produces a binary sequence of the same type—including slices of length 1
  • Although binary sequences are really sequences of integers, their literal notation reflects the fact that ASCII text is often embedded in them. Therefore, four different displays are used, depending on each byte value:
    • For bytes with decimal codes 32 to 126—from space to ~ (tilde)—the ASCII character itself is used
    • For bytes corresponding to tab, newline, carriage return, and , the escape sequences \t, \n, \r, and \\ are used
    • If both string delimiters ‘ and “ appear in the byte sequence, the whole sequence is delimited by ‘, and any ‘ inside are escaped as \‘
    • For other byte values, a hexadecimal escape sequence is used (e.g., \x00 is the null byte)
  • Both bytes and bytearray support every str method except those that do formatting (format, format_map) and those that depend on Unicode data, including casefold, isdecimal, isidentifier, isnumeric, isprintable, and encode. This means that you can use familiar string methods like endswith, replace, strip, translate, upper, and dozens of others with binary sequences—only using bytes and not str arguments. In addition, the regular expression functions in the re module also work on binary sequences, if the regex is compiled from a binary sequence instead of a str. Since Python 3.5, the % operator works with binary sequences again
  • Binary sequences have a class method fromhex, which builds a binary sequence by parsing pairs of hex digits optionally separated by spaces. The other ways of building bytes or bytearray instances are calling their constructors with:
    • A str and an encoding keyword argument
    • An iterable providing items with values from 0 to 255
    • An object that implements the buffer protocol (e.g., bytes, bytearray, memoryview, array.array) that copies the bytes from the source object to the newly created binary sequence
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
s = 'café'
cafe = bytes(s, encoding='utf_8')
print(cafe)  # b'caf\xc3\xa9'
print(cafe[0])  # 99
print(cafe[:1])  # b'c'
print(cafe[0] == cafe[:1])  # False
print(s[0] == s[:1])  # True
cafe_arr = bytearray(cafe)
print(cafe_arr)  # bytearray(b'caf\xc3\xa9')
print(cafe_arr[-1:])  # bytearray(b'\xa9')

b = b"She said \"it's fine\""
print(b)  # b'She said "it\'s fine"'
print(b.replace(b'She', b'He'))  # b'He said "it\'s fine"'
import re
print(re.findall(rb'(\w+) said "it\'s (\w+)"', b))  # [(b'She', b'fine')]
print(b'%s said "it\'s %s"' % (b'She', b'fine'))  # b'She said "it\'s fine"'

print(bytes.fromhex('48 65 6C 6C 6F'))  # b'Hello'
print(bytes(b'Hello'))  # b'Hello'
print(bytes('Hello', encoding='ascii'))  # b'Hello'
print(bytes([72, 101, 108, 108, 111]))  # b'Hello'

# building a binary sequence from a buffer-like object is a low-level operation that may involve type casting
import array
numbers = array.array('h', [-2, -1, 0, 1, 2])  # short integers (16 bits)
print(bytes(numbers))  # b'\xfe\xff\xff\xff\x00\x00\x01\x00\x02\x00'
# building a binary sequence from a buffer-like object will always copy the bytes
mv_numbers = memoryview(numbers)
mv_bytes = bytes(mv_numbers)
numbers[0] = 999
print(numbers.tolist())  # [999, -1, 0, 1, 2]
print(mv_numbers.tolist())  # [999, -1, 0, 1, 2]
print(list(mv_bytes))  # [254, 255, 255, 255, 0, 0, 1, 0, 2, 0]

print(b'\x00' * 5)  # b'\x00\x00\x00\x00\x00'
print(bytes(5))  # b'\x00\x00\x00\x00\x00'
print(bytes([5]))  # b'\x05'
print(bytearray(5))  # bytearray(b'\x00\x00\x00\x00\x00')

1.3. Basic Encoders/Decoders

  • The Python distribution bundles more than 100 codecs (encoder/decoders) for text to byte conversion and vice versa. Each codec has a name, like ‘utf_8’, and often aliases, such as ‘utf8’, ‘utf-8’, and ‘U8’, which you can use as the encoding argument in functions like open(), str.encode(), bytes.decode(), and so on. There are some representative encodings:
    • latin1 a.k.a. iso8859_1: It is the basis for other encodings, such as cp1252 and Unicode itself
    • cp1252: A useful latin1 superset created by Microsoft, adding useful symbols like curly quotes and € (euro); some Windows apps call it “ANSI,” but it was never a real ANSI standard
    • cp437: The original character set of the IBM PC, with box drawing characters. Incompatible with latin1, which appeared later
    • gb2312: Legacy standard to encode the simplified Chinese ideographs used in mainland China; one of several widely deployed multibyte encodings for Asian languages
    • utf-8: The most common 8-bit encoding on the web
    • utf-16le: One form of the UTF 16-bit encoding scheme; all UTF-16 encodings support code points beyond U+FFFF through escape sequences called “surrogate pairs”
char. code point ascii latin1 cp1252 cp437 gb2312 utf-8 utf-16le
A U+0041 41 41 41 41 41 41 41 00
¿ U+00BF * BF BF A8 * C2 BF BF 00
à U+00C3 * C3 C3 * * C3 83 C3 00
á U+00E1 * E1 E1 A0 A8 A2 C3 A1 E1 00
Ω U+03A9 * * * EA A6 B8 CE A9 A9 03
ڿ U+06BF * * * * * DA BF BF 06
“ U+201C * * 93 * A1 B0 E2 80 9C 1C 20
€ U+20AC * * 80 * * E2 82 AC AC 20
┌ U+250C * * * DA A9 B0 E2 94 8C 0C 25
气 U+6C14 * * * * C6 F8 E6 B0 94 14 6C
氣 U+6C23 * * * * * E6 B0 A3 23 6C
𝄞 U+1D11E * * * * * F0 9D 84 9E 34 D8 1E DD

2. Encode/Decode Problems

  • Although there is a generic UnicodeError exception, the error reported by Python is usually more specific: either a UnicodeEncodeError or a UnicodeDecodeError. Loading Python modules may also raise SyntaxError when the source encoding is unexpected

2.1. Coping with UnicodeEncodeError

  • Most non-UTF codecs handle only a small subset of the Unicode characters. When converting text to bytes, if a character is not defined in the target encoding, UnicodeEncodeError will be raised, unless special handling is provided by passing an errors argument to the encoding method or function
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
city = 'São Paulo'
print(city.encode('utf_8'))  # b'S\xc3\xa3o Paulo'
print(city.encode('utf_16'))  # b'\xff\xfeS\x00\xe3\x00o\x00 \x00P\x00a\x00u\x00l\x00o\x00'
print(city.encode('iso8859_1'))  # b'S\xe3o Paulo'
#print(city.encode('cp437'))  # UnicodeEncodeError: 'charmap' codec can't encode character '\xe3' in position 1: character maps to <undefined>
print(city.encode('cp437', errors='ignore'))  # b'So Paulo'
print(city.encode('cp437', errors='replace'))  # b'S?o Paulo'
print(city.encode('cp437', errors='xmlcharrefreplace'))  # b'S&#227;o Paulo'

# check whether the Unicode text is 100% pure ASCII
print(city.isascii())  # False

# the codecs error handling is extensible
def replace_with_question_mark(exception):
    if isinstance(exception, UnicodeEncodeError):
        replacement = b'[?]'
        return (replacement, exception.end)
    else:
        raise exception
codecs.register_error('my_custom_error', replace_with_question_mark)
print(city.encode('cp437', errors='my_custom_error'))  # b'S[?]o Paulo'

2.2. Coping with UnicodeDecodeError

  • Not every byte holds a valid ASCII character, and not every byte sequence is valid UTF-8 or UTF-16; therefore, when you assume one of these encodings while converting a binary sequence to text, you will get a UnicodeDecodeError if unexpected bytes are found
  • On the other hand, many legacy 8-bit encodings like ‘cp1252’, ‘iso8859_1’, and ‘koi8_r’ are able to decode any stream of bytes, including random noise, without reporting errors. Therefore, if your program assumes the wrong 8-bit encoding, it will silently decode garbage. Garbled characters are known as gremlins or mojibake
1
2
3
4
5
6
octets = b'Montr\xe9al'
print(octets.decode('cp1252'))  # Montréal
print(octets.decode('iso8859_7'))  # Montrιal
print(octets.decode('koi8_r'))  # MontrИal
#print(octets.decode('utf_8'))  # UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 5: invalid continuation byte
print(octets.decode('utf_8', errors='replace'))  # Montr�al

2.3. SyntaxError When Loading Modules

  • UTF-8 is the default source encoding for Python 3 across all platforms, just as ASCII was the default for Python 2. If you load a .py module containing non-UTF-8 data and no encoding declaration, you get a SyntaxError. To fix this problem, add a magic coding comment at the top of the file
1
2
3
4
5
# save ola.py with GBK encoding
# 1. print("你好")
#import ola  # SyntaxError: (unicode error) 'utf-8' codec can't decode byte 0xc4 in position 0: invalid continuation byte
# 2. # coding: GBK\nprint("你好")
import ola

2.4. Discover the Encoding

  • You can’t find the encoding of a byte sequence. You must be told. Some communication protocols and file formats, like HTTP and XML, contain headers that explicitly tell us how the content is encoded. You can be sure that some byte streams are not ASCII because they contain byte values over 127, and the way UTF-8 and UTF-16 are built also limits the possible byte sequences
  • Human languages also have their rules and restrictions, once you assume that a stream of bytes is human plain text, it may be possible to sniff out its encoding using heuristics and statistics. Chardet is a Python library to guess one of more than 30 supported encodings
1
2
3
4
5
6
7
import chardet
with open('fluent/04/ola.py', 'rb') as f:
    raw_data = f.read()
result = chardet.detect(raw_data)
print(result['encoding'])  # GB18030
print(result['confidence'])  # 0.95
print(result.get('language'))  # zh
  • The way UTF-8 was designed, it’s almost impossible for a random sequence of bytes, or even a nonrandom sequence of bytes coming from a non-UTF-8 encoding, to be decoded accidentally as garbage in UTF-8, instead of raising UnicodeDecodeError. The reasons for this are that UTF-8 escape sequences never use ASCII characters, and these escape sequences have bit patterns that make it very hard for random data to be valid UTF-8 by accident. So if you can decode some bytes containing codes > 127 as UTF-8, it’s probably UTF-8
  • A Byte Order Mark (BOM) is a special sequence of bytes placed at the beginning of a text file to indicate the byte order (endianness) for multi-byte encodings such as UTF-16. Since UTF-16 stores each character using two bytes, computers may store those bytes in little-endian or big-endian order, and the BOM tells the decoder which order to use. In Python, the default utf_16 encoding automatically includes a BOM, while utf_16le and utf_16be explicitly specify the byte order and therefore do not include one
1
2
3
4
5
6
s = 'El Niño'
# BOM (b'\xff\xfe') indicates little-endian UTF-16
print(s.encode('utf_16'))  # b'\xff\xfeE\x00l\x00 \x00N\x00i\x00\xf1\x00o\x00'
print(s.encode('utf_16le'))  # b'E\x00l\x00 \x00N\x00i\x00\xf1\x00o\x00'
print(list(s.encode('utf_16le')))  # [69, 0, 108, 0, 32, 0, 78, 0, 105, 0, 241, 0, 111, 0]
print(list(s.encode('utf_16be')))  # [0, 69, 0, 108, 0, 32, 0, 78, 0, 105, 0, 241, 0, 111]

3. Handling Text Files

  • The best practice for handling text I/O is the “Unicode sandwich”: Decode bytes to str as early as possible when reading input; Work only with str in the business logic of the program; Encode str to bytes as late as possible when writing output
  • The open() built-in does the necessary decoding when reading and encoding when writing files in text mode, so all you get from my_file.read() and pass to my_file.write(text) are str objects
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
fp = open('cafe.txt', 'w', encoding='utf_8')
print(fp)  # <_io.TextIOWrapper name='cafe.txt' mode='w' encoding='utf_8'>
# return the number of Unicode characters written
print(fp.write('café'))  # 4
fp.close()

import os
# file has 5 bytes
print(os.stat('cafe.txt').st_size)  # 5

# if the encoding argument is omitted, the locale default encoding will be used
fp2 = open('cafe.txt')
print(fp2)  # <_io.TextIOWrapper name='cafe.txt' mode='r' encoding='UTF-8'>
print(fp2.read())  # café

# always pass an explicit encoding argument, because the default may change from one machine to the next
fp3 = open('cafe.txt', encoding='utf_8')
print(fp3)  # <_io.TextIOWrapper name='cafe.txt' mode='r' encoding='utf_8'>
print(fp3.read())  # café

# open file in binary mode (should only use binary mode to open binary files)
fp4 = open('cafe.txt', 'rb')
print(fp4)  # <_io.BufferedReader name='cafe.txt'>
print(fp4.read())  # b'caf\xc3\xa9'

3.1. Beware of Encoding Defaults

  • Several settings affect the encoding defaults for I/O in Python. The default may be UTF-8 on macOS/Linux, but may be a legacy code page such as cp1252 on Windows, and the behavior may also change when stdin/stdout/stderr are redirected. The best advice about encoding defaults is: do not rely on them. Always be explicit about encodings in your programs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
import locale
import sys
expressions = """
    locale.getpreferredencoding()
    type(my_file)
    my_file.encoding
    sys.stdout.isatty()
    sys.stdout.encoding
    sys.stdin.isatty()
    sys.stdin.encoding
    sys.stderr.isatty()
    sys.stderr.encoding
    sys.getdefaultencoding()
    sys.getfilesystemencoding()
"""
my_file = open('dummy', 'w')
for expression in expressions.split():
    value = eval(expression)
    print(f'{expression:>30} -> {value!r}')
#  locale.getpreferredencoding() -> 'UTF-8'  # open() uses it as the default encoding for text files when the encoding argument is omitted
#                  type(my_file) -> <class '_io.TextIOWrapper'>
#               my_file.encoding -> 'UTF-8'
#            sys.stdout.isatty() -> True  # False if redirect to a file, and encoding will use locale.getpreferredencoding()
#            sys.stdout.encoding -> 'utf-8'  # also depends on the font the console is using (should have the glyph to display it)
#             sys.stdin.isatty() -> True
#             sys.stdin.encoding -> 'utf-8'
#            sys.stderr.isatty() -> True
#            sys.stderr.encoding -> 'utf-8'
#       sys.getdefaultencoding() -> 'utf-8'  # used internally by Python for implicit conversions between bytes and str, do not support change
#    sys.getfilesystemencoding() -> 'utf-8'  # used for filenames, not file contents

import sys
from unicodedata import name
print(sys.version)
print()
print('sys.stdout.isatty():', sys.stdout.isatty())
print('sys.stdout.encoding:', sys.stdout.encoding)
print()
test_chars = [
    # write the official name of the character inside the \N{} for Unicode literals, raises SyntaxError if the name doesn't exist
    '\N{HORIZONTAL ELLIPSIS}', # `…` exists in cp1252, not in cp437
    '\N{INFINITY}', # `∞` exists in cp437, not in cp1252
    '\N{CIRCLED NUMBER FORTY TWO}', # `㊷` not in cp437 or in cp1252
]
for char in test_chars:
    print(f'Trying to output {name(char)}:')
    print(char)

4. Normalizing Unicode for Reliable Comparisons

  • String comparisons are complicated by the fact that Unicode has combining characters: diacritics and other marks that attach to the preceding character, appearing as one when printed. For example, placing U+0301 after “e” renders “é”. In the Unicode standard, sequences like ‘é’ and ‘e\u0301’ are called “canonical equivalents”. But Python sees two different sequences of code points, and considers them not equal
  • The solution is unicodedata.normalize(). The first argument to that function is one of four strings:
    • NFC: Normalization Form Canonical Composition, composes the code points to produce the shortest equivalent string
    • NFD: Normalization Form Canonical Decomposition, expanding composed characters into base characters and separate combining characters
    • NFKC: Normalization Form Compatibility Composition, replacing compatibility characters (some characters appear more than once for compatibility with preexisting standards) with preferred representations, even if there is some formatting loss
    • NFKD: Normalization Form Compatibility Decomposition, similar to NFKC
    • Keyboard drivers usually generate composed characters, so text typed by users will be in NFC by default. NFC and NFD are safe for most comparisons. NFKC and NFKD may replace characters with visually or semantically related alternatives, so they can lose or distort information. But they can produce convenient intermediate representations for searching and indexing
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
from unicodedata import normalize, name

s1 = 'café'
s2 = 'cafe\N{COMBINING ACUTE ACCENT}'
print(s1, s2)  # café café
print(len(s1), len(s2))  # 4 5
print(s1 == s2)  # False

print(len(normalize('NFC', s1)), len(normalize('NFC', s2)))  # 4 4
print(len(normalize('NFD', s1)), len(normalize('NFD', s2)))  # 5 5
print(normalize('NFC', s1) == normalize('NFC', s2))  # True
print(normalize('NFD', s1) == normalize('NFD', s2))  # True

# some single characters are normalized by NFC into another single character
ohm = '\u2126'
ohm_c = normalize('NFC', ohm)
print(ohm, ohm_c)  # Ω Ω
print(name(ohm))  # OHM SIGN
print(name(ohm_c))  # GREEK CAPITAL LETTER OMEGA
print(ohm == ohm_c)  # False
print(normalize('NFC', ohm) == normalize('NFC', ohm_c))  # True

# compatibility normalization may change the representation more aggressively
half = '\N{VULGAR FRACTION ONE HALF}'
print(half)  # ½
print(normalize('NFKC', half))  # 1⁄2
for char in normalize('NFKC', half):
    print(char, name(char), sep='\t')
# 1       DIGIT ONE
# ⁄       FRACTION SLASH  # not the familiar 47 character slash /
# 2       DIGIT TWO

four_squared = '4²'
print(normalize('NFKC', four_squared))  # 42

micro = 'µ'
micro_kc = normalize('NFKC', micro)
print(micro, micro_kc)  # µ μ
print(name(micro))  # MICRO SIGN
print(name(micro_kc))  # GREEK SMALL LETTER MU
print(ord(micro), ord(micro_kc))  # 181 956

4.1. Case Folding

  • Case folding is essentially converting all text to lowercase, with some additional transformations. It is supported by str.casefold(). For any string s containing only latin1 characters, s.casefold() produces the same result as s.lower(), with only two exceptions—the micro sign ‘µ’ and the German Eszett ‘ß’. But casefold() is more aggressive and is better for case-insensitive Unicode comparisons
1
2
3
4
5
6
7
8
9
10
11
12
13
14
from unicodedata import name

micro = 'µ'
micro_cf = micro.casefold()
print(micro, micro_cf)  # µ μ
print(name(micro))  # MICRO SIGN
print(name(micro_cf))  # GREEK SMALL LETTER MU
print(micro.lower() == micro.casefold())  # False

eszett = 'ß'
eszett_cf = eszett.casefold()
print(eszett, eszett_cf)  # ß ss
print(name(eszett))  # LATIN SMALL LETTER SHARP S
print(eszett.lower() == eszett.casefold())  # False

4.2. Utility Functions for Text Matching

  • NFC is the best normalized form for most applications. str.casefold() is the way to go for case-insensitive comparisons. We can combine them into small utility functions for Unicode string matching
1
2
3
4
5
6
7
8
9
10
11
12
13
from unicodedata import normalize
def nfc_equal(str1, str2):
    return normalize('NFC', str1) == normalize('NFC', str2)
def fold_equal(str1, str2):
    return normalize('NFC', str1).casefold() == normalize('NFC', str2).casefold()

s1 = 'café'
s2 = 'cafe\u0301'
print(s1 == s2)  # False
print(nfc_equal(s1, s2))  # True
print(nfc_equal('A', 'a'))  # False
print(fold_equal(s1, s2))  # True
print(fold_equal('A', 'a'))  # True

4.3. Taking Out Diacritics

  • Removing diacritics is not a proper form of Unicode normalization, because it may change the meaning of words. But it can be useful for searching, indexing, and building readable URLs. The basic idea is: decompose text with NFD, remove combining marks, then recompose with NFC
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
import unicodedata
import string

def shave_marks(txt):
    """Remove all diacritic marks"""
    norm_txt = unicodedata.normalize('NFD', txt)
    shaved = ''.join(c for c in norm_txt if not unicodedata.combining(c))
    return unicodedata.normalize('NFC', shaved)
order = '“Herr Voß: • ½ cup of Œtker™ caffè latte • bowl of açaí.”'
print(shave_marks(order))  # “Herr Voß: • ½ cup of Œtker™ caffe latte • bowl of acai.”
greek = 'Ζέφυρος, Zéfiro'
print(shave_marks(greek))  # Ζεφυρος, Zefiro

# often the reason to remove diacritics is to change Latin text to pure ASCII
def shave_marks_latin(txt):
    """Remove all diacritic marks from Latin base characters"""
    norm_txt = unicodedata.normalize('NFD', txt)
    latin_base = False
    preserve = []
    for c in norm_txt:
        if unicodedata.combining(c) and latin_base:
            continue
        preserve.append(c)
        if not unicodedata.combining(c):
            latin_base = c in string.ascii_letters
    shaved = ''.join(preserve)
    return unicodedata.normalize('NFC', shaved)
print(shave_marks_latin(greek))  # Ζέφυρος, Zefiro

# replace Western typographical symbols with ASCII equivalents, remove Latin diacritics, and apply NFKC
single_map = str.maketrans("""‚ƒ„ˆ‹‘’“”•–—˜›""", """'f"^<''""---~>""")  # char-to-char replacement
multi_map = str.maketrans({
    '€': 'EUR',
    '…': '...',
    'Æ': 'AE',
    'æ': 'ae',
    'Œ': 'OE',
    'œ': 'oe',
    '™': '(TM)',
    '‰': '<per mille>',
    '†': '**',
    '‡': '***',
})
multi_map.update(single_map)
def dewinize(txt):
    """Replace Win1252 symbols with ASCII chars or sequences"""
    return txt.translate(multi_map)
def asciize(txt):
    no_marks = shave_marks_latin(dewinize(txt))
    no_marks = no_marks.replace('ß', 'ss')
    return unicodedata.normalize('NFKC', no_marks)
print(dewinize(order))  # "Herr Voß: - ½ cup of OEtker(TM) caffè latte - bowl of açaí."
print(asciize(order))  # "Herr Voss: - 1⁄2 cup of OEtker(TM) caffe latte - bowl of acai."

4.4. Sorting Unicode Text

  • Python sorts sequences of any type by comparing the items in each sequence one by one. For strings, this means comparing the code points. Unfortunately, this produces unacceptable results for non-ASCII characters. The standard way to sort non-ASCII text in Python is to use the locale.strxfrm function which transforms a string to one that can be used in locale-aware comparisons. However, there are some caveats:
    • You need to set locale before using locale.strxfrm as the key when sorting
    • Locale settings are global, calling setlocale in a library is not recommended. Your application or framework should set the locale when the process starts, and should not change it afterward
    • The locale must be installed and correctly implemented by the OS, so locale-aware sorting can be hard to deploy reliably
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
fruits = ['caju', 'atemoia', 'cajá', 'açaí', 'acerola']
print(sorted(fruits))  # ['acerola', 'atemoia', 'açaí', 'caju', 'cajá']

import locale
try:
    my_locale = locale.setlocale(locale.LC_COLLATE, 'pt_BR.UTF-8')
except locale.Error as exc:
    print(exc)  # unsupported locale setting
else:
    print(my_locale)  # pt_BR.UTF-8
    print(sorted(fruits, key=locale.strxfrm))  # ['açaí', 'acerola', 'atemoia', 'cajá', 'caju']

# pyuca library is a pure-Python implementation of the Unicode Collation Algorithm (UCA)
# pyuca has one sorting algorithm that does not respect the sorting order in individual languages
import pyuca
coll = pyuca.Collator()
fruits = ['caju', 'atemoia', 'cajá', 'açaí', 'acerola']
print(sorted(fruits, key=coll.sort_key))  # ['açaí', 'acerola', 'atemoia', 'cajá', 'caju']

5. The Unicode Database

  • The Unicode standard provides a database—in the form of several structured text files—that maps code points to character names and metadata, such as whether a character is printable, a letter, a decimal digit, or another numeric symbol. That’s how the str methods isalpha, isprintable, isdecimal, and isnumeric work. str.casefold also uses information from a Unicode table
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
import re
import sys
import unicodedata

chars = ['A', 'é', 'π', '①', '€', '\N{INFINITY}']
for char in chars:
    # unicodedata.category() returns the two-letter category of char from the Unicode database
    print(char, unicodedata.name(char), unicodedata.category(char), sep='\t')
# A       LATIN CAPITAL LETTER A  Lu
# é       LATIN SMALL LETTER E WITH ACUTE Ll
# π       GREEK SMALL LETTER PI   Ll
# ①       CIRCLED DIGIT ONE       No
# €       EURO SIGN       Sc
# ∞       INFINITY        Sm

print('abc'.isalpha())  # True
print('①'.isdecimal())  # False
print('①'.isnumeric())  # True

START, END = ord(' '), sys.maxunicode + 1
def find(*query_words, start=START, end=END):
    query = {w.upper() for w in query_words}
    for code in range(start, end):
        # get the Unicode character for code
        char = chr(code)
        # get the name of the character, or None if the code point is unassigned
        name = unicodedata.name(char, None)
        if name and query.issubset(name.split()):
            print(f'U+{code:04X}\t{char}\t{name}')
find('cat', 'face', start=0x1F400, end=0x1FA00)
# U+1F431 🐱      CAT FACE
# U+1F638 😸      GRINNING CAT FACE WITH SMILING EYES
# U+1F639 😹      CAT FACE WITH TEARS OF JOY
# U+1F63A 😺      SMILING CAT FACE WITH OPEN MOUTH
# U+1F63B 😻      SMILING CAT FACE WITH HEART-SHAPED EYES
# U+1F63C 😼      CAT FACE WITH WRY SMILE
# U+1F63D 😽      KISSING CAT FACE WITH CLOSED EYES
# U+1F63E 😾      POUTING CAT FACE
# U+1F63F 😿      CRYING CAT FACE
# U+1F640 🙀      WEARY CAT FACE

re_digit = re.compile(r'\d')
sample = '1\xbc\xb2\u0969\u136b\u216b\u2466\u2480\u3285'
for char in sample:
    print(f'U+{ord(char):04X}',
          char.center(6),
          're_dig' if re_digit.match(char) else '-',
          'isdec' if char.isdecimal() else '-',
          'isdig' if char.isdigit() else '-',
          'isnum' if char.isnumeric() else '-',
          f'{unicodedata.numeric(char):5.2f}',
          unicodedata.name(char),
          sep='\t')
# U+0031    1     re_dig  isdec   isdig   isnum    1.00   DIGIT ONE
# U+00BC    ¼     -       -       -       isnum    0.25   VULGAR FRACTION ONE QUARTER
# U+00B2    ²     -       -       isdig   isnum    2.00   SUPERSCRIPT TWO
# U+0969    ३     re_dig  isdec   isdig   isnum    3.00   DEVANAGARI DIGIT THREE
# U+136B    ፫     -       -       isdig   isnum    3.00   ETHIOPIC DIGIT THREE
# U+216B    Ⅻ     -       -       -       isnum   12.00   ROMAN NUMERAL TWELVE
# U+2466    ⑦     -       -       isdig   isnum    7.00   CIRCLED DIGIT SEVEN
# U+2480    ⒀     -       -       -       isnum   13.00   PARENTHESIZED NUMBER THIRTEEN
# U+3285    ㊅    -       -       -       isnum    6.00   CIRCLED IDEOGRAPH SIX

6. Dual-Mode str and bytes APIs

  • Some standard library functions accept both str and bytes arguments, but behave differently depending on the type. The re and os modules are important examples
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import re
# if re is built with str, patterns such as \d and \w match Unicode digits or letters beyond ASCII
re_numbers_str = re.compile(r'\d+')
re_words_str = re.compile(r'\w+')
# if re is built with bytes, those patterns only match ASCII bytes
re_numbers_bytes = re.compile(rb'\d+')
re_words_bytes = re.compile(rb'\w+')
text_str = ("Ramanujan saw \u0be7\u0bed\u0be8\u0bef as 1729 = 1³ + 12³ = 9³ + 10³.")
text_bytes = text_str.encode('utf_8')
print(f'{text_str!r}')  # 'Ramanujan saw ௧௭௨௯ as 1729 = 1³ + 12³ = 9³ + 10³.'
print(re_numbers_str.findall(text_str))  # ['௧௭௨௯', '1729', '1', '12', '9', '10']
print(re_numbers_bytes.findall(text_bytes))  # [b'1729', b'1', b'12', b'9', b'10']
print(re_words_str.findall(text_str))  # ['Ramanujan', 'saw', '௧௭௨௯', 'as', '1729', '1³', '12³', '9³', '10³']
print(re_words_bytes.findall(text_bytes))  # [b'Ramanujan', b'saw', b'as', b'1729', b'1', b'12', b'9', b'10']
# for str regular expressions, there is a re.ASCII flag that makes \w, \W, \b, \B, \d, \D, \s, and \S perform ASCII-only matching
re_numbers_str = re.compile(r'\d+', re.ASCII)
print(re_numbers_str.findall(text_str))  # ['1729', '1', '12', '9', '10']

import os
dirname = 'unicode_demo'
filename = 'digits-of-π.txt'
# all os module functions that accept filenames or pathnames take arguments as str or bytes
# if you pass str, Python encodes/decodes filenames with sys.getfilesystemencoding()
os.makedirs(dirname, exist_ok=True)
open(os.path.join(dirname, filename), 'w', encoding='utf_8').close()
print(os.listdir(dirname))  # ['digits-of-π.txt']
# if you pass bytes, os functions return bytes filenames
print(os.listdir(os.fsencode(dirname)))  # [b'digits-of-\xcf\x80.txt']
# os.fsencode() and os.fsdecode() help convert filenames or pathnames manually
name_bytes = os.listdir(os.fsencode(dirname))[0]
print(name_bytes)  # b'digits-of-\xcf\x80.txt'
print(os.fsdecode(name_bytes))  # digits-of-π.txt
print(os.fsencode(filename))  # b'digits-of-\xcf\x80.txt'
os.remove(os.path.join(dirname, filename))
os.rmdir(dirname)

References

  • Fluent Python