程序代写代做代考 scheme c# file system interpreter Java flex Haskell python crawler c++ database Excel data structure javascript chain algorithm The Coder’s Apprentice

12.4.4 remove()

remove() allows you to remove an element from a list. The element you wish to remove is
given as argument. If the element occurs in the list multiple times, only the first occurrence
will be removed. If you try to remove an element that is not on the list, a runtime error is
generated.

fruitlist = [“apple”, “banana”, “cherry”, “banana”, “durian”]
print( fruitlist )
fruitlist.remove( “banana” )
print( fruitlist )

12.4.5 pop()

Like remove(), pop() removes an element from the list, but does so by index. It has one
optional argument, which is the index of the element that you wish to remove. If you do
not provide that argument, pop() removes the last element from the list. If the index is
beyond the boundaries of the list, pop() generates a runtime error.

A major difference with remove() is that pop() actually has a return value, namely the
element that gets removed. This allows you to quickly process all the elements of a list,
while emptying the list at the same time.

fruitlist = [“apple”, “banana”, “cherry”, “durian”]
print( fruitlist )
print( fruitlist.pop() )
print( fruitlist )
print( fruitlist.pop( 0 ) )
print( fruitlist )

12.4.6 del

del is neither a method nor a function, but since it is often metioned in one breath with
remove() and pop(), I place it here. del is a keyword that allows you to delete a list
element, or list slice, by index. It is similar to pop() in functionality, but does not have a
return value. Also, pop() cannot be used on slices. To remove one element from a list, use
del []. To remove a slice, use del [:].

152 Chapter 12. Lists

fruitlist = [“apple”, “banana”, “cherry”, “banana”, “durian”]
del fruitlist[3]
print( fruitlist )

12.4.7 index()

index() returns the index of the first occurrence on the list of the element that is given to
index() as argument. A runtime error is generated if the element is not found on the list.

fruitlist = [“apple”, “banana”, “cherry”, “banana”, “durian”]
print( fruitlist.index( “banana” ) )

12.4.8 count()

count() returns an integer that indicates how often the element that is passed to it as an
argument occurs in the list.

fruitlist = [“apple”, “banana”, “cherry”, “banana”, “durian”]
print( fruitlist.count( “banana” ) )

12.4.9 sort()

sort() sorts the elements of the list, from low to high. If the elements of the list are strings,
it does an alphabetical sort. If the elements are numbers, it does a numeric sort. If the
elements are mixed, it generates a runtime error, unless certain arguments are given.

listing1203.py

fruitlist = [“apple”, “strawberry”, “banana”, “raspberry”,
“cherry”, “banana”, “durian”, “blueberry”]

fruitlist.sort()
print( fruitlist )

numlist = [314, 315, 642, 246, 129, 999]
numlist.sort()
print( numlist )

To do a reverse sort (for instance, sorting numerical items from high to low, or sorting
strings alphabetically from “z” to “a”), you can add an argument reverse=.

fruitlist = [“apple”, “strawberry”, “banana”, “raspberry”,
“cherry”, “banana”, “durian”, “blueberry”]

fruitlist.sort( reverse=True )
print( fruitlist )

12.4. List methods 153

Another argument that you can give sort() is a key. You have to provide this argument
as .sort( key= ), whereby is a function that takes one argument (the
element that is to be sorted) and returns a value that is used as key. A typical use for
the key argument is if you want to sort a list of strings, but want to do the sorting case-
insensitively. So as key you want to use the elements, but in lower case, i.e., you want
to apply the function str.lower() to the element. You call the sort() method as in the
following example:

listing1204.py

fruitlist = [“apple”, “Strawberry”, “banana”, “raspberry”,
“CHERRY”, “banana”, “durian”, “blueberry”]

fruitlist.sort()
print( fruitlist )
fruitlist.sort( key=str.lower ) # case-insensitive sort
print( fruitlist )

Note that for the key argument, you do not place parentheses after the function name. This
is not a function call, it is an argument that tells Python which function to use to generate
the key. You can write your own function to be used as key. For example, in the code below,
numlist is sorted with the digits reversed:

listing1205.py

def revertdigits( item ):
return (item%10) *100 + (int(item/10)%10) *10 + int(item/100)

numlist = [314, 315, 642, 246, 129, 999]
numlist.sort( key=revertdigits )
print( numlist )

Here is another example, that sorts a list of strings by length, then alphabetical order:

listing1206.py

def len_alphabetical( element ):
return len( element ), element

fruitlist = [“apple”, “strawberry”, “banana”, “raspberry”,
“cherry”, “banana”, “durian”, “blueberry”]

fruitlist.sort( key=len_alphabetical )
print( fruitlist )

Note that the len_alphabetical() function returns a tuple. When two tuples are com-
pared, first the first elements of both tuples are compared, and if they are equal, the second
elements are compared. You can use this knowledge to create a key function which sorts a
mixed list, e.g., a list which consists of both strings and numbers. Just let the key function
return a tuple of which the first item indicates the type of the element represented by a
number, and the second the element itself.

listing1206a.py

154 Chapter 12. Lists

def mixed_key( element ):
if isinstance( element, str ):

return 1, element
return 0, element

mixedlist = [“apple”, 0, “strawberry”, 5, “banana”, 2, \
“raspberry”, 9, “cherry”, “banana”, 7, 7, 6, “blueberry”]
mixedlist.sort( key=mixed_key )
print( mixedlist )

At this point I can give a typical example of the use of “anonymous functions,” which I
introduced in Chapter 8. Using an anonymous function to specify the key for the sort()
method keeps the code for the key next to where you call the sort(), instead of elsewhere
in the program. This may improve readability.

listing1207.py

fruitlist = [“apple”, “strawberry”, “banana”, “raspberry”,
“cherry”, “banana”, “durian”, “blueberry”]

fruitlist.sort( key=lambda x: (len(x),x) )
print( fruitlist )

12.4.10 reverse()

reverse() simply puts the elements of the list in reverse order.

fruitlist = [“apple”,”strawberry”,”banana”,”raspberry”,”durian”]
fruitlist.reverse()
print( fruitlist )

12.4.11 Practice

Exercise Write a program that asks the user to enter some data, for instance the names
of their friends. When the user wants to stop providing inputs, he just presses Enter. The
program then displays an alphabetically sorted list of the data items entered. Do not just
print the list, but print each item separately, on a different line.

Exercise Sort a list of numbers using their absolute values; use the abs() function as key.

Exercise Count how often each letter occurs in a string (case-insensitively). You can
ignore every character that is not a letter. Store the counts in a list of 26 items that all start
at zero. Print the resulting counts. As index you can use ord(letter) – ord(“a”), where
letter is a lower case letter (the ord() function is explained in Chapter 10).

12.5. Aliasing 155

12.5 Aliasing

If you assign a variable that contains a list to another variable, you might expect that you
create a copy of the list in the second variable. But you are not doing that. You are actually
creating an alias for the list, i.e., a new variable that is referring to the same list. This means
that the new variable can be treated as a list, but any change that you make to the list it
refers to, is visible in the original list variable, and vice versa. They are not different lists.

listing1208.py

fruitlist = [“apple”, “banana”, “cherry”, “durian”]
newfruitlist = fruitlist
print( fruitlist )
print( newfruitlist )
newfruitlist[2] = “orange”
print( fruitlist )
print( newfruitlist )

Every variable in Python has an identification number. You can see it with the id() func-
tion. The ID number indicates which memory spot the variable refers to. For an alias of a
list, the ID is the same as for the original list.

listing1209.py

fruitlist = [“apple”, “banana”, “cherry”, “durian”]
newfruitlist = fruitlist
print( id( fruitlist ) )
print( id( newfruitlist ) )

If you want to create a copy of a list, you can do so using a little trick. Instead of using
= , you use the command = [:].

listing1210.py

fruitlist = [“apple”, “banana”, “cherry”, “durian”]
newfruitlist = fruitlist
verynewfruitlist = fruitlist[:]

print( id( fruitlist ) )
print( id( newfruitlist ) )
print( id( verynewfruitlist ) )

fruitlist[2] = “orange”
print( fruitlist )
print( newfruitlist )
print( verynewfruitlist )

12.5.1 is

The keyword is is introduced to compare the identities of two variables.

156 Chapter 12. Lists

listing1211.py

fruitlist = [“apple”, “banana”, “cherry”, “durian”]
newfruitlist = fruitlist
verynewfruitlist = fruitlist[:]

print( fruitlist is newfruitlist )
print( fruitlist is verynewfruitlist )
print( newfruitlist is verynewfruitlist )

As you can see, the keyword is manages to determine that fruitlist and newfruitlist
are aliases, but that verynewfruitlist is not the same list. If you compare them with the
== operator, the results are not the same as comparing them with is:

listing1212.py

fruitlist = [“apple”, “banana”, “cherry”, “durian”]
newfruitlist = fruitlist
verynewfruitlist = fruitlist[:]

print( fruitlist == newfruitlist )
print( fruitlist == verynewfruitlist )
print( newfruitlist == verynewfruitlist )

The == operator actually compares the contents of the lists, so it returns True for all com-
parisons. For data types for which == is not defined, it executes an identity comparison,
but for lists it has been defined as a comparison of the contents. I will return to this topic
when discussing “operator overloading” in Chapter 21. .

12.5.2 Shallow vs. deep copies

If (some of) the items of your list are lists themselves (or other mutable data structures
which are introduced in the next chapters), you may get problems if you copy the list
using the = [:] syntax. The reason is that such a copy is a “shallow
copy,” which means that it copies each of the elements of the list with a regular assignment,
which entails that the items in the list that are lists themselves become aliases of the items
on the original list.

listing1213.py

numlist = [ 1, 2, [3, 4] ]
copylist = numlist[:]

numlist[0] = 5
numlist[2][0] = 6
print( numlist )
print( copylist )

In the code above, you can see that the assignment numlist[0] = 5 only has an effect on
numlist, as copylist contains a copy of numlist. However, since this is a shallow copy,

12.5. Aliasing 157

the assignment to numlist[2][0] has an effect on both lists, as the sublist [3, 4] is stored
in copylist as an alias.

If you want to create a “deep copy” of a list (i.e., a copy that also contains true copies of all
mutable substructures of the list, which in turn contain true copies of all their mutable sub-
structures, etcetera), then you can use the copy module for that. The deepcopy() function
from the copy module allows you to create deep copies of any mutable data structure.

listing1214.py

from copy import deepcopy

numlist = [ 1, 2, [3, 4] ]
copylist = deepcopy( numlist )

numlist[0] = 5
numlist[2][0] = 6
print( numlist )
print( copylist )

Note that the copy module also contains a function copy() that makes shallow copies. If
you wonder why that function is included as you can easily create shallow copies of lists
with the = [:] command: the copy module not only works for lists,
but for any mutable data structure. Not for all such data structures there exist shortcuts to
create shallow copies.

12.5.3 Passing lists as arguments

When you pass a list as an argument to a function, this is a “pass by reference.” The
parameter that the function has access to will be an alias for the list that you pass. This
means that a function that you pass a list to, can actually change the contents of the list.

This is important, so I repeat it: when you pass a mutable data structure to a function,
this is a “pass by reference,” meaning that the data structure is passed as an alias and the
function can change the contents of the data structure.

You have to know whether a function that you pass a list to will or will not change the list.
If you do not want the function to change the list, and you do not know if it will, you best
pass a deep copy of the list to the function.

listing1215.py

def changelist( x ):
if len( x ) > 0:

x[0] = “CHANGE!”

fruitlist = [“apple”, “banana”, “cherry”, “durian”]
changelist( fruitlist )
print( fruitlist )

The reason that a list is “passed by reference” and not “by value” is that technically, every
argument that is passed to a function must be stored in the computer in a specific block of

158 Chapter 12. Lists

memory that is part of the processor. This is called the “stack,” and it is pretty limited in
size. Since lists can be really long, allowing a program to place a list on the stack would
cause all kinds of annoying runtime errors. In Python, as in most other programming
languages, for the most part only basic data types (such as integers, floats, and strings) are
passed by value.

12.6 Nested lists

The elements of a list may be lists themselves (which also may contains lists, etcetera). This
is a good way to create a matrix in a program. For instance, you can create a Tic-Tac-Toe
board, where a dash (-) represents an empty cell, as follows:

board = [ [“-“, “-“, “-“], [“-“, “-“, “-“], [“-“, “-“, “-“] ]

The first row of the board is represented by board[0], the second row by board[1], and
the third row by board[2]. If you want to access the first cell of the first row, that is
board[0][0], the second cell is board[0][1] and the third cell is board[0][2]. For ex-
ample, the following code places an “X” in the middle of the board, and an “O” in the
upper right corner. It also displays the board in a nice way (with markers for rows and
columns around it).

listing1216.py

def display_board( b ):
print( ” 1 2 3″ )
for row in range( 3 ):

print( row+1, end=” “)
for col in range( 3 ):

print( b[row][col], end=” ” )
print()

board = [ [“-“, “-“, “-“], [“-“, “-“, “-“], [“-“, “-“, “-“] ]
board[1][1] = “X”
board[0][2] = “O”
display_board( board )

12.7 List casting

You can type cast a sequence of elements to a list using the list() function. The code
below turns a tuple into a list.

t1 = ( “apple”, “banana”, “cherry” )
print( t1 )
print( type( t1 ) )
fruitlist = list( t1 )
print( fruitlist )
print( type( fruitlist ) )

12.8. List comprehensions 159

This is sometimes necessary, in particular when you have an “iterator” available and you
want to use the elements in a list format. An iterator is a function that generates a se-
quence (more on iterators is given in Chapter 23). An example of an iterator that I already
discussed is the range() function. The range() function generates a sequence of numbers.
If you want to use these numbers as a list, you can use list casting.

numlist = range( 1, 11 )
print( numlist )
numlist = list( range( 1, 11 ) )
print( numlist )

You can turn a string into a list of its characters by using a list casting on the string.

12.8 List comprehensions
List comprehensions are a concise way to create lists. They are typical for Python, but you
do not find them in many other programming languages. They are not actually needed,
as you can use functions to achieve the same effect, but as they are often used in examples
(especially by people who want to show off their Python abilities to create short statements
that have extensive effects), I thought it prudent to discuss them. If you are never going to
use them in your own code, that is fine as they are completely optional. But you should be
able to recognize them in other people’s code.

Suppose that you want to create a list consisting of the squares of the numbers 1 to 25. A
function that creates such a list is:

def squareslist():
squares = []
for i in range( 1, 26 ):

squares.append( i *i )
return squares

sl = squareslist()
print( sl )

In Python, you can create that list with one single statement, namely as follows:

sl = [ x *x for x in range( 1, 26 ) ]
print( sl )

Now suppose that you want to create this list, but want to leave out (for some reason) the
squares of any numbers that end in 5. That would add at least two lines to the function
above, but with list comprehensions you can still do it with that single line:

sl = [ x *x for x in range( 1, 26 ) if x%10 != 5]
print( sl )

A list comprehension consists of an expression in square brackets, followed by a for clause,
followed by zero or more for and/or if clauses. The result is a list that contains the

160 Chapter 12. Lists

elements that result from evaluating the expression for the combination of the for and
if clauses.

The results can become quite complex. For instance, here is a list comprehension that
creates a list of tuples with three integers between 1 and 4, whereby the three integers are
all different:

listing1217.py

triplelist = [ (x,y,z) for x in range( 1, 5 )
for y in range( 1, 5 ) for z in range( 1, 5 )
if x != y if x != z if y != z]

print( triplelist )

If you find list comprehensions hard to use, remember that there is absolutely no reason
to use them except for keeping code concise, and that keeping code readable and under-
standable is far more important than keeping it concise.

What you learned

In this chapter, you learned about:

• Lists

• Mutability of lists

• Using + and ∗ with lists
• List methods append(), extend(), insert(), remove(), pop(), index(), count(),

sort(), and reverse()

• delwith lists

• Aliasing

• The keyword is

• Creating list copies

• Creating deep copies of lists using deepcopy()

• Using lists as arguments

• Nested lists

• List casting

• List comprehensions

Exercises

Exercise 12.1 A magic 8-ball, when asked a question, provides a random answer from a
list. The code below contains a list of possible answers. Create a magic 8-ball program that
asks a question, then gives a random answer.

Exercises 161

exercise1201.py

answers = [ “It is certain”, “It is decidedly so”, “Without a \
doubt”, “Yes, definitely”, “You may rely on it”, “As I see it, \
yes”, “Most likely”, “Outlook good”, “Yes”, “Signs point to yes”,
“Reply hazy try again”, “Ask again later”, “Better not tell you \
now”, “Cannot predict now”, “Concentrate and ask again”, “Don ‘ t \
count on it”, “My reply is no”, “My sources say no”, “Outlook \
not so good”, “Very doubtful” ]

Exercise 12.2 A playing card consists of a suit (“Hearts”, “Spades”, “Clubs”,
“Diamonds”) and a value (2, 3, 4, 5, 6, 7, 8, 9, 10, “Jack”, “Queen”, “King”,
“Ace”). Create a list of all possible playing cards, which is a deck. Then create a func-
tion that shuffles the deck, producing a random order.

Exercise 12.3 A first-in-first-out (FIFO) structure, also called a “queue,” is a list that gets
new elements added at the end, while elements from the front are removed and processed.
Write a program that processes a queue. In a loop, ask the user for input. If the user just
presses the Enter key, the program ends. If the user enters anything else, except for a
single question mark (?), the program considers what the user entered a new element and
appends it to the queue. If the user enters a single question mark, the program pops the
first element from the queue and displays it. You have to take into account that the user
might type a question mark even if the queue is empty.

Exercise 12.4 Count how often each letter occurs in a string (case-insensitively). You can
ignore every character that is not a letter. Print the letters with their counts, in order from
highest count to lowest count.

Exercise 12.5 The sieve of Eratosthenes is a method to find all prime numbers between
1 and a given number using a list. This works as follows: Fill the list with the sequence of
numbers from 1 to the highest number. Set the value of 1 to zero, as 1 is not prime. Now
loop over the list. Find the next number on the list that is not zero, which, at the start, is
the number 2. Now set all multiples of this number to zero. Then find the next number on
the list that is not zero, which is 3. Set all multiples of this number to zero. Then the next
number, which is 5 (because 4 has already been set to zero), and do the same thing again.
Process all the numbers of the list in this way. When you have finished, the only numbers
left on the list are primes. Use this method to determine all the primes between 1 and 100.

Exercise 12.6 Write a Tic-Tac-Toe program that allows two people to play the game
against each other. In turn, ask each player which row and column they want to play. Make
sure that the program checks if that row/column combination is empty. When a player has
won, end the game. When the whole board is full and there is no winner, announce a draw.

This is a fairly long program to write (60 lines or so). It will definitely help to use some
functions. I recommend that you create a function display_board() that gets the board
as parameter and displays it, a function getRowCol() that asks for a row or a column (de-
pending on a parameter) and checks whether the user entered a legal value, and a function

162 Chapter 12. Lists

winner() that gets the board as argument and checks if there is a winner. Keep track of
who the current player is using a global variable player that you can pass to a function as
an argument if the function needs it. I also use a function opponent(), that takes the player
as argument and returns the opponent. I use that to switch players after each move.

The main program will be something along the lines of (in pseudo-code):

display board

while True:

ask for row

ask for column

if row/column combination already occupied:

display error message

continue

place player marker on row/column combination

display board

if there is a winner:

announce winner

break

if the board is full:

announce draw

break

switch players

Exercise 12.7 Create a program that is a simplified version of the game “Battleship.”
The computer creates (in memory) a grid that is 4 cells wide and 3 cells high. The rows of
the grid are numbered 1 to 3, and the columns of the grid are labeled A to D. The computer
hides a battleship in three random cells in the grid. Each battleship occupies exactly one
cell. Battleships are not allowed to touch each other horizontally or vertically. Make sure
that the program places the battleships randomly, so not pre-configured.

The computer asks the player to “shoot” at cells of the grid. The player does so by entering
the column letter and row number of the cell which he wants to shoot at (e.g., “D3”). If the
cell which the player shoots at contains nothing, the computer responds with “Miss!” If
the cell contains a battleship, the computer responds with “You sunk my battleship!” and
removes the battleship from the cell (i.e., a second shot at the same cell is a miss). As soon
as the player hits the last battleship, the computer responds with displaying how many
shots the player needed to shoot down all three battleships, and the program ends.

To help with debugging the game, at the start the computer should display the grid with
periods marking empty cells and Xs marking cells with battleships.

Hint: If you have troubles with this exercise, start by using a board which has the bat-
tleships already placed. Once the rest of the code works, add a function that places the
battleships at random, at first without checking if they are touching one another. Once that
works, add code that disallows battleships touching each other.

Exercise 12.8 The “subset sum” problem asks the question whether a list of integers
contains a subset of integers that, when summed, gives zero as answer. For instance, for
the list [1, 4, -3, -5, 7] the answer is “yes,” as 1 + 4− 5 = 0. However, for the list
[1, 4, -3, 7] the answer is “no,” as there is no subset of integers that adds up to zero.

Exercises 163

Write a program that solves the “subset sum” problem for a list of integers. If there is a
solution, print it; if not, report that there is no solution.

Hint: This problem is tackled best using recursion. If you skipped Chapter 9, you better
skip this exercise too.

164 Chapter 12. Lists

Chapter 13

Dictionaries

Strings, tuples and lists are ordered data structures, which entails that they can be indexed.
Not all data is naturally ordered, which is why Python offers dictionaries as a way to struc-
ture unordered data.

13.1 Basics of dictionaries

Dictionaries are unordered collections of elements. To identify an element, you have to
know the element’s “key.”

Basically, dictionaries store “key-value pairs.” Any immutable data type can function as a
key. A very common type to use as key is the string.

You create dictionaries using curly brackets {}, similar to how you create lists using square
brackets. An empty dictionary you create by assigning {} to a variable. You can create a
dictionary with contents by describing every element of the dictionary between the curly
brackets using the syntax :, and commas between the elements.

Here a dictionary fruitbasket is created, that contains three key-value pairs, namely the key
“apple”with value 3, the key “banana”with value 5, and the key “cherry”with value 50.

fruitbasket = { “apple”:3, “banana”:5, “cherry”:50 }

To access the value belonging to a specific key, you use the same syntax as you would use
for a list, except that you write the key in the place where you would write the index for a
list.

fruitbasket = { “apple”:3, “banana”:5, “cherry”:50 }
print( fruitbasket[“banana”] )

You can use a for loop to traverse a dictionary. The variable in the for loop gets access to
all the keys.

166 Chapter 13. Dictionaries

listing1301.py

fruitbasket = { “apple”:3, “banana”:5, “cherry”:50 }
for key in fruitbasket:

print( “{}:{}”.format( key, fruitbasket[key] ))

Trying to access a dictionary element using a key that is not available in the dictionary will
lead to a runtime error. However, adding a new element to a dictionary you can do by
simply assigning a value to the dictionary item identified by the new key. For instance,
adding a “mango” to the fruitbasket you can do as follows:

fruitbasket = { “apple”:3, “banana”:5, “cherry”:50 }
print( fruitbasket )
fruitbasket[“mango”] = 1
print( fruitbasket )

Overwriting a dictionary item works in exactly the same way as creating a new dictionary
item: you just assign a value to it.

Deleting an item from a dictionary you do using the del keyword, just as you can use with
lists.

fruitbasket = { “apple”:3, “banana”:5, “cherry”:50 }
print( fruitbasket )
del fruitbasket[“banana”]
print( fruitbasket )

You can determine the number of key-value pairs in a dictionary by using the len() func-
tion.

By the way, do you understand how the ordering of a dictionary works when looking at
the display of a dictionary? Think about it.

The answer is: there is no ordering. That was what I said at the start: dictionaries are
unordered. In principle I cannot even tell you what ordering you see on your screen when
you run the code above, because it might differ between computers, operating systems, and
versions of Python. There is a certain structure to the ordering of the items, but nothing
that you can (or should desire to) predict. By adding enough items, the ordering might
even suddenly change completely.

Since dictionaries are unordered, many of the concepts that are applicable to lists, do not
work on dictionaries. For instance, you cannot refer to “slices” of a dictionary, and neither
can you “sort” or “reverse” a dictionary. So dictionaries are quite limited, but they do have
their uses.

13.2 Dictionary methods

This section describes the dictionary methods that are most often used.

13.2. Dictionary methods 167

13.2.1 copy()

Just like lists, if you assign a variable that contains a dictionary to another variable, you
are not creating a copy of the dictionary; you are actually creating an alias (if you do not
remember what an alias is, see Chapter 12). You cannot use the trick which is used for lists
to create a copy, as it uses a slice-syntax, and dictionaries do not support slices. Therefore,
there is a method copy() that returns a copy of a dictionary.

listing1302.py

fruitbasket = { “apple”:3, “banana”:5, “cherry”:50 }
fruitbasketalias = fruitbasket
fruitbasketcopy = fruitbasket.copy()

print( id( fruitbasket ) )
print( id( fruitbasketalias ) )
print( id( fruitbasketcopy ) )

Note that this method makes a shallow copy of the dictionary (see Chapter 12 if you do not
remember the difference between shallow and deep copies). If you want to make a deep
copy, use the deepcopy() function from the copy module.

13.2.2 keys(), values(), and items()

The method keys() provides an iterator that lists all the keys of a dictionary. The method
values() provides an iterator that lists all the values of a dictionary. The method items()
provides an iterator that lists all the key-value pairs of a dictionary as tuples.

I specifically say that these methods returns an iterator and not a list. If you want to turn
them into lists, you have to use list casting (see Chapter 12).

fruitbasket = { “apple”:3, “banana”:5, “cherry”:50 }
print( list( fruitbasket.keys() ) )
print( list( fruitbasket.values() ) )
print( list( fruitbasket.items() ) )

At this point you might be wondering when you can use an iterator. You mainly use
iterators for for loops (though you can also use them as arguments for the functions max(),
min() and sum()).

fruitbasket = { “apple”:3, “banana”:5, “cherry”:50, “durian”:0, ”
mango”:2 }

for key in fruitbasket.keys():
print( “{}:{}”.format( key, fruitbasket[key] ) )

print( sum( fruitbasket.values() ) )

Since this code provides an unpredictable order for the keys, you might want to sort them
before looping over them. Since keys() does not provide a list, it cannot be sorted directly,
but you can turn the result into a list using list casting. After doing that, you can sort.

168 Chapter 13. Dictionaries

listing1303.py
fruitbasket = { “apple”:3, “banana”:5, “cherry”:50, “durian”:0, ”
mango”:2 }

keylist = list( fruitbasket.keys() )
keylist.sort()
for key in keylist:

print( “{}:{}”.format( key, fruitbasket[key] ) )

keylist = list( fruitbasket.key() ).sort() does not work, as you cannot apply the
sort()method directly to the list casting. You must first create the list, then sort it. Neither
can you write for key in keylist.sort(), as the sort()method has no return value.

If you wonder why Python seems to prefer iterators instead of lists: the answer is that
iterators are more general and use much less memory. They are “lazy” methods, as they
only provide an item when it is requested.

13.2.3 get()

The get() method can be used to get a value from a dictionary even when you do not
know if the key for which you seek the value exists. You call the get() method with the
key you are looking for, and it will return the corresponding value when the key exists in
the dictionary, or the special value None when the key does not exist in the dictionary. If
you want to return a specific value instead of None if the key does not exist, you can add
that value as a second argument to the method.

listing1304.py
fruitbasket = { “apple”:3, “banana”:5, “cherry”:50, “durian”:0, ”
mango”:2 }

apple = fruitbasket.get( “apple” )
if apple:

print( “apple is in the basket” )
else:

print( “no apples in the basket”)

orange = fruitbasket.get( “orange” )
if orange:

print( “orange is in the basket” )
else:

print( “no oranges in the basket”)

banana = fruitbasket.get( “banana”, 0 )
print( “number of bananas in the basket:”, banana )

strawberry = fruitbasket.get( “strawberry”, 0 )
print( “number of strawberries in the basket:”, strawberry )

Run and study the example above closely, as what it demonstrates about the get()method
is very useful. Suppose that you store a collection of items with corresponding quantities,

13.2. Dictionary methods 169

for instance, the contents of a fruit basket with the keys being the names of the fruits and
the values being the quantities. When you query the fruitbasket using the get()method
with a second parameter zero, you can look for any fruit in the basket without the need to
check first if the fruit exists in the basket, because if you ask for a fruit that is not there, the
get()method returns zero, which is exactly what you want to hear.

13.2.4 Practice

Exercise The code below contains a list of words. Build a dictionary that contains all
these words as keys, and their quantities as values. Print the words with their quantities.

listing1305.py
wordlist = [“apple”,”durian”,”banana”,”durian”,”apple”,”cherry”,

“cherry”,”mango”,”apple”,”apple”,”cherry”,”durian”,”banana”,
“apple”,”apple”,”apple”,”apple”,”banana”,”apple”]

Exercise The code block below contains a string that is a list of words, separated by
commas. Build a dictionary that contains all these words as keys, and how often they
occur as values. Then print the words with their quantities.

listing1306.py
text = “apple,durian,banana,durian,apple,cherry,cherry,mango,” + \

“apple,apple,cherry,durian,banana,apple,apple,apple,” + \
“apple,banana,apple”

Exercise The code block below contains a very small dictionary that contains the trans-
lations of English words to Dutch. Write a program that uses this dictionary to create a
word-for-word translation of the given sentence. A word for which you cannot find a
translation, you can leave “as is.” The dictionary is supposed to be used case-insensitively,
but your translation may consist of all lower case words. It is nice if you leave punctuation
in the translation, but if you take it out, that is acceptable (as leaving punctuation in is quite
a bit of work, and does not really have anything to do with dictionaries – besides, leaving
punctuation in is much easier to do once you have learned about regular expressions).

listing1307.py
english_dutch = { “last”:”laatst”, “week”:”week”, “the”:”de”,

“royal”:”koninklijk”, “festival”:”feest”, “hall”:”hal”, “saw”:
“zaag”, “first”:”eerst”, “performance”:”optreden”, “of”:”van”,
“a”:”een”, “new”:”nieuw”, “symphony”:”symphonie”, “by”:”bij”,
“one”:”een”, “world”:”wereld”, “leading”:”leidend”, “modern”:
“modern”, “composer”:”componist”, “composers”:”componisten”,
“two”:”twee”, “shed”:”schuur”, “sheds”:”schuren” }

sentence = “Last week The Royal Festival Hall saw the first \
performance of a new symphony by one of the world ‘ s leading \
modern composers , Arthur \”Two-Sheds\” Jackson.”

170 Chapter 13. Dictionaries

13.3 Keys

As I said, any immutable data type can be a dictionary key. This means that strings, inte-
gers, and floats can all be used as keys. You may remember that tuples are also immutable,
which entails that you can use tuples as keys. This can occasionally be useful.

A very straightforward example of tuples being useful as keys is a dictionary in which you
want to store information associated with points in two-dimensional space (a discussion
of which was given in Chapter 11). There is no good way in which you can store the
identification of a point in a single number or string. It is not impossible (for instance, you
could store the number-pair as their string-representations, concatenated with a comma in
between) but it becomes ambiguous and convoluted (for instance, the string-keys “2,3”,
“2, 3”, “+2,+3”, and “02,03”would all be representing the same tuple but different keys).

13.4 Storing complicated values

Until now I only considered the case in which a dictionary stores a single value of a simple
data type. However, it is possible to store much more complex values in dictionaries. Val-
ues can be arbitrary Python objects. For example, you can store a list with each key. Below a
dictionary is used to store the students who are following a course. The course is identified
by its course number, while the students are identified by their student numbers.

listing1308.py

courses = {
‘ 880254 ‘ :[ ‘ u123456 ‘ , ‘ u383213 ‘ , ‘ u234178 ‘ ],
‘ 822177 ‘ :[ ‘ u123456 ‘ , ‘ u223416 ‘ , ‘ u234178 ‘ ],
‘ 822164 ‘ :[ ‘ u123456 ‘ , ‘ u223416 ‘ , ‘ u383213 ‘ , ‘ u234178 ‘ ]}

for c in courses:
print( c )
for s in courses[c]:

print( s, end=” ” )
print()

Suppose that you do not only want to store the student numbers for a course number, but
also the name of the course, the ECTS value of the course, and for each student number
also the grade. You can do that (for example) by storing the value for a course number as a
dictionary, with three keys, namely “name”, “ects”, and “students”. The value for “name”
is the course name as a string, the value for “ects” is the ECTS as an integer, and the value
for “students” is another dictionary, which contains student numbers as keys and grades
as values.

listing1309.py

courses = {
‘ 880254 ‘ : { “name”:”RS: Data Processing”, “ects”:3,

“students”:{ ‘ u123456 ‘ :8, ‘ u383213 ‘ :7.5, ‘ u234178 ‘ :6} },
‘ 822177 ‘ : { “name”:”Understanding Intelligence”, “ects”:6,

“students”:{ ‘ u123456 ‘ :5, ‘ u223416 ‘ :7, ‘ u234178 ‘ :9} },

13.5. Lookup speed 171

‘ 822164 ‘ : { “name”:”Computer Games”, “ects”:6,
“students”:{ ‘ u383213 ‘ :6, ‘ u234178 ‘ :4} } }

for c in courses:
print( “{}: {} ({})”.format( c, courses[c][“name”],

courses[c][“ects”] ) )
for s in courses[c][“students”]:

print( “{}: {}”.format( s, courses[c][“students”][s] ) )
print()

Data structures can become a lot more complex than this if you want. However, if you
are really considering designing Python programs for data structures like this, you should
at least investigate object orientation first (Chapter 20 and onward) and probably do a
separate course on databases.

13.5 Lookup speed

Lists and dictionaries are the two most-used data structures in Python. While often it is
clear when you should use which data structure, it is helpful if you know a little bit about
how Python processes these data structures in case you have a choice.

Suppose that you read a large bunch of numbers from a file. The numbers are all different
and can be anything. You later need to compare the numbers on another list to the numbers
that you read from the file.

Should you use a list or a dictionary to store the numbers that you read from the file?
Since they are just numbers, without extra data, a list seems to be the best option. There
is, however, a problem if you use a list here. Check out the following code, in which a list
of 10000 numbers is created, and after that some code checks for 10000 different numbers
whether they are on the list (which none of them are).

listing1310.py

from datetime import datetime

numlist = []
for i in range( 10000 ):

numlist.append( i )

start = datetime.now()
count = 0
for i in range( 10000, 20000 ):

if i in numlist:
count += 1

end = datetime.now()

print( “{}.{} seconds needed to find {} numbers”.format(
(end – start).seconds, (end – start).microseconds ,count ) )

172 Chapter 13. Dictionaries

Here is the code for doing the same thing with a dictionary, where I simply store the value
1 with each number.

listing1311.py
from datetime import datetime

numdict = {}
for i in range( 10000 ):

numdict[i] = 1

start = datetime.now()
count = 0
for i in range( 10000, 20000 ):

if i in numdict:
count += 1

end = datetime.now()

print( “{}.{} seconds needed to find {} numbers”.format(
(end – start).seconds, (end – start).microseconds ,count ) )

You will notice that for a dictionary, the code gives an answer almost immediately, while
for a list it takes quite some time for the code to provide an answer.

The reason is that I use the in operator to check whether a number is in the list, or in the
dictionary. For a list this means that Python searches through the list, sequentially, until it
reaches the number or reaches the end of the list. In this case, it means that Python checks
10000 times 10000 numbers (as it cannot find any of them), which is 100 million numbers.

For a dictionary, the process of finding a key is much faster. Python can quickly decide
whether or not a key is in a dictionary.9 Usually, the checking of just a handful of numbers
suffices. Therefore, the code is much, much faster for a dictionary.

You might think that a couple of seconds for the list search is still negligible, but the search
time increases quadratically with the size of the data. Depending on the problem, using a
dictionary might be highly preferable over using a list.

On the other hand, lists take less memory than dictionaries, and if you can directly access a
list item via its index, lists are faster than dictionaries. For instance, in the problem above,
if the list is sorted you can find numbers on it in a smarter way than using the in operator
(checking about 14 indices would suffice) – in that case, a list may be faster again.

From this, you should remember that a list is fast if you can access its elements directly via
their index, while a dictionary is a much better choice if the main way to find something
is by scanning items. The in operator seems easy and reads well, but if you use it to seek
something in a long list, you better think again.

What you learned
In this chapter, you learned about:

9Technically, Python stores the keys for the dictionary in a so-called “hash table.” I will not explain the details
here, and just tell you that a hash table allows for very fast look-up of keys at the cost of some memory.

Exercises 173

• Dictionaries

• Dictionary keys and values

• Dictionary methods copy(), keys(), values(), items(), and get()

• Complicated dictionaries

• Speed differences between lists and dictionaries

Excercises

Exercise 13.1 Write a program that takes a text (for instance the one given below), splits
it into words (where everything that is not a letter is considered a word boundary), and
case-insensitively builds a dictionary that stores for every word how often it occurs in the
text. Then print all the words with their quantities in alphabetical order.

exercise1301.py

text = “””How much wood would a woodchuck chuck
If a woodchuck could chuck wood?
He would chuck, he would, as much as he could,
And chuck as much as a woodchuck would
If a woodchuck could chuck wood.”””

Exercise 13.2 The code block below shows a list of movies. For each movie it also shows
a list of ratings. Convert this code in such a way that it stores all this data in one dictionary,
then use the dictionary to print the average rating for each movie, rounded to one decimal.

exercise1302.py

movies = [“Monty Python and the Holy Grail”,
“Monty Python ‘ s Life of Brian”,
“Monty Python ‘ s Meaning of Life”,
“And Now For Something Completely Different”]

grail_ratings = [ 9, 10, 9.5, 8.5, 3, 7.5,8 ]
brian_ratings = [ 10, 10, 0, 9, 1, 8, 7.5, 8, 6, 9 ]
life_ratings = [ 7, 6, 5 ]
different_ratings = [ 6, 5, 6, 6 ]

Exercise 13.3 A library contains books. Books have a writer, identified by last name
and first name. Books also have a title. Books also have a location number that identifies
where they can be found in the library. Librarians want to be able to locate a specific book
if they know writer and title, and they want to be able to list all the books that they have of
a specific writer. What data structure would you use to store the books?

174 Chapter 13. Dictionaries

Chapter 14

Sets

Sets are unordered data structures, which contain only unique elements. Few program-
ming languages support sets natively, but Python is one of them. Sets are not used often,
but occasionally they provide a nice solution to a problem that you are trying to solve,
when you need to ensure that you only have unique solution items.

14.1 Basics of sets

Sets are unordered collections of elements. You cannot access specific elements using an
index or a key. The only way to access items in a set is by using a for loop, or by testing
for the existence of elements in the set using the in operator.

You have to think of sets in the mathematical sense. In mathematics, a set is a collection of
elements, all unique, and elements can be part of a specific set, or not part of the set. You
use special set operators to combine sets in different ways.

Python uses dictionaries to implement sets; specifically, it implements the elements of a set
as dictionary keys. Thus, only immutable data types can be set elements. Sets themselves,
however, are mutable.

Since Python uses dictionaries to implement sets, you might think that you can create an
empty set by assigning {} to a variable. That, however, does not work as it creates an
empty dictionary, not an empty set. Instead, you create an empty set by assigning a call to
the function set() to a variable.

To create a set with some elements already in it, you can assign the elements to the variable
between curly brackets. Alternatively, you can call the set() function with a list of the
elements as argument.

fruitset = { “apple”, “banana”, “cherry” }
print( fruitset )

If you want to create a set consisting of the different characters in a string, you can call
set()with the string as argument.

176 Chapter 14. Sets

helloset = set( “hello world” )
print( helloset )

You can use a for loop to traverse a set. The variable in the for loop gets access to all the
set elements. There is no way to determine in which order you get to see the elements.
Sorting them is not possible as long as they form a set. You can, however, use list casting
on a set to create a list of its elements, which can then be sorted.

listing1401.py

fruitset = { “apple”, “banana”, “cherry”, “durian”, “mango” }
for element in fruitset:

print( element )
print()

fruitlist = list( fruitset )
fruitlist.sort()
for element in fruitlist:

print( element )

You can determine the number of elements in a set using the len() function.

14.2 Set methods
To manipulate the contents of sets, the following methods are supported. This is not a
complete list of set methods, but these are the most common ones.

14.2.1 add() and update()

Adding new items to a set you can do using the add()method, to add one new element that
you provide as an argument. If you want to add multiple new elements at once, you can
use the update()method, which you provide with a list of the new elements as argument.
You can also use update()with a tuple as argument, and you can even use it with a string
as argument. If you use it with a string, it will consider each letter of the string as a separate
element to add.

Since sets can only contain unique elements, any duplicate element that you try to add will
be ignored.

listing1402.py

fruitset = { “apple”, “banana”, “cherry”, “durian”, “mango” }
print( fruitset )
fruitset.add( “apple” )
fruitset.add( “elderberry” )
print( fruitset )
fruitset.update( [“apple”,”apple”,”apple”,”strawberry”,

“strawberry”,”apple”,”mango”] )
print( fruitset )

14.2. Set methods 177

14.2.2 remove(), discard(), and clear()

To remove elements from a set, you can use the remove() or discard() method. Both
get the element to remove as argument. The difference between the two methods is that
remove() will cause a runtime error if the element is not part of the set, while discard()
will ignore such errors.

fruitset = { “apple”, “banana”, “cherry”, “durian”, “mango” }
print( fruitset )

fruitset.remove( “apple” )
print( fruitset )

clear() removes all elements of the set at once.

14.2.3 pop()

Calling the pop() method will remove an element from the set and return it. You cannot
predict which element will be removed, as sets are unordered.

fruitset = { “apple”, “banana”, “cherry”, “durian”, “mango” }
while len( fruitset ) > 0:

print( fruitset.pop() )

14.2.4 copy()

Just like lists and dictionaries, if you assign a variable that contains a set to another vari-
able, you are creating an alias. Like with dictionaries (and probably because sets are im-
plemented as dictionaries), you use the method copy() to create a copy of a set.

14.2.5 union()

The union of two sets is a set which contains elements of both of them. You can use the
union()method for one set, with as argument a second set, to return the union of both sets
involved. This does not change the sets themselves. Alternatively, you can use the special
operator | (pipeline) to create the union of two sets. Note: you might suspect that you can
also use the + operator to combine two sets, but + is not defined for sets, and neither is ∗.

listing1403.py

fruit1 = { “apple”, “banana”, “cherry” }
fruit2 = { “banana”, “cherry”, “durian” }
fruitunion = fruit1.union( fruit2 )
print( fruitunion )

fruitunion = fruit1 | fruit2
print( fruitunion )

178 Chapter 14. Sets

14.2.6 intersection()

The intersection of two sets is a set which contains only the elements that they both have.
You can use the intersection() method for one set, with as argument a second set, to
return the intersection of the sets involved. This does not change the sets themselves.
Alternatively, you can use the special operator & (ampersand) to create the intersection of
two sets.

listing1404.py

fruit1 = { “apple”, “banana”, “cherry” }
fruit2 = { “banana”, “cherry”, “durian” }
fruitintersection = fruit1.intersection( fruit2 )
print( fruitintersection )

fruitintersection = fruit1 & fruit2
print( fruitintersection )

14.2.7 difference()

The difference of two sets is a set which contains only the elements that the first set has
that are not also in the second set. You can use the difference()method for one set, with
as argument a second set, to return the difference whereby the elements of the argument
set are removed from the first set. This does not change the sets themselves. Alternatively,
you can use the special operator – (minus) to create the difference of two sets.

listing1405.py

fruit1 = { “apple”, “banana”, “cherry” }
fruit2 = { “banana”, “cherry”, “durian” }
fruitdifference = fruit1.difference( fruit2 )
print( fruitdifference )

fruitdifference = fruit1 – fruit2
print( fruitdifference )

fruitdifference = fruit2 – fruit1
print( fruitdifference )

14.2.8 isdisjoint(), issubset(), and issuperset()

The methods isdisjoint(), issubset(), and issuperset() are all called as methods of
one set, with a second set as argument. All return True or False. isdisjoint() returns
True if the two sets share no elements. issubset() returns True if all the elements of the
first set are also found in the argument set. issuperset() returns True if all the elements
of the argument set are also found in the first set. Note that a set is both a subset and a
superset of itself.

14.3. Frozensets 179

listing1406.py

fruit1 = { “apple”, “banana”, “cherry” }
fruit2 = { “banana”, “cherry” }

print( fruit1.isdisjoint( fruit2 ) )
print( fruit1.issubset( fruit2 ) )
print( fruit2.issubset( fruit1 ) )
print( fruit1.issubset( fruit1 ) )
print( fruit1.issuperset( fruit2 ) )
print( fruit2.issuperset( fruit1 ) )
print( fruit1.issuperset( fruit1 ) )

14.2.9 Practice

Exercise There is also a set method symmetric_difference() which returns a set that
contains all the elements that are in the union of two sets, except those that are found
in both sets. For example, if set 1 contains A, B, and C, and set 2 contains B, C, and
D, the symmetric difference of sets 1 and 2 contains A and D. Can you implement the
symmetric_difference()method by using only some of the methods found above?

Exercise In the chapter on iterations you were asked to write code that determines all
the letters that two words have in common, whereby each letter should only be reported
once. Using sets, you can do this very efficiently. Please write the appropriate code.

14.3 Frozensets

Python supports a variant on the set type, namely the frozenset. You create a frozenset
by using the frozenset() function. The elements of a frozenset, once assigned, can-
not be changed. You therefore have to create the frozenset immediately when you call
the frozenset() function, because it is impossible to add or remove elements later. I.e.,
frozensets are immutable.

All the regular set methods work for frozensets, except for those that try to change the
set. Trying to use such a method for a frozensetwill lead to a syntax error.

fruit1 = frozenset( [“apple”, “banana”, “cherry”] )
fruit2 = frozenset( [“banana”, “cherry”, “durian”] )

print( fruit1.union( fruit2 ) )

What you learned

In this chapter, you learned about:

180 Chapter 14. Sets

• Sets

• add(), update(), remove(), discard(), clear(), pop(), copy(), union(),
intersection(), difference(), isdisjoint(), issubset(), and issuperset()

• Frozensets

Exercises

Exercise 14.1 A famous syllogism says: All men are mortal. Socrates is a man. Therefore
Socrates is mortal. In the code block below you see some sets. The first is the set of all things
(I know a few are missing, but for the sake of argument). The second is the set of all men
(assuming that the first set indeed contains all things). The third set contains everything
that is mortal (again, assuming…). Using set operators and methods, show that indeed (a)
all men are mortal, (b) Socrates is a man, and (c) Socrates is mortal. Also shows that (d)
there are mortal things that are not men, and (e) there are things that are not mortal.

exercise1401.py

allthings = {“Socrates”, “Plato”, “Eratosthenes”, “Zeus”, “Hera”,
“Athens”, “Acropolis”, “Cat”, “Dog”}

men = {“Socrates”, “Plato”, “Eratosthenes”}
mortalthings = {“Socrates”,”Plato”,”Eratosthenes”,”Cat”,”Dog”}

Exercise 14.2 Write a program that first produces three sets of numbers between 1 and
1000, the first all those numbers that are divisible by 3, the second all those numbers that
are divisible by 7, and the third all those numbers that are divisible by 11. It is easiest to do
that with list comprehension, but it is not necessary. Now produce sets of all the numbers
between 1 and 1000 that (a) are divisible by 3, 7, and 11, (b) are divisible by 3 and 7, but not
by 11, (c) that are not divisible by 3, 7, or 11. The shortest solution has only one line of code
for each of the six sets.

Chapter 15

Operating System

Until now, we considered Python programs as self-contained functionalities. Python pro-
grams, however, run on a computer, and occasionally the program must deal with com-
puter intricacies. This will start to play a major role from Chapter 16 onward, when I will
discuss file handling. To deal with the computer, Python offers a series of standard func-
tionalities in the os module, whereby os is a common abbreviation for “operating system.”
This chapter explains the most important functions from the os module.

15.1 Basics of operating systems

A computer consists of hardware, while programs consist of software. The software uses
facilities offered by the hardware. While in the early days of computer programming, pro-
grammers accessed hardware directly (for instance, to make a pixel visible on a computer
screen, a programmer placed a value in a specific memory address that was directly cou-
pled to the screen – an approach called “poking”), nowadays hardware is so complex and
diverse that this is no longer a viable approach. Let alone the fact that if you want to write
a program that runs on multiple computers, you cannot afford to access hardware directly
as hardware differs from computer to computer.

Therefore, programs access hardware functionalities through an “operating system.” An
operating system can be seen as a layer between programs and hardware, that offers pro-
grams high-level functions to get the hardware to work. Typical operating systems in use
nowadays on personal computers are Microsoft’s “Windows,” Apple’s “Mac OS,” and the
open-source OS “Linux” (though there are many more). Each of these exists in multiple
variants, often differentiated by numbers or “builds,” and sometimes (in the case of Linux)
by a company name. Regardless, they all offer functionalities that allow accessing hard-
ware.

The problem is that while all of them offer such functionalities, the functionalities are not
named in consistent ways, and have different parameterizations. This means that if you
want to write Python programs that access hardware by directly “talking” to the operat-
ing system, your program is not portable to other operating systems. This is where the os
module comes in. The os module offers functions that you can use to access the hardware

182 Chapter 15. Operating System

with, regardless of the operating system. Basically, the os module has a different imple-
mentation for each operating system, but your program does not need to know that, as the
functions are always named the same, and have the same parameters.

That does not mean that you can be completely oblivious of OS intricacies. For instance,
when you access a file, on Windows you might need to include a “drive letter,” which
Mac OS does not support. Another example is that security and file access are much more
flexible on Linux than on either Windows or Mac OS, so accessing files on Linux might
generate different kinds of warnings and errors than on other operating systems. There are
quite a few functions that only have an effect for particular operating systems. Still, the os
module is a fine compromise between portability and OS-dependent effectiveness.

15.2 Command prompt

When you are working with a mouse-driven user interface (UI), which is standard for Win-
dows and Mac OS, and is used by many Linux users too, you are actually interacting with
a visual representation of the system, specifically, of the file system. Programs and doc-
uments are represented by “icons,” which have a name. They are grouped by “folders,”
which are actually “directories” of the file system. You can create new folders, delete docu-
ments, rename programs, change security settings, etcetera. All these actions you can also
execute by directly typing commands, in an environment that is often called the “command
prompt” or “command shell.”

Most Linux users are familiar with a command shell, but for many Windows and Mac
users this is not something that they are aware of. Both Windows and Mac actually have
a program that allows you to work in the command shell. On Windows, you find the
“command prompt” as one of the “accessories” or “system tools.” On Macs, this is called
the “Terminal.” If you start that program, you get confronted by a window with a black
background and a blinking prompt. Here you can type commands that the system will
execute for you.

The commands that you can give depend on the system that you are using. This book is
not meant to teach you how to use it, but I want to tell you at least that you can run Python
programs directly from the command prompt by typing the command:

python .py

As long as Python can be found on your system, and the program is actually found in the
current working directory (i.e., the place in the computer’s file system where you currently
are), or you have specified the complete path for the program, then it will run the program.
This can actually be quite handy if you have written a program that processes files and you
want to process many files in a “batch.” Again, this goes a bit too far for this book, but you
might get to a point in your career where this is extremely useful.

The commands that you can give are things like “change the current working directory,”
“make a new directory,” “remove an empty directory,” “list all the files in the directory,”
“delete a file,” etcetera. Again, it depends on the operating system what exactly the com-
mands are that you need to give to achieve these things.

Exercise On your system, find the command shell and run the program. On Windows,
type “dir” to see the files in the current directory. On Macs and Linux, this command is
usually “ls.” After doing this, you can close the command shell again.

15.3. File system 183

15.3 File system

A computer’s file system consists of a tree-like structured organization of directories and
files.

There is one “root” directory, which is the main access point for all other directories. The
root directory is identified by a slash (/) or backslash (\), depending on the operating
system. Under Windows it is a backslash, under Mac OS and Linux it is a forward slash.
However, Windows now also supports the forward slash. I recommend using the forward
slash in most cases, as in strings the backslash indicates a special symbol, so if you want
to use a backslash in a string as a directory separator, you have to use a double backslash.
This tends to be confusing, which is why I recommend using the forward slash.

“Under” the root-directory there are multiple other directories, each identified by a name,
and usually also multiple files, each identified by a name. Under each directory there may
be more directories and files.

Each operating system has certain restrictions on what file and directory names can be
used, but in general most characters are supported. It is convention that regular files have
an extension, which is placed at the end of the file name, and separated from the filename
with a period. The extension identifies what kind of file it is, for instance, an executable
program (.exe), a flat text file (.txt), or a Python file (.py). It is also convention that
directory names do not have such an extension. However, this is not a rule, and you may
certainly encounter files without, and directories with an extension. Note that in the visual
environment, extensions for files are often hidden, but they are there – you just do not see
them.

To uniquely identify a file, you need to know its exact “path” from the
root to the file, following the directories. The path name for the file is
///…/. Under Windows, a drive letter can
be placed in front of this path, making it :///…
/. For instance, if under Windows, on the “C” drive, under the root there is
a directory “Python34,” under which there is a directory “Lib,” in which you can find a
file “os.py,” the path for that file is C:/Python34/Lib/os.py. Under Windows, this path is
case insensitive, so you can use only lower case letters if you like. That is not the case for
all operating systems, though.

When you are working in the file system (and you always are working in the file system,
even if you do not realize that), there is a “current directory,” which is identified by a
period (.). If you want to access a file in the current directory, you do not need to know
the complete path; it is enough to know the file name. One directory “higher” than the
current directory (i.e., the “parent” directory) is identified by a double period (..). The
parent directory of the root is the root itself.

Finally, it should be noted that most operating systems support a method that allows you
to access files, without knowing the path, even if those files are not in the current directory.
Under Windows, for instance, you can set a PATH environment variable that contains a
string that lists all the directories that Windows will search when you use a filename that
is for a file that is not in the current directory. How to adapt such an environment variable
is not part of this book, though.

184 Chapter 15. Operating System

15.4 os functions
The os module supports many functions that allow you to affect the file system. I will
mention only a few of them, as many of them are actually a bit dangerous to use (you can
easily delete files that you wanted to keep) and you do not need them anyway. If you are
really interested in manipulating the file system, you can read up on the dozens of other
functions that os supports.

15.4.1 getcwd()

getcwd() returns the current working directory as a string.

from os import getcwd
print( getcwd() )

15.4.2 chdir()

chdir() changes the current working directory. The new directory is provided as a string
argument.

from os import getcwd, chdir

home = getcwd()
print( home )
chdir( “..” )
print( getcwd() )
chdir( home )
print( getcwd() )

15.4.3 listdir()

listdir() returns a list of all the files and directories in the directory that is given as
argument. The names are given in arbitrary order. Notice that they do not include the full
path name.

from os import listdir

flist = listdir( “.” )
for name in flist:
print( name )

15.4.4 system()

system() gets a string argument that is a command, that Python executes on the command
line. You can use it to do anything that the operating system supports, including running
other programs. There are better ways to execute other programs, though (look for func-
tions that start with “exec”).

What you learned 185

What you learned

In this chapter, you learned about:

• Operating systems

• Command prompt

• File systems

• Functions getcwd(), chdir(), listdir(), and system()

Exercises

Exercise 15.1 Write a program that lists all the files and directories in the current direc-
tory, displaying them with their full path names.

186 Chapter 15. Operating System

Chapter 16

Text Files

One of the most important uses of Python for data processing is the reading, changing,
and writing of text files. Data is often stored in text files, because text files can be easily
transferred between different programs. There are multiple standardized formats for text
files, such as “comma-separated values” (CSV) files. Python supports particular text file
formats through modules, some of which will be discussed later. This chapter focusses on
opening, reading, writing, and closing of any text file, regardless of format.

16.1 Flat text files

When programmers refer to “text files” or “flat text files,” they mean files in which all
characters are meant to be read as regular characters, like you would type on a keyboard.
For instance, Python program files are flat text files, as are HTML files. Word processor
documents, however, are not flat text files, and neither are images. If you want to know
whether a file is a text file or not, you can try to open it in a text editor (such as the editor
for the IDLE environment, which comes with Python). If you see only readable text, the file
is likely to be a text file. Otherwise, it is a so-called “binary file” (binary files are discussed
in Chapter 18).

Text files consist of lines of text. At the end of a line, there is a “newline” symbol, which
in Python is the character “\n”. Different operating systems use slightly different ways of
storing this character in a text file: some Windows programs store it as “carriage return plus
line feed” (“\r\n”), while on Linux it is always stored as a single “\n”. As long as you ac-
cess a file from Python as a regular text file, Python will convert the characters that it reads
to the standard “\n”, and vice versa when it writes. So you do not need to worry about
such differences (except when you need to transfer text files between operating systems).

16.1.1 File handles and pointers

When you work with a file in a program, you have to open the file. Opening a file provides
a so-called “file handle.” A file handle can be seen as an access point to the file. It contains
a “pointer” that indicates a particular place in the file. That pointer is used when you read

188 Chapter 16. Text Files

from or write to the file. For instance, when you read from the file, it starts reading at the
pointer, and moves the pointer forward in the file.

When you open a file, the pointer is placed at a particular spot in the file, depending on
how you opened the file. If you opened the file for reading only, the pointer is placed at the
start of the file. The same is true when you open the file for both reading and writing. If
you open the file for “appending” (i.e., to place new data at the file’s end), the file pointer
is positioned at the end of the file. Lastly, if you open a file for writing only, actually the
file is completely emptied and the file pointer is placed at the start of the, now empty, file.
To create a new file (i.e., a file with a name that does not exist yet), you open it for “writing
only.”

After opening the file, the file handle is the only access point for the file. All actions you
perform on the file, you perform as methods for the file handle.

Note that any operating system only allows a limited number of files to be open simulta-
neously. Therefore you should close files that you no longer need to work with.

16.1.2 Moving the file pointer

The file pointer, that indicates where in a file you are working, is moved automatically.
For instance, when you read 10 characters from a file, the file pointer indicates which is
the first of those 10 characters, and, while reading, moves up 10 characters, so that its new
position is 10 characters further in the file than before. When you deal with text files, the
automatic movements of the file pointer are exactly what you want. You can position the
file pointer manually using specific methods, but such methods are, in general, only used
when dealing with binary files. Therefore, in this chapter I will ignore these methods, but
they will be discussed in the chapter on binary files.

16.1.3 Buffering

When you make changes to files, these often are not stored in the files immediately. Instead,
the operating system “buffers” the changes in memory, and “flushes” the buffers to the
actual files when it sees a need for that. You can force the flushing of the buffers by closing
a file. Buffers are also flushed when your program ends normally.

16.2. Reading text files 189

However, when your program crashes (for instance because of a runtime error), buffers
might not be flushed, and your files will not be updated to the point where the crash took
place. So you cannot take the file contents into account when trying to debug a program.

16.1.4 File processing programs

Most programs that deal with text files follow a process that, in a loop, reads contents of
a file, processes those contents in some way, then writes the contents to another file. For
instance, a program might read lines from a text file, and for each line sort the words, then
write the sorted words to another text file. This is hardly any different from a program
that asks the user the provide, in a loop, a line of text, then sorts the words in the line, and
displays them using the print() function. While students tend to find it easy to write the
version of the program that gets user input and displays output, my experience has taught
me that many students find it very hard to use file input and output for the same purpose.

I never found out why exactly students think it is so much harder to do it with files, though
I can imagine that you feel you have little control over your own program when working
with files. When you provide input to a program manually, and see it displaying outputs,
you always know more or less what lines of your code Python is processing, and you can
make up tests on the fly. If you work with files, you have to prepare your files beforehand,
then run the program and wait until it finishes before you can examine the contents of the
output files.

While working with files might give a sense of lack of control, during development of the
program you can always include print() statements to get an insight in what the program
is doing. For instance, when it reads a line, you can print that line, and when it writes
a line, you can also print that line. That way, your insight in the inner workings of the
program is no different regardless whether you use manual inputs and screen outputs, or
file inputs and outputs.

16.2 Reading text files

To read the contents of a file, you must first open it, then read the contents, then close it.

16.2.1 Opening a file using open()

To open a file, you use the open() function.

The open() function gets two arguments, of which the second one is optional. The first
argument is the name of the file. If the file is not in the current directory, you have to
include the complete path to the file so that Python can find it. The second argument is the
“mode.” The mode indicates how you want to treat the file. The default mode (which is
picked when you do not supply the second argument) is opening the file as a text file for
reading only. How you set other modes is discussed later.

The open() function returns a file handle, which you use for all the remaining functionali-
ties.

Rather than writing = open( ), you will often see Python pro-
grams that write this as open( ) as . These two ways of writing

190 Chapter 16. Text Files

code are equivalent. I myself prefer the first, as that is the way it is done in most pro-
gramming languages. However, the second method has an advantage that I discuss below,
when talking about closing a file.

16.2.2 Reading a file using read()

The simplest way to read the contents of a file is using the read() method, without argu-
ments, on the file handle. This returns a string that contains the complete contents of the
file. read() can get an argument, but I will discuss that in the chapter on binary files.

Reading from a file moves the file pointer to right after the part of the file that was read.
This means that if you use the read()method without arguments, the file pointer is moved
to the end of the file. This entails that if you would try to read() from it a second time,
nothing would be read, as there is nothing to be read after the spot where the file pointer
is. read() then returns an empty string.

16.2.3 Closing a file using close()

To close a file, you use the close() method on the file handle. After that, the handle is no
longer associated with the file. Every file that you open, you should close at some point in
your program.

So, a complete program that opens a file, reads the complete contents, prints them, and
closes the file again, is as follows:

listing1601.py

fp = open( “pc_rose.txt” )
print( fp.read() )
fp.close()

If everything that you need to do with a file is done in a single block, you can write this
block as follows:

with open( ) as :

This syntactic construction has the advantage that the file will be closed automatically after
the block ends, so you do not need to include an explicit close() call. This
construction is typically Python; you do not see it in many other programming languages.

16.2.4 Displaying the contents of a file

Now the first few functions and methods for dealing with text files have been introduced,
I can show some code that reads the contents of a file.

listing1602.py

with open( “pc_rose.txt” ) as fp:
buffer = fp.read()

print( buffer )

16.2. Reading text files 191

This code assumes that a file is available with the name “pc_rose.txt,” and that it is located
in the same directory as the program. Appendix E explains how to get it. If the file is
unavailable, you get a runtime error. How to deal with such errors will be explained in the
next chapter.

Exercise In the code above, change the file name “pc_rose.txt” to something that does
not exist. Run the program and observe the error that you get.

Exercise In the code above, change the file name to “pc_woodchuck.txt” (if you have
that file). Run the program and observe the output.

16.2.5 Reading lines using readline()

To read a text file line by line, you can use the readline() method. The readline()
method reads characters starting at the file pointer up to and including the next newline
character, and returns them as a string. You can recognize that you have reached the end
of the file by the fact that no characters are read anymore, i.e., the string that is returned is
empty.

listing1603.py

fp = open( “pc_rose.txt” )
while True:

buffer = fp.readline()
if buffer == “”:

break
print( buffer )

fp.close()

Notice that the output of the code above has an empty line between each of the lines dis-
played. Where is that extra line coming from? Think about it.

The extra line is there because the readline() method returns a string of the characters
read, up to and including the newline character. So when the buffer is printed, it prints a
newline character too. And since the print() function also moves to a new line after it is
executed, there is an empty line printed after each line of text.

Exercise Write a program that reads the lines from “pc_rose.txt,” and displays only those
lines that contain the word “name”.

16.2.6 Reading lines using readlines()

A corollary to the readline() method is the readlines() method. readlines() reads all
the lines in the file, and returns them as a list of strings. The strings include the newline
characters.

listing1604.py

fp = open( “pc_rose.txt” )
buffer = fp.readlines()

192 Chapter 16. Text Files

for line in buffer:
print( line, end=”” )

fp.close()

Note that the output of the code above does not have the empty lines between the lines of
text, as the print() function includes the “end=””” argument, which entails that print()
itself does not go to the next line after printing.

16.2.7 When to use which file-reading method

Both the read() and readlines() method read a whole file at once. Obviously, for small
files this is acceptable, but for long files you might not have enough memory to store the
file contents efficiently. In such circumstances (or when you do not know the file size), you
should read a file line by line with the readline()method.

It is often a good idea, during code development, to process only the first few lines of a file.
That way you limit the amount of time that the program needs to process a file, and limit
its output, which makes debugging easier. For instance, the code below process the first 5
lines of a file.

listing1605.py
fp = open( “pc_jabberwocky.txt” )
count = 0
while count < 5: buffer = fp.readline() if buffer == "": break print( buffer, end="" ) count += 1 fp.close() Once the program is finished and debugged, I can remove the references to count and change the loop to while True, to process the whole file. Exercise Adapt the code above to count how often the word “jabberwock” (with any capitalization) occurs in the first 5 lines. Print only the number of occurrences of that word. Once it works, remove the count so that you count the number of occurrences of the word in the text as a whole. 16.3 Writing text files Writing a text file is similar to reading. You open the file, write to it, and close it. 16.3.1 Opening a file for writing To open a file for writing, and writing only, you give the value "w" as the second argument to the open() function. If the file does not exist yet, it will create it. If it does exist, it will delete its contents. 16.3. Writing text files 193 Let me repeat that: when you open a file for writing and it already exists, its contents are deleted! There is no warning message saying “are you sure?” The file is simply emptied. So you have to be very, very careful when opening a file for writing. Usually I ask students to write their programs in such a way that they first check if a file exists before opening it for writing, and give an error message when it already exists. Functions for checking if a file exists are discussed later in this chapter. 16.3.2 Writing using write() To write something to a text file, you use the write() method with as argument a string that you want to write to the file. The example code below asks you to enter some strings, and writes them to a file. It stops asking for inputs when you enter an empty line. It then opens the file, reads the contents, and displays them. Run the code, enter at least two lines of text, and see what happens. listing1606.py fp = open( "pc_writetest.tmp", "w" ) while True: text = input( "Please enter a line of text: " ) if text == "": break fp.write( text ) fp.close() fp = open( "pc_writetest.tmp" ) buffer = fp.read() fp.close() print( buffer ) If you did what I asked, you have noticed that all the text that you entered is in the file, but it all is on one line. There are no newlines in between. The reason is that you have to explicitly write newline characters when you want newlines in your file. When you get input from the keyboard using input(), while you stop entering input using the Enter key, that does not result in a newline character in the string that input() returns. So you have to add that to the string that you write manually. Exercise Adapt the code above so that every line of text that you enter, is a separate line in the file that you write. 16.3.3 Writing using writelines() You can write a list of strings at once, by using the writelines()method that gets the list as argument. Each of the strings in the list must end in a newline character if you want those newline characters in the output file. writelines() is the opposite of readlines(); if you use the list that readlines() returns as argument for writelines(), the contents of the output file will be exactly the same as the contents of the input file. 194 Chapter 16. Text Files Note that there is no writeline() method. writeline() would be exactly the same method as write(), so it is not needed. 16.3.4 Practice Exercise Write a program that reads the contents of “pc_rose.txt,” and writes exactly the same contents to the file “pc_writetest.tmp.” Then open the file “pc_writetest.tmp” and display the contents. You can easily construct this program by cobbling together some of the code given above. Exercise Write a program that reads the contents of “pc_rose.txt,” reverses each of the lines, and writes the reversed lines to the file “pc_writetest.tmp.” Then open the file “pc_writetest.tmp” and display the contents. 16.4 Appending to text files “Appending” refers to writing at the end of an existing file. When you open a file for appending, the contents are not erased, but the file pointer is placed at the end of the file, where you can then write new data. You open a file in “append” mode by using "a" as the mode argument when opening the file. The code below first displays the contents of “pc_writetest.tmp” (which should exist by now). It then asks the user for lines which are appended to the file. Finally, it displays the contents of the new file. I took the liberty of creating this little program in a slightly-better structured manner than before, using a constant for the filename that is repeated three times in the program, and using a function to display the file contents as this functionality is needed twice. listing1607.py FILENAME = "pc_writetest.tmp" def displaycontents( filename ): fp = open( filename ) print( fp.read() ) fp.close() displaycontents( FILENAME ) fp = open( FILENAME , "a" ) while True: text = input( "Please enter a line of text: " ) if text == "": break fp.write( text+"\n" ) fp.close() displaycontents( FILENAME ) 16.5. os.path methods 195 16.5 os.path methods At this point you know everything you need to handle text files in Python. However, there are several handy functions that make your life easier when dealing with files. These are collected in the os.path module. As per usual, I am not going to list all of them, but I will list the ones that you will use the most. In these functions, the term “path” refers to a filename or a directory name, complete with parent directories (and drive letter). The parent directories (and drive letter) do not need to be there explicitly, but even if they are not, implicitly they still are as each file and each directory is located in a particular place in the file system. 16.5.1 exists() The function exists() gets a path as argument, and returns True if that path exists, and False if it does not. from os.path import exists if exists( "pc_rose.txt" ): print( "Rose exists" ) else: print( "Rose does not exist" ) if exists( "pc_tulip.txt" ): print( "Tulip exists" ) else: print( "Tulip does not exist" ) 16.5.2 isfile() isfile() tests if the path that is supplied as argument is a file. If it is, it returns True. If it is not, it returns False. If the path does not exist, the function also returns False. from os.path import isfile if isfile( "pc_rose.txt" ): print( "Rose is a file" ) else: print( "Rose is not a file" ) 16.5.3 isdir() isdir() tests if the path that is supplied as argument is a directory. If it is, it returns True. If it is not, it returns False. If the path does not exist, the function also returns False. 196 Chapter 16. Text Files from os.path import isdir if isdir( "pc_rose.txt" ): print( "Rose is a directory" ) else: print( "Rose is not a directory" ) 16.5.4 join() join() takes one or more parts of a path as argument, and concatenates them reasonably intelligently to a legal name for a path, which it returns. This means that it will add and remove slashes as needed. join() is particularly handy in combination with listdir() (see Chapter 15, and the example below). The reason that join() is handy with listdir(), is that listdir() results in a list of file names that do not include the directory names. Usually, when you ask for a list of file names, you intend to open them at some point. But to open a file that is not in the current directory, you need to know the complete path name that leads to the file. When you apply listdir(), you know where you are looking for files, so you know the elements of the path name. To construct the complete path name for each file, you need to concatenate the elements of the path name to the file name. Rather than trying to decide where you need to add slashes, and which kind of slashes they need to be, you can leave all of that to the join() function. The code below looks for all the files in the current directory, and lists them including their complete path name. See how join() is used to construct that path name from the current directory, and the file name. from os import listdir, getcwd from os.path import join filelist = listdir( "." ) for name in filelist: pathname = join( getcwd(), name ) print( pathname ) 16.5.5 basename() basename() extracts the filename from a path, and returns it. from os.path import basename print( basename( "/System/Home/readme.txt" ) ) 16.5.6 dirname() dirname() extracts the directory name from a path, and returns it. 16.6. File encoding 197 from os.path import dirname print( dirname( "/System/Home/readme.txt" ) ) 16.5.7 getsize() getsize() gets the size of the file that is supplied as argument, and returns it as an integer (representing a number of bytes). The file must exist, otherwise you get a runtime error. from os.path import getsize numbytes = getsize( "pc_rose.txt" ) print( numbytes ) Exercise Write a program that adds up the sizes of all the files in the current directory, and prints the result. 16.6 File encoding Text files use an “encoding,” i.e., a system that prescribes how characters in the files are supposed to be interpreted. This encoding may differ between operating sys- tems. You can see the preferred encoding that your system uses with a call to sys.getfilesystemencoding(). from sys import getfilesystemencoding print( getfilesystemencoding() ) If you read a text file which uses a different encoding than your file system prefers, you may get a UnicodeDecodeError. Whether or not you get this error for a particular file, is related to your operating system. An annoying consequence of that is that when you port Python code that reads a file to another system, a file that could be read by your code previously may cause your code to crash after the port.10 An easy way to get around this problem is by adding an extra parameter when opening a file, which indicates the encoding mechanism that you want to use when reading the file. You do this by adding a parameter encoding=, where is
a string that can have a variety of values, for which some typical ones are:

10I have to make note of some Python behavior that seems bizarre when you first encounter it: you may get this
error when your file contains characters in an encoding that is not supported by your system in lines that you are
not even trying to read! E.g., suppose that there is such an erroneous character on line 10 of your file, but you are
only trying to read the first 5 lines before closing the file again – your program may still crash! I suspect that this
is related to the buffering of data: rather than reading exactly what you ask Python to read, Python reads data in
bigger chunks, so that the program is faster when you actually want to go through the whole file. So, by trying to
be smart, Python may saddle you up with problems that you did not expect could arise. It is good to be aware of
such issues.

198 Chapter 16. Text Files

• ascii: 7-bits encoding, characters with values in the range 00-7F

• latin-1: 8-bits encoding, characters with values in the range 00-FF

• mbcs: 2-byte encoding, that is currently getting replaced by UTF-8

• utf-8: variable bytes encoding

Typically, text files are created with ascii or latin-1 encoding. Since ascii is incorpo-
rated in latin-1, you can safely open any text file by specifying latin-1 as encoding. It is
possible that for the characters beyond the ascii range, you get different characters than
the person who created the file wanted you to see – that depends on the encoding mecha-
nism that your file system uses. But at least the UnicodeDecodeError is avoided. So, when
you try to read the contents of a file and get a UnicodeDecodeError, you may try to open
it using open( , encoding=”latin-1″ ). Usually that will solve the problem.

Note that while utf-8 supports a much wider range of characters than latin-1, you may
still get the UnicodeDecodeError when you read a text file that uses latin-1 encoding on
a system that uses utf-8 encoding, as utf-8 has no corresponding characters with values
in the (hexadecimal) range 80-FF.

If you want to see which special characters are supported with values in the range 80-FF
on your system, run the code below. The numerical value of a character in the table can be
derived by calculating 16 ∗ row + col, whereby row and col are the hexadecimal row and
column number, respectively. I do not display the characters in the range 80-9F, as these
are normally not filled in.

listing1608.py

for i in range(16):
if i < 10: print( ' ' +chr( ord( ' 0 ' )+i ), end= ' ' ) else: print( ' ' +chr( ord( ' A ' )+i-10 ), end= ' ' ) print() for i in range( 10, 16 ): print( chr( ord( ' A ' )+i-10 ), end= ' ' ) for j in range( 16 ): c = i *16+j print( ' ' +chr( c ), end= ' ' ) print() More details on UTF-8 encoding will be given in Chapter 19, but for dealing with text files, the information above suffices. What you learned In this chapter, you learned about: • Text files • File pointers Exercises 199 • Opening and closing files with open() and close() • Reading files with read(), readline(), and readlines() • Writing files with write() and writelines() • Appending to files • os.path methods exists(), isfile(), isdir(), join(), basename(), dirname(), and getsize() • Dealing with text files with different encoding mechanisms Exercises Exercise 16.1 Write a program that reads the contents of the file “pc_woodchuck.txt,” splits it into words (where everything that is not a letter is considered a word boundary), and case-insensitively builds a dictionary that stores for every word how often it occurs in the text. Then print all the words with their quantities in alphabetical order. Exercise 16.2 Do the same thing as you did for the previous exercise, but now process the text line by line. This is something that you would have to do if you had to process a very long text. Exercise 16.3 Write a program that processes the contents of “pc_woodchuck.txt,” line by line. It creates an output file in the current working directory called “pc_woodchuck.tmp,” which has the same contents as “pc_woodchuck.txt,” except that all the vowels are removed (case-insensitively). At the end, display how many characters you read, and how many characters you wrote. Exercise 16.4 Write a program that determines how many words of 2 or more letters the files “pc_woodchuck.txt,” “pc_jabberwocky.txt” and “pc_rose.txt” have in common. You have to treat the words case-insensitively, and, as always, any character that is not a letter can be treated as a word boundary. If your program is correct, you will find three such words. Exercise 16.5 For the three files that you used in the previous exercise, count for each of them how often each letter (case-insensitively) occurs. Calculate for each let- ter and each file the fraction /. Write an outputfile (any name, as long as you
can safely overwrite it) with extension .csv, that contains 26 lines, each line for-
matted as follows: ““,,,. The first line should have letter a, the second let-
ter b, etcetera. The fractions should be stored with 5 decimals. Finally, display the contents
of the outputfile. As the outputfile is a CSV file, you should also be able to load it in a
spreadsheet program.

A quick check to see if you did things correctly is that all of the fractions must be between
zero and 1, and the fraction for “e” should be highest for both “pc_jabberwocky.txt” and

200 Chapter 16. Text Files

“pc_rose.txt” (but not so much for “pc_woodchuck.txt”). If you would use longer files
which are all in the same language, you would also find that the fractions are usually more
or less in each others’ neighborhood.

Chapter 17

Exceptions

Sometimes runtime errors occur not because you made a programming mistake, but be-
cause your program encountered a situation that you could not foresee when you wrote it.
This is particularly relevant when you deal with files: for instance, when you access a file
on a USB-stick, and the user pulls out the USB-stick during the file processing, obviously
an error occurs that you cannot really account for in your code. Every runtime error raises
a so-called “exception,” and you can “capture” such exceptions if you want to deal with
them in your program, rather than make the program end abruptly.

17.1 Errors and exceptions

When you try to run a Python program, Python first does a quick check to see if all the
statements in the program meet the basic Python syntax requirements. If they do not,
Python announces a “syntax error” and will not run the program.

If no syntax errors are encountered, Python will run the program, but may encounter state-
ments that generate errors while trying to execute them. Such statements cause a “runtime
error.” You have seen runtime errors many times while writing code (don’t try to deny it).

In general, you try to fix runtime errors by extending or changing your code. For instance,
the following program causes a runtime error when you enter a zero as input:

from pcinput import getInteger

num = getInteger( “Please enter a number: ” )
print( “3 divided by {} is {}”.format( num, 3/num ) )
print( “Goodbye!” )

Python tells you what kind of error it is, namely a ZeroDivisionError. To fix it, you can
change the program:

from pcinput import getInteger

num = getInteger( “Please enter a number: ” )

202 Chapter 17. Exceptions

if num == 0:
print( “Dividing by zero is not allowed” )

else:
print( “3 divided by {} is {}”.format( num, 3/num ) )

print( “Goodbye!” )

ZeroDivisionError is actually the name of an “exception” that Python “raises” (gener-
ates). If you do not handle such an exception in your program, Python interrupts the
program’s execution and splashes its error message on the screen. However, this entails
that you actually can handle exceptions in your program and simply continue running it.

While in the code above you should ensure that no exception is raised on dividing by zero
– because you can foresee that this might happen – it occasionally happens that you have
to accept that exceptions might be raised as you cannot foresee all the circumstances that
your program might have to deal with. This is especially relevant when your program
depends on elements outside its direct control, such as files and user behavior.

17.2 Exception handling

To handle exceptions explicitly in your program, you use a try … except clause. There
are different ways of applying this clause.

17.2.1 try … except

The most basic form of the try … except clause has the following syntax:

try:

except:

When the between try: and except: are executed and raise an ex-
ception, Python immediately jumps to the statements and exe-
cutes those, after which the program continues as normal, at the first line below the
statements. If no exceptions are raised during the execution of
the , the statements are skipped.

Using exception handling, the code at the start of this chapter can be written as follows to
avoid runtime errors:

listing1701.py

from pcinput import getInteger

num = getInteger( “Please enter a number: ” )
try:

print( “3 divided by {} is {}”.format( num, 3/num ) )
except:

print( “Division by zero is not allowed” )
print( “Goodbye!” )

17.2. Exception handling 203

Multiple statements may be part of a single try … except clause. For instance, the fol-
lowing code raises an exception both when a user enters zero and when a user enters 3.
Both exceptions are handled by the same try … except clause.

listing1702.py
from pcinput import getInteger

num = getInteger( “Please enter a number: ” )
try:

print( “3 divided by {} is {}”.format( num, 3/num ) )
print( “3 divided by {}-3 is {}”.format( num, 3/(num-3) ) )

except:
print( “Division by zero is not allowed” )

print( “Goodbye!” )

This is slightly ugly, not only because these errors should have been avoided rather than
handled via exceptions, but also because when an exception occurs, it is unclear which of
the statements caused it (though in this case, if you enter 3, you can see that the first of the
two statements under the try executed correctly). However, this is just a demonstration,
and there certainly can be situations where you say “I do not care where an exception
occurs in this sequence of statements, but if anything happens, I want to do this.”

17.2.2 Handling specific exceptions

Examine the code below. This code can cause at least two different exceptions. Which?

print( 3 / int( input( “Please enter a number: ” ) ) )

The two different exceptions that this code can generate are the ZeroDivisionError when
you enter a zero, and the ValueError when you enter something that is not an integer. Try
it if you did not try it already.

You can handle both these errors with a single try … except clause, but you can dis-
tinguish them by specifying multiple excepts. Each except can be followed by one of the
specific exceptions, and the code below it will only be executed if that specific exception is
raised.

listing1703.py
try:

print( 3 / int( input( “Please enter a number: ” ) ) )
except ZeroDivisionError:

print( “Dividing by zero is not allowed” )
except ValueError:

print( “You have not entered an integer” )
print( “Goodbye!” )

If you want to capture “all remaining exceptions,” you add an except without a specific
exception at the end. Only one of the except clauses will be executed, namely the first one
encountered that applies. It is a lot like an if … elif … elif … else clause.

204 Chapter 17. Exceptions

listing1704.py

try:
print( 3 / int( input( “Please enter a number: ” ) ) )

except ZeroDivisionError:
print( “Dividing by zero is not allowed” )

except ValueError:
print( “You have not entered an integer” )

except:
print( “Something unforeseen went wrong” )

print( “Goodbye!” )

Here is a list of some specific exceptions that are raised often:

• ZeroDivisionError: Trying to divide by zero

• IndexError: Trying to access a list or tuple element with an out-of-bounds index

• KeyError: Trying to access a dictionary element with an unknown key

• IOError: Any error that occurs while trying to access a file

• FileNotFoundError: Trying to open a file that does not exist for reading

• ValueError: Error while trying to type cast a value to another value

• TypeError: Using a value of a type that is not supported for an operation

Exercise The code below can generate several exceptions. These are now handled by a
single try … except clause. Extend this code by handling all exceptions that may occur
explicitly (there are at least three different kinds of exceptions that can be raised). Note:
Let me stress again that I rather have you avoid exceptions occurring than handling them,
but in this case I want you to practice with exception handling.

listing1705.py

fruitlist = [“apple”, “banana”, “cherry”]
try:

num = input( “Please enter a number: ” )
if “.” in num:

num = float( num )
else:

num = int( num )
print( fruitlist[num] )

except:
print( “Something went wrong” )

17.2.3 Adding an else clause

At the end of a try … except clause you can add an else clause. The statements with
that elsewill be executed if no exception occurs. For instance, in the code block below, the
calculated value for num will only be printed if no exception is raised.

17.2. Exception handling 205

listing1706.py

try:
num = 3 / int( input( “Please enter a number: ” ) )

except ZeroDivisionError:
print( “Dividing by zero is not allowed” )

except ValueError:
print( “You have not entered an integer” )

except:
print( “Something unforeseen went wrong” )

else:
print( num )

print( “Goodbye” )

In general, I prefer not to use an else clause with an exception, as it feels like the code
under the except clauses should be code that is only executed in abnormal circumstances.
But if you really want you can use it.

17.2.4 Adding a finally clause

You can add an extra branch to the try clause, which is finally:. The finally clause has
statements which are executed regardless of the manner in which the try clause is exited.
If it ends normally, it is executed, but if you get a runtime error, it is executed too. You can
use such a finally clause to, for instance, make sure that a file gets closed before code is
harshly interrupted.

listing1707.py

try:
fp = open( “pc_rose.txt” )
print( “File opened” )
print( fp.read() )

finally:
fp.close()
print( “File closed” )

17.2.5 Accessing exception information

You can get extra information on the exception by adding and as clause to the except
statement, using the syntax except as . When an
occurs, the variable is filled with an “exception object,” that may provide more information
on the exception. The problem is that there is no standardized way to get the information
out: it depends on the kind of exception what the variable contains.

The variable will always have a tuple of arguments (even if there is only one), that were
provided to the exception when it was raised. You can examine these arguments by means
of the attribute .args. A ValueError gets a tuple with only one value, namely
a string.

206 Chapter 17. Exceptions

listing1708.py

try:
print( int( “NotAnInteger” ) )

except ValueError as ex:
print( ex.args )

If you run the code below, you see that an IOError gets a tuple of two values: an integer
and a string. The integer that is provided for the IOError is actually quite informative, as
it explains what went wrong.

listing1709.py

try:
fp = open( “NotAFile” )
fp.close()

except IOError as ex:
print( ex.args )

17.2.6 General advice on using exception handling

Never capture an exception and then just ignore it. In particular, you should not use a
general try … except clause and then do nothing with the exception. If you think you
can ignore a certain exception, make sure that you capture that specific exception, and
comment in your program why you think you can ignore it. Basically, all exceptions should
either be handled responsibly or should just make the program crash.

17.3 File handling exceptions

Any problem with accessing files, whether it is the inability to find a file, a problem with
reading or writing a file, or trying to open a protected system file or even a directory, leads
to an IOError exception. Since problems with file access are quite common and usually at
least partly outside the realm of control of the program, it is a good idea to handle IOError
exceptions in your programs when you can.

Since many different things can go wrong with files, the args tuple explained above might
be used to provide better information on what you have to do to handle the problem. For
instance, if your program asked the user to supply a filename, and when you open that file
you get an IOError, if the error number (the first element of the tuple) indicates that the
file does not exist (2), then an appropriate response might be to simply report this to the
user and ask for a new name.

The error numbers are defined in the errno module, which you can import in your pro-
gram. The module offers constants that you can use instead of the actual numbers, which
is the convention. The most common error numbers are:

• errno.ENOENT: No such file or directory. You get this when you try to access a file
that does not exist.

17.4. Raising exceptions 207

• errno.EACCESS: Permission denied. You can get this in varied circumstances, such
as when you try to read from a closed file, when you try to open a read-only file for
writing, or when you try to open a directory as if it is a file.

• errno.ENOSPC: No more space left on device. You get this when you try to write a
file and there is no room for it, for instance when you try to write to a USB-stick that
is full.

There is a big list of such error numbers which you can easily find in the reference manuals.
You might not understand what all of them refer to, and actually many of them are archaic
and no longer occur on modern computer systems. The best thing to do when you develop
your program, is to try to capture IOErrors, and when you do encounter an IOError, print
the arguments so that you know the number and the error message. You can then look up
which errno constant that message belongs to, and respond to it in your program if you
can do that in a sensible way.

However, just as with other kinds of exceptions, it is better to avoid them than to capture
them. There is no reason that you should ever encounter a “file does not exist” error, as you
can test whether a file exists with the exists() and isfile() functions from the os.path
module.

listing1710.py

import errno

try:
fp = open( “NotAFile” )
fp.close()

except IOError as ex:
if ex.args[0] == errno.ENOENT:

print( “File not found!” )
else:

print( ex.args[0], ex.args[1] )

Note: Exception FileNotFoundError is a “subclass” (see Chapter 22) of IOError, which, in
Python 3, is actually an alias for yet another exception, namely OSError. This means that
capturing FileNotFoundError is equivalent with capturing IOError (or OSError) as ex
and testing whether ex.args[0] holds errno.ENOENT.

17.4 Raising exceptions

You are allowed to raise exceptions yourself. For that, you use the keyword raise, and
follow that with one of the known exceptions (potentially, you could create your own, new
exceptions if you like, but you need to have studied Chapters 20 and 22 before you are
ready for it). You can give the exception arguments of any kind that you like.

You might wonder why you would want to raise your own exceptions. The answer is that
when you create a module, when an error occurs (for instance because the main program
that uses the module passes arguments to a function that are of an incorrect type), it is bad
form to print an error message. It is much better to just raise an exception, and let the main
program handle the exception. Here is an example of raising an exception:

208 Chapter 17. Exceptions

listing1711.py

def getStringLenMax10( prompt ):
s = input( prompt )
if len( s ) > 10:

raise ValueError( “Length exceeds 10”, len( s ) )
return s

print( getStringLenMax10( “Use 10 characters or less: ” ) )

When you run this code, you see that if you enter a string of more than 10 characters, a
ValueError exception is raised. It has two arguments, which you can see displayed as a
tuple when Python splashes the exception on the screen. You can handle this exception just
as you would handle exceptions that Python itself produces.

The raise keyword has a second function: if you are in an except clause, and rather than
handle the exception there, you want to pass it on to the “next level” of the program, you
can just write the keyword raise, and the exception will be “re-raised.” This can be useful
when you want to do a bit of extra handling before the program “crashes” or the exception
is handled elsewhere. For instance:

listing1712.py

fp = open( “pc_rose.txt “)
try:

buffer = fp.read()
print( buffer )

except IOError:
fp.close()
raise

fp.close()

This code probably runs fine, but if an IOError occurs when reading the file, the file gets
closed before the exception is re-raised.

What you learned

In this chapter, you learned about:

• Exception handling

• except, else, and finally

• ZeroDivisionError, IndexError, KeyError, IOError, ValueError, TypeError, and
FileNotFoundError

• Getting exception information

• File handling exceptions

• Raising and re-raising exceptions

Exercises 209

Exercises

Exercise 17.1 Which exceptions can the code below raise? Extend the code to handle all
of them in a reasonable manner.

exercise1701.py

numlist = [ 100, 101, 0, “103”, 104 ]

i1 = int( input( “Give an index: ” ) )
print( “100 /”, numlist[i1], “=”, 100 / numlist[i1] )

210 Chapter 17. Exceptions

Chapter 18

Binary Files

“Binary files” is the term used to refer to all files that are not text files. Executable programs
are binary files, as are image files, movies, word processor documents, and many other file
types. It is not common to use Python to process binary files (usually binary files are not
handled by general-purpose programming languages, but by special-purpose programs),
but it is possible. This chapter will explain how to deal with binary files.

18.1 Opening and closing binary files

The handling of binary files is quite similar to the handling of text files. You have to open()
a file when you want to access its contents, close() it when you are finished, read() from
the file and write() to the file.

When you open a binary file, you have to indicate to Python that you want to handle
this file in “binary mode.” You do this by adding a letter “b” to the mode argument. For
instance, to open a file in “binary read” mode, the mode argument should be “rb”. You can
also open a file both for reading and writing; reading and writing you indicate with mode
“r+”, so reading and writing in binary mode is “r+b” (while it is also possible to open text
files in “r+” mode, I did not indicate it in Chapter 16, as it seldom makes sense to open
text files in this mode). Just as with text files, if you open a binary file in write-mode, with
“wb”, the file gets emptied. The mode “w+b” will open a file for both reading and writing,
but also empties the file to start with.

When you open a file for both reading and writing, if the file pointer is not at the end of
the file, when you write you actually overwrite.

You can open any file in binary mode, even text files. However, when you open text files
in binary mode, you treat them like binary files, which means that Python does not do the
automatic conversion of newline characters.

Closing a binary file is no different from closing a text file.

fp = open( “pc_rose.txt”, “rb” )
fp.close()

212 Chapter 18. Binary Files

Note: The code above has no output – if it does have output, that is a runtime error, mean-
ing that “pc_rose.txt” is not available.

18.2 Reading a binary file

As binary files do not know the concept of “lines,” the only way to read from a binary file
is to use the read() method. If you use read() without argument, it reads the whole file
(starting at the file pointer). If you give the method an integer as parameter, that integer
indicates the number of bytes that are read from the file (starting at the file pointer, and
reading at maximum until the end of the file).

A “byte,” if you do not know, is an 8-bit character, i.e., a number between zero and 255,
which is stored in the smallest possible memory unit that a computer supports. The regular
characters on a keyboard are each stored in a single byte, and the characters in a string are
also each a byte, though limited to a specific range of numbers.

18.2.1 Byte strings

Here we enter one of the more obscure parts of the Python language. When you read from
a binary file, the read()method does not return a regular string – it returns a “byte string.”
There are some subtle differences between regular strings and byte strings. To show you
these differences, I first have to tell you that you can indicate that a string is a byte string
by placing a letter b in front of it. So “Hello, world”! is a string, while b”Hello, world”!
is a byte string.

18.2. Reading a binary file 213

hw1 = “Hello, world!”
hw2 = b”Hello, world!”

print( hw1 )
print( hw2 )

The difference between a string and a byte string is that a byte string can contain characters
that a string cannot. For instance, if you remember the discussion on the ASCII table, you
may recall that each character has a number associated with it. You saw, for instance, that
“A” has the number 65, and the space has the number 32. The space was the lowest num-
bered character that I showed, and you might wonder which characters are associated with
numbers 0 to 31. The answer is: these are control codes, and are not legal characters that
you can put in a string (barring a few exceptions). You can try to put them in a string using
an “escape code”: the escape sequence \x can be followed by a two-character hexadecimal
code that represents the character with the specified number. For example, the hexadec-
imal code for a space is \x20, i.e., “Hello, world!” is the same as “Hello,\x20world!”
(this was discussed in Chapter 10).

hw1 = “Hello ,\x20world!”
print( hw1 )

But what if you try to put illegal characters in a string that way? They are ignored:

print( “Hello ,\x00\x01\x02world!” )

The problem is that such characters can occur in binary files, so you must be able to read
them from binary files. Since byte strings can contain such characters, reading from binary
files results in byte strings.

print( b”Hello ,\x00\x01\x02world!” )

Characters from a byte string you can access using indices, just like you do with regular
strings. The difference here is that with regular strings you get letters, while with byte
strings you get numbers. The numbers are the codes for the letters, which you would also
get when you use the ord() function on the corresponding letter.

listing1801.py

hw1 = “Hello, world!”
hw2 = b”Hello, world!”

for c in hw1:
print( c, end=” ” )

print()
for c in hw1:

print( ord( c ), end=” ” )
print()
for c in hw2:

print( c, end=” ” )

214 Chapter 18. Binary Files

Since bytes are numbers between 0 and 255, you might want to convert a number to a
single-character byte string, or a list of numbers to a multi-character byte string. You can
do so using a bytes casting on a list of those numbers. Note that if you want to convert a
single character, you still have to use a list, but a list with just one element. Do not forget to
put the list brackets around that element, because if you do not, the result will not be what
you expect.

listing1802.py

bs = bytes( [72,101,108,108,111,44,32,119,111,114,108,100,33] )
print( bs )
bch = bytes( [72] )
print( bch )
wrong = bytes( 72 )
print( wrong )

Can you convert from a byte string to a regular string? You might think that string casting
works, but unfortunately it does not:

hw1 = b”Hello, world!”
hw2 = str( hw1 )
print( hw2 )

The reason that it does not, is that when a string is in the format of a byte string, it uses
an encoding scheme, according to the Unicode standard (discussed in Chapter 16). You
have to “decode” the byte string according to a certain decoding scheme, which usually
is “utf-8″, as that is the most common Unicode format. You decode using the decode()
method, with the encoding scheme as a string parameter. You can also go from a string to
a byte string by encoding using the encode()method, again with the encoding scheme as
string parameter.

listing1803.py

hw1 = b”Hello, world!”
hw2 = hw1.decode( “utf-8” )
print( hw2 )
hw3 = hw2.encode( “utf-8” )
print( hw3 )

In general you have little reason to read text files in binary mode, at least not if you just
want to access the text, and so you do not have to worry about encoding and decoding
byte strings. The exception is when you have to deal with a text file that uses Unicode
characters. Such a file cannot be treated as a text file, and you have to open it in binary
mode.

18.2.2 Binary reading demonstration

To demonstrate how reading a binary file works, I now open the file “pc_rose.txt,” and
read ten times ten bytes from it.

18.3. Writing a binary file 215

listing1804.py

fp = open( “pc_rose.txt”, “rb” )
for i in range( 10 ):

buffer = fp.read( 10 )
print( buffer )

fp.close()

When you run the code, you see the ten byte strings being displayed. You may also notice
that there are certain control characters visible, such as \r and \n. The \r you would not
see if you read this file as a text file, because Python converts it, together with the following
\n, to a single \n. Moreover, in a regular string you would not see the \n, because it is a
newline character which tells Python to move to the next line.

If instead of a text file, you open an actual binary file, you probably will not be able to make
much sense of the output when you display it.

18.3 Writing a binary file

You write to a binary file using the write() method. The difference with writing to text
files is that you have to supply a byte string as argument, rather than a regular string. The
following code creates a binary file with some text in it.

listing1805.py

from os.path import getsize

FILENAME = “pc_binarytest.tmp”
fp = open( FILENAME , “wb” )
fp.write( b”And now for something completely different…\x0A\
\x00\x00\x00\x00\xD4\xE8\xE5\xA0\xD3\xF0\xE1\xEE\xE9\xF3\xE8\xA0\
\xC9\xEE\xF1\xF5\xE9\xF3\xE9\xF4\xE9\xEF\xEE\x00\x00\x00″ )
fp.close()
print( getsize( FILENAME ), “bytes written” )

Run the code above to create the binary file. The code below opens it in text mode (you
can do that, as Python cannot know that it actually is a binary file), reads the contents, and
prints the contents. You will see some readable text and some unreadable characters.

listing1806.py

FILENAME = “pc_binarytest.tmp”
fp = open( FILENAME , encoding=”latin -1″ )
while True:

buffer = fp.readline()
if buffer == “”:

break
print( buffer )

fp.close()

216 Chapter 18. Binary Files

Exercise Change the code above to open the file in binary mode and print the contents.

18.4 Positioning the file pointer
The file “pc_binarytest.tmp” actually contains a few secret words, which you cannot rec-
ognize when printing the file. I am going to use them as an illustration on how to move
the file pointer.

The file pointer indicates where in the file you start reading or writing. You can move
the file pointer with the seek()method. seek() gets two integer arguments, of which the
second one is optional. The first argument is a relative byte position. The second is the
position relative to which you want to move the file pointer.

The second argument can be 0, 1, or 2. 0 means “relative to the beginning of the file,” 1
means “relative to the current file pointer position,” and 2 means “relative to the end of the
file.” If you do not specify a second argument, it is assumed to be 0. In the os module there
are constants for this argument: os.SEEK_SET is 0, os.SEEK_CUR is 1, and os.SEEK_END is 2.

The first parameter indicates how many bytes you move from the indicated position. When
starting at the beginning of the file, it should be a positive number; when starting at the
end of the file, it should be a negative number; when starting somewhere in the middle of
the file, it can be positive or negative. For instance, the statement fp.seek(5) is equivalent
to fp.seek(5,0), which moves the file pointer 5 bytes up from the start of the file, placing
it at the sixth byte.

Should you wish to know at which position the file pointer is currently placed, you can
use the tell()method. Both seek() and tell() can be called for text files too, but are not
very useful then.

Now, the secret words are found starting at position 50, and run for a length of 23 bytes.
The encoding is such that if you subtract 128 from byte values, you get the ordinals for the
letters. So, here is how you get the words out of the file:

listing1807.py

fp = open( “pc_binarytest.tmp”, “rb” )
print( “1. Current position of the file pointer is”, fp.tell() )
fp.seek( 50 )
print( “2. Current position of the file pointer is”, fp.tell() )
buffer = fp.read( 23 )
print( “3. Current position of the file pointer is”, fp.tell() )
fp.close()

print( buffer )
s = “”
for c in buffer:

s += chr( c-128 )
print( “The secret words are:”, s )

The seek() method is particularly useful when you open a file in “reading and writing”
mode (“r+b”). It allows you to move through the file, reading where you need to read, and
(over)writing where you need to (over)write.

What you learned 217

Exercise Open the file “pc_binarytest.tmp” in binary “reading and writing” mode, and
overwrite the encoded secret words with their decoded translation. Once you have closed
the file, open it again in text mode, read the contents, and display them. If you did it all
correctly, you should see two readable lines. Should you mess up the file in some way, you
can always recreate it.

What you learned

In this chapter, you learned about:

• Using binary files for reading, writing, and reading plus writing

• Binary read() and write()

• Byte strings

• Conversion between strings and byte strings using encode() and decode()

• seek() and tell()methods

Exercises

Exercise 18.1 Create a simple file encryption program. Open a file and read it in binary
mode. For each byte, if it is smaller than 128, add 128; if it is bigger than or equal to 128,
subtract 128. Overwrite the byte with new value. Test the program on a copy of a text
file (make sure it is a copy, because you will destroy the file). Check the contents of the
encrypted file: they should be a mess. However, when you run the program again, the
original file should be restored. If it isn’t, you have a bug in your program. Aren’t you glad
you were only working on a copy?

Exercise 18.2 The fourteen most common letters in the English language are:
etaoinshrdlcum. Write a text compression program based on this fact. The compression
program stores these letters in half-bytes. A half-byte can take the numbers zero to 15. If
you only use the numbers 1 to 15, each number can represent one of these fourteen most
common letters, and you can use the number 15 for the space. So you can store two of
these letters (or space) in a byte (the value for the whole byte would be 16 times the value
for the first letter, plus the second letter). If in the text you encounter a letter that is not
amongst these fifteen, you indicate that by storing a zero-half-byte, followed by a whole
byte that represents the unencoded letter. Of course, in this setup it is possible that the full
byte is actually divided over two bytes, namely the second half-byte of one byte, and the
first half-byte of the other byte.

Hint: an easy approach is to build a list of half-bytes. For the most common letters, you
store their index-value for the string “etaoinshrdlcum ” plus 1 (which is a value in the
range 1 to 15; notice that the last character in the string is the space). For the other charac-
ters, you store three half bytes, namely zero, followed by the ordinal value of the character
divided by 16 (rounded down), followed by that ordinal value modulo 16. Once the half-
byte-list is finished, you can turn it into a byte list by taking pairs of half-bytes, multiplying
the first by 16 and adding the second. Create a byte string using bytes() casting.

218 Chapter 18. Binary Files

Figure 18.1: Compression example.

For testing: the string “Hello, world!”, which is 13 characters in length, will become the
11-character byte string b ‘\x04\x81\xbb@,\xf0wI\xba\x02\x10’ if you follow the proce-
dure outlined above (which assigns e the value 1, t the value 2, etcetera).

A note on the translation of “Hello, world!” to the given byte string (see Figure 18.1): You
may remember that a hexadecimal representation of a byte consists of two hexadecimal
digits, i.e., each digit is a half-byte. Using that information, you can see how the translation
has been done. The first byte is \x04, i.e., the first half-byte is zero. That means that the first
character is given literally, i.e., it consists of the second half-byte of \x04, and the first half-
byte of the next byte, which is \x81. That is the byte \x48. If you look up the hexadecimal
code 48 in the ASCII table (given in the chapter on strings), you see that that is the character
H. The following half-byte is the second half-byte of \x81, i.e., it is 1. Since this is not zero,
it is one of the most common characters, namely the first one, which is e. So now you see
how “Hello, world!” is compressed as the byte string provided. The byte string does
contain a few characters that are not displayed as their hexadecimal code; if you really
want to know which hexadecimal code they represent, look them up in the ASCII table.

By the way, despite the long description, the whole program needs less than 30 lines of
code, including comments, empty lines, and testing statements.

Exercise 18.3 As a collary to the previous exercise, write a decompression program for
the produced strings.

Hint: Just do the opposite of what you did in the previous exercise: rebuild the half-byte-
list. That list is then easily converted back to the original string.

Exercise 18.4 This chapter is about binary files, and the previous two exercises were not,
at least, not directly. There simply is not much that you can exercise with where binary
files are concerned; the main problems are with handling byte values, which is what the
previous two exercises were concerned with. But to round off what these two exercises
did, let’s now use what you developed in them to compress files.

Write a program that asks for an input file, that must exist, and an output file, that should
not exist. Then it asks whether you want to compress or decompress. If you choose com-
press, the input file is compressed using the method developed above, and written as the
output file. If you choose decompress, the input file is decompressed under the assump-
tion that it was compressed with the method developed above, and written as the output
file. So you should be able to get the original file again by first compressing and then
decompressing.

You best read the whole file in memory before (de)compressing, so that you do not get into
problems when a byte string ends in half a byte instead of a full byte after compression.
You also best treat both the input file and the output file as binary files.

Chapter 19

Bitwise Operators

Chapter 18 discussed dealing with binary files. When binary files are used, you are no
longer working with characters and numbers; rather, you are working with bytes. To ma-
nipulate information on the level of bytes, Python offers a number of so-called “bitwise
operators.” You will not need these often, but when you delve into binary file manipula-
tion, they might come in handy.

19.1 Bits and bytes

A bit is the smallest size data unit that a computer can handle. A single bit can have only
two different values, namely 1 and zero.

While “prehistoric” computers were indeed programmed by directly dealing with single
ones and zeroes, very quickly computers were introduced that handled groups of bits. The
smallest unit in that respect is the “byte,” which consists of 8 bits. Today, the concept of
a byte still permeates most computer languages, even though computers have been en-
hanced to use larger collections of bytes as smallest data units (notably, most computers
today either deal with 32-bits or 64-bits data units).

19.1.1 Binary counting

A byte consists of 8 bits, which you can display as a sequence of ones and zeroes,
e.g., 11010010. As such, a byte can be used to represent a number in binary code.
If a byte is used to represent a positive number, that number can be calculated by
multiplying the right-most bit by 1, the bit next to that by 2, the bit next to that by
4, etcetera, and adding up all those values. For instance, the sequence 11010010 is
1*128+1*64+0*32+1*16+0*8+0*4+1*2+0*1, which is 210. Note that this is similar to cal-
culating the value of decimal numbers, where the rightmost digit is multiplied by 1, the
digit next to that by 10, the digit next to that by 100, etcetera, and adding up all those re-
sulting values. It is also similar to hexadecimal counting, which was discussed in Chapter
10.

When bits are numbered, by convention numbering starts at zero at the rightmost end, and
numbers are increased when counting to the left, i.e., the rightmost bit has number 0, the

220 Chapter 19. Bitwise Operators

bit next to that has number 1, the bit next to that has number 2, etcetera. The reason is that
the rightmost bit represents the value 20 (which, in case you forgot, equals 1), the bit next
to it the value 21, the bit next to that 22, etcetera.

Byte 1 1 0 1 0 0 1 0
Number of bit 7 6 5 4 3 2 1 0
Represented value 27 26 25 24 23 22 21 20

Byte value 27 + 26 + 0 + 24 + 0 + 0 + 21 + 0 = 210

Exercise Write some code that calculates the decimal number represented by a binary
string of 8 ones and zeroes. The nicest solution uses a loop, a multiplier, and a total. The
total starts at 0. The multiplier (which is the represented value in the example above) starts
at 1, and every time the loop is traversed it is multiplied by 2. The loop processes the string
from right to left (or the reversed string from left to right), and if the character encountered
is a “1,” it adds the multiplier to a total. This will end up with the number represented by
the string as the total.

The lowest number that can be expressed by a byte is 00000000, which equals zero. The
highest is 11111111, which equals 255. Thus, there are 256 different values that can be
expressed by one byte.

19.1.2 Character encoding

The most basic character encoding mechanism is ASCII. The ASCII table was shown in
Chapter 10, including hexadecimal codes. You may have noticed that the codes used ran
from (hexadecimal) 20 to 7E. The codes below 20 are used for special sequences (such as the
newline character). The code 7F usually represent the Del key. No other codes are in use,
which means that all ASCII characters can be represented by 7 bits, or the 8-bit sequences
00000000 to 01111111.

While computers use bytes as basic data unit, the ASCII character set does not use 128 of
all the values that can be stored in a byte. All these unused bytes have a 1 as their leftmost
bit. Naturally, some character encodings were introduced that assign a character to all
256 different values that a byte can take. A typical one is latin-1, which is discussed in
Chapter 16. Unfortunately, not all encoding mechanisms assign the same characters to the
numbers between 128 and 255. However, all encoding mechanisms in use today at least
have the basic ASCII characters for the values 0 to 127.

Python is based on Unicode encoding. Specifically, it uses UTF-8 as character encoding
mechanism (discussed in Chapters 10 and 16). UTF-8 encoding works as follows:

• A byte that has a zero as leftmost bit is an ASCII character.

• A byte that has a 1 as leftmost bit is the start of a sequence of multiple bytes that
represent one character. The sequence consists of a leading byte (the leftmost byte)
and one or more continuation bytes.

• For a multibyte sequence, the leading byte has, from left to right, several bits with
value 1, followed by a bit with value zero, followed by the remaining bits. The length
of the total multibyte sequence is as many bytes as there are bits with value 1 to the
left of the leftmost zero. E.g., if the leading byte has value 1110xxxx (where each x

19.2. Manipulating bits 221

is some bit value), the whole sequence is three bytes long. This includes the leading
byte. The minimum sequence length is two bytes, and the maximum sequence length
is six bytes (the leading byte will then be 1111110x).

• Each continuation byte has 10 as the two leftmost bits.

• In practice, UTF-8 encoding is restricted to at most 4-byte sequences, and some of the
4-byte sequences have been excluded.

This means that UTF-8 can express a great many different characters. However, it also
means that, due to the way characters are encoded, some bit patterns do not express UTF-
8 characters. While any bit pattern expresses a legal string with latin-1 encoding, it is
possible to construct a bit pattern that does not express a legal UTF-8 encoding. This may
cause those annoying UnicodeDecodeErrors when reading files.

19.1.3 Number encoding

The way that numbers are encoded as bit patterns is somewhat tricky, and in general you
do not need to bother with it. You should know that positive integers are always encoded
as multi-byte patterns, that have a zero as their leftmost bit. The rest of the pattern is like
you would expect, and as explained above.

Negative numbers, however, are encoded rather differently. They use the so-called “two’s
complement” system. When a negative number is encoded, first the absolute value of that
number (i.e., the positive version) is taken. From this number, all the bits are “flipped,” i.e.,
every 1 becomes a zero and every zero becomes a 1. Finally, 1 is numerically added to the
result. The bit pattern of a negative number therefore always has a 1 as its leftmost bit.

For example, to encode −1, first the bit pattern of 1 is taken, which is …00000001. All the
bits are flipped, which gives …11111110. Finally, 1 is added the the result, which gives
…11111111. Thus, −1 is encoded as a sequence of only 1s.

As for floating point numbers, these use scientific notation, whereby part of the multi-byte
pattern is used as exponent.

The reason that I am explaining all of this, is to indicate that if you want to handle bit
patterns in a Python program, and you want to treat these patterns as numbers, you best
work only with positive integers, as the bit patterns of those are easily understood.

19.2 Manipulating bits

Python offers a variety of operators that allow the manipulation of data items at the level
of bits. These are the following:

<< shift left >> shift right

& bitwise and

| bitwise or

~ bitwise not

^ bitwise exclusive or

They are used as follows.

222 Chapter 19. Bitwise Operators

19.2.1 Shifting bits

When you have a data item, you can use the << and >> to shift its bits to the left or right.
x<>y shifts
the bits of x by y places to the right, copying the leftmost bit of x at the left while shifting,
and losing the bits of x at the right. x and y must both be numbers.

For example, the exclamation mark ! has decimal code 33, which is written as 00100001 in
binary. Shifting this pattern one place to the left gives 01000010, i.e., 66 in decimal, which
is the code for the capital B. You can reverse this by shifting the pattern of B one place to
the right.

code = “!”
print( chr(ord(code)<<1) ) code = "B" print( chr(ord(code)>>1) )

You might have noticed that shifting a number one place to the left amounts to doubling
the number, while shifting it one place to the right amounts to halving it (while rounding
down). Indeed, you can double the value expressed by a bit pattern by placing a zero to
the right of it – and you can halve it (using integer division) by removing the rightmost bit.

print( “345 quadrupled makes”, 345<<2 ) print( "345 divided by 8 makes", 345>>3 )

19.2.2 Bitwise and

The bitwise and operator (&) takes two bit patterns, and produces a new pattern that is all
zeroes, except for those places where both bit patterns had a 1, which will then also have
a 1 in the output pattern. For instance, if the input patterns are the number 11 (00001011)
and the number 6 (00000110), then the bitwise and operator produces the pattern 00000010,
which is the number 2.

print( 11 & 6 )

Exercise The bitwise and is an easy way to take (positive) numbers modulo a power of
2. For instance, if you want to take a number modulo 16, this is the same as performing the
bitwise and on the number with 15, which is 00001111. Check that the value of 345 modulo
32 is the same as taking 345 & 31.

19.2.3 Bitwise or

The bitwise or operator (|) takes two bit patterns, and produces a new pattern that is all
ones, except for those places where both bit patterns had a 0, which will then also have a
0 in the output pattern. For instance, if the input patterns are the number 11 (00001011)
and the number 6 (00000110), then the bitwise or operator produces the pattern 00001111,
which is the number 15.

19.2. Manipulating bits 223

print( 11 | 6 )

Exercise To set a single bit in a pattern to the value 1 (this is usually called “setting a
bit”), you can use the bitwise or and a pattern that consists of only zeroes, except for a 1 in
the spot where you want to set the bit. An easy way to create a bit pattern with only one
bit set, is to start with the number 1, and use the shift-left operator to shift that bit to the
left as far as you need. Now take a number and set the bit with index 7 (i.e., the eight bit
from the right) to 1.

19.2.4 Bitwise not

The bitwise not operator (~) is placed in front of a bit pattern, and then produces a new
pattern that has all the bits of the original pattern “flipped,” i.e., each zero becomes a 1 and
each 1 becomes a zero. For instance, if the input pattern is the number 11 (00001011), then
the bitwise not produces the pattern 11110100, which is the number −12. If you wonder
why it is −12 and not −11: this is the result of the two’s complement encoding, which I
explained above. Don’t worry too much about it.

print( ~11 )

Exercise To clear a single bit in a pattern (i.e., setting it to the value zero), you can use the
bitwise and and a pattern that consists of only 1s, except for a zero in the spot where you
want to clear the bit. An easy way to create a bit pattern consisting on only ones, except
for a zero in the intended spot, is to start with the number 1, and use the shift-left operator
to shift that bit to the left as far as you need. Then invert the pattern with the bitwise not
operator. Now take a number and clear the bit with index 3 (i.e., the fourth bit from the
right).

19.2.5 Bitwise xor

The bitwise exclusive or, or “xor,” operator (^) takes two bit patterns, and produces a
new pattern that has a zero in all places where the two bit patterns have the same bit, and
a 1 in all places where the two bit patterns have different bits. For instance, if the input
patterns are the number 11 (00001011) and the number 6 (00000110), then the bitwise xor
operator produces the pattern 00001101, which is the number 13.

print( 11 ^ 6 )

Exercise The bitwise xor operator provides an easy way to encrypt numbers. Take a bit
pattern, and call it the “mask.” Apply the mask to a number using the xor. This gives a
new number, which is the encrypted number. Somebody who does not know the mask,
can’t tell what the original number was. However, someone who does know the mask, can
easily get the original number back, by applying the mask once more. Try this.

224 Chapter 19. Bitwise Operators

19.2.6 Precedence of bitwise operators

Warning: the precedence of bitwise operators is not that they are handled before other
operators. Make sure that you use parentheses to order the operators when you use bitwise
operators in a calculation. For instance, you might think that 1<<1 + 2<<1 is the same as 1*2 + 2*2, but in actuality it is evaluated as (1<<(1+2))<<1, or 1*8*2. print( 1<<1 + 2<<1 ) print( (1<<1) + (2<<1) ) 19.3 Usefulness of bitwise operations Anything that you can do with bitwise operators, you can also do with general calcula- tions, with the advantage of general calculations that they can do much more than bitwise operators. So what is the use of bitwise operators? Bitwise operations are incredibly fast. Much, much faster than regular calculations. So should you use them when making calculations, when it is opportune to do so? The answer is no, for two reasons: • Python is already smart enough to recognize that some calculations can be executed using bitwise operators, so it will make the conversion for you. • If you really want a fast program, you should not use Python at all. Another use that is often mentioned, is that they facilitate storing boolean values in a small storage space. For instance, if I have eight booleans that I want to store, I can use a tuple of eight booleans, which amounts to at least eight bytes of space, or encode all eight of them in one byte using bitwise operators. However, in today’s computers space is of little concern, so only if you are talking about huge, huge data collections you might get worried about space. So what is the use of bitwise operators then? They are actually of fairly little use, unless you have to create programs that need to work “close to the machine.” Occasionally you have to deal with data structures that are most naturally handled using bitwise operators. They may also help when you need to manipulate the content of binary files. To give an example: colors are usually encoded as three bytes, for the red, green, and blue channel. A color number is thus a three-byte number. Bitwise operators are a natural way to distinguish the separate color channels from a color number. Here is a function that does that: listing1901.py def getRGB( color ): blue = color & 255 green = (color >> 8) & 255
red = (color >> 16) & 255
return red, green, blue

r, g, b = getRGB( 223567 )
print( “red={}, green={}, blue={}”.format( r, g, b ) )

What you learned 225

For someone who knows about color encoding, such a function reads well.

What you learned

In this chapter, you learned about:

• Binary counting

• Character and number encoding

• Bitwise operators <<, >>, &, |, ~, and ^

Exercises

Exercise 19.1 Encode a string using the bitwise exclusive or (xor) and the pattern
00101010 as mask. Display the resulting string. Then decode it, and display the decoded
string, which should be the same as the original string.

Exercise 19.2 Write a function that gets an integer, a boolean, and a number. The integer
is used to store booleans. Each bit in the integer represents True or False. The bits of the
integer are numbered as the convention indicates, with the rightmost bit having number
zero, the bit next to that number 1, etcetera. If the boolean parameter is True, the function
sets the bit corresponding to the number parameter in the integer to 1. If the boolean
parameter is False, the function clears the bit corresponding to the number parameter in
the integer (i.e., it sets it to 0). The function then returns the integer.

Also write a function that gets an integer and a number as parameters, and returns True if
the bit corresponding to the number is set to 1, and False otherwise.

To test the functions, it helps to create an extra function that displays the bits in the number.
The display function can make use of the function that gets the bit values.

226 Chapter 19. Bitwise Operators

Chapter 20

Object Orientation

The chapters until this point covered an approach to programming that is often referred to
as “structured programming” or “imperative programming,” wherein a program is con-
sidered a sequence of statements, decisions, and loops. You can solve any programming
problem with a structured programming approach. However, in the last decades several
other programming “paradigms” have been coined up, which help designing and imple-
menting large-scale programs. One of the most successful paradigms is “object orienta-
tion,” and most modern programming languages support the object oriented paradigm.
Python is, in fact, an object oriented language.

While object orientation tends to provide a natural way to look at problems and solutions,
designing an object oriented program can be quite hard. The reason that it is hard, is that
you have to really think about your approach to a problem in all of its aspects, before
you start coding. For bigger problems, this can be daunting, especially when you lack
experience with programming. However, for bigger problems you have to spend a lot
of time designing your solution anyway, and an object oriented approach may be quite
helpful in creating it. Moreover, you will find that most modules provide object oriented
implementations, and that object orientation can be helpful for many smaller problems too.

Since object orientation is a broad topic, several chapters will be spent on it, of which this
one is the first. It discusses the basics of object orientation, leaving the more specialized
(and powereful!) aspects of object orientation for later chapters.

20.1 The object oriented world

While I am typing this, I am sitting at my kitchen table. Next to me is a bowl of fruit.
There are some apples in the bowl. While these apples share certain features, they have
their differences too. They share their name, their price, and their age, but they all have
(slightly) different weights. There are also some oranges in the bowl. Like the apples, they
are fruits, but they have a lot of differences with apples: different names, different colors,
different trees that they grow from. Still, they share some things with apples that all fruits
share, and make them different from, for instance, the table I am sitting at. I can eat a fruit,
i.e., I can eat apples and I can eat oranges. I am not going to try to eat a table.

228 Chapter 20. Object Orientation

When I try to model my world in a computer program, I have to model objects: objects such
as apples, oranges, and tables. Some of these objects have a lot in common, for instance,
each apple shares a lot of features with every other apple. It behooves me to define a class
“apple” which contains the features that all apples share, and only fill in the few features
in which apples differ from each other for each individual apple object. The same holds for
oranges, they should get their own class “orange.” And while “apples” and “oranges” are
quite different, they still share some features that entail that I would like to put them in the
same class: the class “fruit.” Every object that belongs to the class “fruit” at least has the
property that I can eat it. Which means that each individual apple object not only belongs
to the class “apple,” but also belongs to the class “fruit” – just like the “oranges.”

Come to think of it: I can eat more things than only apples and oranges. I can eat cakes too.
And mushrooms. And bread. And licorice. So maybe I need another class, which the class
“fruit” also belongs to. The class “food,” perhaps?

What this leads to, is that if I try to model the world, or part of the world, I need to model
objects – and rather than modeling each separate object, I am better off defining classes of
objects, as that means I can make statements about certain groups of objects in general. I
can talk about the relationships between classes, and I can define functions that work on
classes; for instance, I can define a functionality “eat” that works on every object that is part
of the class “food,” which removes the object from the world and assigns its “nutrients” to
the object that does the “eating.” Since I can “eat” objects that belong to the class “food,” I
can eat “fruit.” And since I can eat “fruit,” I can eat any “apple” object.

A computer program is, in essence, a model of a part of the world. As such, there are
many programs that benefit from the ability to deal with objects, classes, relationships, and
functionalities (methods) that work on objects.

20.1.1 Students, teachers, and courses

Many programs deal with persons. The student administration deals with students, who
are persons. These students follow courses, which are taught by teachers, who are also per-
sons. Undoubtedly, the student administration stores information on students and teach-
ers, and probably the programmer who created the software for the student administration
was smart enough to create a single interface that allows entering person data.

What data do all persons share, as far as the student administration is concerned? Well,
probably all persons have a first and a last name. They have an address. They also have an
age and a gender. They all get assigned an administration number, so that for the admin-
istration they have at least one thing that makes them unique. These data elements are all
“properties” or “attributes” of “persons.”

I mentioned the properties first name, last name, address, age, gender, and administration
number. One of these is actually more like a function than a property. Do you see which
one?

The answer is “age.” Age is calculated from date of birth and current date. While you
can consider age a property, it is a property that should be calculated each time that it
is needed. You cannot store it as a value, as tomorrow it might be different from today,
without anything changing but the date. Therefore, if I design a class Person that models
a person, I best make “date of birth” an attribute of the person, while “age” is a method of
the person. Remember that methods are functions that belong to a certain data type: if a

20.1. The object oriented world 229

data type Person is defined, date_of_birth is an attribute of that data type, while age()
is a method of that data type that returns the person’s age as an integer.

Students and teachers are both persons. They share the properties of the Person class.
Yet there are differences. Teachers, for instance, get paid a salary, while students do not.
Students, on the other hand, earn grades in courses, which teachers do not; they teach the
courses. From this follow two obvious observations:

• While students and teachers are both persons, they have clear differences; besides a
class Person I need a class Student and a class Teacher, both of which are derived
from the class Person.

• “Courses” seem to be an inherent part of the student administration world, so a class
Course might be needed too.

Once Course has become a class in the student administration world, relationships become
visible. Students have relationships to multiple courses, and teachers do too, though in a
different capacity. Students “enroll” in courses. It looks like an enroll()method is needed,
that allows a student to get into a relationship with a course. The question is: is enroll()
a method of Student, that gets a course as argument, or is it a method of Course, that gets
a student as argument? What do you think?

The answer is: “it depends.” It depends on how you envision the student administration
world. To me, it feels more natural to make enroll() a method of a course, as I view a
course as a collection of students. However, in principle there is nothing against seeing a
student as an entity who encompasses a collection of courses. You might also decide to
make enroll() a method of each of them, or think of yet another class that contains the
enroll()method that has both the student and the course as arguments.

This illustrates the difficulty of the object oriented view on program design: by designing
the classes that form the world model that the program works with, choices need to be
made that may have a big impact on how the program works. Weak choices may lead to
difficulties in implementation. You need to spend considerable time on designing the ob-
ject oriented model that underlies the program, and try to anticipate all the consequences
of your choices. This is hard even for experienced designers. However, a solid object ori-
ented model makes programs easy to read, understand and maintain. The object oriented
paradigm is often worth the hassle.

20.1.2 Classes, objects, and hierarchies

In the object oriented world, every distinguishable entity belongs to a “class.” A class is a
general model for a specific group of entities. It describes all the attributes that these enti-
ties have, and it describes the methods that the class offers the outside world to influence
it.

A class, by itself, is not an entity. An entity that belongs to the class, is an “object.” The
terminology is that an object is an “instance” of a particular class. While the class describes
its attributes, an object that is an instance of the class has values for these attributes. While
the class describes the methods that it supports, to execute such a method one needs an
object that is an instance of the class to call the method with.

A class is a data type, an object is a value.

230 Chapter 20. Object Orientation

Classes may exist in hierarchies. A general, high-level class may describe properties and
methods that are shared by different subclasses. Each subclass may add properties, add
methods, and even change properties and methods (though in general cannot – and should
not – completely remove them). Each subclass may have further subclasses.

For instance, the class Apple may be a subclass of the class Fruit, which may be a subclass
of the class Food. This means that where in a program an object of the class Food is needed,
you can supply an object that is an instance of the class Food, but also an object that is
an instance of the class Fruit, or an object that is an instance of the class Apple. This
does not work the other way around, though. When, for instance, a function in a program
was designed for instances of Apple, you cannot use it with instances of Fruit, or other
subclasses of Fruit. While an Apple is Fruit, Fruit is not an Apple, and Apples aren’t
Oranges.

Such a hierarchy is implemented using “inheritance,” which is the topic of Chapter 22.

20.1.3 Classes and data types in Python

Most object oriented programming languages have some basic data types, and allow you
to create classes, i.e., new data types. This was the case for Python up to Python 2. Since
Python 3, every data type is a class.

You can recognize some of this by the way that many functionalities of the basic data
types are implemented as methods. Remember that a method is always called as
.(), contrary to functions that work on a variable, which are called
as ( ). The fact that when you want to create a lower case version
of a string, you effectuate that as .lower() already indicates that the string is an
instance of a class.

But not only strings are class instances: integers and floats are too. They even have meth-
ods, though these are seldom used explicitly. Some methods are used implicitly, e.g., when
you add two numbers together with +, that is actually a method call. This will be discussed
in Chapter 21.

20.2 Classes and objects

Now the basic philosophies of object orientation are out of the way, I am going to dis-
cuss how to use object orientation in Python. It starts with creating new classes using the
keyword class.

20.2.1 class

A class can be considered a new data type. Once a class is created, you can assign instances
of the class to variables. To start simple, I am going to create a class that represents a
point in 2D space. I name this class Point (the naming of classes is restricted to the same
requirements as the naming of variables, and it is convention to let the names of classes
start with a capital). Creating this class in Python is incredibly easy:

20.2. Classes and objects 231

class Point:
pass

The keyword pass in the class definition means “do nothing.” This keyword can be used
wherever you need to place a statement, but you have nothing yet to place there. You
cannot just leave it empty or give a comment and nothing else. But as soon as statements
are added, you no longer need pass.

To create an object that is an instance of the class, I assign to a variable the name of the
class, with parentheses after it, as if it is a function call (you can have arguments between
the parenthesis, which will be discussed a bit later in this chapter).

class Point:
pass

p = Point()
print( type( p ) )

Of course, a point is more than just an object. A point has an x and a y coordinate. Since
Python is a soft-typed language, you need to assign values to attributes to create them.
This is done in a special initialization method in the class.

20.2.2 __init__()

The initialization method of a class has the name __init__ (that’s two underscores, fol-
lowed by the word init, followed by two more underscores). Even if the __init__()
method is not defined explicitly for the class, it still exists. You use the __init__()method
to initialize everything that you want to initialize upon creation of an instance of the class.

In the case of Point, __init__() should assure that any Point object has an x and a y
coordinate. This is implemented as follows:

232 Chapter 20. Object Orientation

listing2001.py

class Point:
def __init__( self ):

self.x = 0.0
self.y = 0.0

p = Point()
print( “({}, {})”.format( p.x, p.y ) )

Study the code above closely. You see that __init__() is defined just as you would define
a function, inside the class definition.

__init__() gets one parameter, which is called self. Every method that you define, al-
ways gets at least one parameter, which will get filled with a reference to the object for
which the method is called. By convention, this first parameter is always called self. That
is not mandatory, but everybody always does it like this. If you forget to include that first
parameter, you will get a runtime error. If you forget to include the first parameter self
but you do have other parameters, Python will fill the first of the parameters that you do
list with a reference to the object, and you will probably also get a runtime error (as you
did not expect that that would happen).

In the __init__() method for Point, the object that is created gets two attributes, which
are variables that are part of the object. They are called x and y, and since they are part of
the object, you refer to them as self.x and self.y. They both get initial value 0.0, which
makes them floats.

To refer to these attributes when the object has been created, you use the syntax