Practical Week 01
1. Simple Statistics and NLTK¶
The following exercises use a portion of the Gutenberg corpus that is stored in the corpus dataset of NLTK. The Project Gutenberg is a large collection of electronic books that are out of copyright. These books are free to download for reading, or for our case, for doing a little of corpus analysis.
To obtain the list of files of NLTK’s Gutenberg corpus, type the following commands:
In [1]:
import nltk
nltk.download(‘gutenberg’)
nltk.corpus.gutenberg.fileids()
[nltk_data] Downloading package gutenberg to /home/diego/nltk_data…
[nltk_data] Package gutenberg is already up-to-date!
Out[1]:
[‘austen-emma.txt’,
‘austen-persuasion.txt’,
‘austen-sense.txt’,
‘bible-kjv.txt’,
‘blake-poems.txt’,
‘bryant-stories.txt’,
‘burgess-busterbrown.txt’,
‘carroll-alice.txt’,
‘chesterton-ball.txt’,
‘chesterton-brown.txt’,
‘chesterton-thursday.txt’,
‘edgeworth-parents.txt’,
‘melville-moby_dick.txt’,
‘milton-paradise.txt’,
‘shakespeare-caesar.txt’,
‘shakespeare-hamlet.txt’,
‘shakespeare-macbeth.txt’,
‘whitman-leaves.txt’]
To obtain all words in the entire Gutenberg corpus of NLTK, type the following:
In [2]:
gutenbergwords = nltk.corpus.gutenberg.words()
Now you can find the total number of words, and the first 10 words (do not attempt to display all the words or your computer will freeze!):
In [3]:
len(gutenbergwords)
Out[3]:
2621613
In [4]:
gutenbergwords[:10]
Out[4]:
[‘[‘, ‘Emma’, ‘by’, ‘Jane’, ‘Austen’, ‘1816’, ‘]’, ‘VOLUME’, ‘I’, ‘CHAPTER’]
You can also find the words of just a selection of documents, as shown below. For more details of what information you can extract from this corpus, read the “Gutenberg corpus” section of the NLTK book chapter 2, section 2.1.
In [5]:
emma = nltk.corpus.gutenberg.words(‘austen-emma.txt’)
len(emma)
Out[5]:
192427
In [6]:
emma[:10]
Out[6]:
[‘[‘, ‘Emma’, ‘by’, ‘Jane’, ‘Austen’, ‘1816’, ‘]’, ‘VOLUME’, ‘I’, ‘CHAPTER’]
As we have seen in the lectures, we can use Python’s collections.Counter to find the most frequent words of a document from NLTK’s Gutenberg collection. Below you can see how you can find the 5 most frequent words of the word list stored in the variable emma:
In [7]:
import collections
emma_counter = collections.Counter(emma)
print(emma_counter.most_common(5))
[(‘,’, 11454), (‘.’, 6928), (‘to’, 5183), (‘the’, 4844), (‘and’, 4672)]
Exercise 1.1¶
Write Python code that prints the 10 most frequent words in each of the documents of the Gutenberg corpus. Can you identify any similarities among these list of most frequent words?
In [8]:
austen-emma.txt
[(‘,’, 11454), (‘.’, 6928), (‘to’, 5183), (‘the’, 4844), (‘and’, 4672), (‘of’, 4279), (‘I’, 3178), (‘a’, 3004), (‘was’, 2385), (‘her’, 2381)]
austen-persuasion.txt
[(‘,’, 6750), (‘the’, 3120), (‘to’, 2775), (‘.’, 2741), (‘and’, 2739), (‘of’, 2564), (‘a’, 1529), (‘in’, 1346), (‘was’, 1330), (‘;’, 1290)]
austen-sense.txt
[(‘,’, 9397), (‘to’, 4063), (‘.’, 3975), (‘the’, 3861), (‘of’, 3565), (‘and’, 3350), (‘her’, 2436), (‘a’, 2043), (‘I’, 2004), (‘in’, 1904)]
bible-kjv.txt
[(‘,’, 70509), (‘the’, 62103), (‘:’, 43766), (‘and’, 38847), (‘of’, 34480), (‘.’, 26160), (‘to’, 13396), (‘And’, 12846), (‘that’, 12576), (‘in’, 12331)]
blake-poems.txt
[(‘,’, 680), (‘the’, 351), (‘.’, 201), (‘And’, 176), (‘and’, 169), (‘of’, 131), (‘I’, 130), (‘in’, 116), (‘a’, 108), (“‘”, 104)]
bryant-stories.txt
[(‘,’, 3481), (‘the’, 3086), (‘and’, 1873), (‘.’, 1817), (‘to’, 1165), (‘a’, 988), (‘”‘, 900), (‘he’, 872), (‘of’, 801), (‘was’, 706)]
burgess-busterbrown.txt
[(‘.’, 823), (‘,’, 822), (‘the’, 639), (‘he’, 562), (‘and’, 484), (‘to’, 426), (“‘”, 401), (‘of’, 326), (‘that’, 285), (‘a’, 275)]
carroll-alice.txt
[(‘,’, 1993), (“‘”, 1731), (‘the’, 1527), (‘and’, 802), (‘.’, 764), (‘to’, 725), (‘a’, 615), (‘I’, 543), (‘it’, 527), (‘she’, 509)]
chesterton-ball.txt
[(‘,’, 4547), (‘the’, 4523), (‘.’, 3589), (‘of’, 2529), (‘and’, 2488), (‘a’, 2184), (‘”‘, 1751), (‘to’, 1558), (‘in’, 1355), (‘that’, 1120)]
chesterton-brown.txt
[(‘the’, 4321), (‘,’, 4069), (‘.’, 2784), (‘of’, 2087), (‘and’, 2074), (‘a’, 2074), (‘”‘, 1461), (‘to’, 1378), (‘in’, 1205), (‘was’, 1141)]
chesterton-thursday.txt
[(‘,’, 3488), (‘the’, 3291), (‘.’, 2717), (‘a’, 1713), (‘of’, 1710), (‘and’, 1568), (‘”‘, 1336), (‘to’, 1045), (‘in’, 888), (‘I’, 885)]
edgeworth-parents.txt
[(‘,’, 15219), (‘the’, 7149), (‘.’, 6945), (‘to’, 5150), (‘and’, 4769), (‘”‘, 3880), (‘of’, 3730), (‘I’, 3656), (“‘”, 3293), (‘a’, 3017)]
melville-moby_dick.txt
[(‘,’, 18713), (‘the’, 13721), (‘.’, 6862), (‘of’, 6536), (‘and’, 6024), (‘a’, 4569), (‘to’, 4542), (‘;’, 4072), (‘in’, 3916), (‘that’, 2982)]
milton-paradise.txt
[(‘,’, 10198), (‘and’, 2799), (‘the’, 2505), (‘;’, 2317), (‘to’, 1758), (‘of’, 1486), (‘.’, 1254), (‘in’, 1083), (‘his’, 986), (‘with’, 876)]
shakespeare-caesar.txt
[(‘,’, 2204), (‘.’, 1296), (‘I’, 531), (‘the’, 502), (‘:’, 499), (‘and’, 409), (“‘”, 384), (‘to’, 370), (‘you’, 342), (‘of’, 336)]
shakespeare-hamlet.txt
[(‘,’, 2892), (‘.’, 1886), (‘the’, 860), (“‘”, 729), (‘and’, 606), (‘of’, 576), (‘to’, 576), (‘:’, 565), (‘I’, 553), (‘you’, 479)]
shakespeare-macbeth.txt
[(‘,’, 1962), (‘.’, 1235), (“‘”, 637), (‘the’, 531), (‘:’, 477), (‘and’, 376), (‘I’, 333), (‘of’, 315), (‘to’, 311), (‘?’, 241)]
whitman-leaves.txt
[(‘,’, 17713), (‘the’, 8814), (‘and’, 4797), (‘of’, 4127), (‘I’, 2932), (“‘”, 2362), (‘to’, 1930), (‘-‘, 1774), (‘.’, 1769), (‘in’, 1714)]
Exercise 1.2¶
Find the unique words with length of more than 17 characters in the complete Gutenberg corpus.
Hint: to find the distinct items of a Python list you can convert it into a set:
In [9]:
my_list = [‘a’,’b’,’c’,’a’,’c’]
my_set = set(my_list)
print(my_set)
print(len(my_set))
{‘c’, ‘b’, ‘a’}
3
In [10]:
3
{‘characteristically’, ‘Mahershalalhashbaz’, ‘uninterpenetratingly’}
Exercise 1.3¶
Find the words that are longer than 5 characters and occur more than 2000 times in the complete Gutenberg corpus.
In [11]:
[(‘little’, 2825), (‘before’, 3335), (‘people’, 2773), (‘children’, 2223), (‘should’, 2496), (‘against’, 2255), (‘Israel’, 2591)]
Exercise 1.4¶
Find the average number of words in the documents of the NLTK Gutenberg corpus.
In [12]:
Out[12]:
145645.16666666666
(Optional) Exercise 1.5¶
Find the Gutenberg document that has the longest average word length.
In [13]:
Document with largest average word length is milton-paradise.txt with word length 4.835734572682675
Exercise 1.6¶
Find the 10 most frequent bigrams in the entire Gutenberg corpus.
In [14]:
Out[14]:
[((‘,’, ‘and’), 41294),
((‘of’, ‘the’), 18912),
((‘in’, ‘the’), 9793),
((“‘”, ‘s’), 9781),
((‘;’, ‘and’), 7559),
((‘and’, ‘the’), 6432),
((‘the’, ‘LORD’), 5964),
((‘,’, ‘the’), 5957),
((‘,’, ‘I’), 5677),
((‘,’, ‘that’), 5352)]
Exercise 1.7¶
Find the most frequent bigram that begins with “Moby” in Herman Melville’s “Moby Dick”.
In [15]:
Out[15]:
[((‘Moby’, ‘Dick’), 83), ((‘Moby’, ‘-‘), 1)]
2. Text Preprocessing with NLTK¶
The following exercises will ask questions about tokens, stems, and parts of speech.
Exercise 2.1¶
What is the sentence with the largest number of tokens in Austen’s “Emma”?
In [16]:
[nltk_data] Downloading package punkt to /home/diego/nltk_data…
[nltk_data] Package punkt is already up-to-date!
The longest sentence is
While he lived, it must be only an engagement;
but she flattered herself, that if divested of the danger of
drawing her away, it might become an increase of comfort to him.–
How to do her best by Harriet, was of more difficult decision;–
how to spare her from any unnecessary pain; how to make
her any possible atonement; how to appear least her enemy?–
On these subjects, her perplexity and distress were very great–
and her mind had to pass again and again through every bitter
reproach and sorrowful regret that had ever surrounded it.–
She could only resolve at last, that she would still avoid a
meeting with her, and communicate all that need be told by letter;
that it would be inexpressibly desirable to have her removed just
now for a time from Highbury, and–indulging in one scheme more–
nearly resolve, that it might be practicable to get an invitation
for her to Brunswick Square.–Isabella had been pleased with Harriet;
and a few weeks spent in London must give her some amusement.–
She did not think it in Harriet’s nature to escape being benefited
by novelty and variety, by the streets, the shops, and the children.–
At any rate, it would be a proof of attention and kindness in herself,
from whom every thing was due; a separation for the present; an averting
of the evil day, when they must all be together again.
The mumber of words of the longest sentence is 275
Exercise 2.2¶
What are the 5 most frequent parts of speech in Austen’s “Emma”? Use the universal tag set
In [20]:
Out[20]:
[(‘VERB’, 35723),
(‘NOUN’, 31998),
(‘.’, 30304),
(‘PRON’, 21263),
(‘ADP’, 17880)]
Exercise 2.3¶
What is the number of distinct stems in Austen’s “Emma”?
In [21]:
Austen’s Emma has 5394 distinct stems
(Optional) Exercise 2.4¶
What is the most ambiguous stem in Austen’s “Emma”? (meaning, which stem in Austen’s “Emma” is realised in the largest number of distinct tokens?)
In [22]:
The most ambiguous stem is ‘respect’ with words {‘Respect’, ‘respecting’, ‘respects’, ‘respectful’, ‘respectable’, ‘respective’, ‘respectfully’, ‘respectably’, ‘respected’, ‘respectability’, ‘respect’}
In [ ]: