2021S2-workshop-week5-lab-solution
Elements Of Data Processing (2021S1) – Week 5¶
Regular Expressions (Regex)¶
Regular expressions allow you to match patterns in strings, rather than matching exact characters.
For example, if I wanted to find all phone numbers with form (03) xxxx xxxx, where x is some arbitrary digit, then I could use a regular expression like this: \(03\) \d\d\d\d \d\d\d\d
\(03\) \d{4} \d{4} where \d{4} matches a digit exactly 4 times
Here’s a good tutorial on Python regex: https://www.w3schools.com/python/python_regex.asp
and a website to test your regex expressions with test cases: https://regex101.com/
Regex with Python¶
The re library in Python allows you to use regular expressions.
Methods of note are: .search() (search for a particular pattern given a string)
.findall() (finds all substrings that match a given pattern)
.sub() (replaces all matched substrings with another given substring)
Regex Quantifiers¶
?: exactly zero or one occurrences of the preceding element
*: zero or more occurrences of the preceding element
+: one or more occurrences of the preceding element
{n}: preceding item is matched exactly n times
{,n}: preceding item is matched up to n times inclusive
{n,}: preceding item is matched at least n or more times
{m,n}: preceding item is matched at least m or more times, but up to n times inclusive
Escaping Special Characters¶
Like special characters in Python (i.e \n), you will also need to escape special characters in regex.
For example, if you wanted to match a literal bracket (, you have to type \( to escape it as () in regex is used to capture a literal group of characters
Consider the phone number from the example above.
In [6]:
import re
string = r’Name: Chris, ph: (03) 9923 1123, comments: this is not my real number’
# this is the regex pattern we want
# notice that we need to “escape” the brackets
pattern = r’\(03\) \d{4} \d{4}’
if re.search(pattern, string) :
print(“Phone number found”)
print(re.findall(pattern, string))
else :
print(“Not found”)
Phone number found
[‘(03) 9923 1123′]
Exercise 1 ¶
Modify the example above so that it will also find phone numbers starting with (03) that:
have missing brackets;
instead of a single space, uses hyphens, backslashes, and/or spaces.
Your program should match all elements in strings in the code segment below
In [7]:
# This examples looks for phone numbers that match the format above
import re
strings = [
r’Name: Chris, ph: (03) 9923 1123, comments: this is not my real number’,
r’Name: John, ph: 03-9923-1123, comments: this might be an old number’,
r’Name: Sara, phone: (03)-9923-1123, comments: there is data quality issues, so far, three people sharig the same number’,
r’Name: Christopher, ph: (03)\-9923 -1123, comments, is this the same Chris in the first record?’
]
# change this line
pattern = r’\(?03\)?[\\\s-]+\d{4}[\\\s-]+\d{4}’
for s in strings:
if re.search(pattern, s) :
print(“Phone number found”)
print(re.findall(pattern, s))
else :
print(“Not found”)
Phone number found
[‘(03) 9923 1123′]
Phone number found
[’03-9923-1123’]
Phone number found
[‘(03)-9923-1123’]
Phone number found
[‘(03)\\-9923 -1123’]
Exercise 2 ¶
Write a program that will remove all leading zeros from an IP address
For example, 0216.08.094.102 should become 216.8.94.196
Your program should match all elements in ip_addr in the code segment below
In [8]:
#Exercise 2: Write a program that will remove all leading zeros from an IP address
#For example, 0216.08.094.102 should become 216.8.94.196
import re
ip_addr = ‘0216.08.094.102’
#change this line
revised_addr = ip_addr
# solution
#revised_addr = re.sub(‘\.[0]*’, ‘.’, ip_addr)
pattern = r'(^|\.)0*’
revised_addr = re.sub(pattern, r’\1′ , ip_addr)
#
print(revised_addr)
216.8.94.102
Jaccard Similarity¶
Jaccard similarity (set-based) is a measure of calculating the similarity between two $n$-grams.
Let $A$ and $B$ be two $n$-grams. Then the Jaccard similarity can be computed as:
$$
\text{sim} = J(A, B) = \frac{A\cap B}{A\cup B}
$$
where:
The intersection is the number of common elements between the two sets;
and the union contains the set of all elements in the two sets.
For example, if I had two sets of numbers:
$A = \{0,1,2,5,6\}, B = \{0,2,3,4,5,7,9\}$
Then $A\cap B = \{0, 2, 5\}$ and $A\cup B = \{0, 1, 2, 3, 4, 5, 6, 7, 9\}$
Therefore, $J(A, B) = 3 / 9 = 0.33$
Exercise 3 ¶
Use nltk.util.ngram to produce bi-grams (your device may need to download the punkt toolbox for nltk).
Then calculate the Jaccard similarity for each bi-gram.
In [9]:
import nltk
sent1 = “crat”
sent2 = “cart”
ng1_chars = set(nltk.ngrams(sent1, n=2, pad_right=True, pad_left=True))
ng2_chars = set(nltk.ngrams(sent2, n=2, pad_right=True, pad_left=True))
similarity=len(set (ng1_chars) .intersection (set (ng2_chars)))/len(set (ng1_chars) .union (set (ng2_chars)))
print(similarity, “Jaccard Similarity between sent1 and sent2 with ngram 2”)
jd_sent_1_2 = nltk.jaccard_distance(ng1_chars, ng2_chars)
print(jd_sent_1_2, “Jaccard Distance between sent1 and sent2 with ngram 2”)
0.25 Jaccard Similarity between sent1 and sent2 with ngram 2
0.75 Jaccard Distance between sent1 and sent2 with ngram 2
Case Folding¶
Case folding removes all case distinctions present in a string. It is used for caseless matching, i.e. ignores cases when comparing.
Exercise 4 ¶
Use appropraite functions to covert string “Whereof one cannot speak, thereof one must be silent.” in
(1) Lower case
(2) Upper case
In [3]:
s = “Whereof one cannot speak, thereof one must be silent.”
print(s.lower())
print(s.upper())
whereof one cannot speak, thereof one must be silent.
WHEREOF ONE CANNOT SPEAK, THEREOF ONE MUST BE SILENT.
Natural Language Processing¶
The nltk library provides you with tools for natural language processing, including tokenizing, stemming and lemmatization
In [1]:
import nltk
from nltk.stem.porter import *
# first time:
#nltk.download(‘punkt’)
#nltk.download(‘stopwords’)
#
porterStemmer = PorterStemmer()
speech = ‘Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal. Now we are engaged in a great civil war, testing whether that nation, or any nation so conceived and so dedicated, can long endure. We are met on a great battle-field of that war. We have come to dedicate a portion of that field, as a final resting place for those who here gave their lives that that nation might live. It is altogether fitting and proper that we should do this. But, in a larger sense, we can not dedicate — we can not consecrate — we can not hallow — this ground. The brave men, living and dead, who struggled here, have consecrated it, far above our poor power to add or detract. The world will little note, nor long remember what we say here, but it can never forget what they did here. It is for us the living, rather, to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced. It is rather for us to be here dedicated to the great task remaining before us — that from these honored dead we take increased devotion to that cause for which they gave the last full measure of devotion — that we here highly resolve that these dead shall not have died in vain — that this nation, under God, shall have a new birth of freedom — and that government of the people, by the people, for the people, shall not perish from the earth.’
wordList = nltk.word_tokenize(speech)
# run the line to download it the first time:
#nltk.download(‘stopwords’)
from nltk.corpus import stopwords
stopWords = set(stopwords.words(‘english’))
filteredList = [w for w in wordList if not w in stopWords]
wordDict = {}
for word in filteredList:
stemWord = porterStemmer.stem(word)
if stemWord in wordDict :
wordDict[stemWord] = wordDict[stemWord] +1
else :
wordDict[stemWord] = 1
wordDict = {k: v for k, v in sorted(wordDict.items(), key=lambda item: item[1], reverse=True)}
for key in wordDict : print(key, wordDict[key])
, 22
. 10
— 7
dedic 6
nation 5
live 4
great 3
It 3
dead 3
us 3
shall 3
peopl 3
new 2
conceiv 2
men 2
war 2
long 2
We 2
gave 2
consecr 2
the 2
far 2
rather 2
devot 2
four 1
score 1
seven 1
year 1
ago 1
father 1
brought 1
forth 1
contin 1
liberti 1
proposit 1
creat 1
equal 1
now 1
engag 1
civil 1
test 1
whether 1
endur 1
met 1
battle-field 1
come 1
portion 1
field 1
final 1
rest 1
place 1
might 1
altogeth 1
fit 1
proper 1
but 1
larger 1
sens 1
hallow 1
ground 1
brave 1
struggl 1
poor 1
power 1
add 1
detract 1
world 1
littl 1
note 1
rememb 1
say 1
never 1
forget 1
unfinish 1
work 1
fought 1
thu 1
nobli 1
advanc 1
task 1
remain 1
honor 1
take 1
increas 1
caus 1
last 1
full 1
measur 1
highli 1
resolv 1
die 1
vain 1
god 1
birth 1
freedom 1
govern 1
perish 1
earth 1
Exercise 5 ¶
Modify the example above to use a WordNet Lemmatizer instead of a porter stemmer.
Comment on the differences
In [2]:
#Solution to Exercise :
#Modify the example above to use a WordNet Lemmatizer instead of a Porter Stemmer
#Comment on the differences
import nltk
from nltk.stem.porter import *
from nltk.stem import WordNetLemmatizer
# first time, run the line:
#nltk.download(‘wordnet’)
# Init the Wordnet Lemmatizer
lemmatizer = WordNetLemmatizer()
speech = ‘Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal. Now we are engaged in a great civil war, testing whether that nation, or any nation so conceived and so dedicated, can long endure. We are met on a great battle-field of that war. We have come to dedicate a portion of that field, as a final resting place for those who here gave their lives that that nation might live. It is altogether fitting and proper that we should do this. But, in a larger sense, we can not dedicate — we can not consecrate — we can not hallow — this ground. The brave men, living and dead, who struggled here, have consecrated it, far above our poor power to add or detract. The world will little note, nor long remember what we say here, but it can never forget what they did here. It is for us the living, rather, to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced. It is rather for us to be here dedicated to the great task remaining before us — that from these honored dead we take increased devotion to that cause for which they gave the last full measure of devotion — that we here highly resolve that these dead shall not have died in vain — that this nation, under God, shall have a new birth of freedom — and that government of the people, by the people, for the people, shall not perish from the earth.’
wordList = nltk.word_tokenize(speech)
from nltk.corpus import stopwords
stopWords = set(stopwords.words(‘english’))
filteredList = [w for w in wordList if not w in stopWords]
wordDict = {}
for word in filteredList:
stemWord = lemmatizer.lemmatize(word)
if stemWord in wordDict :
wordDict[stemWord] = wordDict[stemWord] +1
else :
wordDict[stemWord] = 1
wordDict = {k: v for k, v in sorted(wordDict.items(), key=lambda item: item[1], reverse=True)}
for key in wordDict : print(key, wordDict[key])
#The most obvious difference is that the lemmatizer produces actual words rather than stems.
#Lemmatizer will also handle case differences (e.g. eat vs ate) that the Stemmer will not.
, 22
. 10
— 7
nation 5
dedicated 4
great 3
It 3
dead 3
u 3
shall 3
people 3
new 2
conceived 2
men 2
war 2
long 2
We 2
dedicate 2
gave 2
The 2
living 2
far 2
rather 2
devotion 2
Four 1
score 1
seven 1
year 1
ago 1
father 1
brought 1
forth 1
continent 1
Liberty 1
proposition 1
created 1
equal 1
Now 1
engaged 1
civil 1
testing 1
whether 1
endure 1
met 1
battle-field 1
come 1
portion 1
field 1
final 1
resting 1
place 1
life 1
might 1
live 1
altogether 1
fitting 1
proper 1
But 1
larger 1
sense 1
consecrate 1
hallow 1
ground 1
brave 1
struggled 1
consecrated 1
poor 1
power 1
add 1
detract 1
world 1
little 1
note 1
remember 1
say 1
never 1
forget 1
unfinished 1
work 1
fought 1
thus 1
nobly 1
advanced 1
task 1
remaining 1
honored 1
take 1
increased 1
cause 1
last 1
full 1
measure 1
highly 1
resolve 1
died 1
vain 1
God 1
birth 1
freedom 1
government 1
perish 1
earth 1
In [ ]: