exercise1-checkpoint
Exercise one¶
Part A: Text preprocessing¶
Load the json data¶
I use the json library to convert each line to python object.
and then extract the content and convert to lowercase.
In [1]:
import json
news = []
with open(“signal-news1/signal-news1.jsonl”) as newsfile:
for line in newsfile:
news.append(json.loads(line))
In [2]:
# extract the content and convert to lowercase
contents = [l[“content”].lower() for l in news]
Remove URLS¶
My url regular expression is “(http[s]?://\S+)|(www.\S+)|(\S+.(com|org|edu))”.
It will match 3 kind of urls. (http[s]?://\S+) will match things such as “http://www”, ““http://domain”, “https://www.*, (www.\S+) will match things such as ““www.*”. and “(\S+.(com|org|edu))” will
match things such as “domain.com”
Some example matched URLS for the news story.
http://bit.ly/1jd3nq4
https://youtu.be/khlhd2ex8wk
amazon.com
analystratingsnetwork.com
www.twitter.com/russostrib
oma-online.org
Use the re.sub to replace matched urls with empty string.
In [38]:
import re
urlReg = “(http[s]?://\S+)|(www\.\S+)|(\S+\.(com|org|edu))”
# show some of the matched URLS to ensure it is correct
for n in contents:
url = re.search(urlReg, n)
if url:
pass
# print(url.group(0))
savingcountrymusic.com
marketbeat.com
marketbeat.com
analystratingsnetwork.com
confidence:link.reuters.com
india.com
network.com
gstubbe@startribune.com
les.roopanarine@theguardian.com
pic.twitter.com
analystratingsnetwork.com
cars.com
(gmm/inautonews.com
uefa.com
amazon.com
usmagazine.com
facebook.com
tmz.com
oma-online.org
marketbeat.com
zacks.com
philly.com
moschino.com
zacks.com
hollywoodlife.com
stocktargetprices.com
marketbeat.com
stocktargetprices.com
http://bit.ly/1jd3nq4
(zenit.org
zacks.com
goldenderby.com
beat.com
zacks.com
ashleymadison.com
amazon.com
massivechangenetwork.com
http://t.co/hn0ctno9cc
expressandstar.com
https://youtu.be/khlhd2ex8wk
fararu.com
radaronline.com
orlandosentinel.com
india.com
marketbeat.com
espn.com
autoguide.com
maxprep.com
social.com
mlive.com
beat.com
livenation.com
factcheck.org
tnaumetz@hilltimes.com
heatherchristo.com
analystratingsnetwork.com
http://t.co/gdcbjeai7r
ctaloma@reviewjournal.com
network.com
honest.com
motorsport.com
india.com
http://bit.ly/1iktvfi)
amazon.com
reviewed.com
usmagazine.com
mlive.com
tacoheads.com
mmamania.com
www.designengineering.com,
stocktargetprices.com
o.canada.com
stocktargetprices.com
zacks.com
beat.com
analystratingsnetwork.com
booking.com
salesforce.com
scientificamerican.com
(www.albawaba.com
http://mney.co/1fuhjwe
csnne.com
transact.com
admin@camerasport.com
analystratingsnetwork.com
pic.twitter.com
expedia.com
marketbeat.com
apple.com
https://twitter.com/jeneps/status/wasserman
http://twitter.com/share?text=‘fuller
accuweather.com
Tweets by XHNews
mlssoccer.com
marketbeat.com
zacks.com
www.gorancho.com.
https://tools.bcsc-scc.org/bc5yearrisk.
analystratingsnetwork.com
goal.com
usmagazine.com
pregame.com
stocktargetprices.com
ktaylor@entrepreneur.com
elliottbay.com
www.twitter.com/russostrib
pic.twitter.com
www.genworth.com/costofcare,
bizjournals.com
zacks.com
boxing247.com
zacks.com
zacks.com
zacks.com
thiellsdentist.com
ringtv.com
stocktargetprices.com
djcarguy.com
premierleague.com
(www.liverpoolfc.com
www.wisconsinfishingclub.com
seasidemarket.com
marketbeat.com
sparkfun.com
http://twitter.com/share?text=
goal.com
tvguide.com
newsmax.com
clee@post-dispatch.com
alexa.com
tvguide.com
www.wprugby.com
radaronline.com
credit.com
scott.eidler@newsday.com
zacks.com
fall.com
www.lat.ms/1jramb0),
zacks.com
[philly.com
cnstock.com
http://samspades.ca)
t.mccarthy@ibtimes.com
contratados.org
network.com
network.com
analystratingsnetwork.com
http://www.x17online.com/2015/09/jessica_simpson_pantsuit_lax_photos_090715
forbes.com
ongobankingrates.com
idntimes.com
www.storiesonstage.org
geeky-gadgets.com
boxingnews24.com
espn.com
monmouthpark.com
usmagazine.com
frys.com
geeky-gadgets.com
rivals.com
essence.com
http://www.x17online.com/2015/09/ruby_rose_leather_pants_tee_shirt_photos_091515
marketbeat.com
zacks.com
rac1.org
www.shutterstock.com
a.garofalo@ibtimes.com
tickerreport.com
zacks.com
archive.org
americablog.com
offbeat.com
network.com
www.nationalpost.com
gearheads.org
beat.com
marketbeat.com
googlemaps.com
boxing247.com
cosmopolitan.com
(http://creativecommons.org
t.mccarthy@ibtimes.com
analystratingsnetwork.com
chron.com
powerpost@washpost.com
uefa.com
pic.twitter.com
b.com
network.com
comedycentral.com
upi.com
a.pandey@ibtimes.com
marketbeat.com
fashionweekdaily.com
byhours.com
marketbeat.com
analystratingsnetwork.com
dishanywhere.com
http://mney.co/1qmikfb
zacks.com
mlssoccer.com
marketbeat.com
motorsport.com
hollywoodlife.com
(thekyliejenner.com
radaronline.com
jd.com
broadway.com
outlook.com
s.varandani@ibtimes.com
society6.com
trivago.com
stocktargetprices.com
readnewcastle.com
usmagazine.com
monster.com
huffingtonpost.com
network.com
gtagaming.com
marketbeat.com
whoscored.com
outlook.com
boxingnews24.com
india.com
zacks.com
tedcruz.org
zacks.com
nj.com
www.ticketweb.com.
goldderby.com
www.thebroad.org/tickets
http://www.londonfashionweek.co.uk/news/717/london-fashion-week-moves-to-s
http://www.torquenews.com
worstpreviews.com
philly.com
marketbeat.com
rio2016.com
commonfloor.com
zacks.com
http://twitter.com/share?text=ratings:
google.com
stocktargetprices.com
usmagazine.com
network.com
usmagazine.com
network.com
contemporarycraft.org
oddsshark.com
network.com
src_url=”https://spshared.5min.com
zacks.com
beat.com
d.sirota@ibtimes.com
stocktargetprices.com
analystratingsnetwork.com
boxingnews24.com
marketbeat.com
(www.albawaba.com
geekschicago.com
k.flynn@ibtimes.com
internet.org
facebook.com
link.reuters.com
espn.com
zacks.com
marketbeat.com
geeky-gadgets.com
http://bit.ly/1hakd45
shineonstpete.com
latimes.com
hitrecord.org
putitallinaplasticbag.com
(on.ft.com
analystratingsnetwork.com
analystratingsnetwork.com
forbes.com
nj.com
usatoday.com
ckohlruss@fresnobee.com
zacks.com
rvelotta@reviewjournal.com
liverpoolfc.com
tvguide.com
j.glum@ibtimes.com
scientificamerican.com
rottentomatoes.com
www.shutterstock.com
india.com
sciencebasedmedicine.org
zacks.com
(www.milliestrust.com
analystratingsnetwork.com
(gmm/inautonews.com
zacks.com
www.dailyrecord.co.uk…
http://twitter.com/share?text=miss
boxingnews24.com
swimoutlet.com
nj.com
chron.com
impactalpha.com
www.alt-tickets.co.uk/lush-tickets
autoguide.com
marketbeat.com
network.com
tennisworld.org
usmagazine.com
http://twitter.com/share?text=ratings:
analystratingsnetwork.com
pparx.org
today.com
bandonthewall.org
network.com
huffingtonpost.com
www.gizbot.com}
espn.com
nando.com
marketbeat.com
http://president.rutgers.edu/files/final-report.pdf.
analystratingsnetwork.com
(http://creativecommons.org
boxingnews24.com
http://t.co/ymszmrgex0
http://www.x17online.com/2015/09/ashton_kutcher_mila_kunis_sushi_date_night_photos_090915
beat.com
analystratingsnetwork.com
goal.com
zacks.com
analystratingsnetwork.com
hellofashion.com
usmagazine.com
boxingnews24.com
change.org
usmagazine.com
worstpreviews.com
bestoflasvegas.com
sephora.com
squawka.com
a.garofalo@ibtimes.com
http://twitter.com/share?text=‘empire’
https://t.co/20sgksku4o
deadline.com
pic.twitter.com
network.com
www.wearemoviegeeks.com
cbsla.com
beat.com
www.cbc.ca
franchiseplayers@entrepreneur.com
http://mney.co/1o3k0hh
salon.com
stocktargetprices.com
georgiahealthnews.com
network.com
marketbeat.com
espn.com
www.pointzero.ca)
https://t.co/uvqxm1wvmz
www.sportspromedia.com/.new
marketbeat.com
(chinapost.com
tickerreport.com
corruption.com
pic.twitter.com
usmagazine.com
mmamania.com
familysearch.org
analystratingsnetwork.com
calciomercatoweb.com
http://www.goalzero.com.
marketbeat.com
goal.com
zacks.com
pe.com
pic.twitter.com
m.vultaggio@ibtimes.com
www.jonnopromotions.com/events
marketbeat.com
security.openstack.com
marketbeat.com
benzinga.com
petergreenberg.com
o.com
soccerlens.com
pitchfork.com
beat.com
fighthype.com
marketbeat.com
instagram.com
starwars.com
speedcafe.com
arsenal.com
hackteria.org
www.gamasutra.com
unsettle.org
english.com
boxingscene.com
marketbeat.com
freightquote.com
marketbeat.com
tvguide.com
stocktargetprices.com
zacks.com
whoscored.com
src_url=”https://spshared.5min.com
usmagazine.com
petergreenberg.com
(chinapost.com
omaha.com
tucson.com
zacks.com
tucson.com
m.kaplan@ibtimes.com
zacks.com
www.cbc.ca
zacks.com
goal.com
tucson.com
india.com
hollywoodlife.com
zacks.com
geeky-gadgets.com
beat.com
http://t.co/gdcbjeai7r
s.varandani@ibtimes.com
autoguide.com
patpaynedirector@me.com
wired.com
(on.wsj.com
niall.mcveigh.casual@theguardian.com
network.com
http://link.reuters.com/nav65w
siliconrepublic.com
10news.com
www.cbc.ca
whoscored.com
reviewed.com
http://twitter.com/share?text=21st
autoguide.com
marketbeat.com
zacks.com
ufc.com
(on.ft.com
(www.albawaba.com
www.cbc.ca
beat.com
marketbeat.com
indianexpress.com
analystratingsnetwork.com
usmagazine.com
deadspin.com
manutd.com
india.com
rogerebert.com
http://www.journalotohns.com/content/44/1/39
tickerreport.com
uefa.com
jordanstudio.com
marketbeat.com
zacks.com
marketbeat.com
india.com
eonline.com
startrekthecruise.com
beat.com
zacks.com
usmagazine.com
forbes.com
boxingnews24.com
zacks.com
(phys.org
radaronline.com
scientificamerican.com
scientificamerican.com
network.com
n4g.com
marketbeat.com
zacks.com
beat.com
l.tomkiw@ibtimes.com
espn.com
soccerlens.com
analystratingsnetwork.com
stocktargetprices.com
creditcards.com
comicbook.com
https://youtu.be/fa4hup_armk
www.cbc.ca
usmagazine.com
pwronline.org
beat.com
zacks.com
usmagazine.com
xbox.com
marketbeat.com
models.com
mobilesyrup.com
zacks.com
zacks.com
marketbeat.com
jd.com
www.etonline.com
analystratingsnetwork.com
http://mney.co/1pgevht
theeroticreview.com
zacks.com
network.com
zacks.com
(chinapost.com
beat.com
http://twitter.com/share?text=carly
screenrant.com
newegg.com
salon.com
http://www.soulfreshclothing.com
marketbeat.com
(gmm/inautonews.com
zacks.com
kiplinger.com
network.com
www.sportspromedia.com/.as
outlook.com
biscuitsandblues.com
marketwatch.com
http://goo.gl/cdcdtj
usopen.org
depot.http://lifehacker.com
nydailynews.com
canadiens.com
marketbeat.com
http://strib.mn/1urhju8
zacks.com
distribution360.com
(chinapost.com
tenaa.com
marketbeat.com
seewalds.com
http://bit.ly/1mwqs8u
http://francisfactor.com
greg@tv-eh.com
usmagazine.com
tvguide.com
zacks.com
s.varandani@ibtimes.com
whoscored.com
http://townta.lk/1gthiwm
a.garofalo@ibtimes.com
src_url=”https://spshared.5min.com
people.com
analystratingsnetwork.com
s.varandani@ibtimes.com
www.cbc.ca
whoscored.com
www.cbc.ca
customerservice@seattletimes.com
boxingnews24.com
twitter.com
mlb.com
marketbeat.com
http://www.investors.bestbuy.com).
fararu.com
https://www.youtube.com/watch?v=fb4mnemwbsq
bet.com
http://t.co/ivjsvfdox1
mutualfundobserver.com
johnnybsburgersandbrew.com
fraud.org
www.nytimes.com
http://:”
http://bit.ly/1l6cu8g
hollywoodlife.com
marketbeat.com
marketbeat.com
hollywoodlife.com
tickerreport.com
tvguide.com
secondspic.twitter.com
www.sportspromedia.com/.tennent’s,
marketbeat.com
usmagazine.com
beat.com
medicare.com
bishopaccountability.org
(chinapost.com
marketbeat.com
geeky-gadgets.com
zacks.com
axs.com
pic.twitter.com
n.massabrook@ibtimes.com
beat.com
www.infoworld.com
computerworld.com
autoguide.com
www.martynamajok.com
cleveland.com
india.com
geeky-gadgets.com
sleekmoney.com
autoguide.com
grandcanyonracing.com
givenchynyfw15.com
#facebookpic.twitter.com
zacks.com
http://www.organicconsumers.org/articles/article_3697.cfm
qq.com
j.salo@ibtimes.com
http://t.co/grt1hwbx6x
www.pittsburghplayhouse.com
www.teentechtank.com
analystratingsnetwork.com
usmagazine.com
tone.http://lifehacker.com
zacks.com
oddsshark.com
autoguide.com
http://twitter.com/share?text=‘empire’
reddit.com
aceshowbiz.com
ingresso.com
beat.com
beat.com
http://www.x17online.com/2015/09/orlando_bloom_dating_kendall_jenner_revenge_justin_bieber_feud_092415
zacks.com
stocktargetprices.com
zacks.com
zacks.com
zacks.com
theguardian.com
marketbeat.com
trippy.com
zacks.com
(chinapost.com
amazon.com
zacks.com
m.kaplan@ibtimes.com
http://www.bls.gov/ces/).
salesforce.com
tickerreport.com
gizmag.com
zacks.com
http://www.prweb.com/releases/2015/09/prweb12963313.htm
soccerlens.com
tucson.com
stocktargetprices.com
(phys.org
(shivannamagalamaduve.com
play.google.com
marketbeat.com
network.com
marketbeat.com
beat.com
change.org
jacquelinegoldfinger.com
rdivish@seattletimes.com
usmagazine.com
csnnw.com
edmunds.com
marketbeat.com
http://t3live.com/festival
marketbeat.com
kdelacruz@reviewjournal.com
bookarestaurant.com
http://reut.rs/1k2kpn4
nbcsports.com
matthew_simon@wired.com
breastcancer.org
(chinapost.com
zacks.com
beat.com
marketbeat.com
autoguide.com
scientificamerican.com
gizmag.com
htc.com
zacks.com
amazon.com
tripadvisor.com
beat.com
(www.albawaba.com
zacks.com
forbes.com
boxing247.com
www.gizmag.com
tvguide.com
smith/credit.com
beat.com
speedcafe.com
gasbuddy.com
a.garofalo@ibtimes.com
lennyletter.com
pic.twitter.com
amazon.com
chron.com
zacks.com
archive.org
polygon.com
cleveland.com
getawaymavens.com
tucson.com
boxing247.com
marketbeat.com
http://t.co/1fsxgu…
barrons.com
editorial@rttnews.com
www.taxcredits.net
gary.dymski@newsday.com
india.com
http://twitter…
meetup.com
https://www.audiokinetic.com/limbo-redux-contest
http://mney.co/1iohudt
broadway.com
geeky-gadgets.com
stocktargetprices.com
northjersey.com
stocktargetprices.com
poconorecord.com
bestbuy.com
usmagazine.com
boxingnews24.com
foodiggity.com
nj.com
http://twitter.com/share?text=here’s
omaha.com
motorsport.com
steelers.com
www.bulletsoverbroadway.com
petergreenberg.com
analystratingsnetwork.com
twitter.com
bryan.graham@theguardian.com
zacks.com
internet.org
tickerreport.com
j.salo@ibtimes.com
http://lifehacker.com/5803333/how-to…
usmagazine.com
boxing247.com
marketbeat.com
a.young@ibtimes.com
hollywoodlife.com
popefrancisvisit.com
http://www.amazon.com/fitbit-charge
marketbeat.com
goal.com
beat.com
tvguide.com
marketbeat.com
http://twitter.com…
hollywoodlife.com
pic.twitter.com
http://movieweb.com/man-steel-ending-batman-v-superman-zack-snyder/kal-el
cnbc.com
j.mchugh@ibtimes.com
zacks.com
radiotimes.com
marketbeat.com
hellcat.org
mpix.com
uefa.com
www.cbc.ca
m.willens@ibtimes.com
zacks.com
zacks.com
credit.com
marketbeat.com
56.com
amazon.com
www.cityofprescott.net,
beat.com
(www.whatireallywanted.com
hsh.com
stocktargetprices.com
espn.com
drudgereport.com
zacks.com
phil.nobile@newsday.com
www.ipolitics.ca
network.com
(gmm/inautonews.com
http://trib.in/1hnoxf6
geeky-gadgets.com
hollywoodlife.com
analystratingsnetwork.com
b.lee@ibtimes.com
(allafrica.com
marketbeat.com
usmagazine.com
style.com
lesmis.com
http://t3live.com/festival
www.cbc.ca
beat.com
aseng@sacbee.com
beat.com
network.com
zacks.com
free-times.com
change.org
prusa.com
soccerlens.com
pic.twitter.com
marketbeat.com
internet.org
https://www.youtube.com/watch?t=1&v=qc_ivhcgpqc
zacks.com
marketbeat.com
stocktargetprices.com
lamparski/wireimage.com
upi.com
www.cbc.ca
boxingnews24.com
india.com
beat.com
usmagazine.com
www.theintersection.org.
https://youtu.be/ycim_jya1nw
247sports.com
nj.com
usmagazine.com
bishopaccountability.org
facebook.com
zacks.com
zacks.com
www.deeplocal.com
wlex/lex18.com
http://lat.ms/1ulztrd
usmagazine.com
frys.com
transequality.org
salesforce.com
marketbeat.com
amazon.com
marketbeat.com
‘1.widget3.zacks.com
mlssoccer.com
localeateryandpub.com
pic.twitter.com
(zenit.org
tucson.com
nfl.com
marketbeat.com
http://bit.ly/1ky2xfd
india.com
pistons.com
radaronline.com
zacks.com
usmagazine.com
pic.twitter.com
http://twitter.com/xhnews,
coches.com
https://twitter.com/spasepeople/st…
boxing247.com
wyattresearch.com
zacks.com
www.cbc.ca
marketbeat.com
vf.com
indianexpress.com
pic.twitter.com
marketbeat.com
zacks.com
marketbeat.com
usmagazine.com
9to5mac.com
amazon.com
vogue.com
www.realtytrac.com.
substreammagazine.com
http://twitter.com/share?text=‘dancing
tvguide.com
kdelacruz@reviewjournal.com
geeky-gadgets.com
boxingnews24.com
http://techcrunch.com/tag/apple-fall-2015-event/
espn.com
bryan.graham@theguardian.com
marketbeat.com
analystratingsnetwork.com
http://bit.ly/1irfpmd
http://www.teatroallascala.org/en/index.html
forbes.com
marketbeat.com
boxingnews24.com
sleekmoney.com
change.org
ok.com
http://twitter.com/share?text=‘game
(nfl.com
internet.org
thedailyshow.com
zacks.com
jd.com
www.nutraingredients.com
http://www.nasa.gov/1ym.
a.tejas@ibtimes.com
http://twitter.com/share?text=
https://t.co/hb1wxwsjev
http://bit.ly/1onmxrb)
amazon.com
fichajes.com
www.cd-productions.co.uk.
karmaautomotive.com
marketbeat.com
mavs.com
(http://www.wcitt2d.org
salesforce.com
http://www.sridharan.de/)
admin@camerasport.com
www.gizbot.com}
usmagazine.com
techcrunch.com
zacks.com
scientificamerican.com
‘1.widget3.zacks.com
pic.twitter.com
apple.com
beat.com
boxingnews24.com
votepact.org
boxing247.com
http://nwsdy.li/1mgi8ww
marketbeat.com
www.londonshootingclub.com
network.com
www.nancybirtwhistle.co.uk
tvguide.com
analystratingsnetwork.com
network.com
marketbeat.com
itworld.com
hollywoodlife.com
marketbeat.com
analystratingsnetwork.com
network.com
(zenit.org
(on.wsj.com
nopi-restaurant.com
www.sportspromedia.com/.under
oyster.com
network.com
(511nj.org
livenation.com
marketbeat.com
marketbeat.com
beat.com
marketbeat.com
https://t.co/vjkjmt6sqg
https://www.youtube.com/watch?t=5&v=ynmrm7fzvzk
stockmonkeys.com
http://bit.ly/1foxpr2
bishopaccountability.org
beat.com
apayne@sacbee.com
http://twitter.com/share?text=‘meet
zacks.com
n.massabrook@ibtimes.com
commercialappeal.com
http://bit.ly/1noo2zc
noblerotevents@gmail.com
marketbeat.com
marketbeat.com
tvguide.com
network.com
usmagazine.com
uswitch.com
(chinapost.com
radaronline.com
http://www.rubyroxxmodel.com.
http://mney.co/1q1tkoa
hollywoodlife.com
analystratingsnetwork.com
marketbeat.com
amazon.com
http://goo.gl/siif2f
analystratingsnetwork.com
zacks.com
usmagazine.com
espn.com
tampabay.com
edmunds.com
booking.com
expressandstar.com
(gmm/inautonews.com
india.com
analystratingsnetwork.com
s.varandani@ibtimes.com
http://bit.ly/1igihbj
barrons.com
jet.com
zacks.com
www.upi.com
c.harress@ibtimes.com
//platform.twitter.com
network.com
zacks.com
http://www.americangeosciences.org/sites/default/files/education-reports-secondaryes_report.pdf
http://classroom.4teachers.org
stocktargetprices.com
news@hilltimes.com
soccerlens.com
tickerreport.com
zacks.com
soccerlens.com
http://deadspin.com/lawyer-of-patr…
target.com
bankrate.com
www.gizbot.com}
tickerreport.com
www.epicportions.com)
analystratingsnetwork.com
nhl.com
soccerlens.com
http://t3live.com/festival
nj.com
boxing247.com
http://www.x17online.com/2015/09/kim_zolciak_eliminated_dwts_failed_compete_stroke_092915
radaronline.com
network.com
hockeyanalysis.com
www.bushwig.com.
rraihala@pioneerpress.com
usmagazine.com
network.com
d.gilbert@ibtimes.com
radaronline.com
credit.com
c.hitt@ibtimes.com
marketbeat.com
cschreiber@ocregister.com
cinemablend.com
www.audit…
cleveland.com
http://twitter.com/share?text=nbcuniversal,
zacks.com
india.com
zacks.com
worstpreviews.com
http://tsp.plasa.org
hashtags.org
zacks.com
boxing247.com
marketbeat.com
tvguide.com
www.ticketsource.co.uk/whitecobra
www.thisisspotlight.com.
boxingnews24.com
boxing247.com
scientificamerican.com
https://www.youtube.com/watch?v=ikjqdgz9jro
beat.com
hoymercedes.com
oregonlive.com
(www.albawaba.com
(chinapost.com
usmagazine.com
stocktargetprices.com
lawrence.ostlere@theguardian.com
analystratingsnetwork.com
(www.arsenal.com
http://mney.co/1uywuvj
beat.com
zacks.com
http://www.x17online.com/2015/09/kim_kardashian_jokes_kylie_jenner_dethroned_her_092515
today.com
zacks.com
earthtrekkers.com
boxingnews24.com
macrumors.com
style.com
stlhighschoolsports.com
stocktargetprices.com
townhall.com
breastcancer.org
africa.com
geeky-gadgets.com
350.org
hollywoodlife.com
www.pcgamer.com
watchlistnews.com
editorial@rttnews.com
http://bit.ly/1j01ome
network.com
uefa.com
www.foodnavigator.com
zacks.com
parcel2go.com
(allafrica.com
ok.com
scout.com
http://www.x17online.com/2015/09/justin_bieber_shirtless_nature_instagram_photos_090215
usmagazine.com
zacks.com
zacks.com
http://twitter.com/s…
oregonstateparks.org
nhl.com
upi.com
tuenight.com
(www.albawaba.com
beat.com
http://www.saison.or.jp/english/application/index.html
www.fishwrapper.com
cleveland.com
digitaltrends.com
android4schools.com
network.com
s.varandani@ibtimes.com
boxing247.com
marketbeat.com
viperexchange.com
geeky-gadgets.com
http://deadspin.com/roger-goodell
rt.com
marketbeat.com
(drag2zero.com
playbill.com
boxing247.com
hollywoodlife.com
marketbeat.com
theybf.com
analystratingsnetwork.com
http://goo.gl/mqgdeq
http://mney.co/1l6al7z
jd.com
beat.com
http://jobyek.blogspot.ca/2015/09/micro-tut-wheels.html
zacks.com
zacks.com
www.oldfirstconcerts.org.
tvguide.com
www.cbc.ca
(http://creativecommons.org
stocktargetprices.com
marketbeat.com
marketbeat.com
zacks.com
marketbeat.com
facebook.com
http://twitter.c…
www.mercedes.me/she.
(zenit.org
(www.albawaba.com
autoguide.com
google.com
www.cbc.ca
c.hitt@ibtimes.com
masslive.com
marketbeat.com
zacks.com
skeptic.com
agathachristie.com
gdiaz@tribune.com
infphoto.com
boxing247.com
espncricinfo.com
timesofisrael.com
stocktargetprices.com
goldderby.com
(allafrica.com
www.cbc.ca
nj.com
http://mney.co/1ujh1tl
espn.com
(chinapost.com
marketbeat.com
http://mney.co/1vx2izl
abcmouse.com
betterthandieting.com
mlssoccer.com
radaronline.com
(http://on.freep.com
szondy/gizmag.com
greats.com
carolinacuetogo.com
cfletes@post-dispatch.com
stocktargetprices.com
http://bit.ly/1fg9cji.
usmagazine.com
fraternitymom.com
vwdieselinfo.com
mybroadwaybody.com
gog.com
goal.com
(chinapost.com
network.com
latimes.com
stocktargetprices.com
mlssoccer.com
glassdoor.com
sleekmoney.com
zacks.com
rtorres@reviewjournal.com
nhl.com
www.nature.com/articles/srep13368
soccerlens.com
zacks.com
www.cbc.ca
marketbeat.com
zacks.com
http://t.co/jzo2zxfcxw—
briefing.com
espn.com
townhall.com
boxingnews24.com
http://mney.co/1fzroqh
marketbeat.com
marketbeat.com
http://bit.ly/1r2dcsw
http://www.reaganfoundation.org
nytimes.com
marketbeat.com
zacks.com
instagram.com
nj.com
facebook.com
marketbeat.com
people.com
http://t.co/cbuqepogxv
tvguide.com
c.dunn@ibtimes.com
http://twitter.com/share?text=lapd
hollywoodlife.com
http://www.x17online.com/2015/09/bobbi_kristina_brown_dead_boyfriend_nick_gordon_090915
www.sportspromedia.com/.the
amazon.com
mashable.com
usmagazine.com
usmagazine.com
a.young@ibtimes.com
vogue.com
beat.com
espn.com
foxbusiness.com
salesforce.com
hotels.com
skysports.com
marketbeat.com
www.friendsarena.se
vulture.com
zacks.com
www.huffingtonpost.com
tumblr.com
scientificamerican.com
tourdafrique.com
network.com
pic.twitter.com
t.marcin@ibtimes.com
foxsports.com
beat.com
beat.com
pkitagaki@sacbee.com
zacks.com
j.mchugh@ibtimes.com
internet.org
fhwaterman.com
snapdeal.com
analystratingsnetwork.com
http://t.co/0j5tvfu8xo
http://bit.ly/1enlsgf
india.com
zacks.com
stocktargetprices.com
analystratingsnetwork.com
forbes.com
marketbeat.com
theguardian.com
(chinapost.com
zacks.com
tvguide.com
(http://creativecommons.org
marketbeat.com
analystratingsnetwork.com
www.mtv.com
technobuffalo.com
hollywoodlife.com
stockmonster.com
10for10@communitybizbank.com
http://epsteinresearch.com
sosassociates.com
nfl.com
tvguide.com
usmagazine.com
http://mney.co/1fywqep
www.cbc.ca
givenchynyfw15.com
empireonline.com
news@hilltimes.com
www.tradeking.com
vogue.com
http://a/%%30%30
https://twitter.com/shamscharania/status/644294749206257664
www.caiff.org
rbenton@sacbee.com
forbes.com
c.hitt@ibtimes.com
indiespeedrun.com
credit.com
premierboxingchampions.com
(zenit.org
network.com
cleveland.com
radaronline.com
petergreenberg.com
http://mney.co/1mezit2
zacks.com
(http://creativecommons.org
marketbeat.com
boxingnews24.com
http://twitter.com/share?text=james
espn.com
internet.org
internet.org
web.com
espn.com
zacks.com
www.twitter.com/sungilbertson
broadway.com
(chinapost.com
zacks.com
usmagazine.com
barron’s.com
r.schumann@ibtimes.com
geeky-gadgets.com
sfcurran.com
pic.twitter.com
(zenit.org
india.com
https://cycliq.com
boohoo.com
http://oto.sagepub.com/content/153/1_suppl.toc.
force.com
cars.com
answer@pcworld.com
cubs.com
marketbeat.com
(link.reuters.com
www.twitter.com/nhl,
(gmm/inautonews.com
vogue.com
kookauthor.eventbrite.com
myafibrisk.com
huffingtonpost.com
myafibrisk.com
dosomething.org
www.amazon.com/primevideo—the
profootballtalk.com
www.nutraingredients.com
network.com
cleveland.com
boxing247.com
simmons@aquilacommercial.com
twitter.com
coverfox.com
www.india.com
zacks.com
moves@thomsonreuters.com
marketbeat.com
zacks.com
theguardian.com
www.huffingtonpost.com
raysbaseball.com
marketbeat.com
radaronline.com
http://play.it/perez
denverpost.com
marketbeat.com
nj.com
tmz.com
beat.com
sleekmoney.com
phonicfuryclub@gmail.com
marketbeat.com
zacks.com
http://twitter.com/share?text=
analystratingsnetwork.com
boxingnews24.com
soccerlens.com
tickerreport.com
marketbeat.com
india.com
usmagazine.com
marketbeat.com
marketbeat.com
newsday.com
goal.com
http://trib.in/1ftkxlu
https://www.youtube.com/user/aibaproboxingtv
stocktargetprices.com
tvguide.com
(gmm/inautonews.com
boxing247.com
m.brown@ibtimes.com
http://bit.ly/1m1jboo)
http://macaudailytimes.com.mo/boxing-pacquiao-says-street-view-will-help-philippine-tourism.html
youtube/toonfirst.com
radaronline.com
marketbeat.com
r.schumann@ibtimes.com
tvguide.com
tvguide.com
style.com
marketbeat.com
arsenal.com
autotrader.com
marketbeat.com
http://bit.ly/1lplar6
(gmm/inautonews.com
popagami.com
geeky-gadgets.com
scientificamerican.com
nngo@pioneerpress.com
network.com
soccerlens.com
stubhub.com
comicbook.com
9to5toys.com
zacks.com
www.news.com.au
profootballfocus.com
http://:”
nj.com
www.cbc.ca
theuncool.com
customerservice@seattletimes.com
marketbeat.com
beat.com
amazon.com
network.com
a.young@ibtimes.com
tvguide.com
zacks.com
scientificamerican.com
marketbeat.com
zacks.com
hollywoodlife.com
spike.com
(http://www.gamesradar.com
collectspace.com
zacks.com
marketbeat.com
www.cbc.ca
network.com
zacks.com
http://twitter.com/share?text=ratings:
tampabay.com
zacks.com
analystratingsnetwork.com
zacks.com
network.com
aceshowbiz.com
analystratingsnetwork.com
al.com
overstock.com
steelers.com
stopbabypartstrafficking.com
tucson.com
http://recode.net/podcasts,
tennis.com
usmagazine.com
analystratingsnetwork.com
marketbeat.com
(chinapost.com
nba.com
tvguide.com
http://mney.co/1qtr7vo
nhl.com
timbreconcerts.com
dknoop@seattletimes.com
marketbeat.com
analystratingsnetwork.com
hollywoodlife.com
change.org
marketbeat.com
calciomercato.com
marketbeat.com
usmagazine.com
zacks.com
news.com
julia.belskayavontell@fitchratings.com
marketbeat.com
beat.com
marketbeat.com
www.thefillmore.com.
zacks.com
facebook.com
zacks.com
c.hitt@ibtimes.com
network.com
Forest Moon Theater Builds a Better Mousetrap, or Rather “Deathtrap”
zacks.com
tickerreport.com
tvguide.com
www.sja.org.uk
network.com
kburke@bizjournals.com
usmagazine.com
usmagazine.com
breastcancer.org
forbes.com
foliagetrains.com
http://twitter.com/share?text=steven
geeky-gadgets.com
philly.com
zacks.com
realclearpolitics.com
www.chesterrugbyworldcupbeerfestival.co.uk
mlive.com
network.com
khaleesifan3@emmyhost.com
chipotle.com
auction.com
carlilekrem.com
marketbeat.com
moveon.org
huffingtonpost.com
espnfc.com
radaronline.com
analystratingsnetwork.com
boxingnews24.com
(zenit.org
www.gizbot.com}
whoscored.com
cosmopolitan.com
(nonparents.com
amazon.com
zacks.com
marketbeat.com
analystratingsnetwork.com
espn.com
upi.com
autoguide.com
deadline.com
rivals.com
pic.twitter.com
uefa.com
(zenit.org
geeky-gadgets.com
http://twitter.com/share?text=tom
gmckay@post-gazette.com
boxing247.com
tvguide.com
soccerlens.com
stocktargetprices.com
espn.com
marketbeat.com
marketbeat.com
rideshareapps.com
analystratingsnetwork.com
http://bit.ly/1k8tg1w
sleekmoney.com
pic.twitter.com
http://samdani.com.bd/projects/dhaka-art-summit-frieze-2015-london
idigitaltimes.com
oregonhikers.org
marketbeat.com
zacks.com
analystratingsnetwork.com
analystratingsnetwork.com
tmz.com
m.vultaggio@ibtimes.com
www.winnipegfreepress.com
internet.org
uefa.com
network.com
www.aceingautism.com
analystratingsnetwork.com
latimes.com
usmagazine.com
littlethings.com
stocktargetprices.com
jovago.com
http://www.ptcbio.com/en/about-ptc/ptc-strive-awards-program
http://jobyek.blogspot.ca/2015/08/welcome.html
stockmonster.com
asiapacificfilms.com
usmagazine.com
beat.com
usatoday.com
n.massabrook@ibtimes.com
philsteele.com
marketbeat.com
marketbeat.com
zacks.com
bbc.com
usmagazine.com
salesforce.com
winner.com
etsy.com
vegasinsider.com
https://t.co/jhm4dbp9nb
m.hanrahan@ibtimes.com
network.com
google.com
zacks.com
officialcharts.com
marketbeat.com
grantland.com
wpbf.com
wcco.com
geeky-gadgets.com
marketbeat.com
hollywoodlife.com
http://indy.st/1klki0a
gallery@theguardian.com
lmbotofouda@imf.org
(http://tinyurl.com
j.glum@ibtimes.com
videogamer.com
http://mney.co/1uf8qvn
hollywoodllife.com
apple.com
boxingnews24.com
facebook.com
gettyimages.com
boxing247.com
zacks.com
marketbeat.com
hockeynewsnorth.com
(www.alaskarailroad.com
sleekmoney.com
zacks.com
geeky-gadgets.com
hollywoodlife.com
skpl.org
marketbeat.com
amazon.com
usmagazine.com
usmagazine.com
tvguide.com
pornhub.com
sina.com
zacks.com
zacks.com
people.com
m.willens@ibtimes.com
founder.org
paddocktalk.com
beat.com
http://twitter.com/share?text=netflix’s
j.glum@ibtimes.com
forbes.com
beat.com
radaronline.com
stocktargetprices.com
stocktargetprices.com
marketbeat.com
www.parklives.com
www.fifa.com:
zacks.com
(www.albawaba.com
credit.com
cleveland.com
zacks.com
(web.nli.org
ticketmaster.com
www.delta-industries.com
http://twitter.com/share?text=‘arrow’
marketbeat.com
autoguide.com
marketbeat.com
india.com
zacks.com
analystratingsnetwork.com
marketbeat.com
http://www.pcgamesn.com/the-aperture-science-portal-vr-demo-is-not-exclusive-to-vive
espn.com
analystratingsnetwork.com
zacks.com
www.cbc.ca
www.cbc.ca
c.hitt@ibtimes.com
http://twitter.com/share?text=ratings:
tucson.com
http://www.x17online.com/2015/09/arnold_schwarzenegger_to_replace_donald_trump_news_091415
motorsport.com
amazon.com
boxing247.com
housing.com
zacks.com
zacks.com
playbill.com
pic.twitter.com
marketbeat.com
hm.com
marketbeat.com
beat.com
marketbeat.com
www.90min.com.
petergreenberg.com
boxing247.com
analystratingsnetwork.com
www.amazon.c…
http://www.prh.noaa.gov/cphc
network.com
www.vansclassic.com.pe
whoscored.com
marketbeat.com
change.org
thedetroitbureau.com
(chinapost.com
zacks.com
r.schumann@ibtimes.com
(http://creativecommons.org
vogue.com
In [4]:
removedUrl = [re.sub(urlReg, “”, n) for n in contents]
Removeallnon-alphanumericcharactersexceptspaces¶
The non alphanum pattern is “[^a-zA-Z0-9\s]” to indicate that it is not a character
from a-z, A-Z, 0-9 and space character.
Use re.sub to remove matched string.
In [5]:
removedNonalphanum = [re.sub(“[^a-zA-Z0-9\s]”, “”, n) for n in removedUrl]
Split to words¶
Use split method to get word tokens.
In [6]:
words = []
for l in removedNonalphanum:
words.append(l.split())
Remove words with 3 characters or fewer¶
This is simple. Just use list comprehension to filter out the desired words.
In [7]:
removeShortWords = [ [word for word in l if len(word) > 3] for l in words]
Remove numbers that are fully made of digits¶
Regular expression for numbers are “\d+”.
Use fullmatch to test whether it is a full match (fully made of digits).
Also use list comprehension to filter out wanted words. List comprehension is
faster than loops. Thus, I will use list comprehension where it is possible.
In [8]:
digitP = re.compile(“\d+”)
removeNumbers = [[word for word in l if digitP.fullmatch(word) is None ] for l in removeShortWords]
Use an English lemmatiser to process all the words¶
I use the WordNetLemmatizer from nltk library to process the words.
In [34]:
from nltk.stem import WordNetLemmatizer
In [10]:
wl = WordNetLemmatizer()
In [11]:
lemmedWords = [[wl.lemmatize(word) for word in l] for l in removeNumbers]
Part B: N-grams¶
Compute N (number of tokens) and V (vocabulary size)¶
Use set to compute the vocabulary size. Number of tokens is the sum of the count of tokens in each row.
In [12]:
voca = set()
N = 0
for l in lemmedWords:
N += len(l)
for w in l:
voca.add(w)
In [13]:
print(“Number of tokens: %d” %N)
print(“Vocabulary size: %d” %len(voca))
Number of tokens: 3718089
Vocabulary size: 118701
List the top 25 bigrams¶
Use bigrams to get the bigrams and use defaultdict to make computing the bigram counts easier.
After compute the counts for each bigram, sort according to counts and print the top 25 bigrams.
In [14]:
from nltk.util import bigrams
In [15]:
from collections import defaultdict
# compute bigram counts
bicounts = defaultdict(int)
for l in lemmedWords:
for bi in bigrams(l):
bicounts[bi] += 1
In [16]:
sortedBigrams = sorted(bicounts.items(), key=lambda x: -x[1])
In [17]:
print(“top 25 bigrams based on the number of occurrences”)
for item in sortedBigrams[:25]:
print(item)
top 25 bigrams based on the number of occurrences
((‘more’, ‘than’), 3460)
((‘have’, ‘been’), 3316)
((‘hold’, ‘rating’), 2278)
((‘last’, ‘year’), 2266)
((‘this’, ‘year’), 2102)
((‘moving’, ‘average’), 2013)
((‘price’, ‘target’), 1924)
((‘average’, ‘price’), 1808)
((‘research’, ‘report’), 1671)
((‘target’, ‘price’), 1536)
((‘research’, ‘note’), 1390)
((‘nokia’, ‘nokia’), 1352)
((‘united’, ‘state’), 1311)
((‘this’, ‘week’), 1263)
((‘said’, ‘that’), 1206)
((‘price’, ‘objective’), 1201)
((‘they’, ‘have’), 1199)
((‘earnings’, ‘share’), 1186)
((‘premier’, ‘league’), 1167)
((‘that’, ‘will’), 1130)
((‘cell’, ‘phone’), 1116)
((‘last’, ‘week’), 1111)
((‘phone’, ‘plan’), 1076)
((‘plan’, ‘detail’), 1071)
((‘they’, ‘were’), 1046)
Compute the number of positive and negative word counts¶
Read the positve words and negative words from file. Convert them to set,
and for each story, test each word whether they are in postive words set or negative words set.
In [35]:
# function to read words from file
def readDict(fn):
f = open(fn, encoding = “ISO-8859-1”)
lines = f.readlines()
f.close()
return [line.strip() for line in lines[35:]]
# compute counts
def computeCounts(l, s):
return sum([x in s for x in l])
In [19]:
negWords = readDict(“signal-news1/opinion-lexicon-English/negative-words.txt”)
In [20]:
posWords = readDict(“signal-news1/opinion-lexicon-English/positive-words.txt”)
In [21]:
negSet = set(negWords)
posSet = set(posWords)
In [22]:
posNegCounts = [ (computeCounts(doc, posSet), computeCounts(doc, negSet)) for doc in lemmedWords]
In [23]:
# show the results for the first 10 rows
for i in range(10):
pc, nc = posNegCounts[i]
print(“Id %s: positive count: %d negative count: %d” %(news[i][“id”], pc, nc ))
Id 09344ca2-fdcc-4de3-9424-1b201e3da9ea: positive count: 30 negative count: 16
Id 87bc19b8-57df-4111-bf63-a39fcb8ad1a1: positive count: 28 negative count: 30
Id bd45307a-2589-410e-a327-ee7fd2144668: positive count: 0 negative count: 5
Id cb85f346-a961-4e3d-8942-3d2a0e436a85: positive count: 9 negative count: 0
Id f8a65c25-08bb-4b1a-b0a3-9f7994eb33a3: positive count: 14 negative count: 11
Id ecc20144-2d65-43cb-adab-9a9f6559f4f6: positive count: 2 negative count: 0
Id 3c8cc0df-fb5c-433f-b30b-0577e8b29b83: positive count: 17 negative count: 20
Id d60de2e1-3198-476a-85ec-2bd20b380a9a: positive count: 0 negative count: 0
Id 514dca6d-7ca1-4b54-bf27-4c5d8b11adb6: positive count: 13 negative count: 0
Id 5fc5f675-72e6-4cb9-a2a7-8e6dbaeb4ee5: positive count: 8 negative count: 5
Compute the number of news stories with more positive than negative words, as well as the number of news stories with more negative than positive words.¶
Use list comprehension to easily compute the counts.
In [24]:
morePosCounts = sum([x > y for (x, y) in posNegCounts])
moreNegCounts = sum([x < y for (x, y) in posNegCounts])
In [25]:
print("number of news stories with more positive than negative words: %d" %morePosCounts)
print("number of news stories with more negative than positive words: %d" %moreNegCounts)
number of news stories with more positive than negative words: 10548
number of news stories with more negative than positive words: 6597
Part C: Language models¶
Compute language models for bigrams¶
First split the data to training and test part. Compute the unigram counts and bigram counts in the training data.
Same as before, I use defaultdict to facilitate the computing.
In [26]:
train = lemmedWords[:16000]
test = lemmedWords[16000:]
Compute unigram counts¶
In [27]:
uniCounts = defaultdict(int)
for l in train:
for word in l:
uniCounts[word] += 1
Compute bigram counts¶
In [28]:
biCounts = defaultdict(int)
for l in train:
for bi in bigrams(l):
biCounts[bi] += 1
Compute the probability P(current|pre) with Laplace smoothing¶
I use the Laplace smoothing in the computing of the conditional probability P(current | pre). The formulae is $P(current | pre) = \frac {bigramcounts(current, pre) + 1} {unigramCounts(pre) + vocabularySize}$. In this way, when bigram count is 0, we can also give a non-zero probability. To make the computation numerically stable, I use the log probability,
In [29]:
import math
prob = defaultdict(dict)
vocaSize = len(uniCounts)
# Laplace smoothing
for bi, count in biCounts.items():
pre, aft = bi
prob[pre][aft] = math.log( (count + 1) / (uniCounts[pre] + vocaSize))
Produce a sentence of 10 words by appending the most likely next word each time¶
In each step, for the last word $x$ generated, find the word $y$ that has the max value of the $p(y|x)$.
In [30]:
sent = ["they"]
while len(sent) != 10:
h = prob[sent[-1]]
best = None
for k in h:
if best is None or h[k] > h[best]:
best = k
sent.append(best)
In [31]:
print(sent)
[‘they’, ‘have’, ‘been’, ‘given’, ‘rating’, ‘hold’, ‘rating’, ‘hold’, ‘rating’, ‘hold’]
Compute the perplexity on test data¶
Because if we multiply many small float point number, it may underflow. So I use log probability in the computation.
Add up the log probability. Because I use Laplace smoothing, when a bigram in the test data that doesn’t appear in train data, it also has non-probability. Then compute the average log probability. In the last, convert to perplexity.
$perplexity = e^{ – averageLogProb}$
In [32]:
logProb = 0
count = 0
for l in test:
for pre, aft in bigrams(l):
if aft not in prob[pre]:
# process the bigram that doesn’t appear in train data, Laplace smoothing
logProb += math.log(1 / (uniCounts[pre] + vocaSize))
else:
logProb += prob[pre][aft]
count += 1
avProb = logProb / count
perplexity = math.e ** (-avProb)
In [33]:
print(“perplexity: %f” %perplexity)
perplexity: 25685.738581
In [ ]: