Homework 4 – Spark¶
In this homework, we are practicing Apache Spark.
You are required to turn in this notebook as BDM_HW4_Spark_NetId.ipynb. You will be asked to complete each task using Apache Spark. Output can be printed in the notebook.
Task 1 (5 points)¶
You are asked to implement Homework 3 using Spark. The description is provided below for your convenience.
You are asked to implement the Social Triangle example discussed in class. In particular, given the email dataset, please list all “reciprocal” relationships in the company. Recall that:
If A emails B and B emails A, then A and B is reciprocal.
If A emails B but B doesn’t email A, then A and B is directed.
Dataset: We will use a subset of the open Enron Email Dataset, which contains approximately 10,000 simplified email headers from the Enron Corporation. You can download this dataset from NYU Classes as enron_mails_small.csv. The file contains 3 columns Date, From, and To. Their description is as follows:
Column name
Description
Date
The date and time of the email, in the format YYYY-MM-DD hh-mm-ss,
e.g. “1998-10-30 07:43:00”
From
The sender email address,
e.g. “mark.taylor@enron.com”
To
A list of recipients’ email addresses separated by semicolons ‘;’,
e.g. “jennifer.fraser@enron.com;jeffrey.hodge@enron.com”
Note that, we only care about users employed by Enron, or only relationships having email addresses that end with ‘@enron.com’.
The expected output is also provided below. For each reciprocal relationship, please output a tuple consisting of two strings. The first one is always ‘reciprocal’. And the second one is a string showing the name of the two person in the following format: ‘Jane Doe : John Doe’. The names should be presented in the lexical order, i.e. there will not be a ‘John Doe : Jane Doe’ since ‘Jane’ is ordered before ‘John.
Though the dataset only contains email addresses, not actual names, we’re assuming that the email aliases were created based on their name. For example:
Email Address
Converted Name
mark.taylor@enron.com
Mark Taylor
alan.aronowitz@enron.com
Alan Aronowitz
marc.r.cutler@enron.com
Marc R Cutler
hugh@enron.com
Hugh
Please fill the code block with a series of MapReduce jobs using your own mapper and reducer functions. Be sure to include the naming convention logic into one of your mappers and/or reducers.
In [1]:
35
Out[1]:
[(‘reciprocal’, ‘Brenda Whitehead : Elizabeth Sager’),
(‘reciprocal’, ‘Carol Clair : Debra Perlingiere’),
(‘reciprocal’, ‘Carol Clair : Mark Taylor’),
(‘reciprocal’, ‘Carol Clair : Richard Sanders’),
(‘reciprocal’, ‘Carol Clair : Sara Shackleton’),
(‘reciprocal’, ‘Carol Clair : Tana Jones’),
(‘reciprocal’, ‘Debra Perlingiere : Kevin Ruscitti’),
(‘reciprocal’, ‘Drew Fossum : Susan Scott’),
(‘reciprocal’, ‘Elizabeth Sager : Janette Elbertson’),
(‘reciprocal’, ‘Elizabeth Sager : Mark Haedicke’),
(‘reciprocal’, ‘Elizabeth Sager : Mark Taylor’),
(‘reciprocal’, ‘Elizabeth Sager : Richard Sanders’),
(‘reciprocal’, ‘Eric Bass : Susan Scott’),
(‘reciprocal’, ‘Fletcher Sturm : Greg Whalley’),
(‘reciprocal’, ‘Fletcher Sturm : Sally Beck’),
(‘reciprocal’, ‘Gerald Nemec : Susan Scott’),
(‘reciprocal’, ‘Grant Masson : Vince Kaminski’),
(‘reciprocal’, ‘Greg Whalley : Richard Sanders’),
(‘reciprocal’, ‘Janette Elbertson : Mark Taylor’),
(‘reciprocal’, ‘Janette Elbertson : Richard Sanders’),
(‘reciprocal’, ‘Liz Taylor : Mark Haedicke’),
(‘reciprocal’, ‘Mark Haedicke : Mark Taylor’),
(‘reciprocal’, ‘Mark Haedicke : Michelle Cash’),
(‘reciprocal’, ‘Mark Haedicke : Richard Sanders’),
(‘reciprocal’, ‘Mark Haedicke : Twanda Sweet’),
(‘reciprocal’, ‘Mark Taylor : Sara Shackleton’),
(‘reciprocal’, ‘Mark Taylor : Tana Jones’),
(‘reciprocal’, ‘Michelle Cash : Twanda Sweet’),
(‘reciprocal’, ‘Pinnamaneni Krishnarao : Vince Kaminski’),
(‘reciprocal’, ‘Richard Sanders : Sara Shackleton’),
(‘reciprocal’, ‘Rosalee Fleming : Steven Kean’),
(‘reciprocal’, ‘Sara Shackleton : Tana Jones’),
(‘reciprocal’, ‘Shirley Crenshaw : Vince Kaminski’),
(‘reciprocal’, ‘Stinson Gibner : Vince Kaminski’),
(‘reciprocal’, ‘Vasant Shanbhogue : Vince Kaminski’)]
Task 2 (5 points)¶
You are asked to implement Task 2 of Lab 5. The description is provided below for your convenience.
We’ll be using two NYC open data sets: the SAT Results and the NYC High School Directory data sets. Both can be downloaded from the links below, or from online class resources.
Dataset: Please note that each school is uniquely identified by an DBN code, which should be found on both data sets.
SAT_Results.csv Source: https://nycopendata.socrata.com/Education/SAT-Results/f9bf-2cp4
Description: “The most recent school level results for New York City on the SAT. Results are available at the school level for the graduating seniors of 2012.”
DOE_High_School_Directory_2014-2015.csv Source: https://data.cityofnewyork.us/Education/DOE-High-School-Directory-2014-2015/n3p6-zve2
Description: “Directory of NYC High Schools.”
We would like to know how the Math scores vary across bus lines or subway lines serving the schools. Your task is to compute the average Math scores of all schools along each bus line and subway line. You can find the bus and subway lines serving each school in the High School Dictionary as bus and subway columns.
The expected results are two lists:
1. A list of key/value pairs: with bus line as keys, and the average Math scores as values.
2. A list of key/value pairs: with subway line as keys, and the average Math scores as values.
The top ten lines with highest score are shown below.
In [2]:
Out[2]:
[(‘S1115’, 612),
(‘M79’, 594),
(‘Q42’, 582),
(‘M22’, 574),
(‘Bx3’, 571),
(‘B52’, 560),
(‘B63’, 557),
(‘B69’, 548),
(‘B54’, 543),
(‘B25’, 541)]