简介
This question is designed to give your hands-on experience with designing an application using big data and some of the components of Hadoop ecosystem.
This assignment supports the following objectives:
• Design toy applications using Sqoop, Flume, or Storm
• Create a small Hadoop application
题目
ZoomZoomCom, a marketing company, has hired you to design a data analysis project based on Hadoop. They want to determine for each of thousands of product categories if there is a correlation between the sentiment of the context in which a link to a page for a product of a certain category appears and the likelihood of its being clicked on. For example, is an upbeat context of a link to a cosmetic product more likely to be linked on than a downbeat one? Is the opposite true for dieting products?
额外说明
You are provided a database containing the URL for a large number of product pages and the category of each product. (You may assume there is only one product in each product page.) Let us call it the ‘URL DB’.
Assume you have access to a web crawler that crawls web pages. You also have at your disposal a web scraping program that can find out whether a web page has a link to one of the product pages and for each such link it can extract the paragraph (or a roughly 50 words window around the occurrence of the link—don’t worry about this kind of detail).
You have access to a log file which shows for each such page how many times it was served in a given week and how many times any link that points to one of our product pages was clicked on during that week. Based on the data in the log file you can compute the hit ratio of each link anchor (the number of times the link anchor was clicked on divided by the number of times the page was viewed).
You also have at your disposal a sentiment analysis program called Sentimenter for determining whether a paragraph is Upbeat, Downbeat, or Neutral.
So, based on the hit ratio of a link anchor and the sentiment of the context in which each link anchor appears you may come up with something like this: for link anchor ‘XYZ’ the hit ratio when the context was upbeat was 30%, the hit ratio when it was downbeat was 2% and when it was neutral was 7%.
You will have to group the link anchors by the product categories they point to. For example, different occurrences of “truly wonderful device” may point to different products.
Then you will have to aggregate these hit ratios by product categories. So let us say, for cosmetics we found hundred link anchors and we have the hit ratios for each of our 3 sentiment categories. We can then just average them out. Similarly, for other product categories. This will give us the correlations that we seek.
Question 1
First, the URL DB has to be imported into Hadoop.
• Q1a. Which component of Hadoop should your system use to ingest the URL DB into Hadoop? (3 points)
• Q1b. Which component should your system use to interact with the URL DB? (3 points)
• Q1c. Why have you chosen those components? (2 points)
Question 2
Second, the log file has to be ingested into Hadoop.
• Q2a. Which component of Hadoop should your system use to ingest the log file into Hadoop? (3 points)
• Q2b. Why that component? What might be some alternatives. (2 points)
Question 3
Third, the crawler has to retrieve the content of web pages. For simplicity, let us assume that there is only one instance of the crawler operating and it downloads and operates only one page at a time. We can also assume that the web pages to be downloaded belong only to certain media and blog sites (so, not the entire web).
• Q3a. Which Hadoop component should be used to ingest this content? (3 points)
• Q3b. Why that component? (2 points)
Question 4
Fourth, for each page we have to find all the links in each page which point to a URL in our URL DB. Let us say we are provided a program to do that called LinkFinderAnalyzer.
Fortunately, LinkFinderAnalyzer does a number of additional tasks. It extracts the para in which the link appears. Then it invokes Sentimenter to find the sentiment of the para. Then, for each link_anchor it computes the hit score of that link_anchor.
The output of running LinkAnalyzerFinder are tuples of the form
Keep in mind that for each product_type there can be many such tuples because there could be several products in a product type and there could be several URLs pointing to a product in that product type.
By ‘product type’ is meant the type of product it is (e.g., lipstick, of which there can be several brands). This is to be distinguished from product category which may contain many product types. For example, both lipstick and mascara would fall under the product category of cosmetics.
Fifth, for each product category we want to combine the tuples produced by step 4. So, suppose we had two tuples
This may also be a MapReduce job. The Mapper does nothing, but then there is shuffling and sorting in terms of product category, and reducing which combines the lists for each product category.
Sixth, for each tuple of step 5 we want to compute the average hit score for each sentiment type. The output of this step should be
This is the final output of the system.
• Q4a. Which components of the Hadoop ecosystem will you use to do each of the jobs of steps 4-6? Would you use MapReduce or some interface to MapReduce (Hive? Pig? Spark?) (4 points)
• Q4b. Would you use the same component for each of steps 4-6, or would you use different components? You must justify your answer. (4 points)
• Q4c. Keep in mind that we are chaining together the operations of Steps 4-6. How should the output of the previous step in the sequence be passed to the next step? What kind of Hadoop architecture is best suited for that? Will your system store the intermediate results? If so, in what sort of structure? You must justify your answer. (4 points)
Submit your answers to all of the above questions in a single text or Word or pdf file, including any diagrams