RMIT School of Computer Science & IT
COSC2637 Big Data Processing Assignment 2018
Aim: to design, implement, critically analyse and report on a substantial big data project. Due: due 11:59 pm on Tuesday 9 October 2018
Marks: This assignment is worth 30% of your overall mark
Requirements: UsingabigpublicAWSdatasetwriteaMapReduceprogramundertakeaninter- esting new analysis of the data. Compare the performance of your program using different size clusters.
This assignment can be either as a individual or in groups of two students. If you form a team, make sure to include the name of your partner in the submission’s description on Canvas, filenames that are named after your student numbers, the report and the presenta- tion.
Deliverables are:
• Code, output and log files • Written report
• Presentation
Code, output and logs (15 marks)
- YoumustuseAWSEMR(suchasprovidedbyRMITthroughthiscourse)forrunning your MapReduce program (which may be developed on any machine) and accessing one (or more) of the public data sets in S3, such as:
1. s3://datasets.elasticmapreduce/ngrams/books/ 2. s3://aws-publicdatasets/common-crawl/
You do NOT have to use these two datasets. The full list of datasets can be found here:
https://registry.opendata.aws/
- In choosing a dataset(s) and analysis to undertake, a combination of the two factors may contribute to novelty. In particular, for a similar difficulty of analysis task, more novelty will be given for using the common-crawl instead of ngrams, and even more novelty will be given for another similar size dataset available on s3.
- Paths should not be hard-coded.
- As you are required to time your tasks, you should carefully log your experiments on different sized clusters.
1
- Your MapReduce program(s) must be well written, using good coding style and in- cluding appropriate use of comments. Your markers will look at your source code. Coding style will form a small part of the assessment of this assignment.
- You should provide instructions with your code on how to compile and execute it. If your marker cannot compile your programs, you risk yielding zero marks for the coding component of your assignment.
- Depending on the size of output your MapReduce program generates, you may choose or need to submit only samples of the output (in which case you should also provide a complete summary of all the output).
- Depending on your implementation, you may wish to provide additional information about your code (for example, how it is to be compiled and run), output and logs. If so, put this information into a plain text file called readme.txt.
- You can use either Java, Python, Hive or Pig to develop your MapReduce program over AWS EMR.
Results of experiments and written report (10 marks)
- You are required to write a short report in the style of a conference paper on your experiments.
- The text of the report should be only TWO A4 pages (not including references and appendices), using the IEEE conference template for Word or LaTex
http://www.ieee.org/conferences_events/conferences/publishing/templates.html
however you may provide additional material including references, and appendices containing output, tables, graphs in additional pages.
- Your report must be a PDF file, called sNNNNNNN( sNNNNNNN).report.pdf, where sNNNNNNN is your student number and the optional partner’s student number. Files that do not meet this requirement may not be marked. Your report must be well- written.
- Your report should include:
- Explain how you implemented your map reduce program.
- Undertake experiments using your programs and report on the output and tim- ings.
- Discuss your results and critically analyse the scalability of your approach as more nodes are added to the cluster. Are the results as you expected?
- Compare what you have done with published studies using the same data.
- Important: Your report will be marked on the quality of your written explanations and analysis. After writing your report you should carefully revise it checking for clarity of expression and quality of writing.
Presentation (5 marks)
2
• Your presentation must be a one page PDF file, called sNNNNNNN.presentation.pdf, where sNNNNNNN is your student number. Files that do not meet this requirement may not be marked.
• To be conducted during tute/labs in week 12, ONLY students who have submitted their presentations (as sNNNNNNN.presentation.pdf) when due 11:59 pm on Tues- day 9 October 2018 will be eligible to present in class. Students must attend their scheduled tute/lab and be ready to present at their allocated time.
3
Additional general requirements : The following contains information about other require- ments that your assignment must meet. Please read all these general requirements carefully
before you start.
- The “Big Data Processing” Canvas contains further announcements and answers to questions. You are expected to check the discussion section on regular basis. Login through https://rmit.instructure.com.
- Depending on your implementation, you may wish to provide additional information about your code (for example, how it is to be compiled and run). If so, put this information into a plain text file called readme.txt.
- Important: You must run all your experiments on AWS EMR.
- Warning: Assignments will be checked with plagiarism-detection software; any stu- dent found to have plagiarised will be subject to disciplinary action as described in the course guide. Plagiarism includes submitting code that is not your own or sub- mitting text that is not your own — except where your sources are referenced AND it is clear, for example by quotation marks, what is your original work and what has been quoted from others. Submitting one comment in your code or a sentence from someone else’s report is plagiarism, and plagiarism includes submitting work from previous years. Allowing others to copy your work is also plagiarism. All plagiarism will be penalised; there are no exceptions and no excuses. For further information, please see: http://www1.rmit.edu.au/students/academic-integrity.
Marking criteria : an excellent assignment will include the following:
- program code that meets assignment requirements and undertakes an interesting
(non-trivial) original analysis of data, executes correctly, and is well written;
- output that thoroughly demonstrates the program works correctly;
- log files indicating that scalability of results has been extensively explored;
- a written report that provides an excellent detailed explanation of how you imple- mented your mapreduce program to undertake an interesting new analysis of the data, including discussion and interpretation of your output and a thorough set of experi- ments with timings to address the scalability of your approach, as well as a critical analysis of your approach in the context of the theory in the course;
- a presentation using a well prepared single slide such that the oral presentation of the slide clearly presents novel achievements and engages well with the audience.
4
- program code that meets assignment requirements and undertakes an interesting
What to Submit, When, and How What
You need to submit your source code, output, logfiles, written report and presentation. Before you submit anything, read through the assignment specifications again carefully. Check that you have followed all instructions in the general requirements. Also check that you have attempted all parts of all questions. In particular you must submit:
- ONE zip file that contains your MapReduce source files, output, logfiles and the readme.txt file containing any instructions, this should be submitted using the link to Assignment Code, Output and Logs Submission, and
- ONE (A4 portrait) PDF file containing your report, this should be submitted using the link to Assignment Report Submission (it is a turnitin submission), and
- ONE (A4 landscape) PDF file containing a single slide for your presentation (you may use whatever software you choose for producing this, so long as the result is a one page PDF file), this should be submitted using the link to Assignment Presentation Submission.
When
The assignment is due at due 11:59 pm on Tuesday 9 October 2018.
Late submissions should be submitted using the same procedure, but will be penalised by
10% of total possible marks per days for assignments that are late 1 to 5 days late. For assign- ments that are more than 5 days late, a penalty of 100% will apply. See the course guide for further information.
If you are prevented from submitting your assessment on time, by circumstances outside your control, you may apply for an extension by carefully following the instructions at http: //www1.rmit.edu.au/students/assessment/extension
The onus is on you to check that your submission has been received.
How
You need to separately submit three files under Assignments on Canvas
- ONE zip file that contains your MapReduce source files, output, logfiles and readme.txt file, this should be submitted using the link to Assignment Code, Output and Logs Sub- mission, and
- ONE PDF file (A4 portrait) containing your report, this should be submitted using the link to Assignment Report Submission (it is a turnitin submission), and
- ONE PDF file (A4 landscape) containing your presentation (as a one page single slide) using the link to Assignment Presentation Submission.
You will be scheduled for a time during your tutorial/laboratory class in week 12 to do a two minute presentation.
5