Natural Language Engineering: Assessed Coursework
Submission format: You should submit one file that should either be a Jupyter notebook or a zip file containing a Jupyter notebook and the image files that you want to include in the notebook.
Due date: You work should be submitted on the module’s Study Direct site before 4pm on Wednesday 23rd November. This is Wednesday of week 10. The standard late penalties apply.
Return date: Marks and feedback will be provided on Sussex Direct on Friday December 9th for all submissions that are submitted by the due date.
Weighting This assessment contributes 50% of the mark for the module.
Overview
For this assignment you are asked to write a report on the activities cov- ered in lab sessions 8 and 9. During these sessions you extended an opin- ion extractor function provided by us so that it deals with a variety of ways that opinions are expressed in Amazon DVD reviews. For this report you should create a single Jupyter notebook containing the following (further details are given below):
- the python code for your opinion extractor;
- a demonstration that your opinion extractor produces the correct output for the example sentences that we have given for each of the extensions;
- an assessment of the limitations of your extractor when applied to a random sample of the sentences in the corpus of DVD amazon re- views that we have provided; and
- a proposal as to how you would go about creating a website that au- tomatically produces summaries of what people think about various aspects of a DVD.
1
Marking Criteria and Requirements
Your submission will be marked out of 100, based on the following criteria. Please read these guidelines carefully.
Overall quality of report 15 marks available
This concerns issues such as writing style, organisation of material, clarify of presentation, etc.
- Your report should be no more than 3000 words in length ex- cluding the content of graphs and tables and any references.
- You must specify the length of your report. 3000 is a strict limit.
- You must use a formal writing style.
- All figures and tables should be clearly numbered, and have an appropriate caption.
- All graphs should have each axis clearly labelled.
- Use subsections with meaningful headings.
- Do not add external text (e.g. code, output) as images.
- You should include appropriate references to source material. This should not be references to Wikipedia and similar sources, but to proper academic publications.
- Do not include the CONLL representation of the parsed sen- tences as it is equivalent to the tree representation, but harder to read.
2
Quality of opinion extraction algorithm: 15 marks available
This concerns the quality of the Python code implementation (coding style, appropriate use of comments and efficiency).
- Details of the extensions to the basic opinion extractor are given in the notebook Session8.ipynb.
- You are required to do extensions 1 to 5 only. Dealing with the examples given in Section Additional Extensions is optional.
- Donotwriteseparateopinionextractorfunctionsforeachofthe different extension. It is important that all of the different exten- sions are integrated, so that a single opinion extractor function deals with all of the cases you are covering.
- You should state explicitly whether or not you have not been successful in extending your opinion extractor in the ways re- quired. Note that we will be running and testing your code in order to establish whether the claims that you make are valid.
- Do not forget to include comments in your code where neces- sary. Split long functions into smaller coherent sub-functions.
- You should submit a single Jupyter Notebook that include all of the Python code you have written or adapted, i.e. all code that you have written, except code in the NLTK and Sussex NLTK packages.
- Images that appear in your Notebook can be submitted as sep- arate files, zipped up with your Jupyter Notebook to produce a single file submission.
Validating opinion extractor on the examples test set: 20 marks available
This concerns how well you demonstrate that your opinion extrac- tor produces the correct output for the sentences in the examples test set. For each extension we have provided a number of exam- ples that illustrate what we are looking for. These examples can be found in the Examples Test Set. For each extension you need to show that your opinion extractor produces the correct output for those sen- tences that are used as examples in the description of that particular extension.
3
Discussion of performance on Amazon DVD review sentences: 15 marks available
This concerns how well you describe the limitations of your opinion extractor.
- Present an assessment of the performance of your opinion ex- tractor on a reasonable number of randomly sampled examples involving all four aspects of the DVD: plot, characters, cine- matography, and dialogue. It is up to you to decide how many examples to consider. In making this decision, you should be guided by the fact that your goal is convey a realistic sense of how well your opinion extractor is performing.
- IllustrateyourassessmentwithexamplesfromtheAmazonDVD review corpus.
- In cases where an incorrect analysis has been made, make it clear whether the problem has arisen due to: (i) a deficiency in your opinion extractor algorithm (ii) errors made by the part- of-speech tagger, or (iii) errors made by the dependency parser.
- When analysing your option extractor, provide a quantitative breakdown such as “I looked at 50 sentences, 40 were correct. Out of the other 10, 8 were due to parser errors and 2 were due to PoS tagger errors”.
- Discusswaysinwhichyouropinionextractorcouldbeimproved to overcome the problems that you have observed.
Technical understanding: 15 marks available
This concerns how well you have explained each of the methods that you have used in your experiments.
- Youshouldexplainbothpartofspeechtagging(includingHMMs) and dependency parsing in sufficient detail that a well-educated computer scientist who does not know about these particular Natural Language Processing methods will be able to under- stand what you have done.
- Donotjustrepeatdetailsdirectlyfromthelecturenotesorother sources. Your explanations must be expressed in your own words and be related to the specific context of this report.
4
Quality of proposal for a DVD summary website: 20 marks available
This concerns how well you describe the opportunities and chal- lenges involved in creating such a website.
- You should explain what you see as the motivation for having the website.
- You should explain what would be involved in creating the website.
- You should give a diagram showing each of the pipeline of pro- cessing steps involved in processing the DVD reviews.
- You should provide a description of the analytical process ex- pressed as a high-level pseudo-code algorithm, giving sufficient detail that a competent programmer would be able to imple- ment what you have in mind. You can assume that you have access to various language processing building blocks such as modules that can perform tokenisation, lemmatisation, tagging, parsing and opinion extraction. Your algorithm should take a DVD review as input and produce an aspect-oriented analysis as output.
- Discusstheuseofadocumentclassifiertoobtainanassessment of the overall sentiment of the review, including consideration of possible ways to create or obtain training data for the classi- fier.
- You should give a realistic assessment as to how accurate the summaries on the website would be, and whether the idea of having such a website is feasible. In particular, consider ques- tions such as whether the part-of-speech tagger and dependency parser are capably of accurately analysing the informal language found in DVD reviews; and whether the opinion extractor is ca- pable of producing useful outputs.
- Focus on the back-end and underlying NLP system. What prob- lems might you encounter? How could you approach them? How well is your website likely to work? Why would users be interested in your product? We are not interested in issues such as graphic design, implementation language, database back-end, etc. but on what the service does and how it works.
5