SMI205 Replication Project
Assessment Brief (2022)
(Last edited 10 February 2022)
Copyright By PowCoder代写 加微信 powcoder
Assessment summary
Your task in this module is to reproduce a secondary data analysis which has been published in an academic article, in a good quality journal of social sciences (see below) which employs advanced quantitative methods (regression level, not just descriptive statistics), critically reflect on the original model, and design a small research extension.
The task can be broken into two stages:
· (Computational) reproducibility – is understood as obtaining consistent results using the same data, following the same data analysis steps, methods and code. Your Assessment 1 is about computational reproducibility and re-running one model from a selected academic paper.
· Replication can take different forms (see various replication forms in Freese & Peterson 2017) and it does not have to be a step-by-step / exact replication (as above), but the focus is on checking research robustness and further verifying a hypothesis / research question. Following reproducibility check and re-running one model for Assessment 1, in Assessment 2 you design a small extension.
There are two pieces of summative assessment in the module:
Assessment 1 (40% of the final mark)
A 10-minute presentation with notes where you reproduce one model from a selected article. Presentation up to 15 slides – you must use the provided template.
Assessment 2 (60% of the final mark)
A reproducible research report presenting your replication study. The report has to be created in R Markdown and published on Rpubs.com, max. 1,500 words without the code, tables and figures. Use the provided template.
The article you choose must meet the following criteria:
· This cannot be a paper we discussed during any of the seminars.
· The study applies some advanced quantitative methods, not just descriptive statistics, such as a linear, ordered or logistic regression, which could be a one-level or multi-level model specification.
· The paper and your replication cannot use the World Value Survey, as this dataset was used for the assessment in the SMI206 module, semester 1.
· Your original data analysis has to be done using R/RStudio software.
· I have to approve your final choice of an article to make sure it is of a good quality and its replication is feasible within the module duration.
Steps in your replication:
(1) Choosing an article 2
(2) Assessment A – practice critical reading of a paper 3
(3) Criticising the paper and designing the replication study 4
(4) Data analysis and preparing outputs in R 4
(5) Assessment 1 – conference presentation with notes 5
(6) Assessment B – preregister your replication idea [template TBC] 6
(7) Assessment 2 – R Markdown report [to be updated] 6
(1) Choosing an article
Paper for the replication should be published in a peer-reviewed academic journal (see materials on Blackboard → Resources → Academic journals)
Choose a paper that falls within your research interests. You can do it in a few different ways. The process will take time, as you need to read and assess a few papers before you make a decision.
You can do it in a few different ways:
Method 1. Find a paper using search engines for academic outputs:
· Browse key journals of social sciences by subject are via Google Scholar https://scholar.google.com/citations?view_op=top_venues&hl=en&vq=soc
· Skim latest issues of selected journals to get a sense whether they publish papers using quantitative methods
· Type keywords or a name of a dataset into Google Scholar, SCOPUS or Web of Science databases https://scholar.google.co.uk and consider papers, which match your preferences
Method 2. Browse repositories with papers based on data from specific surveys:
· European Social Survey publications http://www.europeansocialsurvey.org/bibliography/
· Understanding Society publications https://www.understandingsociety.ac.uk/research/search
· Comparative Study of Electoral Systems bibliography http://www.cses.org/resources/results/results.htm
· 2011 Workplace Employment Relations Study publications http://www.wers2011.info/publications
· European Election Study bibliography
http://europeanelectionstudies.net/bibliography
· European Value Study
Method 3. Data repositories where you can indientified replication packs for research and check whether the article has been already published:
· Open Science Framework: https://osf.io/
· Harvard Dataverse: https://dataverse.harvard.edu/
· Zenodo: https://zenodo.org/
Method 4. Email authors of the article asking them to share data and, if possible, any data analysis code.
· Be nice and professional! You represent The University of Sheffield.
· See ACRE Chapter 8 “Guidance for Constructive Communication Between Reproducers and Original Authors” https://bitss.github.io/ACRE/comunications.html#for-reproducers-contacting-the-authors-of-the-original-study
By the end of week 3, identify three papers and access the feasibility of the replication project:
1. Is data used in the paper publicly available?
a. Check how to get access to the dataset, as sometimes you have to apply and get an access code or secure access.
2. Are you familiar with all methods used in the paper?
a. Part of the assessment is to exactly reproduce one model before you design your research extension.
b. Even with a code, you might struggle to reproduce a model if it is too complex for you to understand and explain the results.
3. Is replicating one selected model possible within the next 4-5 weeks?
4. Is there a scope to criticise the paper? (see the next stage)
Seek module leader advice and approval during the lab or via email
(2) Assessment A – practice critical reading of a paper
Prepare a 500-word critical reflection on readings for week 3 (my paper – Piekut 2019). Critically reflect on the strong and weak points of the papers. You do not have to cover many issues but focus on 2-3 points.
Please structure the note in the following way:
1. short summary what was the aim of the paper – do not copy/paste bits of the paper and use own words to explain this; practice your of writing;
For example:
2. 1st critical point, e.g. gaps in literature review (use evidence of what literature was missed – reference other papers / research to support your claims)
· Are all arguments convincing?
· Are there any arguments that are not well supported with evidence/literature?
· Which argument is especially well or not well developed?
3. 2nd critical point, e.g. transparency of used methods
· Is the research reproducible?
· Are you able to re-create the research analysis from the information provide in the paper?
· Is the explanation of what is the analytical sample and used methods comprehensive?
(3) Critical reading of the paper and designing the replication study
Read the selected paper carefully and draft a few points of critique – use SMI205 Critical reading of literature – available on the Blackboard in Resources.
Your critique of the original study should drive your replication study design (Assessment 2, the small extension). The critique should be based on the evidence – meaning that you should build your argument by referring to other literature or research to support your claims.
Focus on one research question / hypothesis from the paper which you think can be challenged, improved, or just tested in a different context.
· See replication forms that are discussed in Freese, J., & Peterson, D. (2017). Replication in social science. Annual Review of Sociology, 43, 147-165.
In your replication project you have to use advanced quantitative methods you learnt in semester 1 in the SMI206 module, such as multilevel modelling, logistic regression, or linear regression with (an) interaction term(s).
· Your chosen paper does not have to use these methods, but your replication paper – does, e.g. the original study can have a simple linear regression, but your replication study will have a multilevel structure or an interaction term.
· For your replication study, you can use a different dataset with similar data, or the same dataset, but for a different year or country (replication to check repeatability of the results).
· You can use the same dataset as the original study, but apply a different data analysis method or amend the model specification (replication to check robustness of the results).
· Make sure possibly all variables you use are coded in the same manner as in the original study.
· Make sure you select the right model for the data distribution and structure, and support your choice with a methodological literature.
(4) Data analysis and preparing outputs in R
Work on a reproducible script:
· Use the SMI205 Annotated R template – a draft R script available via the Blackboard.
Add annotations to each step in your R script, so it can be easily understood by others.
Make sure it would be possible to easily reproduce your research by adding links to all datasets.
Remember that the final product of the exact replication will be a PowerPoint presentation (max. 15 slides) and then a publishable research report, so make sure you produce transparent tables and good-looking graphs.
· Focus on 2-3 key outputs, such as
· an overview of a dependent variable (e.g. mean attitudes by countries, crime level by years, maps with %)
· outputs from data modelling (e.g. predicted values, coefficient plots)
· Instead of producing many similar graphs try to contain as much information as possible in a smaller number of graphs.
· Do not embed in your presentation not-formatted tables or just tables, your presentation should be data visualisation driven.
(5) Assessment 1 – conference presentation with notes
On the basis of the chosen paper/article, prepare presentation slides with notes where you will present the paper, reproduce one model and critically reflect on the process.
Slides should be prepared for a 10-minute talk
· Use maximum 15 slides (including a Title slide and a ‘Thank you’ slide)
· A rule of thumb is that you spend about one minute talking over one slide
· Add notes: up to 200 words per slide (maximum 1,500 words in total)
Prepare presentation slides should contain the following sections:
1. Introduction / Why the topic is worth studying (the hook / framing)
2. Summary of the claims made in the paper / hypotheses / research questions / the main arguments of the paper (1-2 slides)
3. Methods (2-3 slides):
· sample description (who, size, sapling method)
· data analysis methods (what kind of the regression(s) were used)
· key variables (preferably presented as graphs)
4. An exact replication of one, selected model from the original paper (1 slide):
· present the model as a table or visualise it (eg. as a coefficient plot)
· do not screenshot R output,
· do not screenshot outputs from the reproduced paper.
5. Methodological reflections coming from the exact replication / reproductions (2-4 slides):
· are the results you got the same?
· if different, what=y it could be so?
· reproducibility assessment: data, code, methods, overall transparency of any replication materials and the paper.
6. Your research extension plans + rationale (1-2 slides).
Remember to make an effort to improve data presentation from the original paper.
In your presentation notes use clear language (avoid jargon) and try to engage the audience in your ‘story’.
Submit as a PDF -> print or save as a PDF, so one page is one slide and its notes below.
Your presentation will be assessed according to these assessment criteria:
· Scholarship – application of quantitative methods.
· Independent learning – quality of the replication and how any challenges were solved.
· Communication – clarity and interesting story/narrative.
· Visual qualities – visual design of the slides, including data visualisation.
(see full details on the Blackboard → Assessment & Feedback → Assessment Criteria).
(6) Assessment B – preregister your replication idea [template TBC]
Use the SMI205 Preregistration form (Blackboard 🡪 Replication) to provide information about the paper you identified for the replication and your research extension design. To proceed you need my approval of the chosen paper and the study design.
(7) Assessment 2 – R Markdown report [to be updated]
Write a replication paper/report in R Markdown and publish it online using Rpubs website:
· your report should be max. 1,500 words long (excluding the code and tables/graphs),
· include parts of R code (R chunks) across various sections of your report,
· make your R Markdown documents downloadable in Rpubs or embed a link to a copy on GitHub or via another repository you created.
· remember to properly embed and describe tables, graphs and other materials.
Do I have to create a data repository?
I would like to have a copy of your R Markdown original file, open it on my computer, and re-run the analysis. Would I be able to do it with just your R Markdown file?
· Create a repository, if you created any new data, such as a spreadsheet with country-level variables.
· Create a repository, if your R Markdown file cannot be downloaded from Rpubs, as you used a theme which does not allow it.
· Do not republish any publicly available data (it is against data access agreement!), such as European Social Survey, WRES etc., but explain how anyone wishing to reproduce your replication paper can access it (link the original data repository or mention R function).
Create R Markdown report as a HTML file
Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents, which allows producing transparent and professionally looking reports and papers. For more details on using R Markdown see http://rmarkdown.rstudio.com
You should embed all your R script in R Markdown, so anyone with the same data could rerun the analysis and reproduce your work. This also allows me to see your code and analysis side-by-side, supported with your narrative why you have run the analysis in this way.
Use SMI205 Replication paper template and other examples I prepared in R Markdown – both available via module’s Blackboard in Resources → Replication tab and on my Rpubs website: https://rpubs.com/AnetaPiekut
Create a new R Markdown output: R Studio → New → R Markdown… or download and use a template created by me and edit it.
You can personalise the look of the report by using a different theme & colours.
Many resources on R Markdown are in Weeks 8 and 9 in the Blackboard.
R Markdown report structure (see details in this Rpubs template):
Reference to the replicated paper
Workspace setup
1. Introduction with the paper critical review
2. Data and methods
3. Results of your replication study
– an exact replication and your research extension
4. Conclusions
References
Submission steps:
Unfortunately, the submission portal – Turnitin – does not allow submitting HTML or R Mardown files, so after you publish your report online, save it as a PDF too.
Steps to follow:
1. Create an account on Rpubs website: https://rpubs.com
a. use your student number as a website name
2. Open your R Markdown report in R Studio
3. Knit your report in R Markdown into HTML
4. Publish R Markdown report on your Rpubs account
a. There is ‘Publish’ button in the top-right in the knitted HTML version
b. Follow Rpubs steps in naming and saving the report online
c. Copy the link that Rpubs generates for your report online
5. Go back to your report in R Markdown in R to edit it, and add your Rpubs link to your report → then Knit & Republish it online
a. Add also a link to a repository of you uploaded any files there
b. See in my example here: https://rpubs.com/AnetaPiekut/SMI205_Replication_nonresponse
6. Print the final knitted HTML document into a PDF for submission:
a. it is better to use the version on your computer, not the published one – although it might depend on the browser which one will be better formatted as a PDF
b. I will not assess the visual design of the PDF file, but of the HTML one published on Rpubs, so it is important to embed the link to your report on Rpubs.com
7. Submit the PDF version with Rpubs link via Blackboard/Turnitin as Assessment 2.
Your report will be assessed according to these assessment criteria:
· Execution and Presentation
· Special attention will be paid to how you present and execute the report in R Markdown; the more professionally you present your work using various R Markdown features, the higher your mark.
· Research Design:
· I will look into your replication argument; the more original and insightful your research design, the higher your mark.
· Methodology and Methods
· I will assess whether methods and their interpretation are correct and advanced.
· Results and Analyses
· Is your research presented in a transparent manner? Is it reproducible?
· Do you provide a good selection of tables, graphs or any other materials to effectively present your findings?
· Discussion and Conclusions
· Have you found the same as the original paper/model? Why not?
· In the discussion you should address any theoretical ideas you introduced in the introduction and demonstrate critical awareness of you project limitations.
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com