AMED3002: Research Project 2

Oral presentation – 10%
Reproducible research report – 10%
Cross-validation critique - 5%
Team process reflection – 2%



General information about Research Project 2
Aim
This project is designed to offer you an opportunity to demonstrate your learning by reproducing
a high-impact manuscripts findings. For this project, you will work in teams of 3-5 students. You
will also gain experience at working in groups and evaluating research results by assessing their
validity qualitatively and quantitatively.
Teams
Students will form teams in Week 9. When required we will allocate students in such a
way as to foster diversity. Successful and sustainable collaboration is dependent on self
and peer awareness of your strengths and weaknesses, a concerted effort in developing
relationships, a shared vision and well-organised strategy and action.
Assessment
Team product will be assessed by both a team oral presentation (10% team mark), a team
reproducible report (10% team mark) with individual cross-validation critique (5% individual
mark), individual team process reflection (2% individual mark) contributing to an individual
mark. Information for each of these assessments follows.
The team process reflection will be assessed from your reflection of yourself and peers
performance in the team. Your course coordinator will use peer evaluations, group
summaries, meeting attendance and minutes to provide individual marks on how well you
contributed to the team. More information on these assessments follow.
Reproducing a “high quality” manuscript from the literature.
Find a high impact manuscript that uses ‘omics data to construct some form of predictive
biomarker for disease response or outcome. Reproduce their findings and comment on those
that cannot be reproduced. On completion of this task you will have communicated whether you
believe the science behind this paper and whether the methods are appropriate, robust and
reproducible.
This task is split into four components.
(10% group mark) Reproducible report to focus on the following 4 aspects:
1. Provide a clear summary of the aim, methods and results of the paper.
2. Describe the experimental design and workflow.
3. Comment on the appropriateness of the methods and analysis used in the paper.
4. Explain the success or difficulties you had reproducing their results.
(10% group mark) Presentation to focus on the same 4 goals in addition to answering some
questions at the end of your presentation.
(5% individual mark) Cross-validation critique:
Critique the cross-validation approaches that were used in the original study. If there was no
cross-validation used in the original manuscript, apply various cross-validation strategies where
appropriate and comment. Assess the robustness of their findings via cross-validation and other
methods of evaluation.
(2% individual mark) Team process reflection:

The team process will be assessed from your reflection of yourself and peers performance in
the team.

Paper selection criteria should include:
1. A high quality paper that made a reasonable impact in the biomedical sciences
published in the last 15 years. Have a look at the number of citations as well as the
departments and institutions of authors.
2. Uses high-throughput biological assays such as high-throughput sequencing, mass
spectrometry or microarray. Are there informative supplementary material and data
online?
3. Conclusions and results that are based on statistical and/or bioinformatics approaches.
4. The outcome or endpoint can be converted into a binary variable (very
important!!!). This will be important for the cross-validation component of the project.
5. It would help if there were at least 10 samples in each outcome category.
6. It is preferable that you choose a manuscript which hasn’t had code made available. If it
has, you will have to perform an alternate analysis to provide a comparison and critique
of the reported results.
Keep in mind that the project is based on reproducing the selected manuscript.
Research Project 2 – Reproducible report: 10%
This assessment is to be completed in teams as part of your project. It is worth 10% of your
final mark in AMED3002 and each student in the team will receive the same mark. Evaluating
and assessing the validity of research results is never trivial with many scientists placing trust
in the peer review system to filter out poor quality research findings. With requirements of
most journals that ‘omics data be made publicly available; in principle it should now be
possible to reproduce a manuscript’s results. The ability to reproduce results is ground-
breaking as it increases transparency in scientific research and facilitates the ability to assess
the robustness of the original findings.

Assessment
As described previously, this project should reproduce, evaluate and assess the analysis
results from a high impact paper that claims to have identified or constructed some biomarker
of disease state. The report should be thorough, succinct and clear and written in language
that would be appropriate for both your classmates and the authors of the original manuscript.

Length: 3 – 10 pages including appendix and references.
File type: html with “hide-able” code chunks.
Due date: 11:59pm Monday 10th of June

Criteria Does not meet expectations Meets expectations More than meets expectations
F (0 – 4.5) P (5 – 6) C (6.5 – 7) D (7.5 – 8) HD (8.5 – 10)
1. Technical Skills

Purpose stated; information accurate,
clear and concise.
Organisation clear, logical and flowing.
Graphics visually consistent and
labelled.
Provides correct citing of all sources
and direct quotes.
Spelling and grammar checked.
Keeps to word limit.

Purpose not stated and or
information not
accurate/concise and/or
organised in a clear manner.
Graphics are poorly labelled
and/or spelling and grammar
incorrect and/or does not
keep to word limit.

Purpose stated, information
accurate and organised in a
clear manner. Graphics are
labelled. Correct spelling
and grammar and keeps to
word limit.

Purpose and information
accurate, concise and
organised in a clear, logical
and flowing manner.
Graphic visually
consistent with the rest of
report and labelled.
Citation, spelling and
grammar correct and keeps
to word limit.

Purpose and information
accurate, comprehensive
and concise, organised in a
clear, logical and flowing
manner. Graphics eye
catching, complement rest
of report spelling and
grammar correct and keeps
to word limit.


Purpose and information accurate,
concise and discerning and
organised in a clear, logical and
flowing manner. Graphics eye
catching and elucidate the rest of
the report and are visually thought
provoking. Citation, spelling and
grammar correct and keeps to word
limit.


2. Communication skills

Promotes the dissemination of
information and findings.
Attracts and engages audience
to discuss/think.
Graphics are clear and digestible.
Analytical methods reported
appropriately for target
readership.

Does not promote the
dissemination of information
or findings. Does not attract
or engage reader to
think/discuss and/or
analytical methods reported
inappropriately for target
readership.

Promotes the
dissemination of
information and findings.
Attracts and engages
reader. Graphics are
clear and digestible.
Analytical methods
reported
appropriately for
target readership.


Comprehensively
promotes
the dissemination of
information and findings.
Attracts and consistently
engages reader to
think/discuss.
Graphics are clear and
digestible. Analytical
methods reported
appropriately for target
readership.

Thought provoking in
dissemination of
information
and findings. Attracts and
continually promotes
engagement of reader to
think/discuss.
Graphics are clear
and representative
of originals.
Analytical methods
reported intuitively
for target readership.

Inspires dissemination of
information and findings.
Attracts and continually promotes
emotional and intellectual
engagement.
Graphics are clear and
reasonable replicas or
improvements of the
originals.
Insightful and intuitive report
of analytical methods for
target readership.

3. Analytics

Provides a summary of the aim,
methods and results of the paper.
Describe the experimental design and
workflow.
Comment on the appropriateness of the
methods and analysis used in the paper.
Outline the success or difficulties you
had reproducing their results.

Does not provide a summary
of the aim, methods and
results of the paper and/or
describe the experimental
design and workflow and/or
comment on the
appropriateness of the
methods and analysis used
in the paper and/or outline
the success or difficulties
you had reproducing their
results.

Provides a summary of the
aim, methods and results of
the paper.
Describe the experimental
design and workflow.
Comment on the
appropriateness of the
methods and analysis used
in the paper.
Outline the success or
difficulties you had
reproducing their results.

Provides a cohesive
summary of the aim,
methods and results of the
paper. Describe the
experimental design and
workflow. Debates the
appropriateness of the
methods and analysis used
in the paper.
Explains the success or
difficulties you had
reproducing their results.

Provides an insightful and
cohesive summary of the
aim, methods and results of
the paper. Debates and
demonstrates the
appropriateness of the
methods and analysis used
in the paper in the context
of the overall experimental
design and workflow.
Explains the success or
difficulties you had
reproducing their results.

Provides a summary of the aim,
methods and results of the paper
that elucidates its findings and
purpose. Is decisive on the
appropriateness of the methods
and analysis used in the paper and
providing insight into plausible
improvements of the experimental
design and workflow.
The success or difficulties in
reproducing their results are used
to critique the findings and
conclusions of the manuscript.

Research Project 2 - Oral Presentation: 10%
This assessment is to be completed as a team as part of your project. It is worth 10% of your
final mark in AMED3002. You will be assessed on your presentation skills, measured by criteria
covering: slides, flow of information, engagement with the audience and clarity of message. The
purpose of this presentation is to provide the background and analytical process of your
reproducibility study while also helping establish the flow of your written report and providing an
opportunity to get feedback. The presentation should be targeted to a scientific audience.
Time: 7 minute presentation + 3 minutes question time
File type: powerpoint, RMarkdown, keynote etc
Due date: Slides to be submitted by May 26th 11:59pm, presentations run on Wednesday
28th May during scheduled workshop times.

The presentation will be given in PechaKucha style with your team presenting 20 slides. Each
slide will be presented for precisely 20 seconds with each member of the team presenting 5
slides (if in a four member team). Questions will follow the presentation. Presentations will run
during the week 13 workshop slots.

Criteria Does not meet expectations Meets expectations More than meets expectations
F (0 – 4.5) P (5 – 6) C (6.5 – 7) D (7.5 – 8) HD (8.5 – 10)
1. Technical Skills

Purpose stated; information
accurate, clear and concise.
Organisation clear, logical and
flowing.
Graphics visually consistent and
labelled.
Provides correct citing of all
sources and direct quotes.
Spelling and grammar checked.
Keeps to time limit.
Enables equitable contributions from
team members.
Purpose not stated and or
information not
accurate/concise and/or
organised in a clear manner.
Graphics are poorly labelled
and/or spelling and grammar
incorrect and/or does not
keep to time limit.

Purpose stated, information
accurate and organised in a
clear manner. Graphics are
labelled. Correct spelling and
grammar and keeps to time
limit.

Purpose and information
accurate, concise and organised
in a clear, logical and flowing
manner. Graphic visually
consistent with the rest of
presentation and labelled.
Citation, spelling and grammar
correct and keeps to time limit.

Purpose and information
accurate, comprehensive
and concise, organised in a
clear, logical and flowing
manner. Graphics eye
catching, complement rest
of presentation and labelled.
Citation, spelling and
grammar correct and keeps
to time limit. Enables
equitable contributions from
team members.


Purpose and information
accurate, concise and
discerning and organised in a
clear, logical and flowing
manner. Graphics eye catching
and elucidate the rest of the
presentation and are visually
thought provoking. Citation,
spelling and grammar correct
and keeps to time limit. Provides
all team members an opportunity
to demonstrate technical
contributions to project.


2. Communication skills

Promotes the dissemination
of information and findings.
Attracts and engages
audience to discuss/think.
Graphics are clear and
digestible.
Analytical methods reported
appropriately for audience.

Does not promote the
dissemination of information
or findings. Does not attract
or engage audience to
think/discuss and/or
analytical methods reported
inappropriately for audience.

Promotes the
dissemination of
information and findings.
Attracts and engages
audience. Graphics are
clear and digestible.
Analytical methods
reported appropriately
for audience.


Comprehensively promotes
the dissemination of
information and findings.
Attracts and consistently
engages audience to
think/discuss.
Graphics are clear and digestible.
Analytical methods reported
appropriately for audience.

Thought provoking in
dissemination of information
and findings. Attracts and
continually promotes
engagement of audience to
think/discuss. Graphics
are, clear and
digestible. Analytical
methods reported
intuitively for
audience.

Inspires dissemination of
information and findings.
Attracts and continually
promotes emotional and
intellectual engagement.
Creative graphics are clear
and digestible.
Content pitched in a
creative and appropriate
way. Insightful and
intuitive report of analytical
methods for audience.

3. Analytics

Provides a summary of the aim,
methods and results of the paper.
Describe the experimental design
and workflow.
Comment on the appropriateness of
the methods and analysis used in
the paper.
Outline the success or difficulties
you had reproducing their results.

Does not provide a summary
of the aim, methods and
results of the paper and/or
describe the experimental
design and workflow and/or
comment on the
appropriateness of the
methods and analysis used
in the paper and/or outline
the success or difficulties
you had reproducing their
results.

Provides a summary of the aim,
methods and results of the
paper.
Describe the experimental
design and workflow.
Comment on the
appropriateness of the methods
and analysis used in the paper.
Outline the success or
difficulties you had reproducing
their results.

Provides a cohesive summary of
the aim, methods and results of
the paper. Describe the
experimental design and
workflow. Debates the
appropriateness of the methods
and analysis used in the paper.
Explains the success or
difficulties you had reproducing
their results.

Provides an insightful and
cohesive summary of the
aim, methods and results of
the paper. Debates and
demonstrates the
appropriateness of the
methods and analysis used
in the paper in the context
of the overall experimental
design and workflow.
Explains the success or
difficulties you had
reproducing their results.

Provides a summary of the aim,
methods and results of the
paper that elucidates its
findings and purpose. Is
decisive on the appropriateness
of the methods and analysis
used in the paper and providing
insight into plausible
improvements of the
experimental design and
workflow.
The success or difficulties in
reproducing their results are
used to critique the findings
and conclusions of the
manuscript.



Research Project 2 – Cross-validation critique: 5%
This assessment is to be completed as an individual in addition to your team’s reproducible
research project. It will be appended to your copy of the groups reproducible report and
submitted to Canvas. It is worth 5% of your final mark in AMED3002.
The purpose of this task is to critique the cross-validation approaches that were used in the
original study. If there was no cross-validation used in the original manuscript, apply various
cross-validation strategies where appropriate and comment. Assess the robustness of their
findings via cross-validation and other methods of evaluation.
You will be assessed on your application of cross-validation and potentially alternate evaluation
techniques. You will also be assessed on your insight into the behaviours, pros and cons of the
approaches and how they relate to your chosen dataset.

Length: 2-4 pages appended to your groups report.
File type: html with “hide-able” code chunks.
Due date: 11:59pm Monday 10th of June.
Criteria Does not meet expectations Meets expectations More than meets expectations
F (0 – 4.5) P (5 – 6) C (6.5 – 7) D (7.5 – 8) HD (8.5 – 10)
1. Technical Skills

Purpose stated; information
accurate, clear and concise.
Organisation clear, logical and
flowing.
Graphics visually consistent and
labelled.
Provides correct citing of all
sources and direct quotes.
Spelling and grammar checked.
Keeps to word limit.

Purpose not stated and or
information not
accurate/concise and/or
organised in a clear manner.
Graphics are poorly labelled
and/or spelling and grammar
incorrect and/or does not
keep to word limit.

Purpose stated, information
accurate and organised in a
clear manner. Graphics are
labelled. Correct spelling and
grammar and keeps to word
limit.

Purpose and information
accurate, concise and organised
in a clear, logical and flowing
manner. Graphic visually
consistent with the rest of report
and labelled. Citation, spelling
and grammar correct and keeps
to word limit.

Purpose and information
accurate, comprehensive
and concise, organised in a
clear, logical and flowing
manner. Graphics eye
catching, complement rest
of report spelling and
grammar correct and keeps
to word limit.


Purpose and information
accurate, concise and
discerning and organised in a
clear, logical and flowing
manner. Graphics eye catching
and elucidate the rest of the
report and are visually thought
provoking. Citation, spelling and
grammar correct and keeps to
word limit.


2. Communication skills

Promotes the dissemination
of information and findings.
Attracts and engages
audience to discuss/think.
Graphics are clear and
digestible.
Analytical methods reported
appropriately for target
readership.

Does not promote the
dissemination of information
or findings. Does not attract
or engage reader to
think/discuss and/or
analytical methods reported
inappropriately for target
readership.

Promotes the
dissemination of
information and findings.
Attracts and engages
reader. Graphics are clear
and digestible.
Analytical methods
reported appropriately
for target readership.


Comprehensively promotes
the dissemination of
information and findings.
Attracts and consistently
engages reader to
think/discuss.
Graphics are clear and digestible.
Analytical methods reported
appropriately for target
readership.

Thought provoking in
dissemination of information
and findings. Attracts and
continually promotes
engagement of reader to
think/discuss. Graphics
are creative, clear and
digestible. Analytical
methods reported
intuitively for target
readership.

Inspires dissemination of
information and findings.
Attracts and continually
promotes emotional and
intellectual engagement.
Graphics are creative,
clear and digestible.
Insightful and intuitive
report of analytical
methods for target
readership.

3. Analytics

Use of cross-validation in manuscript
critiqued.
Cross-validation is used to assess
robustness of findings.
Comments on broader application of
various cross-validation strategies to
the findings in the report.

Use of cross-validation in
manuscript not critiqued
and/or cross-validation is not
used to assess robustness
of findings and/or comments
on broader application of
various cross-validation
strategies to the findings in
the report.

Use of cross-validation in
manuscript identified.
Cross-validation is applied to
analysis from report.
Comments on application of
cross-validation strategies to
the findings in the report.

Use of cross-validation in
manuscript critiqued.
Cross-validation is used to
assess robustness of findings.
Comments on broader application
of various cross-validation
strategies to the findings in the
report.

Insight is provided by the
critique of the cross-
validation used in the
manuscript.
Multiple cross-validation
strategies are used to
assess robustness of
findings and contextualize
the behaviour of the
strategies
Critique of the cross-validation
used in the manuscript is used
to establish validity of results.
Multiple cross-validation
strategies and other
techniques are used to assess
robustness of findings and
illustrate the behaviour of the
strategies in context.
Research Project – Team process: 2%
You are required to submit a report providing information on your team work and reflections as detailed below.
This is worth 2.5% of your final mark in AMED3002 and each team member will receive an individual mark. On
submission of the information below, you agree that your reflections are a genuine and independent account.
You have tried to be as objective as possible. You have stood back and tried your best to view the experience
objectively.
Length: about 2 pages
File type: pdf, word doc, html
Due date: 11:59pm 10th June.

Your Reflection document should contain the following information.

Team Name:
Your Name /Student Number:

Other Group Members: [Name/Student Number, etc]

1. Meetings you attended: dates/time attended + location
[e.g. Meeting 1- ‘Getting started’ Fisher Library, 25.09.17, 2-4pm;
Meeting 2 – ‘Moving forward’ online via Collaborations, Canvas, 4.10.17, 9-11am]
2. List and outline the tasks you completed:

3. Individual reflection [One page in total]
a. What did you learn from being part of this specific group for this task? [maximum half a page]
[Your reflections could include commentary on yours and the group’s teamwork skills, organisational and time
management skills, and collaborative skills]
b. What did you learn from completing the task? [maximum half a page]
[Your reflections should include commentary on yours and the group’s analytic & thinking skills]
4. Complete the team survey for each member of your team.
Provide a number ranging from 0 (no input) to 5 (maximal input /led this component) to evaluate each group
member (including yourself) for the following tasks.
- Organising the team or ensuring thing get done.
- Drafted or constructed the figures and graphics.
- Performed statistical tests.
- Conception of project and identification of data.
- Construction and formatting of media and presentation.
- Interpretation and communication of findings.


学霸联盟