Postgraduate coursework, Information
School, INF6060 Information Retrieval (2022-23)
Review of the University of Sheffield website search system
The aim of this coursework is for you to evaluate how a search system supports users with
finding information through searching, i.e. for the kind of information seeking where the
starting point is formulating a query. This coursework requires you to carry out a form of
‘expert’ assessment of the University of Sheffield website search system (note: make sure
you assess the website search system and not StarPlus, the University library system). You
will consider how well the search system meets the needs of its users undertaking their
typical search tasks. You are required to submit two assignments. For assignment 1, you will
create tasks and queries that you will use to test the search system for assignment 2. For
this second assignment you will produce a portfolio of 4 sets of activities and
recommendations for how the search system could be improved. As a piece of academic
work, you are expected to engage with the research literature and provide references to
justify and support your writing.
If you have questions about the assignments, please contact Dr Sophie Rutter
(s.rutter@sheffield.ac.uk) to discuss. The tasks and queries (assignment 1) are worth 10% of
the overall module mark and the portfolio with recommendations (assignment 2) are worth
90% overall. Submission deadline for the tasks and queries is 10am Monday 31st October
2022 viaTurnitin. Submission deadline for the portfolio with recommendations is 10am
Monday 23rd January 2023 via Turnitin.
Assignment 1: Tasks and queries (10%)
In weeks 2 and 3 you will learn how to create tasks and queries for testing search systems.
For this assignment, you are required to create tasks and queries that you can use to test the
University of Sheffield website search system. There is no word count but you should aim to
be succinct.
Tasks and
queries
What we are looking for Mark
Simulated work
tasks (SWT)
Two SWT that are realistic for the search system (i.e. based
on what typical users might do). The situation should be
different for each SWT.
35
Search tasks
One search task for each of the two SWT tasks. The search
task should be categorised according to the research literature
(e.g. Toms, Kellar etc). The two search tasks should be
different types (e.g. if one is a specific item the other should be
general topical).
35
Queries
10 queries for each search task. These queries should be
categorised and sufficiently different that you could test
different aspects of the search performance.
20
References All use of the research literature should be acknowledged and
cited using the APA style.
5
Presentation
Good use of English and clear presentation style.
5
Assignment 2: Portfolio of activities with recommendations (90%)
For this assignment you will develop a tokenisation and text processing strategy and evaluate
the University of Sheffield website search system for how well the search system meets the
needs of its users undertaking their typical search tasks. In weeks 4-8 you will learn how to
conduct different types of evaluation (heuristic evaluation, measuring retrieval system
effectiveness, and supporting the users’ search process) and how to develop text tokenisation
and processing strategies. Throughout the module, but particularly in weeks 9-12, you will
learn about what is state-of-the-art in information retrieval that you can use to help you develop
recommendations.
You are required to produce a portfolio that includes
1. The tasks and queries you used to test the search system
2. The results of your four activities
3. Recommendations based on your activities and what is considered to be best practice
in the information retrieval research literature.
Each of these sections is described in more detail next.
(1) Tasks and queries. You must state all the tasks and queries that you have used for the
four activities. You can reuse the tasks and queries created for assignment 1. However, as
you become more familiar with the module content and the website to be evaluated you may
wish to revise and extend the assignment 1 tasks and queries. It is also recommended that
you revise your original tasks and queries in response to the feedback received for assignment
1. Additionally, you should also briefly explain why you have made changes to the tasks and
queries.
There is no word limit for the tasks and queries but you should keep your entries succinct. We
have suggested an approximate number of words as a guideline.
Portfolio - section 1 What we are looking for Marks
Tasks and queries All tasks (simulated work tasks and search
tasks) and queries used in the activities.
(guideline up to 100 words per simulated
work task)
1 mark
deducted from
presentation if
not included.
Reflection
A brief explanation of any changes made to
the assessment 1 tasks and queries
(guideline 100 words)
1 mark
deducted from
presentation if
not included.
(2) Activities 1 - 4: You MUST conduct ALL 4 activities. The minimum requirements for
each activity is stated in the table below. You should aim to go beyond this. Some ideas to
help you develop and extend your activities include (i) use more tasks and queries, (ii)
compare against website search systems from other universities, (iii) test on different
devices etc.
There is no word limit for the activities but you should keep your entries succinct. We suggest
500 words per activity as a guideline.
Portfolio - section 2 What we are looking for (minimum
requirements)
Marks
Activity 1: heuristic
evaluation
A table that lists the 10 Nielsen heuristics and
provides a brief description of whether the
search interface deals with the heuristic
effectively for 1 of your SWT.
A brief summary of the findings.
15
Activity 2: text tokenisation
and processing strategy
A logical strategy for processing and tokenising
text documents to be indexed by the search
system. You should justify any assumptions you
make when generating this strategy and give
concrete examples of how your text processing
strategy would transform original raw text
documents.
15
Activity 3: measuring
retrieval system
effectiveness
You should submit 3 different queries for each of
your SWTs (i.e. 6 in total) to the University
search system and assess the objective
performance of the system based on your
chosen queries. This will require you to create
relevance judgements for the results of each of
your queries and to assess the results using
appropriate performance metrics up to rank 10.
You should state any assumptions you made
when assessing retrieved documents for
relevance.
15
Activity 4: supporting the
users’ search process
A table that identifies features of the search
system. Features should be categorised using
Wilson (2011) and the purpose of each feature
should be succinctly described.
Brief description of how the user may carry out
the two search tasks utilising the categorised
features. You should structure this description
around a search process model such as Sutcliffe
& Ennis (1998).
Short conclusion on how well the system
15
supports the two task types.
(3) Recommendations. You should base your recommendations on the activities you have
conducted and supported by findings of the research literature.
IMPORTANT: The word count for the recommendations is 1500. You MUST state this word
count on the front cover of the assignment. References are not included in the word count.
Portfolio - section 3
What we are looking for Marks
Recommendations The strengths and weaknesses of the search
system, making use of the test results to
highlight particular aspects that worked well
and not so well.
Recommendations should be based on your
test results for how the search system could
be improved to better meet the search needs
of its end users.
Recommendations should be made in light of
the best practices for search system design
and current/future developments in IR
research.
30
References All use of the research literature should be
acknowledged and cited using the APA style.
5
Presentation You should write the recommendations in the
style of a corporate report and use
appropriately formal language. You may use
screenshots to aid your description.
5
Information School Coursework Submission
Requirements:
It is the student's responsibility to ensure no aspect of their work is plagiarised or the result
of other unfair means. The University’s and Information School’s advice on unfair means can
be found in your Student Handbook, available via http://www.sheffield.ac.uk/is/current
Your assignment has a word count limit. A deduction of 3 marks will be applied for
coursework that is 10% or more above or below the word count as specified above or that
does not state the word count. It is your responsibility to ensure your coursework is correctly
submitted before the deadline.
It is highly recommended that you submit well before the deadline. Coursework submitted
after 10am on the stated submission date will result in a deduction of 5% of the mark
awarded for each working day after the submission date/time up to a maximum of 5 working
days, where ‘working day’ includes Monday to Friday (excluding public holidays) and runs
from 10am to 10am. Coursework submitted after the maximum period will receive zero
marks.
Work submitted electronically, including through Turnitin, should be reviewed to ensure it
appears as you intended.
Before the submission deadline, you can submit coursework to Turnitin numerous times.
Each submission will overwrite the previous submission. Only your most recent submission
will be assessed. However, after the submission deadline, the coursework can only be
submitted once.
If you encounter any problems during the electronic submission of your coursework, you
should immediately contact the Module Coordinator, and the Information School Teaching
Support Team is-teachingsupport@sheffield.ac.uk . This does not negate your
responsibilities to submit your coursework on time and correctly.