28/09/2023
1
MARK5811 AMR8
Survey (2)
T3-2023
Lecture structure for this week
• Course issues and questions
• Last topic: AMR7: Survey (1)
• Questionnaire layout
• Question content and answer
• Pilot testing questionnaires
• Advantages and disadvantages of using questionnaires
• Next topic: AMR9: Analysing Quantitative Data, Report
Writing, and Wrap up
• Lecture summary
Questionnaires
• One of the most widely used primary data gathering
techniques.
• Questionnaires should be used when they fit the objectives of
the research. In a case study that involves seeking the in-depth
opinions and perspectives of a small number of respondents, a
highly structured questionnaire might be inappropriate.
• Where the audience is relatively large, and where standardized
questions are needed, the questionnaire is ideal, and will allow
an analytical approach exploring relationships between
variables.
1
2
3
28/09/2023
2
Designing questionnaires
Questionnaires reflect the designer’s view of the world, ...
Furthermore, what we choose not to ask about, may just as
easily reflect our world view as what we include in the
questionnaire. It is important, then, that, as a researcher, you
are aware of this and try, as far as possible, to be objective.
Questionnaire layout
• Making questionnaire as attractive as possible may improve
the response rate. Factors such as the general layout, choice of
paper, line spacing and answering directions should be
considered. It is best to avoid making the questionnaire too
cramped as this can be off-putting to respondents.
• The way of answering multiple-choice questions should be
consistent throughout – for example, ticking boxes or circling
numbers.
• Boxes or lines should be provided for open question responses.
4
5
6
28/09/2023
3
Writing a set of instructions (1)
• General instructions, dealing with the purpose of the
questionnaire, assurances of confidentiality, how and when to
return the questionnaire.
• Do not assume the respondents know everything, e.g., tick
only one choice for each question.
• One of the problems with instructions is that they are either not
read or are misread. Be careful and creative when writing and
designing instructions.
General instructions – An example
• (Title of survey)
• This survey is being undertaken by [your name] as part of
MARK5811 research project at the University of New South
Wales to [state purpose].
• It will take approximately x minutes to complete.
• All information provided is completely confidential.
• Thank you for providing your comments.
• Please return the completed survey to ________ by _______.
7
8
9
28/09/2023
4
Writing a set of instructions (2)
• Provide section introductions when the questionnaire is
divided into subsections.
• Question instructions (e.g., tick only one box).
• ‘Go to’ instructions as ‘skip instructions’ are important,
because they avoid respondents reading or completing
questions that do not apply to them. The problem is getting
people to read the skip instructions correctly.
Types of question
Different formats can be used for questions. Using a variety of
such formats adds interest and can even help increase
questionnaire response rates.
10
11
12
28/09/2023
5
Open questions
• Open questions have no definitive response and contain answers
that are recorded in full. The questionnaire must be designed in
such a way that respondents are able to provide a response
without the restriction of lack of space.
• Open questions often begin with words such as ‘How’, ‘Why’,
‘What’.
• Key advantage:
➢ Potential for richness of responses, some of which may not
have been anticipated by the researchers.
• Key disadvantages:
➢Harder to analyse.
➢ Takes longer time for the participant to answer the question.
Closed questions
• A closed question is one to which the respondent is offered a
set of pre-designed replies such as ‘Yes/No’, ‘True/False’,
multiple-choice responses, or is given the opportunity to
choose from a selection of numbers representing strength of
feeling or attitude.
• In contrast to open questions, closed questions may restrict the
richness of alternative responses, but are easier to answer and
analyse.
• They also make it easier to compare the views of one group
with another.
• Closed questions can be useful in providing respondents with
some structure to their answers.
13
14
15
28/09/2023
6
Closed questions - List questions
• List questions provide the respondent with a list of responses,
any of which they can select.
• While the list will influence the direction of people’s
responses, this does not make the approach invalid.
Closed questions - Category questions
• Most category questions are designed so that only one
response is possible.
• For self-administered questionnaires and telephone
questionnaires, it is suggested that a maximum of no more than
five alternative responses.
16
17
18
28/09/2023
7
Closed questions - Ranking questions
• Ranking questions requires the respondent to rank responses in
order.
• It is important to make the instructions for completing the
question clear and explicit.
• More than seven or eight items in the list may make it too
complex for many respondents to complete.
• For telephone interviews, items should be limited to no more
than three or four.
• An ‘other’ category is also provided to catch any features not
mentioned in the list.
Closed questions - Scale questions
• Scale or rating questions are used to measure a variable, and
comprise four types of scale: nominal, ordinal, interval and
ratio.
• A common type is the Likert scale on which respondents are
asked to indicate how strongly they agree or disagree with a
series of statements.
• Common formats e.g. ‘True/False’, ‘Yes/No’, can also be used.

• Another approach would be to get respondents to mark a point
on a continuum.
19
20
21
28/09/2023
8
Drafting the question content (1)
• The researcher has to be clear about the information required
and encode this accurately into a question.
• The researcher must interpret the answer as the respondent had
intended it to be interpreted.
Making use of validated scales
• Measurement instruments (such as questionnaires) contain
collections of items (individual questions) that together make
up a scale. Rather than invent your own set of items in a
measurement scale, a much better approach is to make use of
existing scales. Because:
➢Measurement scales are usually developed by experienced
researchers who have the skills and experience to do this.
➢Scales that are reported in the academic literature (such as
journal articles) have been validated and their internal
consistency measured (e.g. by reporting the Cronbach alpha
of the scale).
22
23
24
28/09/2023
9
Developing questions to measure a construct
• To increase reliability we normally have to use more than one
question to measure a construct accurately.
➢Stage 1: Create construct definitions. They need to be
derived from previous studies or from expert opinion.
➢Stage 2: Create sets of statements that are designed to
represent the definition.
➢Stage 3: Test the statements against the definitions.
25
26
27
28/09/2023
10
Drafting the question content (2)
• Ask yourself the following questions:
➢ Is the question necessary? Just how will it be useful?
➢Are several questions needed on the subject matter of this
question?
➢Do respondents have the information to answer the
question?
➢Will the respondents give the information that is asked for?
28
29
30
28/09/2023
11
Drafting the answer
• Decide on how you want people to respond and stick with it. If
you require respondents to tick their responses, get them to do
this throughout the questionnaire, rather than to also
incorporate underlining and circling.
• In general, people seem to be used to box-ticking. It should be
absolutely clear how the respondent is to complete the
questionnaire.
31
32
33
28/09/2023
12
Providing response categories
• Asking a question like ‘What employment sector did you work
in before your present job?’ is asking for trouble. It might both
confuse the respondent (‘What do they mean – sector?’) or the
respondent might be uncertain as to whether their
classification is acceptable.
• It would be appropriate to provide a list of categories such as:
Finance, Retailing, Education, Commerce, Agriculture, Other
(please specify), etc.
❖Providing these categories also yields a standardized set of
responses that will make the data easier to analyse.
Using classification questions
• Classification questions deal with the name, sex, age, status, etc.
of the respondent.
• These questions can be introduced by statements like: ‘It will
help us in further analysis if you would tell us a little about
yourself’.
➢Do not ask for information that is not needed, e.g., is it
necessary to know the respondent’s exact age?
➢People may also be reluctant to reveal details of their salary. It
may be easier to obtain their response to a question on job
grade that may provide an indirect indication of salary.
Good questions are…
• Simple and easily understood
• As short as possible
• Unambiguous
• Not likely to offend
• Provided with categories such as ‘Don’t know’ and ‘Not
applicable’
34
35
36
28/09/2023
13
“Everything should be made as simple as
possible, but no simpler.”
Albert Einstein
• How often do you do grocery shopping?
➢ Potential answers: a) Not often, b) Often, c) Very often.
• How often do you do grocery shopping?
➢ Potential answers: a) Once a week, b) Twice a week, c)
Three times or more a week.
37
38
39
28/09/2023
14
Sequence of questions
• There should be a logical flow to the sequence of questions,
just as you would expect in a formal written text.
• Personal questions about age, gender, rank, status (work and
marital, etc.) should be kept to the end of the questionnaire.
• Going from easy to more difficult questions.
• Key questions that are obviously key to the purposes of the
survey should come first.
• Special attention be given to the first question since this will
help determine whether the questionnaire is answered or put in
the wastepaper bin.
40
41
42
28/09/2023
15
Checklist (1)
• Is attention paid to detail – e.g., overlapping categories such as
‘age 30–35, 35–40’
• Will the words used have the same meaning for all
respondents, regardless of nationality, language, culture?
• Wording: e.g., is personal wording preferable (e.g. ‘How do
you feel?’), or impersonal (e.g., ‘How do you think people
feel?’). The first is a measure of feelings, the second a measure
of respondents’ perceptions of other people’s feelings.
43
44
45
28/09/2023
16
Checklist (2)
• Do the questions make sense?
• Are all concepts in the questions understandable?
• Are the questions interesting and motivating to answer?
• Are the instructions for completing the questionnaire clear?
• Is the questionnaire of the right length?
46
47
48
28/09/2023
17
Designing email and web-based questionnaires
Given that many organizations, particularly larger ones, have
good connections to the Internet, the use of online surveys is
especially advantageous in terms of convenience and access to
large samples and populations.
Additional tips for email and web-based
questionnaires (1)
• The principles of email questionnaire design are very similar
to paper-based questionnaire design.
• Personalize all email contacts.
• Keep the covering (introductory) text brief, avoiding the need
for the respondent to scroll down the page.
Additional tips for email and web-based
questionnaires (2)
• Introduce the Web questionnaire with a welcome screen that is
motivational, that emphasizes the ease of responding, and
shows how to proceed.
• Provide a login to limit access to the site to the selected
sample.
• Use colour appropriately and not just for the sake of it.
49
50
51
28/09/2023
18
Pilot testing questionnaires (1)
• All questionnaires, whether paper-based, email or Web-based,
need careful piloting before dissemination to a wider audience.
• In the case of Web-based questionnaires, it is best if a paper-
based version is piloted first to produce a final version ready
for putting online. Once online, this Internet version should
also be piloted to see if respondents find it easy to use. While
your questionnaire may appear neat and tidy on your own
screen, words may appear out of place on some users’ screens
if they are using a Web browser different to your own. Piloting
using a variety of Web browsers is therefore necessary.
Pilot testing questionnaires (2)
• What should be tested?
➢ Instructions.
➢Style and wording.
➢Respondents’ names, addresses.
➢Presentation of the questionnaire.
➢Length of the questionnaire – not too long.
➢Sequence of questions.
➢Quality of individual questions.
➢Scales and question format used, for example, Likert
scales, Yes/No responses, etc.
Pilot testing questionnaires (3)
• What should be tested? (cont.)
➢Check the ability of a question to discriminate. It is often
not very useful if everyone responds with the same answer
to a question, since one purpose of using a questionnaire is
to examine the diversity of views on a subject.
➢The use of colour, on-screen instructions, navigational
routes (especially for skip questions) and how respondents
input their responses to questions (i.e. do they know what
to do?).
➢Observation at the piloting stage with respondents actually
using the website questionnaire may also reveal some
entirely unanticipated problems.
52
53
54
28/09/2023
19
Through piloting you may decide to…
• change the wording of individual questions
• add/eliminate some questions
• modify the instructions
• decide not to use the questionnaire!
Maintaining quality: Validity
• Avoid a questionnaire not covering the research area as well as
some questions being asked that are irrelevant to the study.
• For non-response, follow-up interviews can be used for those
who did not reply, and their responses compared with those
who did answer the questionnaire to see if the two sets of
responses are similar. If they are, it suggests that the
responding and non-responding populations are the same.
• Get your questionnaire reviewed by a few (e.g. 5) of your
peers and get them to write down the problems they find on
the questionnaire.
Reliability
• Reliability can be measured by a reliability coefficient using a
scale from 0.00 (very unreliable) to 1.00 (perfectly reliable). In
practice, a score of 0.9 is generally acceptable. There are
several ways in which this coefficient can be calculated. One
of the most common is Cronbach’s alpha, which presents the
average of all possible split-half correlations, and so measures
the consistency of all items.
55
56
57
28/09/2023
20
Advantages of using questionnaires
•Low cost in terms of both time and money.
•The collection of data is quick and from many people.
•Respondents can complete the questionnaire at a time and place
that suits them.
•Data analysis of closed questions is relatively simple, and
questions can be coded quickly.
•Respondents’ anonymity can be assured.
•Lack of interviewer bias.
Disadvantages of using questionnaires
• Low response rates: the response rate can be depressingly low.
This is even more of a danger if questionnaires are too long.
• No opportunity for probing questions.
• Most people find verbal communication easier than using the
written word, yet questionnaires demand a certain level of
literacy.
• Respondents might give inaccurate answers, but the researcher
is not in a position to detect this.
Next Topic: AMR9 – Analysing Quantitative Data,
Report Writing, and Wrap up
• Data preparation, analysis, and reporting
• Wrap-up
• Exam information
58
59
60