Questionnaire construction for quantitative research

A questionnaire is a standardized collection of items (questions) designed to measure a latent construct that cannot be observed directly. Examples of such constructs include attitudes, opinions, and abilities. A comparable example is a written examination, in which standardized procedures are used to assess knowledge or competence.

Strengths and weaknesses of a questionnaire

Questionnaires are indispensable in quantitative social science research. They offer a wide range of strengths:

  • Quantifiable results: The standardized structure makes it possible to express results numerically, which facilitates comparisons and statistical analyses.
  • Representative results: If the questionnaire is carefully designed and the sample is selected correctly, the results can be considered representative.
  • Comparability: Since all respondents answer the same questions, their responses are comparable.

However, questionnaires also have weaknesses that need to be taken into account:

  • Limited flexibility: A questionnaire only captures what was specified in advance. There is little room to address individual differences.
  • No opportunity for follow-up questions: Unlike qualitative interviews, there is no opportunity to address ambiguities or misunderstandings directly.

Factors influencing questionnaire design

Creating questionnaires is a complex process. A variety of preliminary steps need to be taken into account:

  • Research question and hypotheses: The questionnaire should be designed to address the research question precisely.
  • Survey method: Depending on the medium (written, online, telephone), specific adjustments need to be made, such as to the order of the questions.
  • Sampling: The process of selecting participants influences the generalizability of the results.
  • Pretests: Before the final implementation, pretests should be conducted to identify problems with comprehensibility or response logic.

Survey methods and response formats

A questionnaire can be completed in several ways (survey methods):

  • Written (paper or online)
  • By telephone
  • In person (face-to-face)

Each method has advantages and disadvantages in terms of cost, time, anonymity, and data quality.

Within a questionnaire, a distinction can also be made between different response formats:

  1. Open-ended response format: Respondents can formulate their answer freely. One example would be: “What would you like to change about your living situation?”
  2. Fixed-choice response format: Respondents choose from predefined response options, such as a Likert scale (e.g., 1 = strongly disagree, 5 = strongly agree). One example would be: “How satisfied are you with your current living situation?” (1 = not at all satisfied, 5 = very satisfied)

Criteria for good questions

  • Clarity: Questions should be worded clearly and unambiguously.
  • Precision: Short questions are better for avoiding misunderstandings.
  • Comprehensibility: The questions should be understandable to the target group.

Another important point from your notes is the reuse of existing, validated scales from the academic literature. This saves resources and increases the comparability of the results.

Scale development

First of all: The word “scale” is used in statistics in several different ways. Here, a scale refers to a collection of several items. However, we can also use it to refer to a response scale or the level of measurement.

Items and scale construction

An item is the smallest unit in a questionnaire and consists of the item stem (the question or task) and the response format. The item stem describes what the question is about, while the response format defines how respondents can answer. These two elements are crucial to the quality of the data you ultimately obtain.

A good item should be worded clearly and understandably. Here are some typical components:

  • Item stem: The actual question or statement, e.g.: “How strongly do you agree with the statement: ‘I feel comfortable in my current living situation’?”
  • Response format: This determines how respondents answer. There are open and closed response formats.

Examples of response formats:

  1. Open-ended question: Respondents can formulate their answers freely. Example: “What would you change about your living situation?”
  2. Closed-ended question: Respondents choose from predefined categories. Example: “How satisfied are you with your living situation?” (1 = not at all satisfied, 5 = very satisfied)

Scale construction

Choosing the right scale is crucial for the quality of the measurement. Likert scales, Guttman scales, and Thurstone scales are frequently used, depending on the research objective and the type of construct.

Likert scale

The Likert scale is one of the most frequently used scales in survey research. It measures attitudes or opinions by presenting respondents with a statement and asking them to express their agreement or disagreement on a graduated scale.

Example of a Likert scale:

  • “How strongly do you agree with the statement: ‘I feel comfortable in my current living situation’?”
    • 1 = Strongly disagree
    • 2 = Disagree
    • 3 = Neutral
    • 4 = Agree
    • 5 = Strongly agree

The Likert scale makes it possible to capture fine differences in respondents’ opinions or attitudes. It is important that the scale can be either even-numbered (e.g., 4-point, without a neutral midpoint) or odd-numbered (e.g., 5-point, with a neutral midpoint). Odd-numbered scales allow for a neutral response option, whereas even-numbered scales force respondents to indicate a tendency.

Guttman scale

The Guttman scale is less common, but useful when you want a hierarchical ordering of items. The idea is that respondents who agree with an item requiring a high degree of agreement should also agree with all preceding items.

Example of a Guttman scale:

  1. “I sometimes read academic articles.”
  2. “I often read academic articles.”
  3. “I often discuss academic articles with colleagues.”

Thurstone scale

In a Thurstone scale, respondents are presented with a series of statements that vary in intensity. These statements have been rated by experts in advance, and respondents select the statements they agree with. The results are calculated by weighting agreement with specific statements.

Response scales: nuances and special features

The choice of response scale influences how well you can measure the construct:

  • Fully labeled vs. endpoint-labeled: Fully labeled scales give respondents a clearer idea of what the response options mean, whereas endpoint-labeled scales name only the extremes (e.g., 1 = “not at all”, 5 = “very much”).
  • Even vs. odd-numbered scales: As mentioned above, even-numbered scales do not offer a neutral response option, whereas odd-numbered scales provide a neutral midpoint.
  • Number of scale points: Scales that are too broad (e.g., 1 to 10) can overwhelm respondents and lead to false precision, whereas scales that are too narrow (e.g., 1 to 3) allow little differentiation.

Example analysis in R

To analyze the item structure and scale construction in R, you can use Cronbach’s alpha, for example, to measure internal consistency:

# Install the "psych" package if it is not already installed
install.packages("psych")
library(psych)

# Example data for 5 Likert items, answered by 10 people
daten <- data.frame(
  Item1 = c(4, 5, 3, 2, 4, 5, 1, 3, 4, 2),
  Item2 = c(5, 5, 4, 3, 4, 4, 2, 3, 4, 3),
  Item3 = c(4, 4, 3, 3, 4, 5, 1, 3, 4, 2),
  Item4 = c(3, 4, 3, 3, 4, 4, 2, 3, 4, 3),
  Item5 = c(4, 5, 4, 4, 5, 5, 2, 3, 4, 3)
)

# Cronbach's alpha for checking internal consistency
alpha(daten)

This code calculates the Cronbach’s alpha value, which shows how consistently respondents answered different items on a scale.

With a clear structure of items and a well-designed scale, you can ensure that your questionnaire is valid and reliable and effectively captures the data you need.

Implementation in the Unipark survey software

Conclusion

When creating a questionnaire, it is crucial to carefully plan both the content-related and methodological aspects. By taking the points mentioned above into account, you can develop a well-structured and reliable questionnaire that provides valuable data for quantitative research.

Self-assessment questions

Which steps are particularly important when designing a questionnaire for quantitative research?

The key steps in questionnaire construction are:

Finalization and distribution: After validation, finalize the questionnaire and distribute it to the target group.

Definition of the research question and hypotheses: Determine which variables should be investigated.

Operationalization: Translating theoretical concepts into measurable questions.

Question wording: Ensuring that the questions are clear, precise, and easy to understand.

Selection of a suitable response format: e.g., Likert scales, dichotomous responses (Yes/No), and open-ended questions.

Pre-testing: Testing the questionnaire with a small sample to ensure its comprehensibility and functionality.