Self-completion questionnaires: the importance of testing (based on experience from the british nrs)

There are many design factors which can influence the results of self-completion readership questionnaires: the length and nature of written instructions and examples; the question order; the layout of the response grid; typescript vs masthead representation of titles; and so on. Apparently subtle differences can have an impact. Hence the need for rigorous testing before going live. This paper will present the methods and results of cognitive testing of alternative self-completion questionnaire designs for the Great Britain National Readership Survey (NRS). The proposal was that a self-completion questionnaire could be used in circumstances where it would prove difficult to obtain response by the standard face-to-face (CAPI) method. Two versions were proposed for testing – one emulating the question order and filtering of the standard face-to-face NRS questionnaire, the other a shorter version with a different question order and approach. The criteria for assessing the self-completion questionnaire designs were ease of completion, likelihood of generating ‘correct’ results (in terms of reflecting the reality of respondents’ reading behaviour) and comparability of results with the standard CAPI NRS data. The tests described in this paper were qualitative in approach and scale and designed to address the first two criteria. This test would then be followed by quantitative field testing, to assess comparability with the standard NRS data, before the self-completion questionnaire would be adopted on the NRS.

Download Paper (PDF)