Predicting and preventing non-response in cohort studies

This project was funded by the ESRC as part of their Survey Design and Measurement Initiative (SDMI) (which is in turn part of their Survey Resources Network) and directed by Ian Plewis (now at the University of Manchester) with Lisa Calderwood (CLS) and Rebecca Taylor (Natcen) as co-investigators. The research officer on this project was Sosthenes Ketende (CLS). The project ran from 1/1/2008 to 31/12/2009 and was integrated with the fourth sweep of the Millennium Cohort Study.

Longitudinal surveys must overcome a number of methodological challenges. Foremost among these challenges is the attrition problem: the fact that subjects are lost from the surveys every time the sample is remeasured. This sample loss can arise from a failure to locate sample members, or from a failure to contact them, or because contacted subjects choose not to cooperate with the survey. There are two unfortunate consequences of sample loss: the sample becomes progressively smaller; and, often more importantly, the sample becomes less and less representative of the population of research interest because the propensity for subjects to be lost varies in a systematic way. Hence, there is the potential for inferences about change to be biased. The research addressed the attrition problem in two linked ways: by testing whether it is possible to reduce the level of non-cooperation in the field, and by examining the characteristics of the three types of non-responders with a view to improving our ability to predict attrition and hence, in the future, to introduce measures to prevent it.

The prevention component

We tackled the first problem by exploiting the strength of randomised experiments to test hypotheses about non-cooperation. There is some evidence that subjects decline to continue to participate in longitudinal surveys not because of a deep-seated antipathy to surveys but for situational or circumstantial reasons. We tested two ways of increasing cooperation: by using a leaflet that addresses known concerns, and by increasing the numbers of refusals that are reissued by allocating them to a different, and usually a more experienced interviewer. By 'crossing' these two components, we were able to see whether they each have an effect independent of the other or whether it is the combination of the two parts of the intervention that is most effective.

The prediction component

Our approach to the second research question - whether it is possible to learn more about the kinds of subjects that are lost from the survey - was based on sophisticated statistical modelling, supplemented by the collection of data from the interviewers and from the respondents after fieldwork is over. One advantage that longitudinal surveys have is that we can use data collected at earlier occasions to predict how respondents might respond to the prospect of a further contact. Our analyses help to set limits to the predictability of non-response and thus to the benefits of implementing procedures in the field prior to interview to prevent it.

Although it is not possible for one relatively small research project to solve all the problems created for longitudinal surveys by attrition, this project has provided us with increased insights about the reasons for non-response in all its forms and, in turn, this understanding will lead both to better quality longitudinal data and to more powerful ways of adjusting for non-response in analyses.

Outputs and dissemination

The data and documentation for the project will be deposited in the UK Data Archive in summer 2010.

The project’s findings have been presented at several meetings and conferences, including the European Survey Research Association Conference in Warsaw, Poland (June 2009), the Statistics Canada Symposium on Longitudinal Surveys: from Design to Analysis in Gatineau, Canada (October 2009) and the ESRC’s Research Methods Festival in Oxford, UK (July 2010).

A paper based on the results from the experiment has been submitted to a journal for publication.

A paper based on the results from the statistical modelling of non-response at wave 4 will be submitted to a journal for publication later in 2010.

CLS contact:

Lisa Calderwood,

Senior Survey Manager

Lisa leads the teams responsible for survey management and cohort maintenance on the 1958, 1970 and millennium cohort studies. An expert in all aspects of survey design, Lisa’s research interests are in survey methodology, particularly in relation to longitudinal survey design and implementation. Email Lisa.

Project Team:

Ian Plewis , University of Manchester

Rebecca Taylor , NatCen Social Research