Monday - June 9th, 2014
Concurrent Sessions B

Routers, Rivers and Reality: a test of Canadian sampling sources, data quality and reliability

Sample is often treated as a commodity, with price and ease of availability often driving choice. But what about sample quality? What about reliability? Does sample source, how the panel was recruited or routing make a difference?

If sample is variable and unreliable, what are the implications for business decision making?  If results cannot be reliably reproduced, can we claim that market research is based on science? 

This paper raises critical questions and provides provocative evidence of the state of sample quality in Canada. 

In this paper we draw on an extensive body of research on research regarding sample quality and reliability.  Comparing river sample, routed sample and sample from multiple panels with different recruitment techniques, we report on variation between panels in the data collected, variation in answers from wave to wave and the effects of recruitment source on representativeness. 

Since sample is the soil and water of market research, it is critical that we pay attention to quality of the source from which our research springs.

Andrew Grenville


Chief Research Officer, Vision Critical

A thirty year research veteran, Andrew leads Vision Critical in the development of innovative and informative analytics and approaches to research. Andrew is the author of numerous research products and measurement scales, with applications in a wide range of areas including health, branding, social capital, and religion. Andrew has published numerous peer-reviewed journal articles and book chapters, in the areas of sociology, research methods and medicine and has been quoted in numerous publications including The Economist.

His most recently published peer reviewed publication is Chakravarti, A., Grenville, A., Morwitz, V., Tang, J., and Ulkumen, G., “Malleable conjoint partworths: How the breadth of response scales alters price sensitivity”, Journal of Consumer Psychology (2012).  Other publications are currently is press.  I have also spoken at research-related conferences organized by the MRIA, the ARF, the AMA, the ASA, AMSRS, SSR, PMRG and ESOMAR.

Rob Berger


Managing Director of Market Panels, Vision Critical

Rob leads the Global Market Panel division at Vision Critical with the authority to sell, build and distribute our panels and partner panels through our Springboard Network to Sparq clients, End Clients and Agencies around the world. Rob is an industry expert in online panel and sampling with extensive time spent building the industry’s highest quality panels while implementing the latest in sample gathering techniques. With 20 years working in offline/online data collection and leading sales efforts for both access and community panels he has worked with hundreds of MR firms and end clients around the world.