Online questionnaires in market research: beware of convenience samples

Online questionnaires in market research: beware of convenience samples

We have recently devoted an article to the mistakes that should not be made when carrying out online market research (what is commonly called an online “survey”). One of these errors was the use of a convenience sample. In today’s article, we return to this term and explore the subject further to enable you to achieve more reliable results when conducting your market research.

 

Convenience sample Rented panel
the difficulty of the implementation may be carried out by the project leader without any outside assistance the use of a Market research agency is highly recommended
costs free of charge unless you offer a gift by a random selection process. between 2,5 and 10€ per respondent
total budget <500€ regardless of the number of responses and the completion rate from 4000€ for 500 fully completed questionnaires
Timing to obtain 500 completed questionnaires 1 week – 1 month (depending on the size of your network and the selection criteria for respondents) generally, less than one week regardless of the selection criteria for the respondents.
quality of the answers a real risk of obtaining highly biased responses lower risk of bias: participants receive an incentive.
representativity of the sample no control over the characteristics of the respondents possibility to specify socio-demographic categories before the start of the online survey

The place of online surveys in market research

Let’s start by dispelling a long-standing myth. The following equation is wrong: “online survey = market research”

As we have modestly tried to explain in our online market research guide, an online survey is only one method of market research among others. It is only by combining several techniques that you will be able to approach the (changing) reality of a market.

In our 7-step market research methodology (an overview of which is provided below) the quantitative part comes at the very end. Nowadays, online surveys have become the ultimate method for collecting quantitative results, so they should be placed in this category. Its place is therefore at the very end of the market study to reliably quantify the results you have outlined in the qualitative part.

What is a convenience sample?

The convenience sample can be defined as a sample of participants selected by

the ease of recruiting them. They are therefore the most prepared, willing and/or able to participate in the study.

The habit (and also the erroneous information that is conveyed about carrying out market studies and the methods to be used) leads a large number of project leaders (or even professional marketers) to use online surveys exclusively and to favour samples of convenience to minimise their costs. Because if the convenience sample has many defects, it also has a definite advantage, that of not costing anything. By mobilising your first circle of contacts and people you don’t know (via social networks for example), the answers to your questionnaire will cost nothing. But this financial saving has a marketing cost that we explain in this article.

Examples of convenience samples

Although this sampling method is highly questionable, it must be admitted that it is widely used. Marketing studies in scientific publications have in the past been primarily based on answers given by students, and it must be recognised that even if it is less systematic (the use of Big Data in market research is indeed getting more popular), this method is still popular with academic authors.

In psychology, for example, first-year students at American universities still frequently serve as more or less forced respondents for studies conducted by their professors (who would dare say no to their professors?). Dan Ariely’s book on behavioural psychology is full of such studies.

The dangers of convenience samples in marketing research

The use of convenience samples presents 2 types of risks that we explain below.

Risk n°1: the representativeness of marketing targets

We can illustrate the shortcomings of convenience samples by taking the example of first-year psychology students at a particular university. Imagine that the latter have been surveyed on their propensity to exercise. The results obtained will not necessarily be representative of students’ inclination to exercise in a context different from that of the university. On the other hand (and this is the most obvious shortcoming), the results cannot be extrapolated to the population of students in general (only 1st-year psychology students were surveyed) and even less to the population in general. The purpose of market research is to study the needs of an entire target population and its attractiveness for a product or service.

Risk n° 2: the representativeness of opinions

Another danger of convenience samples, often overlooked and yet crucial, concerns the representativeness of opinions. The use of a convenience sample often leads to the expression of particularly motivated respondents who sometimes have extreme views. The risk is therefore that the results may show extremes that are not representative of the general opinion. At the same time, focus groups present a similar risk during the moderation of the discussion group. Some participants have a natural tendency to want to impose themselves (especially when they defend strong opinions), which leads to a risk of monopolisation of speech. For an in-depth discussion of this problem, we invite you to consult the articles we have devoted to this methodology.

In short, the convenience sample is a methodological aberration for those who want to carry out reliable market research.

The solution: how to do without convenience samples in your online surveys

We have already discussed the solution to the problems posed by convenience samples in our article on conducting B2C market research. This solution is called panel rental.

Renting a panel is in itself a low-cost solution (everything is relative of course) compared to the benefits provided. We offer you an overview of the advantages/disadvantages in the table below.

We describe three advantages and two disadvantages of panel rental.

Advantages of panel rental

A panel consists of people who have applied to participate in online surveys. The motivation for participating in the online survey is therefore primarily financial, which reduces the risk of only seeing people with extreme opinions express themselves. The first advantage is consequently to reduce the risk of a biased approach. The second advantage is the speed with which the results are collected. In a panel, the rule is first come, first paid. Respondents, therefore, have an interest in participating and responding quickly to the survey to earn money. Most online surveys can thus be completed in 1 week, and it is not uncommon to see online surveys completed in 48 hours. The third and final advantage is the representativeness of the data. Renting a panel allows you to specify socio-demographic characteristics for your respondents, which guarantees that 100% of the answers to your online questionnaire will be accurate.

Disadvantages of renting a panel

Panel rental does not only have advantages. It also has some disadvantages that project leaders must take into account. The first disadvantage is the cost of renting a panel. Using a convenience sample is free or almost free (sometimes you will offer a gift through a random draw to participants). Renting a panel is expensive. Count at least 2.5€ before tax per complete response for panels that are not or only slightly specified (i.e. without any imposed socio-demographic criteria). These costs can sometimes exceed 10€ per respondent when the selection criteria are numerous or very specific. The second disadvantage is the implementation of an online survey with a rented panel. Specific software must be used, the questionnaire(s) require a programming effort that is not accessible to everyone. In other words, you will need to use a market research firm. The typical budget for an online survey managed by our agency starts at 4000€ for 500 complete answers.

 

Conclusion

If the superiority of the rented panel is no longer to be demonstrated, the use of a convenience sample is not necessarily to be avoided. The convenience sample can be useful in a preliminary phase of the project when the idea of a product/service is not yet clearly defined. The convenience sample can, therefore, be used to invalidate an idea. If your relatives, constituting your sample of convenience, refute your concept, the chances are that the strangers in a rented panel will give you even less chance.

So, use the convenience sample wisely at the beginning of your market research (look at step 1 of our online market research guide: you will be surprised).


Posted in Marketing.