Note: This advice is given by the CAP Executive about non-broadcast advertising. It does not constitute legal advice. It does not bind CAP, CAP advisory panels or the Advertising Standards Authority.

Marketers should give careful consideration to how consumer surveys are designed prior to them being conducted.  The results of the survey should then inform the claims that can be made as a result.

CAP does not provide guidance on survey design methodologies, but this guidance aims to answer some key questions that the Copy Advice team is often asked in relation to survey data and sample claims.

For advice on making comparisons based on consumer surveys, see Comparisons based on surveys or awards.

What sample size should I use?

Given the varying nature of all claims and supporting data, the Code is not prescriptive about the sample size that should be used, nor is there a strict definition of what constitutes a "statistically significant" sample, as this will always depend on the individual scenario.  The ASA will consider claims and supporting data on a case by case basis and marketers are advised to ensure their sample is of a sufficient size to adequately support the claim being made.  Marketers should ensure their claims are an accurate reflection of the supporting data and that they do not exaggerate the results.

If a sample size is unlikely to be large enough to produce statistically significant results to substantiate a headline claim, the headline claim is likely to be considered misleading.  Marketers are therefore advised to amend their headline claim accordingly where this is the case to ensure that it does not misleadingly exaggerate the results. Generally, smaller samples may make it more difficult to demonstrate statistical differences.

Marketers are strongly advised against relying on qualifying small print to disclaim a headline claim where the supporting data is unlikely to be statistically significant.  No small print should contradict the impression given by a headline claim or claim in the body copy.  It should be sufficient to support the relevant claim without adding complexity, additional interpretive information or rendering the headline claim misleading (rule 3.9). See also Qualifications.

Marketers should ensure they use a representative sample, capable of supporting their claim.  Marketers should avoid making a claim – whether direct or implied – about the general population if they are using a sample which is unlikely to be representative of the general population, for example because those surveyed were already users of the product, or because a non-representative age group was sampled.   Where this is the case, the headline claim should be amended; it would not be sufficient to imply the results were representative of the general population and then qualify the basis of the survey in small print.

Marketers are also reminded that ambiguous surveys and questions, or those that lack detail or include leading questions are unlikely to be considered sufficient substantiation by the ASA (On the Beach Ltd, 18 July 2018).

Do ads need to state the survey sample size?

While there isn’t a specific requirement in the CAP Code for ads to state sample sizes, marketers should include this information where omitting it would be likely to mislead consumers.

Ads will be judged by the ASA on a case-by-case basis and it is the marketer’s responsibility to ensure their supporting evidence is sufficient to support their claims.

See also Qualifications and Small print.

How should qualifications appear?

Qualifications should be presented clearly and prominently (rule 3.10).  Marketers should consider the size and colour of text, as well as the background against which it appears.  All small print should be easily legible by the average consumer at a reasonable distance, taking into account the medium in which the ad appears.

If an ad features more than one claim which needs to be qualified, each qualification should clearly relate to the relevant claim it supports; marketers should not rely on one qualification to support a number of claims, unless they can all be supported by that qualification.

Can I make extrapolated conclusions?

The Code doesn’t prohibit the use of extrapolated conclusions, but marketers are advised to exercise caution when doing so (Iceland Foods Ltd, 3 January 2018).

When making a claim based on extrapolated conclusions, the ad should make clear that this is the basis of the claim and marketers must ensure the ad doesn’t mislead by exaggerating the results of the survey from which the conclusion is drawn.  It is the marketer’s responsibility to ensure they have sufficiently robust supporting evidence and extrapolated conclusions should only be made where marketers are able to demonstrate that their claim does not mislead.

Can I support objective claims with consumer survey data?

While a subjective claim, such as “60% of survey respondents loved the taste of our cereal”, can be supported by the results of a consumer survey, marketers are strongly advised against relying on such data to support claims that require objective substantiation.

The claim “80% of survey respondents said they saved money by booking their holiday through us”, for example, would be considered problematic in the absence of robust sales data (their own and competitors’) that demonstrated genuine savings were made by those referenced, even if the marketer could demonstrate that those respondents did state that they made a saving.  Self-reported consumer data will not be sufficient to support a claim that would otherwise require objective substantiation, so marketers should not rely on consumer perception data to support claims for which they do not hold objective substantiation.

See also Substantiation, Testimonials and Endorsements and Beauty and Cosmetics: General.

More on