Background
Summary of Council decision:
Two issues were investigated, both of which were Upheld.
Ad description
The website and a press ad for Vodafone, seen in March 2020:
a. The home page of the Vodafone website featured the headline claim “Awarded the UK’s best mobile data network” alongside an image of a gold medal with the claim “No. 1 Mobile Network Performance. Nperf. 2019”.
b. The press ad featured the headline claim “Switch to 5G. On the UK’s best mobile data network”.
Issue
EE Ltd challenged whether the claim “the UK’s best mobile data network” was:
1. misleading and could be substantiated; and
2. verifiable.
Response
1. Vodafone Ltd said the claim was based on third-party testing data conducted by nPerf. They provided the nPerf report which showed that Vodafone had ranked higher than EE, O2 and Three based on a combined set of measures: success ratio; download bitrate; upload bitrate; latency; browsing; and streaming YouTube. The report stated that Vodafone had both higher download and upload speeds, while it had joint highest latency, browsing performance and streaming performance with EE.
They said nPerf had conducted several million tests around the world and was recognised as a leader in internet performance measurement. They explained that nPerf had developed a new process and algorithm that tested the actual internet experience of the consumer. In contrast, most other tests did not represent a consumer’s actual internet experience as they did not account for the consumer’s distance from the server that hosted the data, which could significantly reduce the download speed. Other tests also did not account for how a consumer was connected and which devices they were testing. Vodafone provided a spreadsheet of the raw data behind the report and explained the principles of nPerf’s scoring and the final rating.
Vodafone explained that the nPerf test featured an app that could be downloaded on the consumer’s mobile device and conducted a bitrate test, latency test, browsing test and a test to measure the quality of watching a video on YouTube. Vodafone said the score for each network was based on a combination of speed and latency tests and “quality of experience” tests (web browsing / streaming).
In response to EE’s concern that the sample size of 35,664 was too small, Vodafone argued that it was sufficient to produce statistically-significant results to substantiate the claim. They said nPerf had experience in over 40 international markets and had been using the same statistical formula in major markets with no issues. They said it was inappropriate to compare sample sizes with other testing bodies as they each used different methodologies. They said that BT/EE made up 28.9% of the testing total while Vodafone made up 28% of the testing total.
Vodafone provided a spreadsheet detailing the different devices tested per network, which showed that each network was tested on a similar proportion of device brands (Apple, Samsung, HUAWEI, etc.) In response to EE’s concern that the sites tested were not weighted by population size, they said that they operated in a crowdsourcing mode, but nPerf had strict rules that eliminated heavy users. For instance, if a person did several tests per day, their tests would be eliminated and too many tests in the same area in a given time, even from separate devices, would be eliminated.
Vodafone said that their testing covered the whole of the UK and the distribution of tests in terms of territory and time of day across all four operators was similar. They said they used over 16,000 cell sites across all operators. They provided a map of the UK which showed the distribution of tests undertaken.In response to EE’s concern that a significant number of data points were recorded close to Vodafone’s offices, Vodafone explained that their coverage checker showed unfiltered tests, prior to automatic and manual checks that nPerf conducted to filter out duplication and abusive or fraudulent use. The nPerf algorithm used to form the results was based on filtered results and therefore used only the relevant tests, eliminating biases related to the overrepresentation of certain terminals, users or test locations. They said they had now updated their coverage checker so it only showed filtered tests. Based on filtered tests, they said there was no overrepresentation of certain terminals, users or test locations.
2. Vodafone believed that the nPerf test report was sufficient to enable consumers to verify the claim.
Assessment
1. Upheld
The claim “the UK’s best mobile data network” in ad (a) appeared directly above an image of a gold medal that included text that clearly stated “No. 1 Mobile Network Performance. nPerf. 2019”. The ASA therefore considered consumers were likely to understand that the claim was based on the results of mobile data network testing undertaken by nPerf. We considered consumers would understand that the testing would be of a technical nature. A range of networks, therefore, had been compared across a range of objective performance measures for a representative sample of mobile data users from across the UK and that Vodafone had received the highest score of all the rated networks.
The claim “the UK’s best mobile data network” appeared centrally in ad (b) with only a small image of a gold medal that included the text “No. 1 Mobile Network Performance. Nperf. 2019” in the top right-hand corner. Although we therefore considered consumers would be less likely to associate the claim with the nPerf award, we considered consumers would again understand that the testing would be of a technical nature. A range of networks had been compared across a range of objective performance measures for a representative sample of mobile data users from across the UK and that Vodafone had received the highest score of all the rated networks. We assessed the nPerf testing in that context.
Four data networks were tested – EE, O2, Three and Vodafone – across a range of criteria that included: download speed; upload speed; latency; internet browsing performance; streaming quality; and 4G connection rate. We therefore considered that a breadth of objective, technical factors had been tested. The final rating was based on weighting together those factors. Although Vodafone explained nPerf’s methodology and rationale for how those factors were weighted together, we were not provided with the calculations that underpinned the 2019 report.
We considered it was important that the comparison was based on a representative assessment of mobile data users across different networks across the UK. The testing was based on a sample size of 35,664 tests, undertaken by over 10,000 users, who during 2019 had at some point downloaded an nPerf app and then used it to test their mobile data performance. However, we understood that a third of those tests were undertaken in London. Vodafone provided a map showing the distribution of tests across the UK alongside the raw data for all testing. They were assured by nPerf that the number of tests would result in a representative picture of mobile data performance. They stated that across the four networks the number of tests per location and device was of a comparable proportion. However, this did not indicate that the distribution of tests was geographically representative. We had not been provided with an explanation as to why the map and data were sufficient to demonstrate that was the case.
As well as being representative geographically, we considered it was also important that the comparison was based on a representative assessment of UK mobile data users and usage more broadly. We understood that the nPerf report had relied on users downloading the app to test their mobile data performance. We understood there was a process in place to filter some tests from the final results to ensure certain participants or locations were not overrepresented. However, we were concerned that participants were self-selected and would therefore be limited to a pool of those who were motivated to proactively download and use the app, which we considered was likely to represent a particular type of mobile data user. For example, a user for whom mobile data quality was low or for whom high mobile data speed was important. We were therefore concerned that the findings would not be based on a sample of participants which was representative of the UK overall, as the claim implied.
We also considered the comparison would need to be based on a representative assessment of mobile data networks taking into account differences in mobile devices. Although we were provided with a breakdown of device brands that had been tested for each network, it was not clear if the results were based on a sample of device types that was representative of those used in the UK. Furthermore, we understood that mobile data performance was likely to be affected by the time of day that users accessed their network. Vodafone provided a distribution of tests by each network based on the time through the day they were conducted. We acknowledged that the pattern of when participants used the app to test their mobile data network through the day was broadly similar for each network. However, it was not clear if that distribution was representative of UK mobile data-use more broadly, and therefore whether the results of the testing were representative of the performance that UK mobile data users would achieve.
For those reasons we considered the evidence was not adequate to substantiate the claim “The UK’s best mobile data network”, as consumers were likely to understand it, and we therefore concluded the ads were misleading.
On that point, the ads breached CAP Code (Edition 12) rules 3.1 3.1 Marketing communications must not materially mislead or be likely to do so. (Misleading advertising), 3.7 3.7 Before distributing or submitting a marketing communication for publication, marketers must hold documentary evidence to prove claims that consumers are likely to regard as objective and that are capable of objective substantiation. The ASA may regard claims as misleading in the absence of adequate substantiation. (Substantiation) and 3.33 3.33 Marketing communications that include a comparison with an identifiable competitor must not mislead, or be likely to mislead, the consumer about either the advertised product or the competing product. (Comparisons with identifiable competitors).
2. Upheld
CAP Code rule 3.35 3.35 They must objectively compare one or more material, relevant, verifiable and representative feature of those products, which may include price. required that comparisons with identifiable competitors must be verifiable. That meant that an ad which featured a comparison with an identifiable competitor or competitors needed to include, or direct consumers to, sufficient information to allow them to understand the comparison, and be able to check the claims were accurate, or ask someone suitably qualified to do so.
The claim “Awarded the UK’s best mobile data network” in ad (a) appeared directly above an image of a gold medal which included text which clearly stated “No. 1 Mobile Network Performance. nPerf. 2019”. We considered consumers would be likely to associate the claim with the nPerf award. A click-through from the ad led to a web page which included further information about the nPerf testing, including a table that summarised the results across each of the performance factors. We noted that a further link from that page led to the nPerf website, where the full report could be downloaded.
The claim “Switch to 5G On the UK’s best mobile data network” appeared centrally in ad (b) with only a small image of a gold medal that included the text “No. 1 Mobile Network Performance. Nperf. 2019” in the top right-hand corner. The ad contained small print, which stated “UKs best mobile data network: Based on nPerf testing during 2019. Vodafone was awarded Best 2019 Mobile Internet Performance vs EE, O2 & Three. Based on 35,664 tests carried out on the nPerf app in the UK. Visit Vodafone.co.uk for verification, details and terms”. We understood the link in the ad’s small print led to the home page of the Vodafone website, where the information shown in ad (a) was presented prominently near the top of the page. We therefore considered that for both ads further information was signposted sufficiently clearly.
We assessed whether the information available to consumers was sufficient to allow them to understand the comparison and be able to check the claims were accurate, or ask someone suitably qualified to do so. We noted that the objective measures which contributed to the findings of the testing were summarised on the Vodafone website and in the nPerf report, including the relative performance of each network against those measures. However, we considered the report contained only basic information about the testing methodology and that greater detail was necessary for consumers to understand fully the comparison. We considered, for example, there was no information on the breakdown of devices tested on each network (including 5G devices which would be able to achieve faster speeds) or on the locations of tested devices. We also noted that nPerf’s final decision to rank Vodafone as best for mobile data performance was based on weighting together the various objective performance measures into a final “nPerf score”. However, we considered that the information about weighting wasn’t available in order for consumers to understand the final part of the comparison. Because we considered information was not available for consumers to understand the claim “the UK’s best mobile data network” in ads (a) and (b), we concluded it was not verifiable and therefore breached the Code.
On that point, the ads breached CAP Code (Edition 12) rule 3.35 3.35 They must objectively compare one or more material, relevant, verifiable and representative feature of those products, which may include price. (Comparisons with identifiable competitors).
Action
The ad must not appear again in its current form. We told Vodafone Ltd to ensure that they had adequate substantiation to support claims, including comparisons with identifiable competitors, in their marketing materials. We also told them to ensure that in future consumers were able to verify the comparisons made in their advertising.
