Leverage A/B testing to evaluate the impact of site speed on your business metrics.
Over the last few years it has been well established that site speed performance is a significant part of the user experience and that improving it benefits different business metrics such as conversion rates and bounce rates. Multiple articles and case studies have been published to back this up, including Cloudflare's How Website Performance Affects Conversion Rates, Deloitte's Milliseconds Make Millions, and Shopping for Speed on eBay.com, to name a few.
Even though the case for speed is clear, many companies still struggle with prioritizing work that will improve their site speed since they do not know exactly how it affects their users and as a result their business.
In the absence of data, it is easy to delay site speed work and focus on other tasks. A common scenario is having some people in the company recognize the importance of site speed and yet not be able to build a case for it and convince multiple stakeholders to invest accordingly.
This article provides high-level guidance on how to leverage A/B testing to evaluate the impact of site speed on business metrics therefore enabling more effective decision-making on the matter.
Step 1: Pick a page to A/B test
Your aim is to test the hypothesis that page speed relates to your business metrics. For simplicity's sake, you can initially limit yourself to identifying a single page for analysis. Future work can expand to multiple pages of the same type to verify findings, and then to other areas of the site. Some suggestions for where to start are provided at the bottom of this step. Several requirements drive the page selection process:
- The A/B test should only be run on the devices of mobile users. Globally, the partner sites we assist see an average of more than 50% (and growing!) of their traffic coming from mobile, but this can increase significantly based on region and vertical. Mobile devices are more sensitive to slower websites due to processing and memory constraints and less stable networks. Also, on-the-go usage patterns mean expectations for speed are higher.
The page you choose for testing should be an important part of your conversion funnel. Every site has a different purpose, so every site tracks different success metrics. These metrics are usually related to a user journey which is analyzed using a funnel. For example, users on an e-commerce website will have to navigate through a home page, category pages, product pages, and a checkout page to complete a purchase. If you're optimizing for conversions, one of those pages would be a good candidate.
The page should have a singular purpose. Unless your site has a very specific mission, it's usually best to avoid using the homepage for your test. Many commercial sites' homepages are portals to a wide variety of functionality that will make your analysis noisy. For example, if you're optimizing for pageviews per session on a news site, it might be best to exclude the non-commercial portions of the site and focus on monetized sections and articles.
The page chosen should get enough traffic so that you don't have to wait long before getting a statistically significant result.
The selected page should be relatively slow. In fact, the slower the better. This not only means that you will likely have an easier time improving the page, it also means that the data should be much clearer. You can do a quick scan through your Google Analytics Speed Report or Search Console Core Web Vitals report to see which of your pages are slowest.
The page should be relatively stable. Don't update pages (anything that would impact business metrics) until the test is complete. The fewer external factors there are to consider, the cleaner the analysis will be.
Using the above as a guide it should be a bit more clear which pages are good candidates for your test. Ad landing pages are also a good choice, because you're likely to have built-in business metrics, A/B testing, and analysis at your disposal. In case you're still unsure, here are some ideas per vertical:
- Content Websites: sections, articles
- Storefronts: category pages, product pages
- Media Players: video discovery/search pages, video play page
- Services & Discovery (travel, rental cars, service registration, etc.): initial form-entry page
Step 2: Measure performance
There are two general ways to measure metrics: in the lab and in the field. We personally have found measuring metrics in the field (also known as Real User Monitoring (RUM)) to be more useful because it reflects the experience of real users. Examples of libraries and services that can help you report RUM data include Perfume, Firebase Performance Monitoring, and Google Analytics Events.
There are many metrics to choose from because they aim to capture different aspects of the user experience. Remember that your goal is to eventually determine if there's a direct correlation between your speed and business metrics, so it might be useful to track a few speed metrics in order to determine which one has the strongest correlation with your business success. In general we recommend starting with the Core Web Vitals. The web-vitals.js library can help you measure some of the Core Web Vitals in the field, although note that browser support isn't 100%. Beyond the Core Web Vitals, the Other Web Vitals are also worth checking out. You can also define custom metrics, such as "Time To First Ad Click".
Step 3: Create speed performance variants
In this stage you will implement changes to create a faster version of the page to be tested against the current version.
A couple of things to keep in mind:
- Avoid making any changes to the UI or UX. Aside from one page being faster than the other, changes must be invisible to users.
- Measuring is also a key aspect of this stage. While developing, lab testing tools such as Lighthouse should be used to indicate the effect your changes have on performance. Keep in mind that changes to one metric can often influence another. Once the pages are live, stick to RUM for a more accurate evaluation.
Creating performance variants can be done in different ways. For the purpose of the test, you would want to do so as simply as possible. Below are a few options.
Create a faster page
- Use a tool like Squoosh to manually optimize the images on your test page
- Use DevTools code coverage to manually eliminate unused JavaScript or CSS just for that one page
- Efficiently load third-party scripts
- Use a tool like Critical to break out and inline critical CSS
- Remove non-critical JavaScript code that does not impact the user experience and which you can do without for the purpose of the test (for example, certain third-party libraries)
- Implement browser-level lazy loading which isn't supported by all browsers but may still improve performance significantly where supported
- Remove non-critical analytics tags or load them asynchronously
Additional optimizations to consider can be found at Fast load times and Frontend Performance Checklist. You can also use PageSpeed Insights to run Lighthouse, which identifies opportunities to improve performance.
Slow down the page
This may be the easiest way to create variants and can be achieved by adding a simple script, slowing down server response times, loading larger images, etc. The Financial Times opted for this option when testing how performance impacted their business metrics: see A faster FT.com.
Speed up page load
For cases where the test page (let's say a product detail page) is mostly linked out from a different page (let's say the homepage), prefetching or prerendering the product page directly from the homepage for the test group will speed up the subsequent load of the page. Note that in this case the A/B test split (step 4) is done on the homepage. In addition, all of this may slow down the first page so be sure to measure that and take it under consideration when analyzing the test results (step 5).
Step 4: Create an A/B test
Once you have two versions of the same page where one is faster than the other, the next step is to split traffic to measure impact. In general there are many techniques and tools to perform A/B tests, but be aware that not all methods are well suited to measure speed performance impact.
If you're using an A/B testing tool like Optimizely or Optimize we strongly recommend setting up a server-side test instead of a client-side test, as client side A/B testing often works by hiding page content until the experiment is loaded, which would mean the A/B testing itself would skew the metrics you wanted to measure. If you can only do client-side testing, consider setting up the experiment on a different page and changing the link outs to your test page to split the traffic. This way the test page itself isn't dragged down by the client-side test.
Example of AB testing performance changes on a given product detail page (PDP) via server-side testing:
The request goes to the backend, which distributes the users to the two different versions of the page. While this is in general a good setup, it often needs IT resources to set up the server-side split.
Here is an example of a client side testing setup, using the prior page (the homepage in the diagram below) to run the testing JavaScript:
The testing JavaScript manipulates the outgoing link to give the two test groups of users links to the two versions of the PDP in question. This is easy to setup and maintain through common A/B testing tools like Optimizely or Optimize and does not influence the performance test because the testing JavaScript runs on a different page.
Alternatively, you can pick two pages which behave and perform very similarly (for example, for two very similar products). Apply your changes to one of them and then compare the difference in metrics over time. This will mean you're not conducting a proper A/B test, yet it can still be quite insightful.
In case your test page is used as a landing page for ad campaigns, it can be handy to use your ad network's built-in A/B testing tools, like Facebook Ads Split Test or Google Ads Drafts & Experiments. If that's not an option, you can use two campaigns with the same setup, and set different landing pages as targets.
Step 5: Analyze A/B test
After you've run your test for long enough and have enough data to feel confident about the results, it's time to put it all together and run an analysis. How you do that really depends on how the test has run, so let's go through the options.
If your test was run on ad landing pages using the above-mentioned tools, analysis should be as straightforward as reading a result. If you're using Google's Drafts & Experiments, take a look at the comparison using the ScoreCard.
Platforms like Optimizely or Optimize also offer easy ways to interpret the results and determine how much of an impact speed has on your pages.
If you're using Google Analytics or a similar tool, you will have to pull together the report yourself. Luckily, Google Analytics makes it pretty easy to build custom reports, so that's where you should start. If you've sent speed data to Google Analytics using a custom dimension, check out the Reporting guide to learn how to set those up and include them in your custom reports. Make sure your report covers the date of the experiment and is configured to display both variants. What should go in this report?
- Primarily, you need to include the business metrics that you most care about: conversions, page views, ads viewed, conversion rate, e-commerce metrics, click-through rate, etc.
- Additionally, other standard page metrics that also make a good case for improving site speed are bounce rate, average session duration, and exit percentage.
You may also need to filter for mobile and make sure to exclude bots and other non-user traffic. More advanced analysis would also filter by region, networks, devices, traffic source, and user profiles and types, such as new users versus repeat visitors. Each group of users may be more or less sensitive to slower speeds and identifying these is also quite helpful.
Looker Studio (formerly Data Studio) or other data visualization tools make it easy to integrate various data sources including Google Analytics. This makes it easy to conduct analysis, and also create dashboards that are shareable with the many stakeholders involved in running a modern website for further buy-in. For example, the Guardian created an automated alert system that warned the editorial team when recently published content crossed their page size or speed thresholds and was likely to result in unsatisfied users.
Step 6: Draw conclusions and decide on next steps
Once you have data that connects performance and business metrics, you can examine the results and start to draw conclusions.
If you can clearly see a correlation between improving performance and improving business metrics, summarize the results and report them across the company. Now that you can talk about speed performance in a "business language" you are more likely to grab the attention of different stakeholders and place site speed performance on everyone's radar. The next step is to set performance budgets based on the results, and plan out work to meet those budgets. Since you know the value such work will provide, you'll be able to prioritize accordingly.
If you can't identify a correlation, take a look at the caveats below and assess whether similar tests should be run elsewhere on the site (for example, through the entire purchase funnel or on a different type of page).
Caveats
There might be several reasons for not finding significant correlation between site speed metrics and business metrics:
- The page chosen does not have enough influence on the business metrics you're examining. For example, a faster product page may not have a big effect on conversion rates if the checkout page is very unfriendly or slow. Consider looking at more relevant metrics such as bounce rates, add-to-basket rates or any other metric that is more directly connected to the page you're testing.
- The difference in speed is not significant enough between the two versions. This should be evaluated according to the RUM metrics you're measuring.
- There's a fault with the A/B testing mechanism. Traffic might not be distributed properly or analytics not reported correctly. In order to rule this out, consider running an A/A test where you test the same version of a page using the same testing mechanism and ensure there's no difference in results when doing so.
- Site speed really doesn't influence your business metrics. This is rare but can occur in cases where your target market is less sensitive to speed (e.g. the site is mostly accessed from powerful devices on a strong network) or user demand is very high and choice is limited (e.g. a ticketing service that exclusively sells tickets for high-demand shows). Note that this does not mean that a faster site will not improve user experience and as a result affect brand reputation.
Conclusion
While it is tempting to launch speed optimizations across the entire site, it is usually more beneficial in the long run to first understand what it means to your users and your company to have a faster website. It is the difference between saying "we improved FCP by 1.5 seconds" and "we improved FCP by 1.5 seconds and that improved our conversion rates by 5%". This will allow you to prioritize further work, get buy-in from different stakeholders and make site speed performance a company-wide effort.