CRO testing helps us identify the best outcome of a website change based on our main goals. It also serves as an analysis of user behavior that can give us valuable insights. But how can we ensure we are following the CRO process correctly and setting up our tests for success? We are going to dive right into the essentials that you need to know prior to making your test live. 

First, we need to determine if you actually need CRO testing. Ask yourself the following questions,

1. Is there a change that I want to implement that can affect user behavior?
2. Am I redesigning my website? 
3. Are my conversion rates low that I want to try different approaches to increase them? 
4. Do I want to increase revenue?

If you answered yes to any of these questions, then you need CRO testing. 

Identify areas of improvement within your analytics

Before you decide if you need to test, your main goal needs to be defined. You can do so by diving into your data to see if there is an area of opportunity to improve your website. From there, you can define your main goal. 

Data analysis can come in many different forms. Examples include heatmaps, customer surveys, session recordings, funnel analysis, Google Analytics, user testing and more. 

A look into your analytics can help you discover what you should focus your efforts on. 

Identifying areas of improvement by assuming or guessing may do more damage to your business than good. It’s important to know what needs change and only data can give you the answer. 

Learn to plan your experience 

Properly identify your main goal

As mentioned above, having a main goal is one of the first things you should have outlined prior to deciding to test. Main goals for CRO testing can be the following but are not limited to:

• Form Submissions
• Purchases
• Average order value
• CTA clicks
• Visits to certain pages like a product, cart of checkout
• Reduction of bounce or exit rates 
• Interactions with elements that exist on the page
• Increase or decrease a behavioral metric 

Let’s dive into an example of a proper CRO test vs one that isn’t. Let’s say you are planning to optimize a hotel website and you want to test out an interactive map on the homepage that shows all the resorts from your chain. You say that your main goal will be map interactions. But if we think about it properly, interactions will only be measurable on the variation. Why? Because you cannot measure map interactions on your control where a map will not exist.  You must have a main goal that is measurable in both the control and the variant.

If you do decide to move forward with a main goal that is only measurable on the variation, your data will look like the following. 

Variant	Sessions	Map Interactions
Control	1005	56
Variant	1008	0

It will contain no valuable insight for you or your hotel website. Now, let’s revisit this example with a proper main goal. 

Let’s say you want to CRO test an interactive map and you want to see if this feature encourages users to start a booking. You select your main goal to be visits to the booking page. This can be measured in both your control and the variation because a booking page exists for both variants. 

Always select a main goal that is measurable in both variants. This will ensure that you are getting the best out of your CRO testing. 

Now that you have identified your main goal properly, let’s move forward with designing your hypothesis. 

Design your Hypothesis

Your hypothesis serves to define what you hope to accomplish in your test. It states what is being tested and what you think the outcome will be. This 1-sentence statement typically follows an If and Then approach. 

If _(we try x approach) _ then___we will increase or decrease main goal

Let’s try one for the hotel example:

If we add an interactive map on the homepage, we will increase visits into the booking page. 

Simple as that. Looking back at your hypothesis as you plan can help you focus on your objective. As your test runs, it will either prove your hypothesis right or wrong. 

It’s important to note that having a hypothesis in your CRO test will not guarantee your test will win. It does promise a great learning experience and an opportunity to iterate if necessary. 

Design your Strategy

What is exactly that you want to do on your CRO test? Following our interactive map example, what we want to do is create an interactive map below the homepage hero that will encourage a user to visit the booking page.

The strategy portion of your planning involves sharing as much detail as possible about what you want to test. 

Rank your testing ideas with the ICE Model

If there is a list of ideas that you want to test, and don’t know which one to start with, then you must prioritize them. The ICE Scoring Model is a relatively quick way to assign a numerical value to different potential ideas to prioritize them based on their relative value, using three parameters: Impact, Confidence, and Ease.  Let’s dive into each of them. 

Impact. Impact helps you define  how impactful the idea is going to be based on the main goal you have set up. Ask yourself: how impactful will this experience be on the main metric?

From a scale of 1-5 rate the idea: 

5 –  Really impactful   

1 – Not so impactful 

Confidence. Confidence is certainty that the experiment will perform well based on the data analysis or industry data that was gathered. 

Ask yourself: how confident are you that this experience will perform well based on industry data or partner data? 

From a scale of 1-5 rate the idea: 

5 –  Really confident   

1 – Not confident at all 

Ease. Ease is defined by the amount of effort that it will take to build out this CRO test. 

Ask yourself: how much time and effort will go into building this experience? Is it complicated? Will it need more resources than usual? 

From a scale of 1-5 rate the idea: 

5 –  Little to no effort  

1 – Lots of effort

Test Name	Impact	Confidence	Ease	Total
Interactive Map	4	4	3	11
Homepage Multivariate	4	3	2	9

Make sure you get an extra set of eyes when ranking your CRO testing ideas. Now, that you have planned your experiences, designed your hypothesis, created your strategy and ranked your ideas, you are ready to set this experiment up. Remember, the more detail your planning has, the more refined your testing will be. 

Know your way around your optimization tool

Setup

If you have chosen an optimization tool to run your experience, the first thing you should do is install it correctly. It’s important to have the appropriate team to do so if you can’t. Installing your CRO tool incorrectly can lead to improper or no data collection. It can also cause your changes not to show up on the site.  We want to receive as much information as possible about our tests and it starts with the proper installation. Always follow the directions on installation from the testing tool that you are using. 

Types of CRO testing

By the time you get to the set up of your CRO test, you must know what type of test you’d like to perform. There are several types of CRO test types that you can set up depending on the tool that you are using. We are going to cover three common ones. 

A/B testing

A/B testing involves testing multiple variations of 1 element on your webpage at a time. It can be as simple as a button color change, image change, copy change, new element addition and so on! The variations help you gather valuable insight on the user’s reaction and can show which one gains the most interactions or conversions. 

Using the resort example, an A/B test would be creating the interactive hotel map and comparing the visits to the booking page against the control. 

Multivariate testing

Multivariate testing involves testing several different elements on your webpage that work towards the same goal. Think of it as performing multiple A/B tests on a webpage at a time. 

Going back to the resort example, let’s say that apart from adding an interactive map on your homepage, you also want to test out the hero image. This is still part of the same objective of pushing users into the booking page so this would work as a multivariate test. What your optimization tool will do when you run this test is that it will create different combinations of the elements 

The beauty of multivariate testing is that the tool you run your test on will tell you exactly which element contributed positively or negatively to your testing goal. 

Multivariate testing is recommended on websites that have high traffic as multiple versions are created that split traffic up. 

Split testing or redirect test 

Split testing involves testing multiple variations of your test but on different URLs. This is typically done when you are redesigning an entire web page or having major changes. 

Your website traffic is divided into the control URL and the variation URL. Depending on you main goal, your optimization tool will tell you which URL lead to higher conversions 

Peeking back into the resort example, let’s say you want to change the entire homepage and compare visits to the booking page. Big changes like these should lean towards split testing. 

(example image)

Define your target audience/segment 

When setting up your CRO test, it is important to know who your target audience will be. Do you want to target all users that enter the website? Do you want to target specific users coming from an ad, users from a specific geographic location, users who use a specific device or engage in certain behavior etc.? Depending on your optimization tool, your audience targeting can include the following but are not limited to:

• Data analytics service audience 
  • Google analytics audience
  • Google Ads account
• Device Category
  • Mobile 
  • Desktop 
  • Tablet
• Geography
  • Country
  • Region
  • City
  • Metro 
• Behavior
  • New or returning user
  • Users who try to exit the webpage
  • Users who are on the webpage for x seconds/minutes
  • Users coming from specific referrers 
• Technology
  • Browser
  • Mobile device version 
  • Operating system 
• UTM parameters
  • Users from specific UTM parameters 

And so much more!

Traffic split has been configured 

It’s important to know how you want to split your traffic on your CRO test. Depending on the tool, that is done for you based on how many variants you add. It’s always recommended to split your traffic evenly between your variants. This assures that users have an equal chance of seeing them. 

You can edit the traffic for each variant based on your needs but keep in mind that if you do, more or less traffic will go to one of your variants. 

Add your Metrics

This routes back to what your main goal is. Your main goal must be measurable. That way you can obtain valuable insights from your test.

Some optimization tools already have built in metrics you can choose from but some require creating custom metrics. A developer or someone who knows their way around the tool should be the one to set these up. 

It’s important that the main metric is set up correctly. The last thing you want to happen is to have a live test not capturing data. 

Device Compatibility

If your test is targeted to all devices, make sure that it is compatible with the user and that they won’t have trouble viewing your experience. 

Your optimization tool should have a preview mode where you can view the test in different sizes that would fall under desktop, mobile and tablet.  

If your test looks off in one of your device previews, then it must be addressed as soon as possible as that is exactly how it will look to the user once it is live. 

QA your Test Before It Is Live

QAing can decide the success or failure of your test. Right before you set your test live, it is important to preview it and check if there are functional or visual errors. Like mentioned in the device compatibility section, your users will see it and it can decrease the chances that they will convert. 

Your optimization tool should have a preview URL that lets you see how the experience will look once it is live.  

Sharing this link with multiple team members will give you an extra set of eyes that can point out possible errors. 

Once your test passes QA, you are ready to set it live! 

Capabilities

As you or the developer is setting up the test, it’s a good time to reflect on the importance of knowing your way around the optimization tool. 

Tool training is essential for your CRO team. Even if your colleagues do not plan on setting up the tests, they should at least know their way around the testing tool. This can help avoid simple mistakes that one can make when setting up a test. 

There are so many benefits of knowing how to use your tool. It can help you expand on your testing and take your business to the next level. Having that knowledge can help you run successful tests. 

If you do end up making a mistake on your setup, use it as an opportunity to learn and expand your tool knowledge. 

Reporting: Monitor the leading variant

Hooray, your test is now live! Now comes the fun part: monitoring the leading variant. At this stage you should be receiving valuable insight from the tool. But what do these numbers mean?

Depending on the optimization tool, you should see the control vs the variant on the left side and metrics on the right focused on your main goal. Pay attention to the statistical significance being measured on the test. This will tell you the likelihood of a result being caused by something other than chance. The lift is there to tell you the difference in the control and variant. Usually when the variant is leading your KPI will contain a positive lift. When the control is leading, you will see a negative lift. 

	Sessions	Time on site	Bounce Rate	Visits to Booking Page	Conversion Rate
Control	150	2m 30s	65%	45	30%
Variant	150	3min 10s	43%	78	52%
Lift					+73%

It’s important to understand what the test is telling you. Doing so will help you decide the next steps for the running test. 

It’s important to set up your CRO test for success. Doing so will help you elevate your testing and reduce any chances for error.