Google Optimize: A/B Test

Welcome back for next instalment of the Google Optimize series! If you hadn't read the introduction and have no idea how to apply for Google Optimize, read my previous entry.

This is one of the three tests available to run on the new testing tool from Google called 'Google Optimize Beta'. If you haven't looked at the previous entry and you're interested in Google Optimize, I'd suggest you'd have a look at that entry first before reading.

Before setting up an A/B test, I'll give a brief explanation of what an A/B test is. An A/B test is when you compare two versions of the same page with slight differences to see which one performs better.

For example, Page A has a green call-to-action button and Page B has a blue call-to-action button.

Whichever brings in a higher conversion rate is the winner! For this test I have decided that I am going to do the exact same as the example above and have a different h1 heading. 

On your experiment page, let's click on the blue button at the top right which reads 'Create Experiment' to begin.

This will open up a window from the right-hand side, where you can begin filling in the details for your first experiment.

When creating an experiment, you have to decide the:

• Name
  
• Site for testing; and
  
• What type of experiment
  

This experiment will be an A/B test on the www.zenfotomatic.com homepage, so let's fill in the details, then click 'Create'. Google Optimize will then take you to your experiment's page.

*hacker voice* I'm in.

So we are inside our first experiment's setup page. On the right we have our experiment information, at the top the status of our experiment which is currently 'Draft', below that our variants which have not yet been completed and our experiment configuration.

Before editing our page, let's decide what our experiment objectives are. In other words, let's decide where we want to track our visitor's movements and see if we can successfully create a conversion with our test. For my experiment, I want the users to navigate to the registered page (so they signed up). If a visitor reaches that page, that's a conversion.

So my primary objective will become 'Registered Page'. And just out of curiosity, I will add a secondary objective for 'Register Page'. Maybe there's something wrong with my sign up page that I need to change to make it easier for visitors to sign up.

Description and hypothesis I will leave empty since I am conducting this test on my own. However, if you have a group of people working on the project or you're going to present it to a manager, it's a good idea fill this in too.

Okay! Now we can finally begin building our A/B test environment... Let's go up to Variants and click 'New Variant'.

A window will appear from the right, so give your variant a name then click 'Add'.

You may have noticed that when you added your variant, the percentage split to 50/50. This is called the 'Variant Weight', which determines the percentage of visitors who see the 'B' to your 'A/B' test. This is entirely up to you, but so there isn't a high chance of frequent visitors noticing a change, I would change this to at least 80/20 or if you have high traffic, 90/10.

For my test, I'm going to change the variant weight to 80/20. Click on the percentage to change the variant weight, then click done. Note: Your percentages always must equal to 100%.

Let's save our changes and click on 'Variant 1'. Note: You cannot proceed without the Google Chrome Extension for Google Optimize previously stated. Please install this before continuing.

This is where you will make all your changes for your A/B Test. The layout is exactly the same for the MVT test, too.

I have numbered the main areas of the Google Optimise layout, so I will give an explanation of numbers 1 to 10.

1. This is the current variant you are working on. You are able to switch between variants, add a new variant and duplicate or rename your current variant.

2. This is the screen type you are currently editing your website on. Right now I am on desktop view, but you can choose to view various mobile devices, too.

This is where things can get frustrating, because the changes made on your desktop view will automatically make changes to your mobile view. The changes made to the desktop version are not mobile friendly, unfortunately. Make sure in your CSS code you have media types included, so you can edit each view without worry. It'd be great if Google Chrome could create an auto mobile-friendly code when you switch between layouts.

3. Here you will be notified of how many changes you have made to your layout. When you click this, a list of all your changes will appear from the right hand side.

4. Diagnostics: Here Google Optimise will notify you of various errors or missing files in your layout. As you can see, I have one error! What is it?

Google Optimise picked up that on my website I have not installed the snippet. Make sure you do this before you begin your experiment!

5. Your selected elements will appear in a line here. You can edit your elements if you click on the small square icon to the right of the '5', or, through the edit window after selecting an element (number 9 in the screen capture). I will give an example of how you can do an edit below. Let's use the h1 as the subject; Click on the square icon on the far left of the element row:

On the bottom right, click on 'add change', and for this example I am going to edit the text, so I'll go on and click 'Text'.

In the window it states which element you are editing, the text you will enter and how you want it positioned.

Under text I have entered what I want it to display. I want my text to replace the title completely, so I will select 'Replace' then 'Apply'.

Great! The title has been replaced! Of course you can do it from the editor palette on the right hand side, but I'll explain this when we reach '9' on our list. Let's go to the next number...

6. CSS: Here we can add, edit and remove CSS that has been created within the Google Optimize app. If there is code that's not displayed but you wish to edit it, you can go ahead and add the changes manually into said CSS editor.

7. Interactive mode: Say if you have edited a video, added animation to your CSS or removed a link, you can preview and interact with your changes here.

8. Move settings: You can decide whether the elements you select snap to the previously set dimensions of the corresponding div, or you can freely move them to any position you wish.

I switched 'Reorder' to off because I prefer to have the freedom of moving my elements to whatever position I like.

9. Editor palette: The editor palette will change its values according to the element selected.

Dimensions, location, typography, background, border and layout can be changed from the palette. If there are some more advanced changes you wish to make, you can do them through the advanced editor. At the top of the editor palette window, you can also undo or redo your changes or hide the editor palette. You can bring it back if click the icon that appears to the left of the CSS editor (shown below).

10. Selected element: Google Optimize highlights which element is currently selected by creating a thick blue border around the element, plus a tag with the element's name.

Now that we have gone through the main elements of the Google Optimize, go ahead and make your changes to your liking. 

Once you're done, click 'save', then 'done' on the top right hand corner.

After clicking 'done', you'll be brought back to your account page. I'm ready to start my experiment, but as you can see Google Optimize says that I'm not quite ready yet. Click 'Show Steps' to see what's left to do…

I've completed everything, all I have to do is save my changes then click 'Start Experiment' to begin. Wait a second, we didn't adjust anything for 'Targeting' did we? Let's go check out what that is...

Beside the 'Objectives' option below, we have the 'Targeting' option. Google Optimize automatically adjusts your target weighting according to the 'Variant Weight' we chose before, or you can edit again here.

1. 'Who': This determines what percentage of users have the chance of being part of the 20% to see the B side of your A/B test. If you want fast results, go for 100%. If you're unsure whether or not your changes are risky, I would lower this percentage.

2. 'When': You can adjust when your A/B test will appear. Of course, the recommended option is to have it appear on page load. Then 'Additional conditions'; Not only could you have your adjusted page appear on the site initially chosen, but you can choose other routes of how the A/B test can appear...

I won't go through every item on the list, but there's options such as only users who use iPhones (Technology), or if the user is located somewhere in particular (Geo).

Once all the changes are finally made and everything has been saved, you can finally start your experiment. Click on 'Start Experiment' at the top of your experiment page to begin.

The 'draft' bar will change colour to a nice bright blue, indicating that your experiment is running. Google suggests at least 2 weeks for your test to run, or if the probability to beat baseline reaches 95%+. 

Now I've actually had this test running for a few days before writing this entry, so let's see how the test has been going...

In your container's page, look to the right of your running experiment. You can see a bar chart icon which reads 'Reporting' on hover.

Wow! Only 5 days in and the improvement overview, the likelihood of a user navigating to the 'Registered Page' improves by 423.3% and 'Register Page' by 118.87%! That's a bit surprising to see just from changing a button colour and heading.

Scrolling down, you can view the report on each experiment objective. Although the change has dramatically improved likelihood of conversion, the probability to beat baseline still has not reached 95% for both objectives. With the original design, the register page received one conversion but the user stopped there. So we now know that there is room for improvement on the register page as well.

10th Day of testing...

We have hit over the 95% baseline for our secondary goal! The registered page was going to be a tough cookie to crack, but this information is enough for us to know that changing the home page's heading and button colour is the way to go (along with other changes too).

So let's go ahead and navigate to the 'Details' page and end the experiment to get the full report.

Click on 'end', where you will be brought back to your experiment page.

If we click on the 'Reporting' tab, we are able to see a run down of the experiment over the past 10 days.

If you look to the right hand side of your window, you will see your experiment information. Now that the experiment is over, you can see the start and end date. As well as this, Google Optimize can take you to the Google Analytics report for this experiment as well.

Anyway! We're done! That's it for the A/B Test. Are you going to give it a go? It does seem a little complicated and overwhelming at first, but once you get the hang of the Google Optimize layout it becomes very easy!

If you have any questions, feel free to email me at tari-inacio@glamscorp.jp!

Thanks!