This post is designed to show advertisers how to leverage multivariate ad testing to discover an optimal ad schedule. Within AdWords, there is no easy way to run a test that treats “schedule” and “keyword” as measurable variables for a scaled experiment, right?
That is not completely true. Advertisers need to do some manual set up but the results can be drastic.
What Will You Possess if You Decide to Try This Ad Testing Approach?
- An optimal ad for every timeslot, for every day of the week, for each keyword
- Data that will help you understand your how and why your customers buy your products at various times of day
- A step toward deeply understanding your customers’ response to your messaging and ad mix
- A leg up for your next ad test
It is important to note, this experiment type involves three conversion layers: scheduling (targeting), keywords (targeting) and ad copy (message). Let's get started.
Who Would Use This?
These tactics are especially useful for agencies or enterprise organizations that have an approval process for ad copy distribution. If you’re an advertiser and you’re looking for ways to optimize your already approved ads, optimizing the schedule and keywords for each ad is a must. In most organizations, you won’t need to cut through red tape in order to “tweak” targeting variables, which can help save you time and hassle.
What’s The Situation?
Let’s pretend for a second that you are an advertiser, either internal or at an agency. You have received approval for your ad copy and you are looking to squeeze the absolute most revenue out of this approved copy.
Let’s say you have four approved ads, you have two keywords that drive most of your conversions, and you want to make sure you are running an optimal ad variation for each keyword at all times.
Here are the steps:
- Divide each day of the week into two timeslots that make sense for your business (example: mornings vs. evening). Doing this will result in 14 AdWords campaigns for this experiment (each timeslot for each day will have its own AdWords campaign). Yeah, it’s obnoxious but the optimization granularity you will achieve will be worth it. We promise.
- Create two single keyword ad groups (SKAGs) for each timeslot. The purpose of this is to isolate which combination of timeslot and ad copy work best for each keyword. Remember, we only have two keywords and four copy variations for this experiment.
- Insert all four of your approved ads into each Single Keyword Ad Group. Be careful not to duplicate or leave any variations out of the ad groups.
- Deploy the ads and expose each ad variation to a statistically significant number of people.
- Determine a winning ad for each timeslot based on the data.
- Spend on these winners, increase your ROI.
Things to Remember:
- Make sure your ad copy variations are drastically different
- Make sure you are spending enough on the experiment to gain impressions
- Keep your existing ad campaigns running, experimental ads will take some impressions away but not many.
Your experiment set up for each day of the week will look this…
After each experiment collects a statistically significant amount of data — most advertisers will tell you 90 to 95% confidence levels are appropriate — shut down spending on suboptimal ads and reallocate ad dollars to the winning ads. It is probably best to re-inject these winning ads back into your previous account structure in order to benefit from your existing Quality Scores.
It is also important to delete the suboptimal ads from your AdWords account. These low performers can pull down your average CTR and as a result have a negative impact on the Quality Score for a specific keyword.
You may potentially see a temporary decrease in CTR during the experiment. Experiments will naturally yield sub-optimal ads that may temporarily pull down your overall account performance (not by much). This decrease will be offset by finding better ads and over time your average ROI per timeslot will rise.
After your experiment finishes collecting data, it will yield:
28 winning ads for the 28 potential timeslots.
To achieve this, multivariate regression analysis is required.
Analyze What Happened and Understand Why
Performing a multivariate analysis within AdWords can prove to be a difficult task especially when you are comparing numbers across ad campaigns. We would recommend exporting your data to Microsoft excel or check out a controlled testing and analysis environment.
After your data is ready to view, what should you look for? Here are some questions to ask and example calculations that will help explain your results.
- Calculate the average CTR and conversion rate for your “winning” ads for each time slot. Compare this average to your current averages. Did it beat it? If so, you are the best advertiser alive.
- What titles were you using for the winning ad variations?
- What is the variable distribution for amongst the winning ads?
- Was there any ad copy that worked well for a particular timeslot?
- Does a certain keyword perform best in a certain timeslot?
- Note the differences in search volumes, clicks and conversions for every day and every timeslot. Are there times you should eliminate?
- Is there an ad variation that won for multiple time slots?
These are the sorts of quantitative questions that can help feed your qualitative answers.
- Use the answers to the above questions to create your next round of ad tests.
- Once your next round of ad messages has been approved, create a new experiment and beat the current winners.