There was a very clear moment, as I was being briefed on writing this blog, when I realised what I was in for.
It was when I heard the words ‘frequentist’ and ‘Bayesian’. They resounded like claps of doom. Horsemen of the miserable-blog-writing apocalypse.
“Are you familiar with them?” asked Joe, my Head of Content. “No,” I scoffed. “Because I have something called a life.”
Thankfully, at Don’t be Shy, we have an Alex Coe. His life revolves around this stuff, getting into the weeds of techniques, methods and processes that benefit campaign performance. Which is great for our clients. He and the Performance team can advise on exactly how every asset in a campaign is running – and, the bit any client really cares about, how they could run better.
But when’s the best time to ask? How long should you wait to see results and change your approach? Is it ever too soon to optimise a campaign? A/B testing is more than running a couple of variants against each other and then just opting for the best-performing. Here’s what goes into different optimisation strategies – and you can guess what they’re called…
The frequentist method, or ‘See it through to the end’
The first requires you to let the activity run its course. You don’t go in with any preconceived, defined assumption that a particular variation will have an impact; you’re purely testing to see what, for example, changing a subject line does. Shorten it for a month, and at the end of that testing period, you’ll have an answer as to its success. No shifts midway through. No sudden ‘scrap that, let’s try something else’.
What that means is you get a full view of data from which to draw conclusive findings. It’s the most common approach for CRO agencies.
The Bayesian method, or ‘Tweak and test as you go’
The alternative allows for intermittent adjustments. You go into your test with a hypothesis, such as: shortening a subject line will increase open rates. You can then look at the data early into your test, make a change and repeat the process. If the results are positive, you can roll it out more; if negative, you might lengthen the subject line, and keep going back and forth until you find the optimal.
With these less strict parameters, however, you can change tack without giving enough time to decisively prove an optimisation hasn’t worked.
Campaign optimisation at Don’t be Shy
With both methods being imperfect, we’ve cherry-picked the best of each for our sweet-spot optimisation approach.
The Bayesian model allows us to test quicker and be more flexible to clients’ needs. While the frequentist model might work for some, others want to see results much sooner. In those cases, we can inform the client of early signs from the data and how we’ll react to big swings – that, ideally, we’d want to run the test for a month, but here’s what the data is showing and how we’re going to optimise the campaign. This also makes us more responsive and reactive for clients who don’t have a huge amount of data available – niche B2B businesses whose web traffic is small compared to big B2C companies.
One of our recent paid LinkedIn campaigns puts all this into context. We’d set out our methodology to attract quality leads over quantity, but after two weeks of steady conversions from suitable businesses, our client wanted to pivot to quantity instead. So we went back to our previous experience of quick lead gen, setting them as lead-gen ads instead of pointing to a landing page; changing the targeting so it was greater volume; and reducing the number of ads to increase the odds of users seeing them and converting.
While we can have these kinds of conversations with clients where we can make compromises based on data and learn from them, holding out for robust data and a long-term approach will give you better results. That way, there’s sufficient data to back decisions, instead of making them too soon and risking detrimental changes. Bayesian principles enable us to operate at speed, but even with this agile approach, you need a bit of time. Hold your nerve, trust the process and the Alex Coesian – or Don’t be Shysian – approach will come through.
Campaign performance your greatest concern right now? Something else keeping you awake at night? Don't be Shy asked marketers across the country, 'What's the matter?' Here's what they had to say.
