Meta · Published 2026-07-28 · 2 min read
A/B tests for creatives in Meta
On the “META: launch and scaling” stream there were several questions about A/B testing creatives. I do not compare creatives with A/B tests in Meta. Here is why.
🧪 What an A/B test is. Users are split at random into a control and a test group, and one element differs between them. Then you test the null hypothesis “there is no difference”: you measure how far the variants diverge and how likely such a divergence is by chance if the null hypothesis holds.
That is a more or less correct definition; simplified, the test answers the question “how different are the groups” — in this case, the creatives.
📱 Where you need it. A new onboarding, prices, paywall, a new feature in the app. Inside an app there is no algorithm that decides on its own which version performs better and serves it to the right users. A human makes the call, which means you need correctly computed numbers and statistical methods of varying complexity, sensitivity and precision. There is no other way to judge a feature's performance properly.
🤖 How Meta is different. Meta has exactly that algorithm: matching a creative to a user is the core function of an ad network. Even at the impression stage Meta sees engagement — who stopped scrolling, who watched to the end, who clicked.
💡 My take
There are two tasks
- measure the quantitative difference between variants, and
- pick the creative the budget will go to
An A/B test solves the first. My problem is the second.
Meta makes that choice on data I do not have, and faster than I could gather a sample. I see no reason to distrust it on this particular task.
So I load creatives into the ad set and simply watch which one META picks.
🙌 Leave a plus in the comments and I will send the recording of the stream — plenty of other interesting things were discussed there.