272: Why your app store A/B tests are misleading you with Scott Masters, Director of Strategy and Partnerships at Gummicube

Today we're talking about mobile A/B testing - and about the moment the app store hands you a winner.

Because that result is directional, not the truth. And if most of your traffic is paid, you may have measured the wrong users entirely.

In this episode, we speak with Scott Masters, Director of Strategy and Partnerships at Gummicube, about why mobile A/B testing is never set-and-forget.

Scott explains why most developers and marketers under-invest in testing, what a sensible cadence looks like - once or twice a month, with each test running at least two weeks to reach statistical significance - and why changing one element at a time is the only way to know what actually moved the needle.

If you work in app store optimization, user acquisition, or app growth, this is a practical guide to testing your product page properly.

Today’s topics include:

• Why testing is never one-and-done - trends, user preferences, and app features all shift, and a static product page goes stale
• Cadence and statistical significance - test once or twice a month, run each test at least two weeks, with traffic volume determining how long
• One variable at a time - change the screenshot copy and the imagery together and you won't know which one worked
• What to test beyond screenshots - title, subtitle, short and full descriptions, icons, and feature graphics all feed the strategy
• Measuring what's real - platform confidence intervals are directional, so validate with pre- vs. post-deployment conversion rates across every channel

Links and Resources:

• Scott Masters on LinkedIn
• Gummicube website
• Business Of Apps - connecting the app industry

Quotes from Scott Masters

“If you're not A/B testing at least once or twice a month, you're leaving users out there for other competitors to take from you.”

"It's not a one-and-done. You can't set it and forget it. It's just like with any other brand - you create a package for your product and you put it on the shelf. If it doesn't change over time, if you're not attracting those users with new materials or new approaches, it's going to get stale."

“The platforms will give you an estimated conversion impact and confidence interval for each test you run. But that data is directional. True validation of performance really comes from launching a test winner into the store and then measuring the pre-deployment conversion rate versus the post-deployment conversion rate.”

Host

Business Of Apps - connecting the app industry since 2012