Volume 92: Death by a thousand A/B tests.

1. Death by a thousand A/B tests.

tl;dr: Franken-Ad optimization is coming for creative. Oh goody.

Whenever a technology solution comes along, it typically brings a hugely hyped promise of future improvement - what Gartner refers to as “the peak of inflated expectations.” However, once the smoke clears, and the technology matures to the point where results can be analyzed, this promise generally turns out to have been vastly overhyped. For example, CRM didn’t transform customer relationships, the cloud isn’t always better, and machine learning-based HR services haven’t radically improved hiring (possibly quite the opposite).

Now, this same reality is catching up to the AdTech Industrial Complex, where the hype that surveillance-based tracking and targeting would radically transform advertising performance is proving to be more aspirational than promised.

As a result, rather than admit they’ve been overhyping this stuff for years, they’ve decided to shift gear and say tracking and targeting alone isn’t enough; what we need now is quality advertising creative too. Excuse me for a second while I cough out my coffee because anyone with a brain has been telling them for years that no matter how accurately targeted, terrible ads don’t work.

But, better late than never on the quality creative front, right? Not so fast, because the means of delivering “quality creative” appears to be through the testing of hundreds of ad-variants, where every single aspect - from headline to font choice to color to imagery, etc. - is independently tested until you get to the perfectly optimized ad. Sigh.

There are multiple things wrong with this. Let’s walk through a few of them.

First, this “separate the elements and optimize” mentality is the exact same mentality that drove the direct mail business into oblivion. Long before the Internet put the final nails into its coffin, response rates had plunged somewhere south of 0.5%. Why? Because it didn’t matter that you could perfectly optimize the shades of blue or orange, or the type size, or the font, or any of that stuff because the optimization mentality turned the whole category into something so predictably awful that nobody bothered opening the envelope it came in, let alone read it and respond.

Second, this perpetuates one of the biggest myths in A/B testing, which is the idea that tiny optimizations lead to big results. They don’t. Tiny optimizations generally result in tiny results. Does anyone remember Google famously A/B testing every shade of blue they could think of just because they could? It delivered almost nothing and caused their design leader to quit. (It’s not that A/B testing is inherently bad because it can be a powerful tool. It’s how it gets used and abused that’s the problem here. The real potential is when you use it to test big differences that might not otherwise see the light of day, but that’s a different story for another day)

This neatly brings me to my third point, which is that A/B testing hundreds of pieces of dogshit won’t alchemically turn that shit into gold. It’ll still be shit even after you’ve finished optimizing it. This really only works if the creative is of a high quality to begin with, which is a problem when advertising creative is getting worse rather than better.

It’s been widely observed that the quality of creative work in advertising has been declining for years, partly as a byproduct of the AdTech Industrial Complex. So the idea that these same AdTech purveyors will fix this systemic issue by “optimizing” hundreds of atomized ad variants so the piece parts can be reconstructed into a Franken-Ad? Well, that looks like some pretty fantastical thinking to me.

What strikes me about this whole thing is that it isn’t really about making better ads at all. It’s about selling a technology solution to a problem it’s singularly ill-suited to solve.

Sadly, the opportunity costs are deadly when you pursue an optimize everything just-because-you-can approach. Because all the wasted energy spent dancing on the head of a pin could’ve been better spent moving the needle instead.

2. The B2B product delusion is real.

tl;dr: And not just in B2B, if I’m honest.

Of all the marketing think-tank-like objects out there, the LinkedIn B2B Institute is one of the best. Over the past year and a half or so, they've consistently published interesting, well-evidenced, and thought-provoking reports that tend to fit closely with my own real-world experience. And while it's clear they subscribe to the Ehrenberg Bass view of the world, they don't seem to push the more off-piste theories promoted by that merry band of Australian marketing scientists (The claim that differentiation is irrelevant being more a product of poor research methodology and a desperate desire for a soundbite than an actual insight in my not at all humble opinion).

Anyway, in a new article, they point out that B2B companies tend to suffer from what they refer to as a "product delusion," which nets out as a false idea that you win or lose based on product performance rather than brand strength. To paraphrase the article, product performance is irrelevant if nobody knows who you are.

I can't tell you how right this is. While it's easy to picture smaller B2B brands being unknown, I've seen this happen firsthand at even very large companies. For example, when working with GE, they’d invested heavily in building a healthcare business that we found prospective customers didn't even know existed; hospital administrators being more likely to say GE made lightbulbs and kitchen appliances than MRI scanners.

Secondly, the observation that B2B advertising singularly fails the creative test because it emphasizes rational product speeds and feeds is largely accurate. B2B companies are generally driven by sales cultures that think product performance is the key to sales success. After doing many customer interviews across multiple categories, I can confidently state that this is not true. Clients rarely buy based on product performance because they understand that product leadership changes hands more quickly than they intend to change partners. They want a shortlist of products that can meet their needs, but after this is established, the purchase decision is almost always made on the basis of other factors.

Finally, because sales cultures tend to treat marketing as nothing more than bottom of the funnel demand gen, a common complaint is wasting cycles explaining to prospective customers who they are before they even get a chance to talk about the product at all. Meanwhile, a better-known competitor with a stronger brand is sucking all the oxygen out of the room, even though they have what the salespeople view as an inferior product. If this sounds familiar, it's because it's really, really common. (I often joke that salespeople love having a strong brand because it makes it easier to sell, but they're the least likely people in the world to invest in building one).

Anyway, it's well worth reading, although I do have one nit I'd like to pick. While 77% of B2B ads ranking at a 1 out of 5 for creative effectiveness is truly abysmal, it's perhaps even more abysmal that 53% of consumer ads score equally poorly. I mean, most of those B2B companies weren't even trying to build a brand. What's the consumer marketer's excuse?

3. Peloton. A modern-day Icarus.

tl;dr: Probably (maybe) not as bad as it looks.

Peloton isn’t in a particularly good place. During the height of lockdown panic, spiking demand combined with supply chain woes meant they couldn’t meet demand, which meant wait times extending to many months, and notoriously terrible buyer communication. Now, they face the opposite problem, sagging demand and ramped up production, leading to warehouses full of product twiddling its thumbs. In the meantime, Peloton stock, which had soared like Icarus, has crashed to earth and is now ignominiously trading below its IPO price.

The rumor mill has gone wild: McKinsey is slashing and burning, the problem is the unattainable brand image, the product is priced way too high, Big died on a Peloton. You name it, the talking heads are saying it. What on earth is going on?

Well, first, there’s no doubt Peloton is rather more poorly managed than they might like us to believe. Supply chain woes hit them hard, they overproduced in response, and the subsequent price cuts smack more of desperation than an actual strategy (nothing kills a premium brand faster than slashing prices because you quickly trash the price premium you worked so hard to establish) But, the piece-de-resistance of leadership failures has to have been their utterly reprehensible victim-blaming response to toddlers being killed by their products.

Now, while this poor leadership isn’t going to change anytime soon (a dual-class ownership structure means the founder & CEO controls the board and the company), I’m not sure it’s their biggest issue. No, the biggest issue was investors valuing them way too high, way too fast, which created a cascading set of incredibly messed-up incentives.

You see, while Peloton makes little sense as a huge, loss-making, multi-billion $$ company, it makes a lot of sense as a smaller company focused on growing a high-end home fitness niche. But when the market values you at that kind of scale, the expectation is you’re going to keep throwing down the growth numbers to justify it. Which…creates all sorts of weird incentives, like wildly overestimating demand, producing too many bikes, and throwing a last-minute Hail Mary pass of discounting to try and meet targets you probably never could’ve met in the first place.

So, what next? Well, while an activist investor is pushing for a sale and for the CEO to step down, neither of these things is likely to happen. It seems to have enough cash in the bank to weather the storm (at least for a bit), the CEO controls the board, and who wants to sell at the bottom?

Here’s what’s more likely. Peloton weathers the next few quarters and drops out of the media spotlight, then sees its stock trend back up slowly as it matures into the at-home fitness leader it always promised to be. Maybe, if a deal comes along that’s good enough, they sell. But at the end of the day, no matter what’s said right now, it has a good product, quality programming, a well-known brand, a subscription business model with low churn, and plenty of people who seem to love it. This means it’s probably not a bad business as long as we don’t judge it against a crazy valuation that never made any sense in the first place.

Longer-term though, as the market matures? Yeah, then these continued leadership failures will likely cause some bigger issues.