Volume 129: Silicon Sampling
1. Silicon Sampling: Pushing The Boundaries of Research.
tl;dr: New ways to fake your own insights.
Long ago, as we discussed a particular project we were working on, a colleague who’d taken me under his wing proclaimed:
“Never believe the market research you didn’t fake yourself.”
I remember it because it was both funny and, to a large extent, true, not in the sense that there’s a diabolical conspiracy of people wandering around faking research willy-nilly, but because with research, like so much else in business, there’s often a hidden agenda at play. This means that if you’re working with research you didn’t commission yourself, there’s a good chance there’ll be some hidden bias deliberately tilted toward a particular perspective or point of view.
This underlying bias is particularly apparent in research used by consultancies and agencies to support the sale of their own products. Porter Novelli, for example, presents itself as being “built on purpose,” which is why it’s no surprise to see they have research promoting the brand purpose thing, even though we now know that the most commonly shared data points on the subject were largely bullshit. (Or, more accurately, the convenient fruits of a suspect research methodology, but that’s a story for another day).
So, I was most intrigued to learn this week about a new research approach that relies on “silicon sampling” rather than actual people. Essentially, this means asking your questions of a generative AI rather than a sample of the consumer population.
As far as I can tell, you start with an audience definition you want the AI to inhabit and then ask it questions. Today, this primarily focuses on product research, but it’s not hard to see it applied more widely toward market research.
I must admit that I laughed out loud when I first read about this. It seemed so bizarrely science fiction. But then I thought to myself, “not so fast; there’s probably a huge market for this.” Just think of the benefits compared to research of the more traditional variety:
Superfast results. You don’t have to design, field, and analyze your research results. You simply ask the AI and get instant feedback. Should you want a more robust research report, I’m sure it could spit that out too.
Iterative questioning. You don’t have to wait to ask new questions. With an AI on tap, the whole process can be more iterative and panel-like. However, unlike with a panel, you can easily go back and ask new questions whenever you like in the future, which has interesting implications for longitudinal studies.
Cheap. Questioning an AI rather than a sample of human beings will likely rip the bottom out of the research price floor. And if there’s anything we know in business, it’s that most executives will happily accept lower fidelity outcomes if it costs less to get there.
Easy. It eliminates your need for a professional research agency or dedicated research employees. Instead, a research-focused AI can augment pretty much anyone across the organization. As for effectiveness, see point 3 above.
This, of course, brings me to the 800lb gorilla in the room and back to my original point. How accurate is this method of research likely to be? Especially when we consider how good generative AI has proven to be at lying. Initially, I figured this would be the dealbreaker. How can you trust the insight from an AI pretending to be, for example, a suburban mom?
And then I reminded myself just how bad most market research is. How few people willingly volunteer to be a research subject (it’s a dirty little secret that there are professional research subjects out there who’ll pretend to be whoever you want them to be in return for a $50 gift card, and they’ve become highly adept at giving researchers exactly the answers they think they want to hear so they’re invited back again next time).
And then I reminded myself that much research isn’t actually conducted with anything more important in mind than covering your own ass. This makes the AI approach even more appealing, as it’ll become even easier and cheaper to generate the answers you want to see.
And finally, I remembered just how little corporations interrogate the data they rely on, especially quantitative. So it almost doesn’t matter how flawed the research methodology is once the data points are spat out. (70% of consumers wishing to do business with brands whose values mirror their own, for example).
So, yeah. I think synthetic research will explode, and quickly.
At large corporations, a tonne of research is already being conducted at high cost. Yet, it’s often of marginal value, and the research groups tend to operate as independent republics rather than truly serving the needs of the business. Here, synthetic research could well be used to drop the cost precipitously, shift the research process closer to the decision maker, and, perhaps most interestingly, embed the “consumer perspective” into ongoing decision-making.
At very small companies, on the other hand, there’s often little or no research being done at all. Startups, in particular, are notorious for solely utilizing behavioral data. Here, I could see material value in using synthetic techniques to increase customer-centricity cheaply.
But, of course, this will do nothing to solve our core problem with all research today. And that is, everyone already has access to the same research and insights as everyone else. This means it’s less important what the research says and more important how you choose to interpret it and which lines you choose to read between.
2. CYA Bowl ZZZZZZZZZ.
tl;dr: Celeb-fest, crypto desert. Boring as watching turnips grow.
Like some of you, the Super Bowl is the one game of (American) Football I watch every year. It’s not that I don’t like the sport as such. On the contrary, this year’s game was quite exciting. It’s just that not having grown up with it, I’m not emotionally vested in any of the teams, and I find the constant interruptions for commercials to take me entirely out of the moment.
However, one of the things the NFL has done so well is to turn these interruptions into a feature rather than a bug when it comes to the Super Bowl, where the ads are almost as discussed as the game itself. And in some years, when the game may have been terminally dull, the ads become the narrative.
As a result, there’s an entire universe of Super Bowl ad commentators seeking to rank them, rate them, and discuss them ad nauseam. If that’s what you’re into, take a look here.
I don’t find Super Bowl ads to be the window into American society that some do, but rather a reflection of the emotional state of marketers and their agencies at the moment the ads were made.
So, if this year is anything to go by, while plenty of money was thrown around, marketers and their agencies were notably risk-averse and inward-looking in their motions. As a result, this wasn’t so much the Super Bowl of advertising as it was the CYA Bowl.
The most notable thing this year wasn’t that brands were trying to stand out and do something different but that they were doing the exact same thing as each other in a desperate attempt to minimize risk. (In the process, running straight into the trap that often the biggest risk is to take no risk at all).
As a result, I’d go so far as to say that this year’s ads were so awful that it’s hard to say who is at greater fault. Did marketers crush creativity under a wall of corporate bureaucracy, or did every ad agency happen to have the same basic idea at the same time that all their clients decided to run with simultaneously?
Because this year was undoubtedly formulaic:
Hire a celebrity or several.
Make an ad referencing 1990s pop culture, sometimes relatively obscure ‘90s pop culture.
Rinse/Repeat.
I suppose it’s good that the Ponzi scheme masquerading as an investment opportunity so active at last year’s Super Bowl was conspicuous by its absence. Last year’s rather desperate attempt to attract millions of new marks having been chastened by the spectacular implosion of FTX, among other things.
I’m not sure there’s ever much that you can read between the lines on vis-a-vis Super Bowl ads. But if this year was saying anything, it wasn’t just saying that 90s celebrity nostalgia is in; it was saying that marketers are extremely risk-averse right now.
For me, I truly don’t get the whole celebrity advertising thing. For years the data has pointed to people remembering the celebrity rather than the brand they were promoting. Made especially apparent this year when GOAT Serena Williams promoted cognac and light beer in the same ad break.
To prove my point, I dare you to remember which cognac and which light beer. And Google searching doesn’t count.
3. Measurement Shmeasurement.
tl;dr: Terminally dull, yet really important.
If you work in or around the branding field, it’s easy to deceive ourselves about how clients make decisions, why brands manifest the way they do, and why what we do matters or doesn’t, as the case may be.
We like to think it’s based on our creativity, superior insight, culture, ideas, strategy, design, experience, and…all sorts of things, really.
But, and it’s a big but, this neatly deflects from the reality that oh-so-often in the marketing field, the tail is wagging the dog.
What do I mean by this? Well, it’s a reflection on the old saying that “what gets measured gets managed” (Or my preferred version, “What’s easily measured gets manipulated, what’s hard to measure gets ignored entirely.”)
And in recent years, measurement, or more specifically, the manipulation of the data we use to target, track, and measure advertising performance, has been the singularly defining feature of the modern approach to marketing. Leading us to a whole generation of marketers who know very little about creativity, insight, or strategy and a lot about things like attribution, campaign ROI, and ROAS.
So, we should all pay attention to how advertising and marketing are measured when a change is mooted. Because if these new measures stick, they’ll wag the dog and directly influence our work.
Here’s an example. While brand and brand building has become sexy again in the past couple of years, it followed a twenty-year dead zone where brands and branding were largely pooh-poohed because what mattered was data and performance marketing and its close cousin, programmatic advertising, which were “proven” to work by attribution models that (entirely coincidentally I’m sure) were provided for free by the corporations with the most to gain from our becoming reliant on them. Notably, Google and Facebook, whose business models we now realize look a lot more like that of a drug dealer than a business partner.
Somewhat amazingly, it’s taken us nearly the entirety of these twenty years to figure out that much of the marketing attribution we’ve come to rely on is, in fact, nonsensical. While attribution models are fast, cheap, and provide the illusion of precision, they’re also horribly inaccurate to the point that they’ll tell you profitable things are unprofitable and vice versa.
So, with this rather long exposition out of the way, I bring you this rather thoughtful article on shifting marketers from attribution to econometric measurement as it becomes harder to track consumer behavior (for various reasons).
I’m going to be honest here for a second. Any change, no matter how necessary or valuable, is going to be an extremely hard lift.
First, it requires a generation of marketers who’ve grown up with an explicit belief in attribution dashboards to admit they were wrong, embrace a whole new approach to measurement, and shift budgets wholesale from things they’re comfortable doing to things they’re uncomfortable doing.
Second, this shift is made doubly hard by the fact that while attribution is fast and cheap and provides the illusion of precision, econometric analysis is slow, expensive, and…fuzzy at best.
Third, marketers are simply addicted to consumer tracking data. I don’t know about you, but I’ve yet to meet a marketer who doesn’t think there will be an imminent workaround vis-a-vis privacy regulation, ad-blocking software, changes to Apple’s privacy rules, and the web going cookieless. Simply put, nobody wants to think that what they’ve become comfortable doing will change. The data, no matter how inaccurately utilized, has become the shared crutch that marketers and their financial overlords simply cannot imagine themselves without because it’s the only shared language they have.
Fourth, and finally, it would require marketers to engage in a major effort toward internal education, where they’d need to educate their peers in finance and the c-suite more broadly on why the fast, cheap, and seemingly precise measurement frameworks they’ve become comfortable with aren’t cutting the mustard and need to be replaced with an alternative that’s slow, expensive and fuzzy. And, I’ll be honest, I don’t think there are many CMOs out there who’d be willing to expend political capital on starting down that path, let alone committing to it.
But yes, there will be some that embrace this shift fully, and I suspect they’ll outperform their peers in the process. Yet, I can’t help but think this will be a niche rather than a wholesale shift.