A mainstay of the modern product creator’s marketing toolkit is the case study. They typically walk you through step-by-step what the creator did and show how you too can do this.
They typically come with a title that looks something like:
- I made this many thousand dollars in this many hours whilst doing surprising thing.
- I got this much traffic whilst not doing thing you expect me to do and in this timeframe.
After all – after you’ve made said achievement it’s easy to milk it a little bit extra by sharing the tantalising secrets to your incredible success.
The problem with these case studies is they make it look easy. Whatever “it” is, it’s always just a straight road from a blank page all the way to watching the money flood in.
I have a problem with this: very very rarely is any of this easy. If it was easy, someone else would have already executed your idea.
The notion that I made twenty thousand dollars in twenty four hours is endlessly appealing – who doesn’t want to do that – but grossly misrepresents the reality that this is hard.
In misrepresenting reality, I argue these case studies are of exceptionally limited use. They satisfy a curiosity to check up on how other people are doing but fail to offer anything actionable you can take away and implement.
This is classic survivorship bias. Survivorship bias is the tendency to focus on only those processes which have “survived” and overlook those which failed. It’s typically seen in medical and investing fields, but applies well to marketing – and especially mega case studies.
Ian Lurie explains the concept nicely:
In marketing:…We only write about the winners. If we only write about the winners, then you only read about tactics as part of success stories.
Survivorship bias often makes the wrong tactics look right. We see those tactics and stop digging…. Always ask ‘Why, really, did this work?’
Given learning from people with a track record of success is a pretty good idea and these articles have the potential to be useful, how should one analyse them? How do you account for the bias and take away something actually useful?
Here are some of the questions you should be asking.
What preconditions for success are there?
Start by asking what role preexisting audience, network and influence play. How does this compare to your situation? Will this work as well if you have none of the preconditions the author is able to leverage?
If preconditions are important (they probably are) then can you get to the same point? If you can’t (say the author leverages five years of networking and you have three months) then is this a deal breaker? Probably not, a deal breaker but may have an impact. If that’s the case, ascertain the impact this is likely to have on your own execution.
At the same time, if a post is promising to show how I increased x by xx%, what was the starting point? Going from 20 visitors to 100 is a 500% increase, but 100 visitors is rarely something to write home about. Where stats are relative, be cautious.
What’s the one thing success is reliant on?
This is a great thing to be asking: work this out and you can establish where to focus your efforts. There’s often one pivotal point which accelerates the success of the project, but it’s rarely highlighted.
Social media posting is often pivotal here. Prominence on any one of a number of social sites can put rocket boosters on all sorts of metrics, but there’s often a huge amount of luck involved here.
Social sharing was a big problem we ran into with MasterWP. We produced a free email course and needed to get it in front of people. Week one posted on a host of social sites and got nothing. A week later tried again and hit the top of the /r/wordpress subreddit. Without this we wouldn’t have had enough interest to move forwards. It seemed whether our social posting was successful or not was random.
Chance and luck play a big role and it’s hard to account for them. Were you just unlucky, or does your product suck? Still, working out social sharing isn’t a level playing field gives you time to work out how to get an advantage.
Social media posting is a good example but by no means an exhaustive list. Working out where those pivotal moments are lets you plan and mitigate for them.
Is this ethical?
The most sensational case study headlines often rely on the sketchiest of tactics:
- Pre-order numbers that rely on deception.
- Signup numbers that rely on really annoying popups and/or other misleading tactics.
When startup speak meets a desire to make the numbers work no matter what, you end up with (actual) case studies which say this:
“we ultimately relied on a bit of deception to validate the idea”
It may be possible to achieve sensational numbers with the tactics described in these case studies, but do you actually want to use them? Think about the long term damage to your brand.
How much work went into this? Am I prepared to commit and make sacrifices?
This is one of my favourites. Let’s call it the “totally unreasonable amount of work required hack”. Case studies often hide behind phrases such as we wrote the whole course over an intensive weekend or repeat these steps with everything you publish.
Often these are disguising the truth: the “intensive weekend” turns out to be ten days without doing anything else at all (and conveniently not having a regular job or bills to pay in that time) and the “couple of steps” you have to repeat takes up an entire day’s work which I’m sure you happened to have spare.
It’s so easy to take the advice of those who are already successful as the gold standard, but you’ve got to ask if the level of commitment required is desirable.
A year ago whilst I was doing my degree I experimented with getting up at 6am to write blog posts (#6amclub) on the basis of Sean McCabe’s recommendation. I’d watch Casey Neistat and be super inspired to work unreasonably hard. It fell to my girlfriend to remind me: these people are workaholics. Their lifestyle resolves solely around work and that’s totally undesirable.
You know what – she was right. I have things other than work that are important to me. When a case study is making something look easy, take a close look at how much work they’re advocating.
What am I not being told?
It’s fun to portray a story of vast success in which there were small challenges but these were triumphantly overcome. The big challenges detract from the narrative of I know precisely what I’m doing and are thus glossed over with some literary gymnastics.
Working out what you’re not being told is important as when you come to do it, it’ll never go to plan. There are going to be difficult challenges and you need to know what those are. There may be points where it looks like the project is fucked and it’s not worth continuing and you need to know how to respond.
The reason you’re reading a case study from someone who’s done something successfully is they’ve had the tenacity to keep going, even when shit got tough. Sure, they’re under no obligation to share the difficult details with you, but it’d be a hell of a lot more practical use if and when you come to use the case study and feel like giving up yourself if you knew how those offering themselves as models reacted when things got really shit.
Take inspiration from the one-sentence clues that things might get really difficult and if and when things don’t work out, use these clues to help you decide how to respond.
Product launch case studies that are useful
Case studies can be useful and inspiring resources. They can also, however, be thoroughly misleading and give the impression of ease when it’s definitely not the case. A case study offering only the good bits is showing off, not sharing anything useful.
To demonstrate my point about well-done case studies being helpful, these are a couple I got a lot of value from:
- Nat Eliason’s mega behind the scenes on building Programming for Marketers.
- Nathan Barry’s launch-sequence guide how to launch anything.
- Robbie Richards’ content marketing case study (falls into the how much work category, but still useful).
Take off the rose-tinted glasses, settle down to the reality that hard things are hard and remember if what you’re doing was easy, someone else probably would have gotten there first.
