Summary:
- When building an online learning experience, you can decide whether to beta test or go straight into a full product launch.
- Both options have high stakes tradeoffs. You can use the Beta Test Spectrum to help you analyze your market and product.
- The Beta Test Spectrum looks at five vectors: demand, competition, brand moat, price, and promise.
- Use the Beta Test Spectrum to maximize the value you add to your students’ lives and value to your business.
Six months after the Writer’s Strike, HBO placed a $60 million bet on a pair of TV neophytes. Neither had worked as a series staff writer, let alone run a show.
In November 2008, actors were cast, sets built, and principal photography of the pilot began.
Later, the writers screened the pilot for a few colleagues. Consensus was quickly reached: that screening was one of the most painful experiences of their lives.
The plot lost itself like a bad alibi. The costumes felt like a medieval Dolly Parton cosplay. On a legal pad, someone wrote, in all caps: “MASSIVE PROBLEM”.
In a stroke of extreme foresight or extreme oversight, HBO gave them a mulligan (foreign pre-sales, so the hypothesis goes).
Do it again, the executives told them. You got this.
It was a bold but not uncommon move. This is exactly WHY you shoot a pilot, isn’t it? To test the show before greenlighting it?
The writers, D.B. Weiss and David Benioff reshot the pilot. In April 2011, nearly three years later, the first episode of Game of Thrones premiered on HBO.
Ordering a pilot (buying and producing one episode) is traditionally how Hollywood made TV shows. Then came House of Cards, Netflix’s foray into scripted television. Not the first straight to series order but certainly the most publicized in the “golden age of TV”. Netflix paid $100 million for two seasons (26 episodes), committing it to series without testing a pilot.
Its success didn’t prove whether one model (pilot orders versus straight-to-series) was better or worse. But it did prove that both models were viable.
Online Learning Products: Beta Test or Full Product Launch?
The same can be said for building an online learning experience. When building an online learning product, you arrive at a similar fork: do you beta test (pilot order) or run a full product launch (straight-to-series)?
As a product developer (course creator), I always pushed to beta test. Why not? I was insecure and the upsides appeared clear and immutable:
- Increased creation speed. Both company and customer accept tradeoffs: customers get a lower quality product, faster, for a discounted price. In exchange, the company receives feedback and/or testimonials.
- Increased feedback cycles. The feedback loop is tighter, which is possible because of the low fidelity product (see above). The customer gets specific questions addressed immediately. The company builds a better product.
- Decreased risk. The company can strategically prune different flavors of risk: demand risk (do people want this?), pricing risk (will they pay at this price?), value promise risk (does it solve their problem)?
Sitting up in the ivory tower of “creative”, the product developer is insulated from the nasty business of, you know, making money. But if you own the P&L, you face the downsides of beta testing:
- Increased time to (final) product. You’ve run your beta; now what? You collect, synthesize, and massage the feedback into the final product. This additional cycle can double the time to final; the time you’d save by committing to a “final” version from the get.
- Increased time to revenue. It also adds an acquisition cycle. Running two launches (one for the beta and one for the final) increases strain on your team, your list, your cash flow, and future products.
- Null hypothesis risk. A beta is an experiment and every experiment runs the risk of a null hypothesis: an experiment with no learnings. If you learn nothing, you gain nothing. Naturally, there are best practices to avoid a null hypothesis — more on that in another article.
Lastly, some products you can’t beta test. Complexity, budget, compliance, regulation, and cash needs are all reasons to go directly to a full product launch. In these cases, you need great project management, not a beta.
So there are tradeoffs between beta testing vs. doing a full product launch. How do you know which is right for your upcoming product or course?
At the end of this article, you’ll know how to analyze your situation with confidence. You’ll know when you have a GoT on your hands or a HoC, and you’ll be able to pick the best strategy for your customers and your business.
The primary tool we’ll use is the Beta Test Spectrum. The Spectrum looks at five vectors: demand, competition, brand moat, price, and promise.
Demand
Demand exists on a scale from low to high. The lower the demand, the more you should beta test. Creating an online learning product takes a lot of work– don’t build something no one wants.
On the flip side, the higher the demand, the more signal the buyers are out there. So give the people what they want.
Demand is your first gate: start here. You want some level of confidence before proceeding to the next vectors.
How do you quantify your demand? Demand testing is its own article, but here are a few options to consider (with T-shirt sizing the effort required):
Qualitative feedback on content (small): Your public content is like a boxer’s lead hand jab: always out there, always gauging distance and probing for opportunities. Your earliest signal of any “there” there is people engaging with your content. If you produce a blog, turn comments on. Read YouTube comments. Reply on Twitter.
Survey your audience (medium): Create a short survey and ask, “what are you interested in learning next?” Send this out in an email or include the link in blog posts, and just leave the survey open. Reduce friction as much as possible: 2-3 questions at most, provide options for them to choose from, and ask for an email so you can follow up.
Ask them to buy (large): The clearest signal of willingness to pay (WTP) is getting the (credit card) digits. You can offer a scoped-down version of your product as a tripwire or an add-on, or give them the option to pre-order with tiered pricing (Steph Smith did this with Doing Content Right; so did Mayweather vs. Logan Paul).
Recently, I worked on an online learning product with Byte-by-Byte. Byte-by-Byte’s flagship program, 6 Weeks to Interview Ready, helps software developers crack the interview process. My product focused on improving a developer’s career. To test demand, we surveyed the list and followed up with 1-on-1 interviews. We found demand was soft — people wanted promotions. They wanted raises. They wanted fat signing bonuses from FAANG companies. They didn’t necessarily want a process to create a meaningful career, which was one reason why we beta tested at the end of 2020.
Three years earlier, I worked on my first Reforge program, the Retention + Engagement Deep Dive. This was also a follow-up to the original flagship program (Growth Series), but we had alumni cold emailing, demanding to know when we’d release a new program. We published a series of blog posts about how Retention was the key to growth, and it generated a lot of buzz. That product went straight to a full launch. Eight months later, Retention + Engagement Deep Dive outsold Growth Series by 17% and earned an NPS of 70.
Now do this: How strong is your demand? How do you know (what are the signals)? If you’re not sure, which strategy will best help you gauge demand: content marketing, surveying, or asking them to buy?
Use this: I really like surveying. Here’s a template you can use.
Competition
Generally speaking, the more competitors you face, the more you should beta test. Specifically, test your positioning. Positioning is:
“An organized system for finding a window in the mind. It is based on the concept that communication can only take place at the right time and under the right circumstances.”
A beta helps you understand how customers perceive your product and brand relative to other products and brands. Are you HBO (prestige programming) or Lifetime (programming for women 35+)? Are you Apple (premium and elegant) or Dell (functional and great value)?
If you look out in the market and see nothing but blank space, then make like Taylor and write your name. Get to market and grab as much mindshare ASAP.
Today, there’s a surplus of courses teaching you to create courses. If that’s your niche, you have to throw elbows and strategically box out competitors with strong positioning. Beta testing can be helpful.
In 2020, Nat Eliason’s published Effortless Output, a course that teaches you how to use the productivity tool, Roam Research. Nat was an early adopter and promoter and wrote extensively about how he uses the tool. Based on feedback from the article (a demand signal) he built an online learning product that he put into the market immediately. The course did better than expected and in the same year, he used the profit to help purchase a second home.
Now do this: Would you describe your market as crowded or open? List 3-5 competitors and write a one-liner about their positioning.
For example, online learning products: Udacity (tech skills with certification) Khan Academy (free online math courses), Masterclass (premium infotainment from celebrities).
Or for a non-edutech example: Planet Fitness (affordable gym for suburban moms), Gold’s (for “real” weightlifters), Equinox (above average Lululemon leggings to person ratio).
Brand Moat
Your brand moat is your unique economic advantage that defends your mind- and market share from competitors. Weak brand moats (Joe Anybody’s Facebook ads Skillshare course) should beta test; strong brand moats (MasterClass) can do full product launches.
At I Will Teach, we had a powerful brand moat in the personal finance and info products space. When we extended into fitness and food niches, however, it was like standing outside the energy shield during the Battle of Wakanda. Our moat didn’t extend that far. So we probed the market for two years with a number of beta products, and it was still challenging for customers to cross that bridge with us.
For four years, Reforge created a brand moat around teaching growth marketing to mid-career tech professionals. We focused only on that audience (mid-career PMs, PMMs, designers, etc. at tech companies) and maintained a high-quality bar on all free and paid material (if it doesn’t meet the bar, we do not publish).
This brand moat allowed us to launch 4 new courses in 2020, all at premium price points, without a single beta test.
Now do this: What companies or brands successfully used their brand moat to transition into an adjacent market? What do these companies have in common? What’s different?
Examples: Credit Karma (credit score to taxes), The Hustle (daily newsletter to Trends product), Uber (ride-share to food delivery).
Price
Beta test premium priced products whenever possible. Premium pricing creates higher perceived value and increased cognitive friction to purchase; beta testing gives you the opportunity to refine the marketing, offer, and content before finalizing the product. With value-priced products, there’s less cognitive friction to overcome and lower perceived value.
Last I Will Teach example here (we shipped a lot of programs): flagship programs like Find Your Dream Job and Zero to Launch were priced at $1,995 or higher, and all were tested for months before going to market. Value-priced products ($299) like Success Triggers and Finisher’s Formula went directly to market
Now do this: Work backwards from revenue to help you decide to beta or launch. You can estimate revenue: pick a conversion rate (1% of your list is a typical benchmark), then a price point that’s meaningful to your strategy. If it’s a higher price point ($1,000+) consider building in time for beta testing. If it’s a lower price point, consider going to market sooner.
Promise
The value promise (“promise” for short) is what the user, or the learner, gets from the online learning product. Promises sit on a spectrum between a process-based (you’ll be more confident in interviews, be healthier, improve your chess game) and an outcome-based promise (you’ll get offers from FAANG companies, lose 20 pounds, become a Grandmaster in 3 months). The more specific the outcome and timeframe, the more you should beta test. The beta test is how you make sure you deliver and how you capture testimonials.
Udacity experimented with a certification program that guaranteed job placement within 6 months or 100% money back. It was a bold promise. They tested it for 2 years across 4 programs in the US, before quietly sunsetting the program.
Compare this to MasterClass, which is less about the specific outcomes and more about general learning from celebrities (masters). I’d happily pay $15/month to watch Natalie Portman teach me acting. I also would not ask for my money back if I didn’t land a co-star role afterward.
Now do this: Write down your outcome. Keep it to one or two bullet points. If it’s a specific outcome (lose 8 lbs, ace your technical interview) consider beta testing. If it’s a process (learn Notion, become more productive) consider doing a full product launch.
Case Studies
Let’s look at two case studies and how they fit into the Beta Test Spectrum. Then we’ll see what the creators ultimately decided to do for their products:
Top Performer by Cal Newport and Scott H Young was an 8-week program that taught a process for deliberate improvement in their fields and careers.
- Demand: Low-to-medium. Both Cal and Scott had popular blogs, and Cal had published one book at the time, So Good They Can’t Ignore You. There was a market for their ideas, but I categorize this as low-to-medium demand because “deliberate improvement” is not an easy sell. People want promotions, not a process for deliberate improvement.
- Competition: Open space. Today there’s more of a market for products focused on deliberate improvement (possibly in reaction to the influx of “Learn X in Y days” type of info products) but when Top Performer launched in 2015, there was even more open space.
- Brand moat: Medium. As mentioned above, both Cal and Scott had popular blogs and Cal was already a published author. Collectively they’d spent over a decade focused on topics of personal development and mastery.
- Price: Moderate. They priced Top Performer at $500.
- Promise: Process-focused. No guarantee you’d earn a promotion or land a fat raise; you’d learn the process to get those things.
My take: Given the above, I would have done a full product launch with Top Performer. Considering the strong brand, the promise, and the moderate price point, I’d cut the extra cycles and get a finished product into the wild ASAP.
What happened: Cal and Scott tested with a 4-week beta called Career Mastery. It was an email course with a graduation webinar / Q&A. The beta was priced at $100 and sold out immediately. The beta launched in April 2013; they launched the finalized product in October 2015, 2.5 years later.
Product Strategy by Reforge is a 6-week program that shows Senior Product Managers how to level up to Group or Director level roles. In addition to the Reforge team, the subject matter experts for this program were Ravi Mehta (former CPO at Tinder), Fareed Mosavat (former Director of Product at Slack) and Casey Winters (CPO at Eventbrite).
- Demand: Medium. Reforge courses focused on growth topics at companies with a strong product-led growth (PLG) motion. This meant a large segment of our alumni were product managers and senior PMs. Through extensive surveying and customer conversations (over the course of a couple of years) we had medium confidence in the demand.
- Competition: Moderately crowded. Companies that teach product management (General Assembly, Product School, Udacity) have been in this market longer than us. Our audience was actually different (late vs early career-focused) but it was uncertain if we’d get the positioning right.
- Brand moat: Strong. We had 4 years of regularly selling out our flagship Growth Series program. In addition, we tapped into the credibility and brands of our program subject matter experts Ravi, Fareed, and Casey.
- Price: High. $3,495 per seat.
- Promise: Process-focused. Similar to Top Performer, Product Strategy taught the process to become a product leader, but we didn’t guarantee promotions or job placement.
My take: The strong demand, brand, and process-focused results put Product Strategy in the full product launch bucket. Reforge programs were premium-priced from their inception so the price was anchored with those familiar with our brand.
What happened: Reforge started developing the Product Strategy program in January 2020. We went immediately to a full launch of the product and ran the first cohort in September 2020, 9 months later. It was the best-selling program that season with over 500 students and had an NPS of 68.
Using the Beta Test Spectrum is both an art and science. If you develop online learning products, use them to help maximize the value you can contribute to your students’ lives and value to your business.
1 Comment
Pingback: Chris Ming’s 2022 Goals and 2021 Recap