What did we learn after running 2,000 experiments for Fortune 500 product teams?

The ‘Lean’ movement has taken the corporate world by storm, however there are still numerous obstacles for product groups that search to adopt its experiment-driven ethos and make selections knowledgeable by buyer knowledge. That’s why two years in the past we started building Alpha, a platform for Fortune 500 product groups to show hypotheses into buyer perception inside 24 hours with out having to tap any inner capabilities or navigate compliance obstacles. In the process, we’ve discovered a considerable amount about corporate culture, the character of consumer research, and product administration processes.

As we speak, our shoppers embrace forward-thinking product teams from AT&T, Capital One, PwC, Aetna, and lots of others. Lately, they collectively surpassed 2,000 experiments on our platform! After generating roughly 660 prototypes, they obtained feedback from almost 400,000 customers. The end result: 46,000 minutes of video from moderated and unmoderated interviews, 6,500 charts, and lots of of undoubtedly sensible — and informed — product selections.

We spent some time mining our databases (for what we name ‘experimetrics’) and reflecting on shopper conversations to extrapolate what we’ve discovered alongside the best way. Under are the seven most meaningful and actionable insights we discovered:

1. Change is troublesome. The previous adage is painstakingly true. As a startup, we need to maintain in examine our myopic perspective of the world — talking to clients could also be an organic part of our job, however, as we discovered, that’s not often the case at a large group.

Regardless of lengthy believing in the worth of speedy prototyping and experimentation, Fortune 500 product managers usually operate in environments with many competing priorities. Consumer research is usually costly and executed by inner groups or businesses in monthly or quarterly cadences. The power to turn round analysis in less than every week, yet alone a day, is totally unprecedented.

And whereas ‘on-demand user insights’ sounds interesting, in apply it challenges many corporate conventions, probably the most entrenched of which is the bias to overplan. When analysis cycles take months, it’s critically essential to make it possible for every facet is rigorously crafted and vetted. However once you speed up that course of to a matter of hours or days, iteration eliminates the need for exhaustive planning.

Our knowledge illustrates how troublesome this modification in mindset and conduct may be. At full capability, particular person product teams execute about 8–12 experiments per thirty days on our platform. Even with workshops and in depth onboarding, it takes anyplace from three to 6 months for shoppers to succeed in that bandwidth. Positive, a few of that point is spent determining the best way to shortly turn knowledge into selections. But the overwhelming majority is consumed as a product group culturally and virtually shifts from waterfall to agile experimentation, recognizing that planned research pales compared to iterative analysis. Spending two weeks outlining buyer analysis that may inevitably be flawed is not any match for six iterations that can be executed in the same timeframe.

On our podcast, That is Product Administration, Cindy Alvarez, Director of Consumer Experience at Yammer, echoed one of the crucial widespread sentiments about training ‘Lean’ and ‘Customer Development’ within a big group. She urged listeners to cease planning and just go begin speaking to clients because it’s unattainable to get higher at doing it in any other case.


She’s absolutely proper and it’s a technique we are heavily invested in. We began pre-populating new shopper accounts with research executed for them, including customer insights into aggressive benchmarks and usefulness throughout their respective products. Up to now it’s been a helpful ignition for product teams to begin iterating.

2. Typically, formality trumps informality. Persevering with the theme from the previous insight, we’ve discovered that, even as soon as shoppers hit full velocity, it doesn’t quite resemble the cadence of how startups follow experimentation. We initially designed the product so that any stakeholder might simply submit an experiment on an advert hoc basis, which is analogous to how we function. As an alternative of running impromptu experiments although, our shoppers submit experiments in batches, typically weekly.

And it turns on the market’s a superb purpose for this. Whereas a fluid workflow is sensible in a startup, it sometimes doesn’t inside a big organization that has numerous stakeholders with totally different (and sometimes competing) goals and tasks. Product managers diligently consult these stakeholders when explaining buyer suggestions and deciding on next steps. A predictable and recurring cadence is usually essential to hold everyone on the same web page.

That’s why concepts like the ‘design sprint’ have taken off: they allot time for stakeholders to get aligned. We’re embracing the position that formality plays right here, and now encourage shoppers to arrange ‘experiment sessions’ on a daily and consistent basis, as long as these periods end with testable hypotheses.

three. Product experiments may be grouped into discrete categories. Before we might create a platform and workflow to accelerate consumer analysis processes, we had to better understand the varieties of research product teams want within the first place. That’s why, earlier than writing a single line of code, we carried out the first 500 or so experiments manually using third-party instruments.

We found that consumer research experiments involving prototypes (as opposed to doing experiments in a manufacturing setting) usually fall into one among six discrete classes. Considered one of them, usability testing, has a extensively accepted definition. We had to delineate the others although, and whereas our definitions are not at all gospel, they suffice surprisingly nicely, requiring solely modest ongoing revisions. Every class is accompanied by ‘rules of thumb’ and a set of configurable experiment templates, which you’ll be able to examine in ourguide to prototyping, however right here is an summary of every:


Here’s a breakdown of the popularity of each check run on our platform:


We’ve lots extra analysis to do, but these working definitions enable consumer researchers in our change to take nearly any shopper request and turn it into an executable research inside minutes.

4. All research is biased. Our offering primarily consists of testing in what we call a ‘simulated environment.’ The customers who provide feedback know that they’re a part of a research and are paid for their time. They interact with high-fidelity, interactive prototypes, and usually understand that the products have not been engineered and launched to the market.

We focus on this sort of testing as a result of product groups can learn an incredible amount from it whereas complying with their group’s present processes and danger tolerance. No inner engineering or design assets are required; no valued customer turns into the sufferer of a half-baked product; and no authorized department needs to be consulted. In fact, the info just isn’t as dependable as what you’d learn from delivery a product.

All analysis, including ours, suffers from a level of bias. However acknowledging such isn’t an excuse to keep away from doing consumer analysis altogether. It’s an argument for the other: to fervently do even more analysis and attempt to attenuate the bias across it. Considering in any other case is lacking the forest for the timber.

One of many core rules of the scientific technique is the concept of replicability — that the outcomes of any single experiment could be reproduced by one other experiment. We’ve far too typically seen a product staff wielding a single ‘statistically significant’ knowledge point to defend a dubious intuition or pet venture. However there are a selection of factors that would and virtually all the time do bias the outcomes of a check with none intentional wrongdoing. Mistakenly asking a number one query or sourcing a sample that doesn’t adequately characterize your goal customer can skew individual check outcomes.

To derive value from individual experiments and customer knowledge factors, product groups have to follow substantiation by way of iteration. Even if the outcomes of any given experiment are skewed or outdated, they can be offset by a strong consumer analysis course of. The safeguard towards pursuing insignificant findings, if you’ll, is to be aware not to think about knowledge to be an actionable perception until a sample has been rigorously established.


That’s why we make it possible for for virtually every experiment, qualitative and quantitative analysis is carried out. Further, we attempt to generate insights which are comparative — it’s not often enough to learn what customers think of a prototype in a vacuum. In the actual world, customers have an array of choices to satisfy any given want, so we make it possible for suggestions on a solution is all the time relative to an alternate. Combining and optimizing these two approaches has tremendously minimized bias, and sometimes leads to a plethora of knowledge from which to determine patterns and insights. And, in fact, we stress the significance of incorporating other knowledge inputs, like conventional market research and in-app analytics.

5. Consumer suggestions never ceases to shock us. You’d assume that after producing knowledge from a whole lot of hundreds of customers, we’d have ‘seen it all’ relating to suggestions and insights. However that isn’t even close to true. We proceed to be stunned by what we see each day, primarily with regard to…

…the distinction between what customers say and what they do.

It’s been properly established that humans are fairly dangerous at predicting their future conduct. We’ve researched the psychology of that dynamic extensively. Nevertheless it’s still shocking when we find nearly unanimous help for a function in a survey and subsequently discover completely little interest in the function as soon as it’s prototyped. Placing a visual stimuli in entrance of your goal market is completely essential for substantiating findings.

…the honest feelings expressed.

Market developments change quickly and product teams are in a continuing hustle to maintain up. Few things get them to drop what they’re doing and sit silently in addition to watching a video of an emotional consumer interview. We’ve witnessed a senior citizen cry profusely as they work together with a prototype that invokes nostalgia. We’ve giggled as a Millennial described how much they hated a product idea and all the things they’d quite use as an alternative of it. We’ve been shocked by a gentleman who opened up about how a brand new product might assist him rebuild relationships together with his youngsters. Consumer research is actually an emotional rollercoaster.

…the validation of passionate enthusiasm.

Some of the widespread questions our shoppers ask is: “How do we know when we’ve validated a product concept with customers?” While we don’t have any hard-and-fast guidelines, we’ve half-joked about making use of the “Pokémon GO Benchmark.” For enjoyable (and because we’re hooked on the sport), we executed analysis towards a number of hundred users of the cellular recreation. The responses have been impressively enthusiastic and exemplified patterns to look for when assessing validation. Players gave detailed suggestions to open-ended questions, spent vital time partaking with prototypes, and routinely provided to pay for new options we designed. Obviously, each product doesn’t must be a meteoric hit to seek out success, but evaluating outliers like Pokémon GO serves as a strong anecdote.

The important thing takeaway is that even when we assume we know a consumer phase rather well, research findings are not often predictable or apparent. You merely can’t underestimate how troublesome and rewarding having empathy could be.

6. Shorter iteration cycles unlock deeper insights. When our initial shoppers finally starting rapidly running experiments on Alpha, it turned clear why generating significant customer insights is usually so elusive for corporations that take months to execute analysis. Velocity in and of itself is the key.

When iteration cycles are sluggish, product teams prototype and experiment until they generate promising outcomes. The moment they get the slightest sense that they’ve struck gold, they begin engineering a solution (if they haven’t already began). In essence, they learn ‘what’ resonates nicely, however they don’t have the time to learn ‘why.’

But when we accelerated the analysis course of to days, we discovered that shoppers have been not content as soon as they validated a product concept. They finally had the time and bandwidth to ask ‘why’ a prototype was perceived as extra beneficial that earlier iterations or options. To keep up, we needed to construct out an in depth qualitative workflow in order that we might return to a sample of users that tested a product and ask them open-ended questions. In doing so, we have been capable of unlock ‘deep insights.’

We define a deep perception as a comprehension of a customer persona that’s so strong that its worth transcends the individual challenge that a product workforce is working on. It is useful to anybody within the group who is concentrated on delivering worth to the identical market. As an alternative of merely understanding that clients want your prototype with an costly one-time purchase compared to an affordable monthly subscription, you conduct interviews to learn why and uncover that clients are literally afraid of forgetting to cancel their subscription. That’s an insight that’s so significant, it may be utilized to different products in your group’s portfolio. And it’s made attainable by velocity.

7. Knowledge is a way to an finish. It’s straightforward to get lost in the buzzwords du jour somewhat than to do the exhausting work of discovering worth and driving ROI. We discovered shortly that to build a successful platform, we’d need to deliver to product teams greater than the power to be ‘data driven.’

Initially, our assumption was that knowledge that shoppers generated within Alpha would translate instantly into better product decision-making. That’s true to an extent and it definitely issues to the group as an entire. But when we really investigated what was happening, we discovered that being knowledge pushed isn’t actually what product managers need or want.

We pay attention intently to how our shoppers talk the value of our platform and experimentation to peers at other organizations. Often, they point out how it aligns their group around hypotheses quite than opinions. As an alternative of two hour conferences full of debates, the group spends 15 minutes putting hypotheses into Alpha after which 15 minutes reviewing the findings once they’re prepared. One product supervisor discussed how he uses Alpha just because the info provides him a cause to e mail his director an replace as soon as every week. Another spoke about how thrilled he is to affect other departments to acknowledge the value of iteration and learning.

In fact, knowledge is important to enabling all of those benefits, however it’s a way quite than an end. And that issues as a result of it informs our product roadmap. For example, early on we didn’t put much effort into the info visualization of research findings. But now we understand that presentation is simply as, if no more, necessary than the underlying info, because it’s going to be shared and used to influence stakeholders. Recognizing how product managers must manage upward, sideways, and downward led us to prioritizing features like reporting and sharing.

We’ll continue to replace this listing as we learn extra. In the event you’re as passionate as we are about experimentation and buyer insights, be a part of our workforce. Or give Alpha a spin and start making smarter product selections.