Experimentation Works: The Surprising Power of Business Experiments

At the search engine Bing, for example, an idea to change the way ads were displayed had been put on the back burner. But then an engineer decided to run a simple experiment, to test its impact with users. The result was a surprise 12 percent revenue spike – representing a $100 million a year. Bing's best-ever revenue-generating idea was discovered through this one, simple, "either/or" test.

And it's not just online where we need help predicting people's behavior. This book shows that we never really know how someone's going to respond to something until we test it out on them. And, thankfully, there are lots of different ways to do that – cheaply, quickly, and pretty much risk-free. In fact, it's not experimenting that often costs money, slows progress, and causes companies serious damage.

We need to be ready to try out lots of ideas – and learn lessons from the results every time. What we really can't afford to do is rely on experience, intuition or beliefs. This book shows that the truly innovative ideas often prove every assumption made about them to be completely wrong.

So, Stefan Thomke is a good person to have at our side. He's an expert in making innovation work, by creating the systems and conditions that support it. Alongside his role as a professor at Harvard Business School, he's an advisor to business leaders around the world. And his book benefits from many case studies and real-life examples, often involving well-known organizations and individuals.

So, keep listening to find out what experimentation means in modern business; why experimenting will benefit every type of organization; and how you can start building an experimentation culture where you work.

Chapter One begins with the story of Ron Johnson. After huge success designing stores for Apple, he left to lead a large-scale revamp at department store chain J.C. Penney. There, he based his decisions on his previous work, and on Penney's own high-quality transaction data.

But his plans flopped – spectacularly. After 17 months, with sales way down and losses soaring, he was out of a job. Neither his experience, nor the existing data, correctly predicted what would work next.

Compare that to the New Zealand team that won the America's Cup for yachting in 1995. They used experimentation to great effect, working out exactly what to test to improve their chances. They ran disciplined experiments – and, each time, they drew meaningful insights. Their "marginal gains" approach of incremental progress took them to an unexpected victory.

This book is peppered with examples of experimentation at its best, featuring household names such as Amazon, Toyota and Expedia. But, despite Thomke's positive tone, there are warnings, too.

For instance, he says some readers will find the "test everything" approach extreme at first. Especially if they're used to planning what to implement and then coping with the consequences. Not all decisions can be tested, either. And not all test results should be followed. We may have very sound ethical, legal or strategic reasons for taking a different course of action. Experimentation shouldn't strip an organization of other useful decision-making tools.

But the biggest warning is about how we define "success" in tests. With careful design and implementation, every experiment can be successful – if it tells us whether an idea will work or not. We just have to get used to receiving lots of negative results. Even with perfectly run experiments, we should expect around 90 percent of our ideas to be proved wrong.

So, we've got to learn the importance of knowing what doesn't work, and the value in avoiding costly mistakes. We should celebrate small wins. And we should always be on the lookout for those golden results that can change everything. As Thomke puts it, we may have to kiss a lot of frogs before we find a prince.

Here's how the book helps us get it right. Chapter One reveals the difference between good and bad experiments. An undisciplined, trial-and-error approach may occasionally toss up results that seem useful. But we should be cautious about trusting them, as they may lead us down blind alleys.

Instead, we should plan our experiments with the precision of New Zealand's victorious sailors. They came up with ideas for improvements, then tested them in the most controlled conditions possible, often on two boats for comparison. When they found something that worked, they implemented it. But they never stopped looking for ways to make it even better.

People are harder to predict than boats, of course. That's why experimentation is so important, and instinct so often wrong. Back in 2003, Apple CEO Steve Jobs famously predicted that people wouldn't want to rent music. In 1946, movie producer Darryl F. Zanuck was confident that TV would never catch on.

So, instead of trying to guess what people will want, or how they'll react to a product or service, many companies have discovered the value of experimenting – and on a grand scale. At Google, for example, 13,000 algorithm tests, 8,000 A/B tests, and nearly 3,000 user-click evaluations in a single year led to around 500 changes being made. Clearly, far more of the ideas turned out not to be right – but they still found 500 great ways to move forward.

And here's an invaluable tip: organizations that get the most out of experimenting actually plan to fail. Early and inexpensively. They do experiments that are quick, cheap, and risk-free.

In addition, they look for ways to get fast feedback. They often have several experiments going at the same time. And they try to use controls – like Team New Zealand's second yacht – to compare their new idea with not doing anything different at all!

There's more on the secrets of successful testing in Chapter Two, including examples of failure as well as success. For instance, we can learn a lesson from animal supplies giant Petco, to avoid setting up inaccurate comparisons. Petco's mistake? To use its worst-performing stores as a control group, and only try out new ideas in its most successful branches. It ended up with data that was dangerously flawed. If Petco hadn't realized that, it could have rolled out worthless or even damaging changes across the whole organization.

In another example, a famous restaurant chain tried to improve customer experience by installing LED lights, and was surprised to see sales drop. But instead of rejecting the new bulbs, the experimenters dug deeper, and discovered the real reason for the downturn. Previously, some store managers had ditched company guidelines, and installed brighter lights than they should have. So, the new LEDs looked dimmer – making some customers think that a restaurant was closed! The lesson here? Don't blindly follow the results of your tests. Investigate anything that doesn't feel right. And, if necessary, test it again.

"Experimentation Works" has the look and feel of a practical manual. So. it's no surprise that Chapter Three is about finding a real-world approach that works for you and your business. It has to be manageable. It needs to produce easy-to-understand results that can be implemented by the right people. And everyone needs to trust the process, so they feel confident to act on their findings – and to keep building on them.

British price comparison website MoneySuperMarket knew it had found the right approach when it could set up, run, and react to tests within just three hours. Someone could have an idea over breakfast, go to work, test it out, and potentially implement a verified improvement by lunchtime! And Thomke is confident that an equally efficient system can be established in any organization.

The trick is to keep things simple. Design clear, unambiguous tests. Whenever possible, don't let people know that you're testing them, as that can affect their choices. And don't end up with so much data that you risk drowning in it. After all, you're looking to make small steps forward, but with large amounts of confidence.

If you're in a leadership role, Chapter Four is where you find out how to start creating a culture of experimentation. Begin by learning the difference between an experiment that fails and a failed experiment. Remember: a great experiment can fail, but still teach you valuable lessons. So, you'll also need humility, along with the confidence and resilience to keep pushing on.

Be prepared to face challenges to your own ideas – from people at any level of the company. As a leader, you'll need to balance steadiness and efficiency, on one hand, with what Amazon boss Jeff Bezos calls "wandering" on the other. It's only by exploring your curiosity carefully that you'll avoid risking things that are working just fine.

But what does this sort of experimental culture look like in practice? Chapter Five shows us, taking us inside Booking.com, and the Lotus Formula 1 racing team. These contrasting companies have both seen the benefits of continual experimentation.

They've both had problems to overcome, too. At Booking, for example, everything runs in a live environment, so they've found it harder to do radical tests. And Lotus have had to master high-speed decision making, in order to thrive in the fast-moving world of Formula One.

But, in their different ways, both companies have created "experimentation cultures." Through their success, they've proved that experimentation works. And the title's double meaning comes out here because these companies have also become "experimentation works", bustling centers of testing, innovation and productivity.

So how did they do it? Chapter Six reveals the five steps to success that Thomke says will work anywhere.

Handily, these steps go from A to E. A is for awareness, of the need for continual testing. B is for belief in the approach you choose, at all levels of the business. C is the commitment to keep testing, through countless negative results. D is for diffusion, as you widen the scope of your experiments, and offer your people more tools. And you get to E when experimentation is embedded in your whole organization, driving things forward year after year.

By Chapter Seven, if you're still in any doubt about the experimental approach, you'll likely find reassurance. The author runs through his seven most-often-heard objections – and presents evidence to counter each one.

Perhaps you're worried that experimenting kills intuition, for example. Don't be. Intuition still has an important role to play – if it's used to make experimentation better.

Or, if you think that small experiments might keep you from taking big steps forward, Thomke shows how quickly incremental changes can accumulate, to have a major impact on your success.

So, what's our verdict on the book overall? We like its energetic argument in favor of experimentation – from an author with a wide business background as well as a strong academic pedigree. His approach is broken down into manageable steps, and his theories are backed up by interesting and detailed examples throughout.

Many of these are based on large organizations, often ones that have developed their testing systems over many years. So, some readers may feel they're a long way off from this – and not in a position to make the kinds of strategy decisions required. But we think the author does enough to show how any company can at least start building an experimentation culture. And how people at every level can play their part, even if it's just discussing a testable idea with their boss.

We're left with a strong sense of opportunity – particularly by the epilog, which makes some intriguing predictions about the future of experimentation. Automation in testing will increase, Thomke says. Artificial intelligence will play an ever bigger role. We may even see "closed-loop experimentation" – done on autopilot, with the system itself spotting things to explore, designing and carrying out the tests, and even acting on the results.

The sooner our current organizations are onboard with experimenting, the better equipped we'll be to understand and benefit from this brave new world.

However, Thomke believes human beings will still be the most important part of the process. It will always be people who come up with the most creative ideas to test. And they'll always be better than machines at questioning and refining the results, and judging their true value.

But he admits these are only his predictions. Ultimately, as his book makes clear, the best way to see the future is to start experimenting right now!

"Experimentation Works: The Surprising Power of Business Experiments," by Stefan H. Thomke, is published by Harvard Business Review Press.

That's the end of this episode of Mind Tools Book Insights, from Emerald Works. Thanks for listening.