How to successfully set up a pilot

The short answer is assumptions. You want to test a number of assumptions that you are making about an idea or concept. Whether or not we are aware of it, our brains are always making assumptions, so we must first get a clear picture and then select the important ones to test. It helps to organize your thinking by using three distinct categories:

• Desirability (people wanting your idea)
• Viability (being worthwhile to pursue)
• Feasibility (being possible to produce your idea)

Desirability Assumptions

The McDonald’s example above is one of testing desirability. You might visit every hamburger joint in town and find that no one wants to go see your movie. Not only will you save yourself fruitless time and energy, but it will also give you a chance to learn how the story-idea may be successfully tweaked. Another way of talking about the desirability test is, “will the dog eat the dog food?” Your great idea for an original kind of dog food will first need to pass a crucial test: put your new strawberry and pumpkin flavored potato peels in front of a dog, sit back, and watch.

Viability Assumptions

Saying an idea is viable can mean three things: One is that the economics work, so that the value the idea generates is greater than the investments needed to create it. The second meaning is that the risk is worth the reward. Maybe selling ice cream in the Sahara is desirable, and you could sell it for more than it costs to get it on location. However, with one power cut you could lose your entire inventory. How viable is it then? Think thoroughly about the risks and rewards, and what need to be tested. Thirdly, viability refers to whether something is sustainable in the long run. Will initial customers buy again? Will the social value sustain itself over time? These assumptions are often the hardest to test as they, by definition, involve a projection over a longer period, making it even more important to get them on the table at an earlier stage.

Feasibility Assumptions

So is it possible to produce your idea and when you do, will it work? By now you assume yes, or you would not be taking it this far. But what exactly do you assume and can you get some data on it through testing? The assumption of Feasibility is listed after Desirability and Viability for a reason. If no one wants it or it is not worth making, you will save yourself the effort of finding out if it can be made. By coming from the other side, and knowing your idea’s worth and desirability, odds are you will find a way to get it done.

There are two principles to adhere to when testing. The first is to test behavior, not opinion. People are notorious for saying one thing and doing another. Put them in a situation where they show their behavior, or, better yet, where they have to pull out their wallet. Money talks, after all.

The second principle is that you can -behaviorally- test your assumption in much less time and cost. This is where the instinct of delaying failure needs to be overridden. You can be quicker, leaner, and lower in resolution. Push yourself to come up with quick behavioral tests, where people make a real decision rather than give an opinion. The quicker and cheaper you are with a pilot, the more you can test ideas and aspects. It is the ABT of innovation: Always Be Testing.

Let’s look at a selection of methods for low resolution, each drawing on the works by Alberto Savoia and Eric Ries.

No Resolution. The Fake Door: Create a fake “entry” for a product that doesn’t exist yet. For instance, create a website inviting people on a fancy cruise to the North Pole just to see if someone will bite. Apologize when they do and say that it will be there in the near future, then quickly get on with your next steps towards building it.

Low Resolution. Parts are missing – the Flintstone: this refers to Fred Flintstone’s foot-propelling car. This is where we have the essential criteria ready to test and the rest is made up by legwork. In 1990, Nick Swinmurn wanted to pilot whether people would buy shoes online. He took photos of shoes in the local shoe shop and posted them on a very basic website called Shoes-online. When an order was placed, he quickly went to buy them from the shop and hand-delivered them to each buyer. Just two or three sales were enough to confirm his assumption: people would indeed buy shoes online without having fitted them first. He closed the rough website and went on to launch Zappos.com, today worth $5 billion.

Minimum Viable Product: a lean startup concept popularized by Eric Ries. The basic idea is to maximize validated learning with the least amount of effort. Create a “functional” version of the product that is stripped down to its most basic functionality. As Ries says: “The minimum viable product is that product which has just those features and no more that allows you to ship a product that early adopters see and, at least some of whom resonate with, pay you money for, and start to give you feedback on.”

Mastering ambiguity is an important skill. Becoming comfortable with conflicting insights, paradoxes, and needs allows you to discover new opportunities and ways forward. Ideally, you bring this skill to the table when learning from a pilot, because you need to learn in seemingly opposite ways from your tests: both focused and unfocused. Using closely monitored metrics and keeping your eyes open will bring unexpected outcomes and learnings.

1. Focused: What are the make-or-break hypotheses? By which metrics can they be tested? What pilot or experiment will provide me with data on these metrics? What are my expectations? Most importantly, what are my green light criteria?
2. Unfocused: How will I maximize the opportunity for unexpected learning? How can I ensure it happens? Where should I be? For how long? What should I observe and with whom should I speak?

It is sensible to spend some time thinking about your green light action and your red light action. Nick Swinmurn knew that as soon as his tested assumption had a green light, he would pitch the idea to venture capitalists. If your pilot proves your assumption to be valid, it is important to use that momentum to go into the green light action. Planning this in advance ensures that you won’t lose time and helps your mind focus on what you are piloting for. Sometimes it can be tempting to add extra tests, but you can quickly lose sight of your purpose. You risk adding time and costs to what should be a very focused, quick, and cheap test. By knowing exactly what your next step is, depending if the color of the light is green or red, your pilot stays lean and mean- just like it should.

Red lights do not need to mean stop. Though shelving your idea is one of the possible red light actions, we prefer to shelve it rather than bin it. Having an attic or a cellar with lots of old ideas can at times be a source of inspiration when you are stuck. Though shelving is a legitimate red light action, more often the red light can be used to pinpoint where creativity is needed in your concept. Here you can formulate a new “How Might We?” question and apply elements of the Creation Flow to this part of the idea that is clearly not yet working.

Ever heard of the online shopping app called Tote? Their pilot showed no one was ready for it, but the way people enjoyed putting their favorite items together in groups led to Pinterest. Or the podcast directory Odeo- it also may have eluded your attention, but their group-send SMS function took on a life of its own, known to the world today as Twitter.

History has shown that an entire new direction can come from a pilot. So know what to test and test it against behavior. Be quicker and more low-res than you think, measure and be open-minded, and finally be ready to shelve, add creativity, or even pivot your idea.