Definition of Wizard of Oz Testing in Eric Ries’ book “The Lean Startup”: “where customers believe they are interacting with the actual product, but behind the scenes human beings are doing the work”
The biggest pro is that you can move quickly.
The biggest con is that you still have to build something eventually.
I wrote about this a bit in another post (What are the best-kept secrets about startups?) but I call it Man/Woman behind the machine/curtain (yes, from the Wizard of Oz):
Man Behind the Machine (aka artificial artificial intelligence): You hypothesize that you can grow your customer base by give existing users rewards for tweeting at your Twitter account when they achieve the next level? (or whatever, just an example) Go manually give rewards to the users instead building a complex feature with many moving parts (back end integration, API integration, cron jobs…etc). Sure, it’s hard work, but it’s smarter than building a feature based on your gut.
Pros:
- You can move very quickly. You can typically make changes, get feedback, and learn from your customers in minutes instead of weeks or months.
- Inexpensive. Time is almost always your most expensive input here…unless you’re in college (when your time is typically close to free).
- You can find real (paying) customers. This is a great way to find early customers. You might even stumble on an investor!
Cons:
- It’s not real. You still have to build something eventually.
- Mixed signals. It’s pretty easy to get a lot of mixed signals with this (and most rapid prototyping) method. That’s okay, your focus should be on a single value proposition in the early days anyway. Learn to listen well and you’ll eventually learn what feedback to take and what feedback to throw away.
- Typically not polished. It can sometimes be difficult to fake a product or service and still make it look polished. This isn’t always a bad thing as it’s easier for people to give you high-level feedback if you don’t have what looks like polished product. It’s also easier for you to take feedback if you don’t have a polished product.
* The wizard getting caught with his pants down… (Wizard of Oz, 1939)
Side Note:
The funny thing is that sometimes, the man behind the machine becomes the product. Or that’s what happened to a company called CardMunch at least. CardMunch is a service that lets you take a photo of a business card from your iPhone and then magically, a few hours later, the contact info has been transcribed into a contact that you can save to your friend (of connect with after LinkedIn acquired the company in early 2011 for a (rumored) few million)
The Bottom Line: This is a great technique to get feedback on your product/service idea before you actually build it. The downside is that you still have to build it and that you often get confusing or mixed feedback. Test users can also be upset when they (inevitably) realize that the product is fake! But if they’re really upset, then they might make a great customer one day.
Have you used Wizard of Oz Testing? Tell me in the comments.