Book Review: Inspired by Marty Cagan

This book is a collection of blog posts cobbled together into a book that has no real thesis and often contradicts itself.  This contradiction is dangerous because it emphasizes putting creative control of the product in the hands of those with titles containing "UX" and "Product" and devalues the creative and innovative capabilities of the rest of the orginization.  Due to this centralization of roles and internally inconsistent nature, it could be treated like a religious text where the high priests of Product can cherry-pick whatever ideas they want to build any product they imagine so long as they can get customer to say it makes them happy.

The advice in this book isn't bad, per se, it just isn't great.  The author, Marty Cagan, takes the waterfall model of product design and extends it into product, defining a dozen or so different specific job titles and outlining when those titles are necessary in what phases of a project.  Unfortunately, in a highly creative and collaborative environment, this advice could potentially slow down development of a product and stifle the creativity of people who can do more than exist in their pre-defined roles.  Worse, he doesn't appear to understand the creativity of engineering and what kind of innovation takes place in the technical side of the product development lifecycle.

Cagan provides some excellent advice.  Namely:

• Product owner sees the product from start to finish and participates in all user testing.
• The Engineering Lead is involved in the entire process to provide information about what is possible and how hard it is.
• Focus groups are bad because customers don't know what they want.
• Sometimes the customer asks for x and the best thing for the business is to give them y.
• Don't charge customers who are in the beta.
• Don't lead the witness.
• Don't confuse customer requirements with product requirements.

I've personally seen the benefit of following these rules and have seen that NOT following these rules can lead to direct negative consequences.  

Let's take a look at the closest thing I could find to a short thesis in the way of a summary of how to better run a startup.  It's basically a copy/past of this blog post into the book.

Here’s how it typically works. Someone with an idea gets some seed funding, and the first thing he does is hire some engineers to start building something. The founder will have some definite ideas on what he wants, and he’ll typically act as product manager and often product designer, and the engineering team will then go from there. The company is typically operating in “stealth mode” so there’s little customer interaction. It takes much longer for the engineering team to build something that originally thought because the requirements and the design are being figured out on the fly.
After 6 months or so, the engineers have things in sort of an alpha or beta state, and that’s when they first show the product around. Things rarely go well in this first viewing, and the team starts scrambling. The run-rate is high because there’s now an engineering team building this thing as fast as they can, so the money is running out, and the product isn’t there. Maybe the company gets more funding and a chance to get the product right, but often they don’t. Many startups try to get more time by outsourcing the engineering to a low-cost off-shore firm, but it’s essentially the same process with the same problems.
Here’s a very different approach to new product creation, one that costs dramatically less and is much more likely to yield the results you want. The founder hires a product manager, a product designer, and a prototyper. Sometimes the designer can also serve as prototyper, and sometimes the founder can serve as the product manager, but one way or another, you have these three functions lined up - product management, product design, and prototyping – and the team starts a process of very rapid product design and iteration.
I describe this process in detail in “How To Write a Good PRD,” but there are two keys: 1) the idea is to create a high-fidelity prototype that mimics the eventual user experience – it’s just fine if the back-end processing and data is all fake; and 2) you need to validate this product design with real target users.
In this model, it is normal to create literally dozens of versions of the prototype – it will evolve daily, sometimes with minor refinements and sometimes with very significant changes. But the point is that with each iteration you are getting closer to identifying a winning product. This process typically takes between 3 weeks and 2 months, but at the end of the process, you have a) identified a product that you have validated with the target market; b) a very rich prototype that serves as a living spec for the engineering team to build from; and c) you now understand at a much greater degree what you’re getting into and what you’ll need to do to succeed.
Now when you bring on an engineering team, they’ll start off with a tremendous advantage – a clear understanding of the product they need to build and a stable spec – and you will find that the team can produce a quality implementation much faster than they would otherwise.
This model of prototype-based product experimentation is increasingly becoming the norm in the manufacturing world, but for some reason this hasn’t taken off in software. I think we’re such an engineering-driven culture that we just naturally start there. But any startup has to realize that everything starts with the right product – so the first order of business is to figure out what that is before burning through $500K or more in seed funding.
I believe this model applies beyond startups to much larger companies as well. The difference is that bigger companies are generally able to underwrite the several iterations it takes to get to a useful product, but startups often can’t. But there’s no reason for the inefficiencies that larger companies regularly endure.

Allow me to nitpick this idea.

1. Stealth mode is bad.  Cagan and I agree that stealth mode deprives the team of feedback, but another solution to this problem is to just not do stealth mode.
2. It takes longer to develop than originally thought. This is a widespread problem related to an engineering team's poor estimation skills.  Skills can be improved.  A typical Agile engineering team is bad at this for the first few iterations of its existence but can quickly refine those estimates as it retrospects on those estimates after each iteration.  My engineering team is typically off estimate by roughly 10% or less, and is quick to refine estimates as we discover new things.  Sometimes we discover an open source tool that lets us re-use rather than rebuild and the estimate drops, and sometimes we learn from our users that we are building the wrong thing and must adjust accordingly.  But, I argue that the total time to market is faster this way than prototyping without engineers first and then bringing in engineers - something I elaborate more on shortly.
3. Six months to the first beta.  Cagan and I agree that spending six months without any user feedback is bad.  But, another solution is to just develop the most important thing first and start the user testing against the actual product rather than a prototype.  My engineering team typically has a beta of the core functionality of a new product ready for user testing after two to four weeks.  I'll get into why this is optimal shortly.
4. High Run Rate.  I suspect that the run-rate wouldn't be that much higher to prototype using real engineers building a real product rather than paying for prototypers to build something that will be thrown out when the engineers show up and tell you the prototype needs to be re-designed because it didn't take into account technical costs.  In fact, I would bet that the total cost to get to market with Cagan's solution to this problem would be much higher than if a real engineer worked on the prototype because UX innovation would be taking place in parallel with engineering innovation rather than delaying the engineering innovation until after the UX design was considered "done" (something that's not totally possible until the engineering innovation has taken place anyway).
5. High Fidelity Prototype.  In the Agile world it would not be easy to create a high-fidelity prototype without consulting with the engineers about what is technically possible for the first version.  At best, this prototype is a close approximation to the perfect world given unlimited engineering budget.  Remember, given a complete waterfall-style specification, you're going to spend far more than 80% of your engineering effort on far less than 20% of the product and without engineers present for the prototyping phase, small changes to that 20% could have a dramatic effect on that 80% of the budget.  I'm not arguing against the high-fidelity prototype.  I'm arguing that the engineers should be building that prototype rapidly using test-driven development, and with very short (sub-weekly) iterations and frequent (sub-weekly) releases.
6. 3 weeks to two months.  In the solution I propose, after those 3-8 weeks, you not only have yourself a high-fidelity prototype, but you have yourself a few engineers who are intimately aware of the customer's needs, and a suite of tests that document the specification of the user-facing part of the application.  Swapping out the fake backend with something that contains security and persistence is simply the next phase of the already-in-progresss project.
7. Clear understanding and a stable spec.  If this were a waterfall-style shop, Cagan makes a good point.  If you want the engineers to passively consume the specification and build a system that meets that specification, Cagan proposes a great solution to the outlined problem.  But, if this system were being built by true software craftsmen, then they would look at the specification and begin to suggest small changes that could drastically reduce the cost of the system.  Those proposals would need to be run through the same UX design process and suddenly that 3-8 weeks spent in pre-planning starts not to be so valuable.  What's missing is that the UX and Product team do not have a clear understanding of what makes sense to build technically until the engineers receive and respond to that specification.  And, when that starts to happen, that specification will no longer be stable.
8. Prototype-based experimentation hasn't taken off in software.  The fact that the author has no idea why the idea hasn't taken off in software strikes directly at the heart of what is wrong with the thesis of this book.  The author has little to no understanding of the state of the art in software engineering processes.  Lean Startup, Scrum, Kanban, and the host of other Agile philosophies have their roots in the manufacturing world and describe themselves as ways to create small "experiments" or continuous evolution of a prototype to create software.  Software engineering is exactly the thing Cagan is proposing we do, but he's proposing we do it without the software engineers.  I suspect this has something to do with the companies he has worked for: HP, AOL, Netscape, eBay - companies that aren't known for their ability to rapidly innovate through software engineering.  Earlier in the book he advises not to make the actual product the prototype (copied from this blog post) because costs to build test and deploy are too high, but another solution to this problem is to crush those costs down so this is no longer a barrier - something that we have done fairly well at the company I am working for now.

I think #7 and #8 are the most important problems with Cagan's thesis.  First, it's unfounded and inaccurate, and second, it's a hypothesis that the author himself hasn't appeared to have tested (or at least he doesn't state he has any experience testing this theory).  Finally, it appears to contradict advice he gave earlier in the book and taken from a different blog post:

Maybe the biggest missed opportunity in [old-school product management], is the fact that engineering gets brought in way too late. We say if you’re just using your engineers to code, you’re only getting about half their value.  The little secret in product is that engineers are typically the best single source of innovation, yet they are not even invited to the party in this process.

Which I whole-hardheartedly agree with, speaking as a self-promoting engineer.

Cagan measures the success of a product based on customer happiness, but he never shows how to measure happiness.  In fact, he never shows how to measure anything.  The word data-driven doesn't appear in the book once.  Granted, a very important way to measure the success of a product is to simply ask customers if they are happy, but even Cagan conceeds that simply asking customers isn't the best way to get requirements from them.  Early on in Google's life, they tried to sell their search technology to Yahoo.  Yahoo didn't want to pay for it because search was considered a solved problem and users already loved Yahoo.  Google had measurably better technology, but no users loved Google, so Yahoo passed.  Upon launching, Google didn't simply ask users if Google Search made them happy.  It measured every time people navigated to the second page of search results and considered that click a "failure" of the search.  Google set a goal to reduce the average number of "next" clicks per search every year.  This allowed Google to innovate in a scientifically measurable way and stay ahead of the competition even though users already loved Yahoo.  True innovation isn't simply putting a fresh coat of paint on a mediocre product or giving the user a horse that's easier to ride.  True innovation happens when creative people solve an important problem using a solution everybody else thinks is impossible or dumb.  The best innovation, rarely developed through UX, is disruptive.  The first automobile was considered slow, difficult to operate, and expensive to maintain compared to the horse.

Cagan's approach is a great way to take a waterfall organization building mediocre products to another level.  But, he isolates the creative process to the few people with the right job title and gives them the tools of subjective psychology-driven decision-making.  The customer may end up loving your product, but it won't necessarily be the best product you could have built.