危 wēi  danger

Great businesses continually improve, prototyping new offers and services before rolling them out to the relevant customer base.  But, success criteria for innovations are often vague and ‘success’ is therefore highly subjective, particularly when innovation teams only prototype to small groups of customers.  The alternative, rolling out to large numbers of customers immediately is too risky though?  And anyway, given how quickly markets change how could the team say with confidence that any change in performance was attributable to the change in strategy?

机 jī opportunity

In my previous post about Netflix I sang the praises of the company placing a top-level strategy summary in the public domain – you can see it here.  Slide 20 particularly caught my eye, a brief slide on customer satisfaction:

I love the idea of breaking customer bases into equal sized groups, allocating each to different employees or teams and then giving them free reign to attempt to improve their group’s experience.  I imagine that this enables more offers to be prototyped and creates a competition internally to deliver quantifiable results.  I also admire the size of these groups in this case – many companies would be tempted to limit the trial to a handful of customers at most, going with groups of 10,000 enables Netflix to really quantify the benefit and assess relative performance.

The approach establishes what would be called a ‘control set’ in a science experiment and encourages agreement in terms of what success metrics might be in advance.  In fact, my colleague Elizabeth said in a talk recently that she likes to approach piloting using the scientific method:

set a hypothesis -> agree variables -> agree metrics -> agree duration -> feedback and conclusions

That makes sense, and contrary to many people’s perceptions it doesn’t need to stifle creativity. After all, the development of new approaches to pilot and the spotting of interesting new patterns in any results depend on creativity.

How About…

• Taking a scientific approach to piloting new offers?
• Thinking how you might divide your business into comparable units for benchmarking?
• Tasking teams with innovating the offer to different units, then rolling out the most effective new approaches?

—