Our clients treat org and process changes as complex problems to solve with experiments, or probes. But unlike, say, chemistry or physics experiments, the measurement part of an org change can be tricky to observe and quantify. There’s a real risk of measuring what’s easy to measure instead of what actually validates or invalidates your assumption.
Quite often, the best measure is some flavor of, “Do the people involved think the new way works better?”
So, how do you measure that in a useful way? Borrow a concept from market research.
Net Promoter Score (NPS) is a tool used by companies to measure customer loyalty to a product or company. NPS is based on two key ideas: (1) The opinions of people who feel strongly matter the most, even if they’re not the majority, because they’ll talk about it with others; and (2) Among those, you need more promoters than detractors.
NPS asks people some flavor of, “How likely are you to recommend XYZ?” on a scale from 1-10. And then, based on research into how the results correlate with actual customer loyalty, the responses are divided into three groups: promoters who voted 9 or 10, passives who voted 7 or 8, and detractors who voted 6 or lower. The score in the end is % promoters – % detractors.
The “would you recommend?” format asks people to imagine how they’d talk about the subject of the question with someone else. That’s more useful data than, “How much do you like XYZ?”
The three buckets adjust for people who’d never give a 10, account for people who don’t have strong feelings either way, and remove excessive precision in the scale.
Over the years, we’ve experimented with NPS-inspired 1-question surveys to evaluate org and process changes. For example,
- “How likely are you to recommend Scrum to another team in your org with similar work?”
- “How likely are you to recommend a working agreement to another similar team?”
- “How likely are you to recommend cross-functional teams for similar work?”
Sometimes, we use a “keep it” vs “recommend it” framing if it makes more sense in the context. Like, “How likely are you to recommend we keep doing X?” That flavor can feel more realistic in contexts where there wouldn’t be any actual recommending practices outside the team. But it carries a little less weight because recommending something to someone else involves spending a little bit of personal credibility.
The original NPS includes a formula to get the actual Net Promoter Score. NPS is a number, 71 or 93 or whatever. We don’t usually bother with that in this context; the sample size is too small.
Instead, we’ll survey the group anonymously and use the counts in the three buckets. On a ten person team, knowing there’s one detractor at a 3, three promoters at 9 and 10, and six neutrals all at six or seven is more useful than knowing your NPS is 20.
Depending on the experiment design, validating a hypothesis might require a certain ratio between categories, no or very few detractors, or a certain percentage of promoters. To set those thresholds, ask, “What result would convince someone to change their mind about this hypothesis?”
Changing how your organization works? We help organizations successfully navigate complex changes through practical experiments and feedback. Schedule a free consultation with us to talk about what you’re working on.
Last updated