Not Particularly Surprising

pH scaleHave you heard people telling you their NPS number? (perhaps with their chests puffed out…or maybe somewhat quietly – depending on the score). Further, have they been telling you that they must do all they can to retain or increase it?1

NPS – what’s one of those?

‘Net Promoter Score’, or NPS, is a customer loyalty metric that has become much loved by the management of many (most?) large corporations. It was introduced to the management world by Fred Reichheld2 in his 2003 HBR article titled ‘One number you need to grow’.

So far, so what.

But as most things in ‘modern management‘ medicine, once introduced, NPS took on a life of its own.

Reichheld designed NPS to be rather simple. You just ask a sample of subjects (usually customers3) one question and give them an 11-point scale of 0 to 10 to answer it. And that question?

‘How likely is it that you would recommend our company/product/ service to a friend or a colleague?’

You then take all your responses (which, incidentally, may be rather low) and boil them down into one number. Marvellous…that will be easy to (ab)use!

But, before you grab your calculators, this number isn’t just an arithmetic average of the responses. Oh no, there’s some magic to take you from your survey results to your rather exciting score…and here’s how:

  • A respondent scoring a 9 or 10 is labelled as a ‘Promoter’;
  • A scorer of 0 to 6 is labelled as a ‘Detractor’; and
  • A 7 or 8 is labelled as being ‘Passive’4.

where the sum of all Promoters, Detractors and Passives = the total number of respondents.

NPS calculation.jpgYou then work out the % of your total respondents that are Promoters and Detractors, and subtract one from the other.

You’ll get a number between -100 (they are all Detractors) and +100 (all Promoters), with a zero meaning Detractors and Promoters exactly balance each other out.

And, guess what…a positive score is desirable…and, over the long term, a likely necessity if you want to stay in business.

Okay, so I’ve done the up-front explanatory bit and regular readers of this blog are probably now ready for me to go on and attempt to tear ‘NPS’ apart.

I’m not particularly bothered by the score – it might be of some interest…though exceedingly limited in its usefulness.

Rather, I’m bothered by:

  1. what use it is said to be; and
  2. what use it is put to.

I’ve split my thoughts into two posts. This post deals with the second ‘bother’, and my next one will go back to consider the first.

Qualitative from Quantitative – trying to ‘make a wrong thing righter’

The sane manager, when faced with an NPS score and a ‘strategic objective’ to improve it, wants to move on from the purely quantitative score and ‘get behind it’ – they want to know why a score of x was given.

Reichheld’s NPS method covers this obvious craving by encouraging a second open-ended question requesting the respondent’s reasoning behind the rating just given – a ‘please explain’ comments box of sorts. The logic being that this additional qualitative data can then be provided to operational management for analysis and follow up action(s).

Reichheld’s research might suggest that NPS provides an indicator of ‘customer loyalty’, but…and here’s the key bit…don’t believe it to be a particularly good tool to help you improve your system’s performance.

There are many limitations with attempting to study the reasons for your system’s performance through such a delayed, incomplete and second-hand ‘the horse has bolted’ method such as NPS.

  • Which subjects (e.g. customers) were surveyed?
  • What caused you to survey them?
  • Which subjects chose to respond…and which didn’t?
  • What effort from the respondent is likely to go into explaining their scoring?
  • Does the respondent even know their ‘why’?
  • Can they put their (potentially hidden) feelings into words?…and do they even want to?

If you truly want to understand how your system works and why, so that you can meaningfully and sustainably improve it, wouldn’t it just be soooo much better (and simpler) to jump straight to (properly5) studying the system in operation?!

A lagging indicator vs. Operational measures

One of my very early posts on this blog covered the mad, yet conventional, idea of ‘management by results’ and subsequent posts have delved into ‘cause and effect’ in more detail (e.g. ‘Chain beats Triangle’).

My ‘cause and effect’ post ends with the key point that:

“Customer Purpose (which, by definition, means quality) comes first…which then delivers growth and profitability, and NOT the other way around!”

Now, if you read up on what Reichheld has to say about NPS, he will tell you that it is a leading measure, whereas I argue that it is a lagging one. The difference is because we are coming from opposite ends of the chain:

  • Reichheld appears to be concerned with growth and profitability, and argues that NPS predicts what is going to happen to these two financial measures (I would say in the short term);

  • I am concerned with customer purpose, and an organisation’s capability at delivering against its customers’ needs. This means that I want to know what IS happening, here and now so that I can understand and improve it …which will deliver (for our customers, for the organisation, for its stakeholders) now, and over the long term.

You might read the above and think I am playing with semantics. I think not.

I want operational measures on the actual demands coming in the door, and how my processes are actually working. I want first hand operational knowledge, rather than attempting to reverse engineer this from partial and likely misleading secondary NPS survey evidence.

“Managers learn to examine results, outcomes. This is wrong. The manager’s concern should be with processes….the concentration of a manager should be to make his processes better and better. To do so, he needs information about the performance of the process – the ‘voice of the process’. “ [‘Four Days with Dr Deming’]

Deming’s clear message was ‘focus on the process and the result will come’ and, conversely, you can look at results all you like but you’d be looking in the wrong place!

NPS thinking fits into the ‘remote control’ school of management. Don’t survey and interrogate. ‘Go to the gemba’ (the place where the work occurs).

 “But what about the Lean Start-up Steve?”

Some readers familiar with Eric Ries’ Lean Start-up movement might respond “but Eric advocates the use of customer data!” and yes, he does.

But he isn’t trying to get a score from them, he is trying to deeply engage with a small number of them, understand how they think and behave when experiencing a product or service, and learn from this…and repeat this loop again and again.

This fits with studying demand, where it comes in, and as it flows.

The Lean Startup movement is about observing and reflecting upon what is actually happening at the point of customer interaction, and not about surveying them afterwards.

To close – some wise words

After writing this post I remembered that John Seddon had written something about NPS…so I searched through my book collection to recover what he had to say…and he didn’t disappoint:

“Even though NPS is completely useless in helping service organisations improve, on our first assignment [e.g. as system improvement interventionists] we say nothing about it, because we know the result of redesigning the system will be an immediate jump in the NPS score…and because when this is reported to the board our work gets the directors’ attention.

It makes it easy to see why NPS is a waste of time and money. First, it is what we call a ‘lagging measure’ – as with all customer satisfaction measures, it assesses the result of something done in the past. Since it doesn’t help anyone understand or improve performance in the present, it fails the test of a good measure5 – it can’t help to understand or improve performance.” [Seddon, ‘The Whitehall Effect’]

Seddon goes on to illuminate a clear and pernicious ‘red herring’ triggered by the use of NPS:  the simple question of ‘would you recommend this service to a friend’ mutates to a hunt for the person who delivered the particular instance of service currently under the microscope. Management become “concerned with the behaviour of people delivering the service” as opposed to the system that makes such behaviour highly likely to occur!

I have experience of this exact management behaviour in full flow, with senior management contacting specified members of staff directly (i.e. those who handled the random transaction in question) to congratulate or interrogate/berate them, following the receipt of particularly outstanding6 NPS responses.

This is to focus on the 5% (the people) and ignore the 95% (the system that they are required to operate within). NPS “becomes an attractive device for controlling them”.

Indeed.

The title of this post follows from Seddon’s point that if you focus on studying, understanding and improving the system then, guess what, the NPS will improve – usually markedly. Not Particularly Surprising.

My next post called ‘How good is that one number’ contains the second part of my NPS critique.

Footnotes

1. This post, as usual, comes from having a most excellent conversation with a friend (and ex-colleague) …and she bought me lunch!

I should add that the title image (the pH scale) is a light-hearted satire of the various NPS images I found i.e. smiley, neutral and angry faces arranged on a coloured and numbered scale.

2. Reichheld has written a number of books on customer loyalty, with one of his more recent ones trying to relabel ‘NPS’ from Net Promoter Score to Net Promoter System (of management) …which, to put it mildly, I am not a fan of.

It reminds me of the earlier ‘Balanced Scorecard’ attempting to morph into a system of management. See ‘Slaughtering the Sacred Cow’.

Yet another ‘management idea’ expanding beyond its initial semblance of relevance, in the hands of book sellers and consultants.

Sorry, but that’s how I feel about it.

NPS is linked to the ‘Balanced Scorecard’ in that it provides a metric for the customer ‘quadrant’ of the scorecard …but, as with financial measures, it is still an ‘outcome’ (lagging) measure of an organisation’s people and processes.

3. The original NPS focused on customers, but this has subsequently been expanded to consider other subjects, particularly employees.

4. Being British (i.e. somewhat subdued), I find the labelling of a 7 or 8 score as ‘Passive’ to be hilarious. A score of 7 from me would be positively gushing in praise! What a great example of the variety inherent within customers…and which NPS cannot reveal.

5. For the ‘tests of a good measure, please see an earlier post titled ‘Capability what?’

6. Where ‘outstanding’ means particularly low, as well as high.

Advertisements

Polishing a Turd

turd polishWhen I was growing up, I remember my dad (a Physicist) telling me that it was pointless, and in fact meaningless, to be accurate with an estimate: if you’ve worked out a calculation using a number of assumptions, there’s no point in writing the answer to 3 decimal places! He would say that my ‘accurate’ answer would be wrong because it is misleading. The reader needs to know about the possible range of answers – i.e. about the uncertainty – so that they don’t run off thinking that it is exact.

And so, with that introduction (and flashback to my school days) this post is about the regular comedy surrounding business cases, and detailed up-front planning…and what to do instead.

A seriously important concept to start with:

The Planning fallacy

Human beings who desire something to be ‘a success’ (e.g. many an Executive/ Senior Manager) tend to:

“make decisions based on delusional optimism rather than on a rational weighting of gains, losses, and probabilities. They overestimate benefits and underestimate costs. They spin scenarios of success while overlooking the potential for mistakes and miscalculations. As a result, they pursue initiatives that are unlikely to come in on budget or on time or deliver the expected returns – or even to be completed.” (Daniel Kahneman)

This isn’t calling such individuals ‘bad people’, or even to suggest that their actions are in some way deliberate – it is simply to call out a well-known human irrationality: the planning fallacy.

We all ‘suffer from’, and would be wise to understand and guard against, it.

I’ve worked (or is that wasted time) on many a ‘detailed business case’ over the years. There is an all-too-common pattern….

“Can you just tweak that figure till it looks good…”

models are wrongLet’s say that someone in senior management (we’ll call her Theresa) wants to carry out a major organisational change that (the salesman said) will change the world as we know it!

Theresa needs permission (e.g. from the board) to make a rather large investment decision. The board want certainty as to what will happen if they sign the cheque – there’s the first problem1.

Theresa looks around for someone who can write a great story, including convincing calculations…and finds YOU.

Yep, you are now the lucky ‘spreadsheet jockey’ on this proposed (ahem) ‘transformation programme’.

You gather all sorts of data, but mainly around the following:

  • a ‘base case’ (i.e. where we are now, and what might happen if we took the ‘do nothing’ option);
  • a list of ‘improvements’ that will (supposedly) occur if the board says ‘Yes’;
  • assumptions relating to the potential costs and benefits (including their size and how/when the cash will flow); and
  • some ‘financial extras’ used to wrap up the above (interest rates, currency rates, taxes, the cost of capital…and so on)

You create an initial broad-brush model and then, after gaining feedback from ‘key’ people, you work through a number of drafts – adding in new features and much detail that they insist as being essential.

And voila! We have a beautifully crafted financial model that has a box at the end with ‘the answer’ in it2.

You show the model to Theresa.

Wow, she’s impressed with the work you’ve put in (over many weeks) and how sophisticated the model is…but she doesn’t like this initial answer. She’s disappointed – it’s not what she was looking for.

You go through all of the assumptions together. Theresa has some suggestions:

  • “I reckon the ‘base case’ comparison will be worse than that…let’s tweak it a bit”
  • “Our turnover should go up by more than that…let’s tweak it a bit”
  • “Nah, there won’t be such a negative productivity hit during implementation – the ‘learning curve’ will be much steeper!…let’s tweak it a bit”
  • “We’ll save more money than that…and avoid paying that…let’s tweak it a bit”
  • “Those savings should kick in much earlier than that…let’s tweak it a bit”
  • “We’ll be able to delay those costs a bit more than that…let’s tweak it a bit”

…and, one of my favourites:

“Mmm, the ‘time value of money’3 makes those upfront costs large compared to the benefits coming later…why don’t we extend the model out for another 5 years?”

And, because you designed a nice flexible model, all of the above ‘suggestions’ are relatively easily tweaked to flow through to the magic ‘answer’ cell

“now THAT looks more healthy! The board is going to LOVE this. Gosh, this is going to be such a success”.

Some reflections

John Dewey quote on learningSome (and perhaps all) of the tweaks might have logic to them…but for every assumption being made (supposedly) tighter:

  • one, or many, of the basic assumptions might be spectacularly wrong;
  • plenty of the assumptions are being (conveniently4) ignored for tweaking…and could equally be ‘tightened’ in the other direction (i.e. making the business case look far worse); and
  • there are many assumptions that are completely missing…because you simply don’t know about them….yet…or don’t want to know about them.

With any and every tweak made, nothing has actually changed: Nothing has been learned about what can and will actually occur. You have been ‘polishing a turd’…but, sadly, that’s not how those around you see it. Your model presents a highly convincing and desirable story.

Going back, your first high-level draft model was probably more useful! It left many ‘as-yet-unknowns’, it contained ranges of outcomes, it provided food-for-thought rather than delusional certainty.

We should reflect that “adding more upfront planning…tends to make the eventual outcome worse, not better” (Lean Enterprise). The more detailed you get then the more reliant you become on those assumptions.

The repercussions

Theresa gains approval from the board for her grand plan and now cascades the (ahem) ‘realisation of benefits’ down to her direct reports…who protest that the desired outcomes are optimistic at best, and sheer madness at worst (though they hold their tongues on this last bit).

Some of the assumptions have already proven to be incorrect – as should be expected – but it’s too late: the board approved it.

The plan is baked into cascaded KPIs…and everyone retreats into their silos, to force their part through regardless of the harm being caused.

But here’s the thing:

“Whether the project ‘succeeds’ according to [the original plan] is irrelevant and insignificant when compared to whether we actually created value for customers and for our organisations.” (Lean Enterprise)

The wider point…and what to do instead

validated learningIt’s not just financial models within business cases – it is ‘detailed up-front’ planning in general: the idea that we should create a highly detailed plan before making a decision (usually by hierarchical committee) as to whether to proceed on a major investment.

The Lean Start-up movement, led by Eric Ries, makes a great case for a totally different way of thinking:

  • assumptions aren’t true! (it seems daft to be writing that…but the existence of the planning fallacy requires me to do so);
  • we should test big assumptions as quickly as possible;
  • such testing can be done through small scale experimentation (which doesn’t require huge investment) and subsequent (open-minded) reflection;
  • we will learn important things…which we did not (and probably could not) predict through detailed up-front planning. This is a seriously good thing – we can save much time, money and pain, and create real customer value;
  • we may (and often will) find a huge flaw in our original thinking…which will enable us to ‘pivot’5 to some new hypothesis, and re-orientate us towards our customer purpose.

The big idea to get across is what has been termed ‘validated learning’.

Learning comes from actually trying things out on, and gaining direct feedback from, the end customers (or patients, citizens, employees etc.), rather than relying on our opinions about them.

Validated is about demonstrating what the customer (or patient, citizen, employee etc.) actually does (or doesn’t do), not what they say they would do when asked (i.e. from external market research or internal survey). It is to observe and measure real behaviours, rather than analyse responses to hypothetical questions.

…and to do the above rapidly by experimenting with ‘minimum viable products’ (MVPs).

Delay (whilst writing a beautiful document, getting it approved, and then building a seemingly golden ‘solution’) prevents the necessary feedback from getting through.

Caveat: Many an organisation has read ‘The Lean Startup’ book (or employed a consultant who has) and is using the above logic merely at the start of their legacy ‘investment decision’ process…but, through grafting new labels (such as Lean) onto old methods and retaining central hierarchical approval committees, their process remains ostensibly the same.

You don’t do validated learning merely at the start of an investment process – you re-imagine what ‘making investments’ means!

“It’s moving leaders from playing Caesar with their thumbs up and down on every idea to – instead – putting in the culture and the systems so that teams can move and innovate at the speed of the experimentation system.”

“The focus of each team is iterating with customers as rapidly as possible, running experiments, and then using validated learning to make real-time investment decisions about what to work on.” (Eric Ries)

 Notice that it is the team that is making the investment decisions as they go along. They are not deferring to some higher body for ‘permission’. This is made possible when:

  • the purpose of the team is clear and meaningful (i.e. based around a service or value stream);
  • they have meaningful capability measures to work with (i.e. truly knowing how they are doing against their purpose); and
  • all extrinsic motivators have been removed…so that they can focus, collaborate and gain a real sense of worth in their collective work.

Nothing new here

You might read the above and shout out:

  • “but this is just the scientific method”; or
  • “it’s yet another re-writing of the ‘Plan – Do – Study – Act’6 way of working”

…and you’d be right.

Eric Ries’ thinking came about directly from his studying of Deming, Toyota etc. and then applying the learning to his world of entrepreneurship – to become effective when investing time and money.

His book, ‘The Lean Startup’, and the ‘validated learning’ concept are an excellent addition to the existing body of work on experimentation towards purpose.

Footnotes

1. We should never present a seemingly certain picture to a board (or merely hide the caveats in footnotes)…and we should coach them to be suspicious if they see one.

2. For the financially aware: this will likely be a net present value (NPV) figure using a cost of capital (WACC) provided by the finance department, or some financial governance body.

3. The ‘time value of money’ reflects the fact that $1 now is worth more to you than $1 in a year’s time.

4. Conveniently doesn’t mean intentionally or maliciously – it can just be that lovely planning fallacy at work.

5. Pivot: This word has become trendy in many a management conversation but I think that its original (i.e.intended) meaning is excellent (as used by Eric Ries, and his mentor Steve Blank).

Eric Ries defines a pivot as “a structured course correction designed to test a new fundamental hypothesis….”

6. PDSA: Popularised by Deming, who learned it from his mentor, Walter Shewhart. A method of iterative experimentation towards your purpose, where the path is discovered as you go, rather than attempted to be planned at the start. Note that, whilst the first step is ‘Plan’, this DOESN’T mean detailed up-front planning of an answer – it simply means properly planning the next experiment (e.g. what you are going to do, how you are going to conduct it, and how you are going to meaningfully measure it).