Does NPS Deserve All This Praise?
Does Net Promoter Score deserve such recognition?
Drifting blindly behind trends is both wrong and misleading. I recently read many articles criticizing the effectiveness of the famous loyalty metric, Net Promoter Score (NPS), and questioning its credibility. I had previously written an article about this metric, which you can find here, without discussing its drawbacks. So I decided to practice critical thinking and analyze, from a neutral standpoint, whether this famous metric truly deserves all this praise and limelight. I came across many articles on blogs concerned with marketing research and its applications, and in this article I will summarize my experience.
It seems the inventors of this metric succeeded brilliantly in marketing it, relying on the claim that there is a direct correlation between the metric and an organization’s growth and profitability. The use of this metric spread like wildfire among many leading institutions and companies around the world a few years ago. The question that arises is this: did everyone who used this metric succeed in proving the relationship between the metric’s results and their own profits? And let us assume that the inventors of this metric did in fact prove that relationship in some companies — does that mean the relationship necessarily exists in all companies and all industries? The answer is: certainly not. Generalization is a flawed rule, and the burden of proof lies with whoever makes the claim.
Several reasons contributed to the spread of this metric, including, but not limited to: the ease of using and applying it; the fact that its inventors granted permission to use it without financial obligation to the rights holders; and that it is among the least costly metrics, since it involves a single question, compared with other models that require asking several questions — which means a longer survey and a higher cost. Human nature wants to achieve the greatest gains at the lowest cost and as quickly as possible, and so this metric was the answer. The article published in Harvard Business Review also lent strong credibility to this measure, even though the article’s author is himself the creator of the metric. The article’s title carries within it a blind confidence: “The One Number You Need to Grow” — by “the number” he means the output of this metric. And when I asked many users here in the Kingdom of Saudi Arabia, it became clear to me that their decision to use it was based on others using it, and that some of them were under the impression that the metric was validated by academic research, though they had never verified that themselves.
The Problems with Net Promoter Score
The First Problem
The problem of proving the relationship between the metric’s results and expected profits or expected growth. Despite the many cases presented to prove this, there are, on the other hand, many companies that used it and proved exactly the opposite — that no relationship exists at all — when this was tested using certain statistical tools to study the relationship.
The Second Problem
The problem of differing cultures. Some peoples will never answer with a 10, saying that perfection belongs to God alone, leaving us with only the 9 that enters into the metric’s equation — and this in itself is a considerable problem. Not to mention that some cultures, especially in Arab countries, exaggerate in their answers and do not give responses that reflect reality. A customer might sympathize with you and tell you, “Of course I’ll recommend you,” giving you the highest point on the scale — but in reality you would find them to be the complete opposite. This is why this metric depends on attitudes, and these attitudes do not necessarily reflect customers’ actual future behaviors, especially those behaviors tied to customer loyalty, such as: customers defecting to competitors, increased purchasing, and so on. Add to that the metric’s inability to distinguish between satisfied and dissatisfied customers — a customer who, according to the metric, promotes positively is not necessarily satisfied, and vice versa.
The Third Problem
The metric’s lack of ability to identify the causes of declining or rising loyalty. Despite asking an open-ended question after the main question, that is not enough to provide the detailed answers that help decision-makers identify points for improvement and prioritize them; nor does the measure help provide a clear analysis of competitors. And when analyzing the answers to the open-ended question, we find it difficult to identify the strengths or positives that positive promoters will speak about to their relatives or friends.
The Fourth Problem
The metric’s result by itself adds no value for decision-makers in the absence of benchmark metrics comparing it against competitors’ performance — and even when the latter are available, dealing with them is very sensitive for several reasons we discussed in a previous article.
The Fifth Problem
The metric relies primarily on (what is passed along among consumers about a particular product or service), which is technically referred to in English as:
(WOM) Word of mouth
We note that there are some products or services to which this principle does not apply at all, and about which people do not pass along any negative or positive talk — while acknowledging, on the other hand, that there are products and services that people talk about a great deal. It is worth noting that what consumers pass along is influenced by several factors, not only by the performance or level of service provided. For example, successful marketing campaigns affect this even if the level of service has not changed during that period. And although word of mouth has a tangible role in raising sales, the process of actually measuring it differs from measuring the likelihood of it occurring by asking customers about it using the metric we are discussing. By actual measurement we mean, for instance, monitoring social networks, analyzing their content, and classifying that content to study the volume and quality of what is passed along among customers or potential consumers.
The Sixth Problem
There is also a problem related to the design of the measure and the design of the question itself, which is considered one of those questions that assume the respondent will recommend — and from the standpoint of survey design science, this is wrong, and using this type of question is not advised. There is no benefit in detailing this in this article, as the discussion of this aspect would go on at length.
The Seventh Problem
Looking at the equation, it may produce numbers that are difficult to interpret. Consider the following example of two competing companies:
Company A: scored an NPS of 20%, with no detractors at all (those who answered between 0 and 6), while it had 80% who answered between points 7 and 8, and 20% who answered between points 9 and 10.
Company B, on the other hand, scored the same result of 20%, with a detractor share (those who answered between points 0 and 6) of 40%, 0% who answered between points 7 and 8, and 60% who answered between points 9 and 10.
Both companies scored the same result of 20%. What does that mean? You may not be able to interpret it as anything. Could both obtain the same market share? Which one will pull ahead of the other, and why? Unfortunately, you cannot interpret that in any scientific way.
The Eighth Problem
The idea of asking a single general question is in itself a misleading idea. When I think about a particular company at the overall level, I personally might be among its most negative detractors — but if you asked me to talk about the details, this company that I dislike might have a particular product that I prefer over all the products of its competitors. That means users of this metric will be forced to ask this question about all their main products that are offered by other competitors, and they must also ask about all the competitors’ products in order to reach a result that can be compared. This means a long survey, which strips this metric of its advantage related to low cost and a short questionnaire.
The Ninth Problem
In business research (B2B), using this metric can be highly misleading. For example, let us assume that a company sold an information system to another company, and the selling company posed this question to the users of their system. You might be optimistic about their answers — but thinking realistically, these users do not hold the purchasing decision; rather, the purchasing decision rests with a very small group of people in each organization. If you posed the question to those people, you might be shocked by the stark contradiction between their results and the results of their colleagues who use the service. And even if we assume, for argument’s sake, that the users will indeed promote positively to many friends and colleagues at their own level, that will not have as great an impact as if the decision-makers themselves promoted to friends or relatives at their own level.
In 2007, Tim Keiningham, a senior vice president of the loyalty division at the global company Ipsos, along with a group of researchers, conducted a study on the measure and on the validity of the claim that it is the best number to rely on for predicting growth or profits. But the results of his research concluded that the relationship between the metric and a company’s ability to grow cannot be proven in any way. After this skeptical research, many researchers and specialists began conducting experiments and published their conclusions, which were close to the conclusions of the researchers at Ipsos.
You will find a summary of this study at the following site:
The people behind the American Customer Satisfaction Index also concluded, through certain experiments, that their index proved its relationship to be stronger than the one claimed for Net Promoter Score.
At the end of my article, I want to point out that there are many other metrics that have proven, through experience, the relationship of their results to companies’ growth and profit indicators — and perhaps these metrics did not receive the limelight that Net Promoter Score received. Some experts have advised using several metrics at once and then arriving at the best metric to rely on. The idea I want to convey to readers is that there is no metric among these that is 100% right or wrong; rather, you should start with the metric whose proponents and testers have proven that relationship. In the end, investing in applying several metrics to determine which is most suitable for you, each one separately, is a waste of money, time, and effort. Speaking of the most important of those metrics, around which much research has been conducted to prove their ability to help decision-makers predict growth and profit rates: the American Customer Satisfaction Index, which I discussed in a previous article here, as well as one of the most recent of these metrics, the Customer Effort Score (CES).
This metric, which has begun to replace Net Promoter Score in many organizations, will be the subject of my next article, where I will try to track the pros and cons of this metric and also delve into the critics’ opinions to learn how worthwhile it is and whether its claims are valid or not. And may God guide us to the right path.
It would be my honor and pleasure to have you visit my other articles here.
