Net Promoter, Reliability "Oversold"

BRUCE_COOIL_1In 2003, noted loyalty consultant Frederick Reichheld introduced the customer service loyalty metric Net Promoter, proclaiming it to be “the single most reliable indicator” of a firm’s growth compared to other measures of customer satisfaction and retention. “It is the one number you need to grow,” he asserted.

PHOTO: Bruce Cooil, Professor of Management (Statistics)

Since then, many companies – including some of the world’s largest corporations such as GE, American Express, Schwab, Intuit and Allianz – have adopted Net Promoter to predict the future revenue growth of their businesses, even citing its findings for analysts and investors. The metric is based on the principle that growth rates of firms within virtually any industry can be accurately predicted based on a “score” tabulated from whether customers would recommend the company to someone else.

Despite these bold, provocative claims for Net Promoter, the metric had not been subjected to rigorous scientific scrutiny and peer review nor had researchers attempted to replicate Mr. Reichheld’s research methodology. Was it really the superior, high-powered tool it was cracked up to be?  According to new research, Net Promoter falls far short of these claims.

The Metric is Put to the Test

To test the Net Promoter concept, a team of researchers from Vanderbilt Owen Graduate School of Management, Norwegian School of Management, Koç University and the global market research firm Ipsos Loyalty conducted a comprehensive study of the metric’s effectiveness. According to Owen Professor Bruce Cooil, a co-author of the study and statistics expert, he and his colleagues “found no evidence to support Net Promoter’s claim of superiority over other commonly used satisfaction/loyalty metrics.” 

The study’s findings (forthcoming in the Journal of Marketing) have significant implications for companies that employ the widely used metric. “The consequences,” the study reported, “are the potential misallocation of resources as a function of erroneous strategies guided by Net Promoter on firm performance, company value, and shareholder wealth.”

Cooil and other academic and industry experts in loyalty, statistical analysis and modeling developed two experiments that comprised the first cross-industry, longitudinal examination of the association between Net Promoter and company revenue growth.  Both analyses sought to replicate Reichheld’s own research methodology.

In the first experiment, the researchers used a comparable dataset representing over 15,000 consumer interviews and 21 firms across five industries tracked by the Norwegian Customer Satisfaction Barometer (NCSB) to calculate each firm’s Net Promoter score. The NCSB – produced and sponsored by the Norwegian School of Management – tracks trends in customer satisfaction for the Norwegian economy, and is based on national satisfaction indices produced in Sweden and the U.S.  Also examined were industry-level correlations between the firms’ rates of growth (i.e., relative change in revenue) and Net Promoter, as well as other commonly used satisfaction/loyalty metrics for the various time periods available.

The results from this first experiment showed there was no significant correlation between Net Promoter – or, for that matter, any of the satisfaction metrics – and relative changes in firm revenues in any of the industries analyzed, including banking, gas stations, home furnishing retailers, security systems and transportation.

The ACSI Holds Up

For the second experiment, researchers replicated a subset of Reichheld’s data for three different U.S. industries – airlines, personal computers and life insurance – which he cites as exemplars of Net Promoter’s prowess in his best-selling book, The Ultimate Question.  “We painstakingly replicated the scatter-plot diagrams for these three industries so that we could directly compare the relative performance of Net Promoter with that of the American Customer Satisfaction Index (ACSI),” explained Prof. Cooil.  What made this comparison so interesting was Reichheld’s assertion in his book that ACSI – a national economic indicator of customer evaluations of the quality of products and services available to consumers in the U.S. – had no correlation whatsoever with company revenue growth.

The study’s findings?  To the surprise of researchers, their apples-to-apples comparison using Reichheld’s own data showed “strikingly similar predictive results” between Net Promoter and ACSI.  Even more startling, in two of the three U.S. industries the ACSI measure actually showed a stronger relationship to revenue growth than Net Promoter. “We had expected dramatically different results,” acknowledged Prof. Cooil, “given that these industries were presented as prime examples of the relationship between Net Promoter and growth, and the fact that ACSI was singled out as having no connection to a firm’s growth.”

Assumptions are Potentially Erroneous

“The clear implication is that managers have adopted a metric for tracking growth based upon the belief that solid science underpinned the findings, and that it was superior to other metrics,” the study reported.  Based on its comprehensive work, however, the research team warned that “the presumptions of these managers appear to be erroneous,” and that potential “misallocation of resources” could result from strategies driven by Net Promoter.

“There are many factors that influence why people buy,” noted the study’s lead author Tim Keiningham, senior vice president at Ipsos Loyalty and a 1989 Owen grad. “For example, a more accurate metric for loyalty would be to take into account customers’ attitudes toward a brand and their actual buying behavior. There is no silver bullet for predicting growth based on customer loyalty – despite Net Promoter’s claims to the contrary.”

Now that Net Promoter has been shown to be over-promised, skeptics can be expected to discount efforts to improve customer loyalty in favor of more “tangible” management efforts, according to Keiningham, as memories of feeling misled tend to last.  “Managers have been consistently frustrated in their attempts to build customer loyalty, as measuring and managing it to achieve growth hasn’t proven an easy task,” he said. “On the positive side, Net Promoter has forced us to examine more closely how customer loyalty links to firm performance. Unfortunately a quick rush to adopt new big ideas often leads to remorse.” These researchers believe that the pursuit of customer loyalty can be a highly profitable strategy, but that there is no simple solution to linking loyalty to growth; if it were easy, everyone would be doing it.

Send comments to Tim.Keiningham@Ipsos-NA.com and bruce.cooil@owen.vanderbilt.edu

Published 2/15/07 in OWENintelligence
© Vanderbilt Owen Graduate School of Management