In Small-Sample Research - How to Reach the Fairest Satisfaction Score

Introduction

It was a long journey that I went on to answer this question. The problem lies in the methodology that should be used to obtain the most accurate measurement of satisfaction for customers whose sample is small. I encountered this issue when I joined Elm Company, which had many services with very few customers (fewer than ten). The statistical formulas and margins of error don’t apply here at all because of the small sample size. The corporate-performance-management department wanted indicators to play their own role in evaluating the relevant departments.

I began my journey by reading a book titled Researching Customer Satisfaction and Loyalty by Paul Szwarc. I didn’t find a satisfying answer to this question after reading it, so I reached out to the author to share my personal experience and my search for a clear answer. He helped me and offered his professional opinion. Several exchanges took place between us in which I explained the nature of the work. After reaching a conclusion together, I sent the question and the conclusion to three veterans working in marketing research here in Saudi Arabia to confirm the conclusion we’d reached. Last but not least, I posted the question and conclusion in the most popular marketing research groups on LinkedIn for peace of mind (this is one of them).

The product of this journey I now place in your hands, in brief. To begin, I’d like to clarify that the customer I’m referring to in this discussion is the stakeholder — not the end beneficiary of the service. The case study related to a company providing electronic services, so the customers of this service were of two types as I mentioned: end beneficiaries (numerous) and stakeholders at the parties commissioning the programming and preparation of these services — and the latter are the ones concerned in the problem of this discussion. I clarify that this discussion isn’t about questionnaire design or analyzing its results — only about answering the question below.

The Question

When conducting a customer-satisfaction study on a small sample, do we rely on the average satisfaction from the direct question at the start of the questionnaire, or do we calculate average satisfaction across several aspects of the service?

The Conclusion

There are two main methods that can be used to obtain a main satisfaction index in this type of study — whose flaw is the small number of targeted respondents. To begin, let’s clarify that the margin of error in these samples is very high, so some specialized books recommend that if you want to obtain results representing the target sample where the number of targets is fewer than 30 individuals, you must interview at least 80% of those targets.

Back to the two methods that can be used:

Method 1

The direct satisfaction question typically placed as the first question in the questionnaire, asked as follows: “What is your overall level of satisfaction with company …… in project/service ……? Please use the 5-point scale below…” Placing the question at the start of the questionnaire will give you the overall impression about the service whose satisfaction is being measured. According to experts, this is a viable, fair method for providing an accurate customer satisfaction indicator. This method is called Overall Satisfaction (Direct Question).

Using this method helps obtain the customer’s initial, spontaneous impression — unbiased by any influence. But this method is a double-edged sword: if there has been a recent problem, that problem greatly affects respondents’ initial impression — and an injustice may occur by overlooking performance levels in other aspects of the service.

Method 2

I find this method more appropriate and fairer, and I try to use it in most cases. The idea: adopt a weighted satisfaction index. It can be obtained by measuring satisfaction levels across several aspects related to the service, then asking another question to measure the degree of importance of each of these aspects, and finally projecting importance onto satisfaction to obtain a weighted, fair satisfaction level. If one aspect’s satisfaction is very high but its importance is very low, the impact of this aspect on the final index becomes small — and vice versa.

The advantages of using this method are that it is more useful from a strategic perspective. Knowing satisfaction and importance levels for all aspects related to the service gives an overall picture of the performance delivered and helps decision-makers draw priorities to raise future performance levels. The weighted satisfaction index is also more stable over time compared to the index extracted with the first method, which fluctuates over time for the reasons mentioned related to bias toward the most recent experiences. However, the researcher must keep in mind that the stability advantage of the weighted satisfaction index decreases the smaller the sample size.

Example

Q1. Using a 5-point scale, how would you rate your level of satisfaction with the following aspects:

Restaurant cleanliness, staff courtesy, service speed, food quality, meal price.

Q2. What is the importance of each aspect to you? Importance has several measurement methods, and I may write a separate article on the question of importance. I prefer using the ranking method from most to least important.

For readers outside the marketing-research industry, you may not fully grasp the example, and I apologize for that.

The important thing: after obtaining answers to the two questions, we place the satisfaction results alongside the importance results, multiply each aspect’s average satisfaction by its importance, then sum the results for all aspects to end up with a single number. This resulting number is called a Composite Score or Weighted Score.

Why Don’t We Rely on Marketing Research Agencies and Their Specialized Models to Derive the Fair Index?

Most of the statistical models offered by marketing research agencies cannot process the data unless 30 or more questionnaires are available — for statistical considerations. So you can’t benefit from these models under any circumstances.

Closing

Some readers might wonder, “Why all this effort, and who cares about this anyway?” I tell you: these assessments went to the corporate-performance-management department, and these results affected — positively or negatively — the evaluation of employees working in these departments. For fear that one of them might be wronged, this research was conducted to ensure the evaluation being raised to corporate performance management is fair and accurate. And to God belongs the right path.

For those wanting to share this discussion in English, you can find it at the link below: https://www.linkedin.com/grp/post/1772348-242747960

← Articles MOC