Response Rates by Data Collection Method

Preamble

A question constantly comes up in the minds of those working in research, without a clear answer: what response rate you can expect when using a particular data-collection method in quantitative research. In this article I’ll dig into the details of this topic, beginning with a note that response rates across all traditional data-collection methods have been in continuous decline from the 1990s to today.

Why Does Response Rate by Data-Collection Method Matter?

In any quantitative study, there is a target sample representing the total population that must be surveyed to obtain reliable results within a specific margin of error. One of the main reasons for choosing one data-collection method over another is the response rate that method will achieve. The nature of the research is what most strongly dictates the data-collection method, but decision-makers may give up partial objectives of their research in exchange for collecting data via a specific method. In the end, there are trade-offs that must be made — sacrificing one thing for another.

Definition and Calculation

Response Rate: The percentage of targeted respondents who answered the questionnaire, divided by the total number of targeted respondents that the research-executing party attempted to reach.

Completion Rate: The percentage of targeted respondents who answered the questionnaire fully, divided by the total number of targets who started the questionnaire but did not finish it for one reason or another.

Example: Suppose a client has a customer-satisfaction survey collected via phone interviews, with a target list of 1,000 respondents.

Respondents the company attempted to call: 1,000.
Respondents successfully reached and who started the questionnaire: 500.
Respondents who completed the questionnaire to the end: 400.
Respondents who declined to participate: 300.
Respondents whose phones were switched off or out of service: 150.
Respondents whose numbers were wrong: 50.

Response Rate = 400 / 1000 = 40% Completion Rate = 400 / 500 = 80%

Note that the calculation method may differ slightly between data-collection methods, and we’ll flag that in this article where relevant.

What Influences Response Rate?

Some mistakenly think the only factor affecting response rate is the data-collection method. It’s a primary factor — but not the only one. There are several others: the nature of the research and the subject it addresses (the more important the topic is to the target segment, the higher the response rate); the level of relationship development between the respondent and the company conducting the research (companies with more loyal customers see higher response rates on their surveys); the location where the survey is conducted (companies running global research in several countries will notice the response rate vary from country to country despite using the same instruments and the same data-collection method); and finally — and not least — the nature of the target segment itself heavily affects response rates (the elderly respond differently from teenagers; the less educated differently from the highly educated, etc.).

Completion rate has its own influences too: questionnaire length is a very important factor in its rise or fall, and the rule is direct — the longer the questionnaire, the lower the completion rate. Questionnaire design is another factor — the simpler and less complex the questions, and the more standardized the scales used in it (in terms of number of points), the higher the completion rate.

Speaking of these two rates: applying the results of the questionnaire in follow-up studies is very important and has a big effect on both rates as well. For example, employee-satisfaction or customer-satisfaction studies often show very high response and completion rates the first time the study is run, but these drop noticeably in subsequent waves. The reason: the results aren’t being applied. The participant who took part doesn’t see the impact of their participation on reality, so they don’t bother to participate again in the future. In developed countries, part of the results is shared with participants, along with the plan for using the results and the decisions that will be taken based on their participation. They are kept informed of how those decisions evolve, so the participant gets excited, feels the value of their participation, and looks forward to the next wave of the study.

Response Rate by Data-Collection Method

Based on the previous section, it’s clear it’s difficult to set fixed standards and fixed rates around response, since these vary with multiple influences. Still, I’ll put these rates in approximate ranges, to be used as indicators rather than facts.

1. Face-to-Face Interviews

Under the umbrella of face-to-face interviews fall paper questionnaires and surveys via smart handheld devices. It’s hard to specify a response rate for this type of methodology because of the wide variation discussed in the influences section. These rates can usually be estimated during the questionnaire-testing phase, when real interviews are conducted to test the questionnaire and target-segment responsiveness. Generally, this method has the highest response rates — the presence of a human in the equation and the interview being conducted as a conversation lifts response rates in most cases to 50% and above.

2. Telephone Interviews

The formula is the same as in the example at the start of the article. As for the response rate on phone interviews (assuming the questionnaire doesn’t exceed 10 minutes), it ranges between 10% and 40%, depending on the type of study and the target segment.

3. Online (Ad Form) on Web Pages

The response rate on this type of survey ranges between 5% and 10%, while the completion rate ranges between 75% and 85% in most cases.

There’s another factor affecting response rate in this method — and the one that follows — which is the day the survey is presented or sent, as well as the time of day. This is studied behavior among social-media users: every social network has hours of peak activity for various reasons. Presenting or sending the survey on active days or peak hours increases the likelihood of a higher response rate.

4. Online (Email Invitation) to the Target Segment

In this method the response rate is higher than presenting the survey as an ad. Email is more personalized — it addresses the recipient by name, and the recipient was only targeted based on certain demographics, meaning the survey topics may be more relevant to them. The response rate in this data-collection method ranges between 10% and 20% — and here I’m talking about the kind sent by companies to a database they own. For surveys sent by the researcher to their own list of customers or employees, the response rate becomes much higher and can sometimes exceed 50%. The completion rate ranges between 75% and 85% in most cases.

There’s another factor affecting response rate in this method: sending a reminder to those who haven’t yet participated to encourage participation. Even in the best case, a reminder won’t lift the rate by more than 10–15%; those genuinely interested in participating will do so on the first day they get the survey or the day after.

5. Interactive Voice Response (IVR)

In this data-collection method, there’s a practice that lowers the response rate and also affects the realism of the results: call-center employees (this method is most often used in call centers) only transfer the caller to the survey if they feel the caller will give them a high rating. Mechanisms have been developed to help the center supervisor or manager monitor the rate at which callers are transferred to the survey, so they can track calls that weren’t transferred and listen in to determine whether the non-transfer was intentional or not. There’s another method too: as soon as the customer ends the call, they’re automatically called back and the IVR starts the moment the customer picks up.

The way response rate is calculated in this method is the percentage of targeted respondents who answered the questionnaire fully, divided by the number who agreed to start it. I don’t know why they calculate it this way when it could be calculated the same way as the other methods. Using the same method as the others, the response rate ranges between 30% and 40%, and the completion rate ranges between 40% and 50% in most cases — assuming the questionnaire is very short.

6. Mobile Research (SMS) on Target Mobile Phones

Surveys are shared via short SMS messages containing a participation link, or via SMS that asks the target to reply with their answer in the same message — or via what’s known as Unstructured Supplementary Service Data (USSD). USSD is similar to SMS, but appears as a window on the screen and is responded to directly within the same window without needing to send paid messages. The response rate in this type of data-collection peaks at around 20% in the best case, according to some research, and has dropped sharply in recent years. This is related to customer annoyance from the volume of promotional and advertising messages they get on their phone every day. That said, the response rate can rise when the survey is with customers and when those customers see the impact of their participation.

Closing

The continuous drop in response rates across all data-collection methods raises the cost of research — the operations side consumes more time and effort to achieve the target sample. This tendency is pushing global agencies to develop alternative methodologies and data-collection methods, and to begin using neuroscience methods that predict target behavior without interviewing them. It’s a situation very similar to the search for alternative energy sources: the information is needed for decision-making, and if you can’t get it from certain sources, you have to develop and innovate alternatives. We may detail these methods in a separate article. Note that most of them rely heavily on technology and its applications.

← Articles MOC