The State of Marketing Research in Saudi Arabia - Field Researchers

Preamble

There is something we call the eloquence of numbers — the precision that numbers give us in grasping phenomena is unmatched by anything else. But we must note two important things:

The clarity of what numbers convey makes demand for them very high and their circulation very wide, and this leads to two consequences:

First: Numbers are subject to serious distortion. For example, the figure cited in one text that the average Arab reads only 6 minutes a day was distorted in some books, as it was passed around, to 6 minutes a year — a very dangerous distortion!

Second: The trafficking in numbers. Detecting lies in numbers is often extremely difficult, and so some parties try to gain profits, advantages, and interests by inflating certain numbers and deflating others. Hence we must be very careful — otherwise our awareness and understanding will be the next victim.

In the Kingdom of Saudi Arabia, there are more than thirty agencies offering marketing research services — but the question remains: what about quality? That is what I want to address in today’s article, focusing specifically on field researchers and the quality of what they deliver. To begin, let’s clarify that the need for field researchers arises when a marketing research agency wishes to conduct certain types of quantitative or qualitative studies.

The Reality of Field Researchers

All marketing research agencies rely on contract (part-time) researchers, because these researchers also hold full-time jobs elsewhere. A large portion of them are expatriates, which conflicts with Ministry of Labor regulations — expatriates are only allowed to work for the sponsor that brought them in. So they work in a hidden, cautious manner for fear of accountability. These agencies also engage Saudi nationals and university students, but these latter two groups don’t stay in the role for long because of better opportunities they pursue later. So research agencies prefer not to invest in them, and the cooperating expatriate researcher ends up being the most enduring. I’ve seen with my own eyes researchers who have been working in this field for more than 20 years. And I won’t hide from you that a number of researchers deal with many agencies simultaneously. You’d be amazed if you saw one of their cars full of questionnaires and wondered how on earth they finish such a quantity in a record timeframe that no one could realistically achieve even working 24/7.

Let’s discuss the downsides we face because of this reality, broken down by the different data-collection methods used by research agencies, and then I’ll talk about proposed solutions.

1. Face-to-Face Interviews — Paper Questionnaires

In this methodology, I’ve encountered many cases of deliberate cheating by researchers. Research agencies know this and take measures to detect fraud and tampering in order to exclude those cases. What do I mean by field-researcher cheating?

I chose the word “cheating” (ghish) because of its breadth in Arabic — anything that falls under falsification, fabrication, or failure to perform the work as required by any person counts as cheating and a betrayal of the trust placed in them. In English there are several scientific terms describing this process; the most well-known is Interviewer Falsification.

The definition of interviewer falsification: cheating is not limited to a specific activity; it can be defined as anything that leads to distortion of the results that should have been collected from the targeted respondent in a study, including deliberate failure to comply with the instructions and guidance the interviewer received during training.

The data-collection method most prone to cheating is face-to-face paper-questionnaire interviews, because the quality-control mechanism is weak — relying solely on call-backs to those interviewed to confirm that the interview went well. Many field researchers have found crooked ways to bypass this mechanism successfully. One way: the researcher has a group of friends whose phone numbers they use in every study, instructing them on what to do if the research agency calls. In some cases, they use their own personal number.

In this article I am not generalizing that all field researchers in Saudi Arabia are cheats — generalization is a flawed rule under any circumstances. Among them are the honest who want to earn lawful income, but some have been overcome by greed and don’t care whether what they earn is lawful or not. They don’t even grasp the impact of their cheating on the companies running the research — it’s like the butterfly effect, where the consequences ripple out to many parties.

I’ll mention a short anecdote I personally witnessed. One agency was conducting a study for a multinational tobacco company. I asked the supervisor of the field-researcher team a spontaneous question: “Don’t you feel embarrassed conducting research for a company that kills people with its products?” He said, “Yes, of course I’m embarrassed, and I’ve told the field researchers not to hesitate to distort the results so that this company gets misleading information.” … Professionally, what he should have done was refuse to conduct the research, not run it and distort its results.

2. Qualitative Research

Qualitative research requires assembling a group of targets at a specific time and place — for example, to run focus groups, the most popular data-collection method in qualitative research. Practically, you have what are called “recruiters” — people the research agency gives the target-segment criteria so they can search for and recruit qualifying respondents. The cheating I’ve observed in this type of research is that there is a fixed pool of people who get rotated across all the agencies. The recruiters know them and keep them ready for crunch time. For example, a focus group requires 8 people; let’s say the recruiter successfully recruits 6 acceptable ones. They’ll get the remaining 2 from this reserve pool, briefing them on the characteristics they should claim to have if asked — that they belong to a certain income bracket, or work in a certain industry, etc. Unfortunately, there is no supervisory or regulatory body over research agencies. If there were, the solution would be simple: maintain a database of all participants and block their participation for a defined period.

3. Exit Interviews and Mystery Shopping

These two methods require field researchers to go to specific locations to conduct interviews at the exits of the entities being evaluated — such as the exit of a retail outlet or a government office — or to conduct a mystery-shopping mission in which the researcher plays the role of a customer, lives through the experience, and evaluates it later. Many tampering tactics have been spotted and curtailed over time in these two methodologies as well. I recall a funny case in exit interviews in which the researcher was caught red-handed: there was an assignment to conduct interviews in the Qassim region at the exit of a car dealership. Instead of conducting them objectively, the researcher went to the branch manager of that dealership and told him the whole story, saying, “Fill out the questionnaires however you like. That way you give yourself high scores and earn promotions from your company thanks to the quality of service from your branch employees.” The branch manager’s integrity, however, prevented him from doing that. He told the researcher he accepted, then contacted head office and told them what had happened. Head office contacted the research firm, and a major dispute erupted between them. Yes, this was an isolated case — but it could have been prevented under the principle that prevention is better than cure.

4. Other Data-Collection Methods

There are methodologies in which tampering is harder — for example, studies sent to respondents by email. Some agencies, like YouGov and Ipsos, have huge databases in many countries. Questionnaires are sent electronically to the respondents’ inboxes for participation. Still, in some cases the target segment is narrow and hard to reach, and not available in their databases. During a visit to one global agency, I witnessed the following case: one of the recruiters was given a large list of survey links to send to a difficult target segment. Her main task was simply to ensure these links got sent to that segment. But what happened was completely unexpected: this recruiter was opening the links one after another and filling them out randomly herself until she had completed all of them, without sending anything to the target segment. This was a failure on the agency’s part for placing such trust in her.

Another methodology where cheating is harder, thanks to technology: computer-assisted telephonic interviews. Some agencies in Saudi Arabia provide field researchers with specific handheld devices to enter survey answers when interviewing the target segment. These devices capture the GPS coordinates of where the interview took place and record the interview audio without the researcher’s knowledge — there’s no privacy violation here, since the purpose is quality control. Many cases of cheating have been spotted, and the researchers involved have been blacklisted to prevent the agency from working with them in future.

We also have phone interviews, where technology has dramatically improved quality control. Some systems don’t even give the researcher the option to choose whom to call — it’s an automated dialing system that routes the call to the researcher. The interview is fully recorded for QC purposes, and the requesting company is given access to these recordings to listen in and judge their quality.

Proposed Solutions

Knowing the cheating tactics researchers use is only the first step in building a wall to resist these acts. Many cases have been documented in academic research, and counter-tactics have been developed in countries that resist cheating not just because it is religiously forbidden but because it is unethical. As in schools and universities, every time a cheating method is exposed students invent another — and the same goes here. What I’ll mention below is, by no means, more than 50% of the methods out there. As the saying goes: “What’s hidden is greater.”

The simplest tactic: accepting a person outside the target sample (one to whom the screening questions don’t apply) but continuing the interview anyway — either because they know that person, or out of laziness in searching for the right target.
Interview termination cover-up: they meet a respondent who starts the interview and then suddenly refuses to complete it. Instead of canceling, they complete it themselves and submit it as a finished interview.
The most common method: they meet someone who matches the required criteria, take their name and number, ask a few of the survey questions, then continue the interview off the questionnaire as a casual chat. After that, the researcher fills in the rest of the answers from their own head.
Duplicating real interviews: they interview one person from the target sample, then copy the results onto other forms with the names and numbers of other people they’ve pre-arranged with — telling them that someone will call to ask whether one of “our team” interviewed them. This indicates poor QC operations if such lies get past them.
Fully fabricated: the questionnaire is filled out entirely by the researcher, using names and phone numbers from previous studies related to or similar to the current one — or they put a friend’s number and ask them to claim, if QC calls, that someone interviewed them.

The person primarily responsible for what happens is the researcher. But there’s another primary party — the people assigning these tasks: the field-research supervisors or the field-research manager at the agency, all of whom are full-time employees. They face one of two situations: either they don’t know that the field researchers from whom they collect questionnaires will cheat — that’s a disaster — or they know with certainty and still hand them the work — that’s an even greater disaster. In that case they bear the burden of the researcher’s sin and every illicit penny entering their stomach. Especially because the matter is in their hands — they have the power to hire whomever they choose for these missions and, given their long experience in the same field, they can distinguish the upright from the corrupt. But as the saying goes, “If the head of the household plays the tambourine, the household’s custom is to dance.” So I hope the parties involved take this matter more seriously, knowing that God will ask them about their negligence. Here we recall the famous saying of Caliph Umar ibn al-Khattab: “If a ewe stumbled in the lands of Iraq, I would know that God would hold me accountable for it.”

Practical Solutions for Research Agencies

The people running field-research operations in research agencies should put the right researcher in the right place and verify there’s no cheating — rather than just patching up errors and covering gaps. The executive responsible for the study should follow up, supervise the fieldwork, and personally review a number of questionnaires — with the condition that they receive them randomly, not pre-selected from among the best interviews.

Lack of follow-up and strictness from supervisors is one of the main reasons cheating happens. Reviewing questionnaires as they come in and trying to correct mistakes the moment the questionnaire arrives from the field affects researcher performance. If the researcher finds the supervisor lax with their work, the researcher will be lax with theirs.

Providing a suitable environment for researchers is one of the most important points. True, most researchers work part-time, but that doesn’t mean they shouldn’t enjoy a satisfying work environment. The organization can onboard any new researcher by testing them, assessing their familiarity with the subject, and giving them training that ends with a certificate. After that, the employee signs a code of ethics — prepared by the company — covering the ethics of the researcher’s work, including the consequences of questionnaire falsification, so every researcher knows their rights and obligations. I’ll mention some common mistakes agencies make toward their researchers.

Low pay. When you pay researchers little for conducting interviews, you’ll get little care in return. The same goes for delays in paying for completed work. Abdullah ibn Umar reported that the Prophet (peace be upon him) said: “Give the worker his wages before his sweat dries.”

Poor training. Before starting any project, training is conducted for all researchers on the nature of the questionnaire and how it flows. If this is neglected at the start of the project, many things may happen that don’t serve the study’s objectives.

Isolating researchers from the agency. Preventing them from entering the company except when needed for training or to hand over and pick up questionnaires; not allocating them a specific space; not giving them business cards; treating them as strangers or outsiders — even though they are the bedrock of the agency’s work. Without a sense of belonging, the quality they deliver will drop.

Distribute the quota (the target sample to be completed) across the largest possible number of researchers, and do so evenly.

Giving some researchers more questionnaires than others just because they’re faster is a serious mistake. Even if the researcher is honest, they can only complete so many questionnaires per day. The work pressure you place on them leads to errors — whether in collecting the data, forgetting questions, or shortening them.

Supervisors should split the sample evenly across as many researchers as possible. Many supervisors know that the researchers they have are part-time and that many of them probably work for more than one agency. Imagine if the researcher works for you and others — how many questionnaires can they finish in a single day? On top of any other studies the researcher might be doing at the same agency, since agencies may run more than 15 projects simultaneously. This must be managed by the supervisors, who need to know how to manage their researchers, how free or busy each one is. The matter is purely arithmetic: an agency can put a list in Excel of all its projects and the number of questionnaires required for each, the expected time to complete each questionnaire per project, then a parallel list of researchers and the hours each has available per day. From this, the agency knows the productive capacity and can assign suitable quantities. The remaining problem is that some of those researchers, as mentioned, work for more than one agency. If a supervisory regulatory body existed as an umbrella over all the agencies, then quality could be controlled — and controlled arithmetically — much better.

Contacting respondents to verify the interview was conducted (Call Back).

This is one of the most commonly used and oldest methods. A number of QC staff call the respondent’s number on the questionnaire and ask a few questions to judge whether the questionnaire qualifies. But even this method isn’t sufficient — some researchers have gained experience in the kinds of questions QC staff are likely to ask, so they can fool QC by telling the real or fictitious respondent how to respond. Still, an experienced QC officer who chooses good questions can pick up signals of dishonesty and disqualify the questionnaire entirely, denying the field researcher the fee for it.

Conducting periodic researcher evaluations.

I’ve read about some companies that evaluate their researchers on a periodic basis. They prepare cumulative reports on each researcher’s record across all studies, showing the number of interviews rejected in each study and across all studies. Based on the number of violations, the researcher can be reprimanded, warned, or even terminated if necessary. Those with clean records can be rewarded and given priority for more questionnaires than others. This kind of evaluation strongly motivates researchers to give their best.

Practical Solutions for Companies Commissioning Research

Quality control doesn’t fall on the agencies alone — it also falls on the companies commissioning the research. The commissioning company should have a research specialist with sufficient hands-on experience in research agencies themselves so that they can pick good agencies, avoid bad ones, and tighten quality control even if the agency falls short. This is done by accompanying field researchers during face-to-face interviews, listening to recorded phone interviews to evaluate quality, ensuring that the latest technology is used for tighter QC, verifying the quality of the recruited sample in qualitative research, and excluding anyone suspected of being unqualified to participate.

There is a trend among many entities that need marketing research for decision-making or for measuring customer satisfaction and experience: building internal infrastructure to perform this measurement themselves. This is highly effective when the cost of outsourcing exceeds the cost of in-house execution, and I support it heart and soul. However, some companies’ policies — and the laziness of those running research in them — prevent them from making this decision, in order to avoid the operational burden of running this infrastructure. Running it isn’t as easy as some imagine.

There’s a method called Centralized Location Tests (CLT). The idea is to bring the target segment to a specific location for the interviews. This gives the commissioning company more control over QC by being present and inspecting all respondents brought in — but it requires a representative of the commissioning company to dedicate their time fully.

We can rely heavily on technology and the data-collection methods built around it to tighten QC. Paper-questionnaire interviews are on their way out in many developed countries, replaced by advanced methodologies like neuromarketing research, which require expensive technical equipment — though some of these applications have begun to spread through certain agencies in Saudi Arabia.

Questionnaire length is a major factor in getting bad results, and a major driver of field-researcher cheating. The respondent will not, in the end, tolerate spending much time on a survey. The commissioning party must keep this in mind and focus only on what’s most important rather than padding the questionnaire with junk and pearls alike — they’ll bear the consequences later.

Some Researcher Excuses

“They don’t pay us much per interview.” Their words may be true if they compare what they receive to what the agencies make. But this doesn’t justify the wrong. If their claim is true, the right course is to file a collective grievance with the responsible body that can raise the per-interview fee — not to cheat and turn the results upside-down with this heinous behavior.
“All research companies are the same.” They mean that field researchers at other agencies don’t do better work either. This excuse is worse than the sin itself. The Prophet (peace be upon him) said: “Do not be impressionable, saying ‘If people do good, we do good; if they wrong, we wrong.’ But hold yourselves to this: if people do good, you do good; if people do bad, do not wrong” (narrated by al-Tirmidhi).
“The questionnaire is too long.” Here I agree with them. Many books note that the maximum length of a face-to-face interview survey should not exceed 30 minutes.

The Impact of Researcher Cheating on Life Generally

I wonder if any of you has ever reflected on the magnitude of harm that field-researcher cheating causes in life in general? When a field researcher cheats, in those moments their mind is consumed only by greed for more money — blind to the great harm cascading down from their reckless strokes of the pen. When a company is handed false, fabricated data that doesn’t reflect reality, and that client builds huge decisions on those research results, the problem snowballs. Existing problems the research was commissioned to solve can turn into bigger disasters. The harm can reach consumers themselves, and employees can be harmed if the research measures customer satisfaction — or some may be promoted and praised when they don’t deserve it. The biggest impact is long-term: when decision-makers act on the recommendations of research agencies and later find those decisions were wrong, the agency’s reputation in the market will plummet, and they’ll be known for delivering results far from reality and credibility.

Closing

My goal in this article isn’t to demolish the efforts of agencies — as I mentioned, some are constantly working to find better QC mechanisms by various means, one of which is harnessing technology. All I wanted was to raise awareness of this dangerous dimension among both agency staff and the commissioning parties. Apologies for the length — this is only a drop in the bucket. I hope readers will share their opinions and personal experiences.

[1] The preamble is adapted from Forming the Thinker by Dr. Abdul-Karim Bakkar, pp. 99–100.

← Articles MOC