Can Qualitative Research be Rigorous? Part 1: What is Qualitative Research?

Although this view is rapidly changing, qualitative research sometimes carries a reputation of being a less reliable or trustworthy approach than say, econometrics. More often than not, this stems from a belief that qualitative research is fundamentally subjective and therefore prone to bias. For some reason which I’m not adequately trained to explain or justify, the human psyche seems to have a tendency to interpret numbers as being more precise, reliable and objective than, let’s say, words. I’d like to explore the role of qualitative research and its potential for rigor in more detail. But first, I think it’s necessary to consider what qualitative research actually means.

(Photocredit: Vidya Ananthanarayanan)

But, really, what is qualitative research?

The answer to this question may be less obvious than it seems. We tend to think of qualitative research as being data collection and analysis associated with interviews, focus groups, observation and ethnography; while quantitative research gets grouped with surveys and administrative data analysis. This is an easy enough distinction to make, but is admittedly a little vague and simplistic. When one tries to create a more specific definition, the answer becomes more elusive.

In its most basic form, research is simply the process of gathering information and then using that information to make conclusions. The more reliable and accurate the information and the more objective and certain the conclusions, the more rigorous the research is considered to be. Fundamentally, data is simply information that researchers use to answer questions – this information can come from many different sources and present itself in numerous forms. An analytical technique or design approach is how researchers use that information to draw conclusions. We can classify research as either quantitative or qualitative by the source or form of data that it uses and the research design/analytical approach. But, often, the distinction is not entirely obvious.

Defining data as qualitative is fairly straight-forward – essentially, qualitative data is anything that does not exist in numerical form in its raw state. Most commonly in research, qualitative data takes the form of words, but it can also be images, sounds, events or any other form that serves as a potential source of information. Qualitative data can be obtained from interviews, which we typically think of as providing data in the form of words. But interviews can also be video-taped and the information available can include body position, tone of voice, pauses between conversations, laughter, sounds that are made or gestures. In some cases, these types of information can be useful – or even central – to answering a research question. Similarly, qualitative data can be abstracted from photographs, court transcripts, emails, written documents like diaries or medical case notes, blog posts, film, advertisements and any other of the numerous sources of recorded or recordable information that exist in the modern world.

Taken quite strictly, the vast majority of data is in fact qualitative in its purest form, except data which conveys information about quantity – e.g. frequency or counts, monetary figures, time elapse, size or dimension, etc. Yet, we often define quantitative data more broadly than this.

Data is typically considered to be quantitative if it exists in a dataset in numerical form, but most of the time, we are actually forcing qualitative data to take on a numeric form. This is most obvious in Likert-scale type questions, which assign numerical values to various states of satisfaction or other qualitative (and usually subjective) measures. But other types of survey questions frequently assign numbers, sometimes fairly meaningless numbers, to qualitative responses that get stored as numerals in the raw dataset. For example, we might want to know about categories of loan type that a household has taken or the type of health care professional from whom an individual usually seeks care. Often, these categories are arbitrarily assigned a numeric value and stored as such in the raw dataset. Then we have ordinal data, which is typically used to express order or ranking – information that is best conveyed numerically, yet isn’t quite describing a quantity either.

I believe I can say, without much controversy, that most researchers who primarily work with data in numerical form would consider themselves quantitative researchers – even if that data represents a qualitative construct (e.g. category, subjective measure, etc.) So perhaps a better approach is to classify research as qualitative or quantitative by the type of analysis used. Yet, this definition has its limitations as well.

Consider a researcher who conducts a series of interviews, then tallies the frequency and distribution of certain words. Maybe this researcher constructs a word map or word cloud, which graphically displays words or themes according to their frequency and co-occurrence with other words. Is this qualitative research? Some of these analyses and representations of qualitative data require rather sophisticated algorithms. What about the econometrician who conducts a difference in difference analysis of physician diagnoses to determine an intervention’s impact on disease detection – whose single data source was a series of hand-written case notes, obtained from stacks of medical records? Both use qualitative data, but apply quantitative analytical techniques. If we define qualitative research by analytical method alone, then by definition, both of these examples qualify as quantitative research. Yet, it’s likely that most of us in the research world would consider the first example a qualitative researcher and the second, a quantitative researcher.

If we classify qualitative versus quantitative research purely by analytical approach, the above example of researchers would fall closer to, or well within, the quantitative research spectrum. On the other hand, if we consider qualitative research to be collection and/or interpretation of non-numerical data, then both of these researchers are conducting qualitative research. Neither definition of qualitative research seems quite suitable to adequately classify or distinguish the two examples of research. I would argue that this is because trying to classify research as quantitative or qualitative is itself an arbitrary and meaningless exercise. Information is useful and valuable, regardless of its native form. We can analyze this information in n number of ways, some more “rigorous” than others, but the analytical approach we use does not itself define the research as either qualitative or quantitative, nor does the source of data.

Yet, to this day, we do distinguish between qualitative and quantitative research and as long as this distinction exists, it is helpful to have a working definition of qualitative research. I would say that qualitative research is distinguished by the deliberate intention of collecting data in qualitative form, conducting analyses on this data in its natural qualitative form, and ultimately incorporating qualitative information into the results.

To revisit the case of the researcher who conducts tabulations and generates word maps with her interview data, under this definition she would qualify as engaging in qualitative research if: she initially intends to collect content (words, themes, etc.) through interview as qualitative data, records the information as words (or video), analyzes the words (or other qualitative data) directly as such (for example, word counts) and incorporates the content into her results (e.g. as a list of words, or themes, with associated frequencies or perhaps a word cloud.) The econometrician would not qualify as a qualitative researcher because she is not interested in the medical case note content as such, rather she is only interested in diagnoses and would find any other data source for diagnosis equally useful, so long as it was reliable and accurate. She abstracts the diagnosis information, converts it to a number and reports her results as a numerical figure – perhaps, for example, an odds-ratio or percent increase. Thus, the two researchers can be adequately distinguished from each other, even though they both use qualitative data sources and perform statistical analysis with these data sources.

Recently, there has been a movement to encourage research which combines both quantitative and qualitative approaches. Policy research funding organizations are increasingly emphasizing the need to incorporate qualitative approaches into impact evaluation plans. Evaluation associations internationally have devoted attention to supporting mixed methods approaches. I applaud this movement, because I think it embraces the value of both approaches and moves us further in the direction of loosening the distinction between qualitative and quantitative research. Ultimately, I think the distinction between qualitative and quantitative as two separate types of research is unnecessary and creates false assumptions about the inherent value and sophistication of one approach over the other. Truthfully, we can do sophisticated and complex analyses of words and rather clumsy, unsophisticated computations with numbers. I’d rather call myself a researcher, who uses the data collection tools most appropriate to address the research question at hand, gathers the information most relevant to my needs, strives constantly to achieve accuracy and precision with data, and always approaches my analyses with rigor and objectivity at the forefront.

Can Qualitative Research be Rigorous? Part 1: What is Qualitative Research?

Follow Us