How to Find Out How to Do Qualitative Research

In 2003, the NSF Sociology section, whose program many had criticized as making it difficult, if not impossible, for qualitative research projects to receive funding, hosted a meeting of 32 social scientists, seven affiliated with NSF, to discuss standards for such research, how NSF could implement them, and how qualitative research in general could be strengthened. I attended this meeting, wrote a short paper for it and, a few years later, collaborated with Robert R. Faulkner on a qualitative study of musical activity funded by an NSF grant.

Charles Ragin, who organized that first meeting, produced a report (I have read the original) whose final (and somewhat different) version was prepared in collaboration with Joane Nagel and Patricia White of NSF (this is the first of the documents mentioned above, referred to hereafter as the 2004 report). The meeting itself was contentious. Many participants took issue with long-standing NSF policies, which favored research proposals containing a strong theoretically based statement of a research question transformed into hypotheses testable by well-formulated methods of data gathering and analysis. These critics insisted that qualitative research came in many varieties, some of the most important of which (particularly long-term field observation) could not be formulated this way because (what researchers considered a sound scientific reason) you never knew what ideas you would have to investigate and test until you began the research. The rules on the protection of human subjects evoked particularly strong reactions, many critics contending that, for that reason and others, they could not realistically produce the typical “human subjects” documents required by existing procedures. A few suggested that the NSF sociology panel seemed particularly averse to funding field research outside the United States. The bulk of the report consisted of short “appendices” written by attendees on a variety of subject, for the most part thoughtful explorations of various problems by such experienced researchers as Jack Katz, Mitchell Duneier, Kathleen Blee, and Elijah Anderson.

The NSF staff, perhaps surprised by the tenor of the discussion, convened a second meeting in 2005, chaired by Michèle Lamont and attended by twenty-nine people. This conference (the second item mentioned above is the report co-authored by Lamont and Patricia White, NSF staff person for sociology, referred to hereafter as the 2009 report) included people from political science, law and society, and anthropology, as well as sociology, in order to make its results useful in these other fields which also produce qualitative research. Only four participants had participated in the earlier meeting, the lack of overlap perhaps due to the inclusion of people from the other fields. Still, it is worth remarking that none of the strongest critics of NSF policies were invited back. I read the second report as an attempt to undo the damage caused by the first one, even in its revised form.

The bulk of the text again consists of short appendices written by the attendees. As in the 2004 report, these vary considerably. Some report on specific topics: anthropologist Linda Garros’ instructive discussion of the development of her research on illness in everyday life, Susan Silbey’s thoughtful and realistic and discussion of how to improve training in qualitative methods, and Jody Miller’s account of the politics of research methods in criminology. John Comaroff presents some thoughtful remarks on ethnographic method: “[A]nthropology always rests on a dialectic between the deductive and the inductive, between the concept and the concrete, between its objectives and its subjects, whose intensions and inventions frequently set its agenda. The failure to grasp this may account for the autonomic dismissal of ethnography as unrigorous, unreplicable, unfalsifiable, and the other (non)u words with which it is regularly damned.” (p. 37)

Many more of the papers, however, repeat the message delivered by Lamont and White in the 15 page executive summary and short introduction, which might be summarized as “Quit whining and learn to do real science by stating theoretically derived, testable hypotheses, with methods of data gathering and analysis specified before entering the field. Then you’ll get NSF grants like the real scientists do.” Less contentiously, you could say that the report recommends an unnuanced and incomplete version of the King, Keohane and Verba Designing Social Inquiry (1994) message: start out with clear, theoretically anchored hypotheses, pick a sample that will let you test those ideas, and use a pre-specified method of systematic analysis to see if they are right.

The participants in the earlier meeting criticized that method profoundly, but the 2009 report ignores the fundamental questions and criticisms raised there, deriving its principles and recommendations by analogy from the model of the natural sciences, not as those sciences are actually practiced but as philosophers of science and their acolytes in methodological specialties recommend, by deduction from first principles. The sociology of science, one of sociology’s most productive fields in the last two decades, and the related specialties in history and anthropology, have shown repeatedly that these recommendations do not reflect how scientists actually work (Latour and Woolgar’s Laboratory Life (1986) and Peter Galison’s How Experiments End (1987) support this conclusion). We get better understanding of how to construct our own practice by studying what natural scientists in fact do (as Thomas Kuhn (1970) described it, and many empirical researches have since confirmed), inspecting recognized exemplary works and seeing how they did what made them exemplary. The countless recommendations offered over the years to improve qualitative research by imitating quantitative research designs never use this empirical method to arrive at their recommendations, nor explain why they don’t.

We don’t lack qualitative works most sociologists recognize as adequately scientific (I don’t think there’s anything about which we could say most sociologists agree). Whyte’s Street Corner Society (1955) stands as a model of excellent methodological practice, as do Goffman’s Asylums (1961), Duneier’s Sidewalk (1999), Stack’s All Our Kin (1975), or Laboratory Life (referred to above). These recognized classics (and others less well known today) give us the raw material from which we can derive some methodological guidelines which inform successful qualitative research. Inspection of these exemplars suggests that the empirically tested methods they use differ substantially from the non-empirically supported (thus – can we say? –unscientific) principles recommended by the 2009 report. Whyte’s famous Appendix on methods (pp. 279-358, esp. 283-6) describes how he prepared a research proposal in 1936 that was far removed from the reality of his proposed research site, how he finally realized that he didn’t know what he was talking about, and then developed and tested the ideas in his book against evidence gathered in the field over a period of several years. He couldn’t have known what the final subject of his research was going to be or how to study it until he had been in the community for a few years.

A first observation provoked by inspection of these classics: successful qualitative research is an iterative process, in which the data gathered at T1 inform data gathering operations conducted at T2. Successful researchers recognize that they begin their work knowing very little about their object of study, and use what they learn from day to day to guide their subsequent decisions about what to observe, who to interview, what to look for, what to ask about. They interpret data as they get it, over periods of months or years, not waiting (in the fashion of a survey analysis, for instance) until they have it all in to start seeing what it means. They make preliminary interpretations, raise the questions those interpretations suggest as crucial tests of those ideas, and return to the field to gather the data that will make those tests possible. (It’s the method mathematician George Polya (1954) suggested not only as appropriate but as the only possible one for the empirical sciences. See Becker (1998, 151-7) and Faulkner (2009, 82-86) for extended examples.)

Doing research that way is a systematic, rigorous, theoretically informed investigative procedure. But researchers can’t know ahead of time all the questions they will want to investigate, what theories they will ultimately find relevant to discoveries made during the research, or what methods will produce the information needed to solve the newly discovered problems.

As it happens, the 2009 report contains an excellent example of this research style, Linda Garros’ report on her studies of illness in two Mexican communities. She quotes, approvingly, Agar’s description (1996, 62) of ethnographic fieldwork: “You learn something (‘collect some data’), then you try and make sense out of it (‘analysis’), then you go back and see if the interpretation makes sense in light of new experience (‘collect more data’), then you refine your interpretation (‘more analysis’), and so on. The process is dialectic, not linear.” Then she describes a simple interview technique – asking people to list all the illnesses they knew about and then asking about the nature of the illness, what you could do to cure it, and so on – which worked perfectly in the first community she studied, providing the basis for a more structured interview guide (thus demonstrating how you use beginning knowledge to guide later data gathering). An even more instructive event occurred when she put this now tested method to work in a second community – where it failed spectacularly, because the second population didn’t talk about illness the same way, and couldn’t or wouldn’t provide a tidy list from which she could proceed to Step 2, forcing her to invent a new method she could not have known she would need before circumstance forced it on her.

Some specifics in the 2009 report thus refute Lamont and White’s conclusions, and demonstrate that the criticisms made in 2004 were on the money. To be a little repetitious, inspection of the research classics cited above shows that researchers don’t fully specify methods, theory or data when they begin their research. They start out with ideas, orienting perspectives, even specific hypotheses, but once they begin they investigate new leads, apply useful theoretical ideas to the (sometimes unexpected) evidence they gather, and in other ways conduct a systematic and rigorous scientific investigation. Each interview and each day’s observations produce ideas tested against relevant data. Not fully prespecifying these ideas and procedures, and being ready to change them when their findings require it, is not a flaw, but rather one of the great strengths of qualitative research, making possible efficient development and testing of hypotheses.

Donald Cressey’s Other People’s Money (1953) prompts a second observation (argued on somewhat different grounds by Garfinkel, Cicourel and many others): you can’t expect other people’s categories to produce reliable knowledge about a sociologically defined problem. Cressey wanted to study embezzlement, defining that sociologically (rather than by the simpler device of sampling people convicted of that crime as legally described) as the criminal violation of financial trust. But he found, when he interviewed people convicted of the crime of embezzlement, that prosecutors, seeking convictions, charged suspects with the crime whose legal definition they could be made to fit rather than the act Cressey was interested in. Some people who violated trust were convicted of confidence game or forgery, while people convicted of embezzlement, whatever else they had done, might not have violated financial trust. He couldn’t know whether a case would test his hypotheses until he began the interview. His work shows that researchers can use statistics others have gathered, but only when they have independently investigated their adequacy for a theoretically defined purpose, something that can never be taken for granted. Taking such precautions leads to a rigor often absent from studies less critical about the more easily gathered date they rely on.

Jane Mercer’s Labeling the Mentally Retarded (1973), another classic work, shows how school personnel generate and use students’ IQ test scores in ways that create systematic discrimination against minority students. She developed her methods as she uncovered the complexity of the process, eventually gathering data in innovative ways she hadn’t earlier suspected would be necessary in order to test hypotheses developed in the course of her research. In the end, she convincingly demonstrated that borderline mental retardation was a disease black and Latino children in Riverside, California got when they entered school, and were cured of by leaving school.

More generally, researchers discover in the field what they can gather and count that will be useful for testing ideas generated empirically, in the course of the work. This in no way means that qualitative researchers never use numbers. But they do insist that the numbers make sense and stand up to critical inspection. Peneff (1995) found many possibilities just by observing how the people he observed gathered and used numbers, whose meaning he found by watching how their users gave them meaning.

Reading over the classic works cited earlier leads to a third observation. We make best use of theory when we refuse to base our research designs on what organizational personnel tell us or on what “everybody knows” (the likely sources for a priori theorizing of the kind Lamont and White recommend) and instead build theories on unexpected observations made in the field. Latour (Latour and Woolgar 1986, 45-9) began his fieldwork in the biological lab he studied by refusing to take anything for granted. Which led to an initial “naive” discovery: some workers wore white lab coats, others didn’t. Instead of treating this as an unimportant fact of laboratory life, he wondered who wore the coats and who didn’t, and what kinds of work the two differently dressed groups did. And discovered that:

“One area of the laboratory (section B . . .) contains various items of apparatus, while the other (section A) contains only books, dictionaries, and papers. Whereas in section B individuals work with apparatus in a variety of ways: they can be seen to be cutting, sewing, mixing, shaking, screwing, marking, and so on; individuals in section A work with written materials: either reading, writing, or typing. Furthermore, although occupants of section A, who do not wear white coats, spend long periods of time with their white-coated colleagues in section B, the reverse is seldom the case. Individuals referred to as doctors read and write in offices in section A while other staff, known as technicians, spend most of their time handling equipment in section B.”

This suggested ideas about the lab’s division of labor which he then tested against more observations, leading several chapters later to a theory of the cycle of activity by which scientific papers produce money (from grants) which allows more research to be done, leading to more papers, leading to more money, etc. (Latour and Woolgar 1986, 187-234)

Pace the 2009 report’s emphasis, less theory at the beginning of the work typically leads to good social science. Researchers usually don’t know enough to formulate good hypotheses until they are well into their work (this results from the iterative nature of qualitative social science). It follows that they should deliberately not accept the common understandings on which such theorizing would have to rest.

Melville Dalton’s study of several large business organizations, reported in Men Who Manage (1959), provokes a fourth observation. Exemplary qualitative research typically shows that conventional ways of framing and answering research questions often rest on an acceptance of conventional understandings that hide what we should be studying. Analyzing the internal politics of business organizations, he showed (among many other things) that employee theft was not the “individual crime” conventional analyses supposed, but instead an informal reward system by which companies paid employees (at every level of the organization right up to the top) for what they couldn’t legitimately order them to do (pp. 194-218). But he could not have formulated this hypothesis, let alone tested it, until long observation showed him the extent and import of the conventionalized systems of exchange conventionally called “employee theft.”

A final observation. These classic works usually analyze directly observed processes, chains of events that produce the outcomes we want to understand. Thus, Alfred Stanton and Morris Schwartz’ The Mental Hospital (1954) traces the origins of psychiatric symptoms and behavior to regular changes in the hospital’s social organization. Alfred Lindesmith’s Opiate Addiction (1947) shows how processes of collective definition produce addicts’ apparently individualistic behavior. Diane Vaughan’s The Challenger Launch Decision (1996) unravels the mundane events that produced this famous disaster, which in turn leads to a general theory of the way organizational cultures normalize deviance. Everett C. Hughes’ French Canada in Transition (1943) traces the ramifications of industrialization, from demographic and financial data to the intricacies of family and organizational life, as parts of the interactive processes set loose when industry moves where it has not been before. Tamotsu Shibutani’s extensively documented The Derelicts of Company K (1978) analyzes group morale as it responds to the continually changing contingencies of organizational life. Process analysis presents substantial difficulties for quantitative studies, which generally have to substitutes such makeshifts as panel studies, population measurements at several selected times, etc. for realistic, more or less continuous empirical observation.

Both reports deal at length with topics I haven’t discussed here: how NSF should award grants and spend its money, how qualitative researchers might reorient their work so as to improve their chances of getting NSF grants, and so on. I think it’s likely that NSF will continue to award grants as it has in the past, in response to organizational and political pressures sociologists, individually or organizationally, cannot easily counter or control. So I have ignored these questions to concentrate on what I think the more serious and immediate danger posed by the 2009 report: that its sponsorship by the National Science Foundation may lead unwary and uncritical readers to think that all the problems of qualitative research methods have now been definitively settled and the 2009 report, and especially its introductory sections, can be recommended to unsuspecting colleagues (and, worse yet, to students) as an authoritative settling of all the outstanding contentious questions. Nothing could be further from the truth. Only by dealing with the many questions qualitative researchers and their researches have raised (which I have summarized here) frankly and open-mindedly, relying on the many results of empirical research available in the extensive literature of qualitative work, can we arrive at workable research protocols we can with confidence recommend to our students.

On the other hand, it may well be – time will tell – that the methods recommended in the 2009 report will produce one result many people have long hoped for: an NSF grant for their research. Anyone wishing for such good fortune should remember one of the other criticisms many times repeated during the earlier meeting. NSF has an apparently inflexible rule that grants will not be given for faculty time released from teaching. But the chief expense of any qualitative research is always the researcher’s time. To do what Whyte and Goffman and Duneier and Hughes and Vaughan and the others cited above did doesn’t really cost much. The materials for recording, storing and analyzing interviews and field notes are cheap. Qualitative researchers need money to pay for their time, so that they can make observations and conduct interviews and get those data down in a permanent form. And NSF won’t pay for that.