JAC Volume 4

In recent years composition studies has begun to establish legitimate identity as an academic field worthy of genuine faculty appointments and professional rewards. Corresponding­ly, the inherent diversity and complexity of the field have become more and more apparent to those who take it seriously. If composition is not  something we do while waiting to be deemed worthy to teach upper-division or graduate courses in literature, then it is  something we all need to know more about. The pursuit of richer understanding of and higher status for composition studies has generated something of a boom in theory, research, and pedagogy that has indeed advanced the borders of knowledge in the field. All of this has been healthy, and there is no imaginable reason at this writing for the boom to subside. But there is one important area of recent research that I would like to address here an area that I might call science-based or science-conscious research, particularly protocol anlaysis in studies of student composing processes because I think there is some chance that composition studies risks a detrimental borrowing of prestige from science in general, a borrowing that is not only potentially misleading but also too limiting if it causes us to turn our attention away from the broad spectrum of con­siderations inherent in composition studies. I raise these cautions not to belittle either the research or the researchers who have worked and are working in this important area, but to encourage the maintenance of a broad perspective about our field so that it remains in the matrix of rhetoric, philosophy, philology, literature, and other areas of study commonly classified within the humanities.

By no means am I alone in raising these general cautions. In the two years that I have been working with some of the material in this piece, at least two other excellent discussions in the same vein have appeared. Robert J. Connors has taken a broad look at the question of science in our field in “Composition Studies and Science” (College English, 45, January 1983, 1-20). And Michael Holzman has assessed the sentence-combining movement in terms of its scientific bases in “Scientism and Sentence-Combining“ (CCC, 34, February 1983, 73-79). Their general conclusions about the applicability and reliability of social sciences methodology are the same that I draw here, but neither article focusses on composing process research, particularly protocol analysis.

For writing teachers, the majority of whom were actually trained in the humanistic study of literature, it may seem unnecessary to raise a caution about borrowing prestige from science. Nonetheless, we live in an age that reveres science, particularly the “hard” sciences whose empirical rigor and technological advances have provided dazzling, life-changing results. Science enjoys a cachet unmatched by any other type of study in our society, a cachet deservedly earned in the laboratory sciences, but not in the social sciences. I mean no slight of social sciences here, but want only to remind that research methodology in the social sciences is not empirical in any strict sense and therefore not capable of generating the kind of objective and precise resultant knowledge possible in the laboratory sciences. And the “science-based” research in composition studies has derived from social science. Research on the composing processes of students, particularly protocol analysis, is based upon case-study methodology from psychology.

In case-study research of students’ composing processes, no one study has been more influential than Janet Emig’s The Composing Processes of Twelfth Graders (NCTE, 1971). Emig’s study provides the general case-study model followed by many researchers of student composing processes since 1971. Essentially, this method involves tape-recording student answers to questions about their composing procedures on various writing projects, or, in later studies, taping their utterances while they are actually in the process of composing (when they have been asked to “compose aloud”). Emig and her many successors have tried to avoid any sort of prompting or other intervention, in order to be able to record evidence that derives as much as possible solely from the students as they deal with the writing task at hand. Trying not to intervene, however, is not the same as avoiding actual intervention. The case-study situation itself, the tape recorder, and the unnatural business of “composing aloud” create contextual variables that make empirical objectivity in any “pure science” sense impossible. I have discussed the contextual limitations of Emig’s method elsewhere, so need not elaborate them here.¹ My point is that these limitations must be kept in mind as more and more case-study research is generated and more and more individuals in our field read the results of that research.

Case-study methodology, of course, makes no claim of its own to the same objectivity of the laboratory sciences. The problem lies in the claims incautious researchers might makefor it, and the profession-wide perception of it in an atmosphere that reveres almost anything “scientific.” Two of the most well-known recent case-study researchers are Linda Flower and John R. Hayes, whose work using protocol analysis while studying students’ composing processes has been quite influential. Their work also serves to illustrate my general point about prestige-borrowing and my particular point about the limitations of case-study research of student-composing processes.

Flower and Hayes borrow the technique of protocol analysis from cognitive psychology, where it is often used to identify psychological processes in problem-solving tasks. Initially in their research, Flower and Hayes’ protocols included “a transcript of a tape recording made by writers instructed to verbalize their thinking process as they write, as well as all written material the writer produced.”² In the February, 1980 issue of CCC, Flower and Hayes claim that protocol analysis lets them “describe what writers actually do as they write, not just what we, as theorists, think they should do.”³ That is a startlingly comprehensive and assured statement for researchers who, in another 1980 essay, which appears in a collection on Cognitive Processes in Writing, say: “Typically . . . protocols are incomplete. Many processes occur during the performance of a task that the subject can’t or doesn’t report.”⁴ Flower and Hayes go on in this latter study to compare the study of protocols to tracking a porpoise “which occasionally reveals itself by breaking the surface of the sea.” The brief surfacings of the porpoise, they say, “are like the glimpses that the protocol affords us of the underlying mental process. Between surfacings, the mental process, like the porpoise, runs deep and silent.” They conclude that they must “infer the course of the process from these brief traces.”⁵ Given their own porpoise-tracking metaphor, their certainty about what protocol analysis tells them is puzzling.

Yet they become even more sure of the efficacy of protocol analysis in a December 1981 CCC article, in which they pare their definition of protocol down to the transcript of the taped composing session itself, the transcript becoming what they call a “thinking aloud protocol.” According to Flower and Hayes in this later article, “Thinking aloud protocols capture a detailed record of what is going on in the writer’s mind during the act of composing itself.” They go on to explain that they ask the writers “to compose out loud near an unobtrusive tape recorder. We then ask them to work on the task as they normally would—thinking, jotting notes, and writing—except that they must think out loud.”⁶ Even though the tape recorder may be unobtrusive and the researchers not present, the recorder’s presence is known and therefore intrusive; moreover, composing aloud is not working on the task “as they normally would.” This artificiality for experimental purposes is not prominently acknowledged, nor does it become a clearly-stated qualifying factor in the cognitive process theory the article presents. I’m sure Flower and Hayes understand the limitations implicit in their work, but I am not sure that they have made these limitations clear to others. Much of their work, after all, is not written for readers in cognitive psychology but for readers in composition studies, readers for whom some explanation for mediating between fields is appropriate.

Sondra Perl is another more recent composing process researcher whose work seeks to build upon Emig’s promising beginning. In one study, which she reports in a collection of essays on basic writing, Perl examines the composing processes of five different writers in her basic writing class at a Community College of the City University of New York. On one particular page of this study, Perl negates the possibility that her work could be considered truly empirical. First, she says she chose her subjects from her own class, thus complicating the context because she fills the role of teacher as well as investigator. She tells us she made this choice for two reasons, the first of which makes good empirical sense: “I would know firsthand what they were studying in their content courses and thus would be able to construct topics for the writing sessions that legitimately reflected classroom work.”⁷ Her second reason, however, renders objectivity virtually insignificant:

Even granting the perhaps insoluble “objectivity” problems inherent in case-study methodology, Perl’s further revelations about her method are troubling:

These remarks raise several unanswered questions if Perl’s study is to be considered even quasi-empirical. Because these were Perl’s students involved in a collaborative effort, how did she incorporate or allow for evaluation, that inevitable aspect of teacher-student relationship? “All fifteen students in my class volunteered,” Perl writes, even though they understood “that there would be five sessions, each taking place outside of class time.” Isn’t it possible that they volunteered to in-influence their standing with the teacher? And if fifteen volunteered, Perl needs to explain why only five were studied. Did any of Perl’s students think it odd or react negatively to the idea of their teacher admitting that she did not know “how students actually compose”? Such questions are important. Finally, Perl tells us, still on the same page, that once the study was completed, she paid the students for their time and participation. Although she had not mentioned remuneration to the students before or during the study, I think that for some students, remuneration would not have to be monetary; teacher favor or good grades are also types of compensation. And what if Perl wants to conduct more studies among the same student body? Wouldn’t word of the pay circulate, and possibly influence future selection and performance of students?

Of course, Perl does make some careful disclaimers about her method and evidence, though she provides no answers to the above questions; and she does offer considerable insight regarding the special problems of basic writers. I am emphatically not saying here that the case-study, quasi-empirical work of such researchers as Emig, Flower, Hayes, and Perl is of no value. On the contrary, their work has given us a wealth of insight and understanding about how students write, and much stronger theoretical and methodological bases for employing process pedagogy in our classes. But it is a mistake to give their work the prestigious imprimatur of hard science, either because of somewhat careless claims that they make, or because we are preconditioned to be unquestioningly favorable toward that which is even remotely scientific. Since Janet Emig’s landmark work, innumerable studies of student composing processes have been undertaken. The Fall, 1981 Research in Composition Newsletter alone selectively reviews some 47 such studies, all of which enable the researcher “to derive conclusions based on empirical evidence.”⁸ I think we need to keep in mind that we can continue to learn more about our field from other kinds of evidence as well.

There is that about writing which is essentially vitalistic, having to do with aesthetics, creativity, talent, memory, or genius. More precisely, there is that about writing which brings to bear a host of writer options, including rhetorical choice (this analogy, that statistic, or what order things will be in, or how their relationship will be established); stylistic prerogative (the skillfully-placed fragment, the carefully-marshaled parallelism, the subordination of a clause); selective use of long-term memory (this historical incident, that personal experience, a particular character or symbol in a work of literature); and other analysis-and-decision situations related to audience and purpose. These matters are functions of the writer’s intellect interacting with experience and they come into play at both conscious and subconscious levels.⁹ Such matters are extremely difficult, if not impossible, to bring under the kind of scrutiny which will give us reliable information in the best scientific tradition.

I am sure that some important considerations of the composing process in particular and our field in general do not lend themselves to reliable study by empirical method. One such important consideration is the rhetorical nature of writing itself. If, as James C. Raymond recently suggested, rhetoric is the methodology of the humanities, then we ignore a major part of the writing process if we do not also concern ourselves with the lines of thinking in human experience that no science can handle or prove.¹⁰ The processes of analogy, of value judgment, of emotional argument, have little to do with the kind of certitude that truly empirical proofs can bring. Traces of these matters and the wellspring thinking that accompanies them may “surface” in a taped protocol—but they are just as likely to remain “deep and silent.” It is hard to imagine a method of observation and interpretation that could wholly reveal them, and categorize or quantify them. My point here is that we should neither insist upon discovering such a method, nor consider ourselves benighted if we can not.

Something is not necessarily better, or right, or more reliable, just because it is “scientific” or seems to be. In such matters, we need better empiricism—less intrusion, more naturalistic observation, and certainly more caution. And we need a stronger sense of what science willnot tell us. As Raymond observes, to limit ourselves to empirical, particularly quantitative, research “would be to deprive ouselves of the mainstream of insight in rhetorical theory from antiquity to the present day.”¹ And I’m sure I’m not the only writing teacher who agrees with Richard Lloyd-Jones, who recently made brief note of some of the limitations I’ve elaborated here, and stated that “the home of scholarship in writing is still philosophy, philology, rhetoric, and literature.”¹² Like Lloyd-Jones, I’m not sure where any boundaries among those areas could be drawn, or even that any boundaries should be drawn. But it seems clear that while expanding our research “territory” to the imprecise, interpretive, inferential realms of psychology and social science, we should be careful not to borrow the prestige fostered by the precision and exactitude found in the more purely empirical realms of laboratory science.

Ironically, Janet Emig provides a precursor of this problem in the introduction to her study: “The writer’s hope and ambition for this study is that it may provide one rung of a ladder up from alchemy, so the learning and teaching of composition may someday attain the status of science as well as art.”¹³ We need to ask if an equation of composition and science is possible, or even desirable. We need to ask if the rich legacy of rhetoric, philosophy, philology and literature, from which the learning and teaching of composition spring, needs to come “up” any ladder at all to attain the status of science.