Humanities Core Course Study of Student Writing 1999-2000

Principal Investigator: Michael P. Clark, Associate Executive ViceChancellor in Charge of Academic Planning

Report to the Division of Undergraduate Education

Elizabeth Losh, Writing Director, Humanities Core Course





HCC Writing Pedagogy and the Sequence of Assignments (See AppendixA – "Writing Prompts")

One of the fundamental assumptions of this course and its directors is that students learn the skills of academic discourse most effectively when faced with discrete tasks and step-by-step instruction. In 1999-2000students wrote ten essays for their writing grades on the following academic skills: Definition, Counterargument, Rhetorical Analysis, Analysis of Imageor Narrative, Textual Analysis, Textual Explication, Application, Evaluation,Causal Analysis, and Documentary Research.<1>

Students began with a highly structured "template" assignment, in which most of the text was supplied by Philosophy professor Terence Parsons, and gradually, over the course of the year, had more and more freedom in "independent" composition to prepare for the capstone research paper that finished the course. Many of these assignments had corollary assignments in the Writing 39 series, which the majority of students at UCI take to fulfill their lower division writing requirement.<2>  The effort to create a more coherent sequence and more coordination in Lower Division Writing can be traced directly to a 1998 external review, which judged the Humanities Core Course at the time to be a course that "uses writing" rather than a course "in writing."

Two years of feedback from faculty and administrators indicates that the post-review approach to the HCC writing curriculum has produced significant progress in meeting goals for reform. Quarterly instructor evaluation forms are significantly more positive about the writing curriculum since the introduction of the new sequence and its corresponding articulated weekly pedagogical goals. Unlike the 1998 review, the recent 2001 WASC report described HCC as a "superbly conceived" course with "carefully sequenced" writing assignments. In the words of the reviewers: "The result for students is a sustained, supervised voyage of discovery into the realm of academic discourse: they are given both the form of academic writing, and that the form itself is investigation of significant, even existential human matters. UCI can congratulate itself on creating, within this sequence, an actual‘culture of writing.’"<3>

This study was not designed to assess whether the current HCC approach was more effective than other pedagogical approaches in Rhetoric and Composition, such as approaches to student discourse that do not attempt to isolate individual writing skills in academic discourse and are currently not favored in Lower Division Writing on this campus, or approaches that emphasize writing in particular disciplinary genres, which generally is emphasizedin Upper Division Writing on this campus.<4>  It was intended rather to facilitate program review and provide a campus-specific model for assessment that could be transported to other lower division and upper division writing courses. If funds become available to gather and analyze portfolios, this method also allows these students to be "tracked" through their upper division writing experiences that begin in 2001-2002.

Although there were previous studies of student writing performed on campus, to our knowledge, this is the first truly "longitudinal" one, in that it samples randomly selected students and tracks work done over thecourse of the entire year with the same students.<5>

The Sample (See AppendixB – "Demographic Information on the Sample")

The base sample was derived from randomly selecting two students from each section in Fall quarter.

Gathering random samples of written work from consenting students for long-term tracking continues to be a significant problem on the UCI campus that merits further attention. Based on the experiences of other academic programs engaged in similar work, it is the opinion of the Directors of HCC that student participation is the key problem and that the efforts of HCC administrative staff to gather student work use an appropriate variety of means to improve coordination and communication between all parties and would be effective if students were more willing to participate consistently.  Some programs have gone so far as to pay students to participate.<6>  Monetary incentives to tracked students for HCC would severely tax our budget, given the size of the course, and would be contrary to the pedagogical philosophy of a course that only rewards students with cash awards for the merit of student writing, with the Nora Folkenflik Award and the UROPresearch paper prizes.

Although the sample of complete folders with all ten essays was considerably smaller than planned (31 students), significant amounts of written work from over 10% of enrolled HCC students were collected in the 1999-2000 study and were therefore available to serve as a rich resource for material for outside reviewers, samples for grade norming, models of student writing, and data for analysis of student response to individual prompts.

Using data about campus-wide freshmen enrollment, Dr. Judy Shoemakerof DUE provided a brief statistical analysis of the initial core sampleof thirty-seven students and found that whites, females, and Biology majors were "slightly overrepresented" and that Asians and males were "slightly underrepresented."  The sample had a "comparable, although slightly higher," grade point average than the campus as a whole.<7>  (HCC enrollment figures for 99-00 show higher percentages of Biology majorsand white students than the campus as a whole, and anecdotal evidence suggests that there are more female students in HCC than in the Writing 39 series.)

Because HCC is a demanding 8-unit class that also serves students in the Honors program, it generally attracts students with higher mean verbal SAT scores than other freshmen courses. Although the average SAT verbal score of students in the sample (595) was significantly higher than theaverage of all incoming freshmen (556), it matches the SAT verbal average for all enrolled 1999-2000 HCC students (595).

Like the undergraduate population of UCI as a whole, members of the study tended to be from homes where a second language is spoken. There seemed to be no correlation between ethnicity and degree of linguistic immersion for non-white students in the study: members of the same self-identified non-white ethnic group often identified their linguistic backgrounds verydifferently (from "Only English" to "English and Foreign" to "Foreign").

Grade Data and Programmatic Concerns (See AppendixB – Demographic Information on the Sample)

Grade Data on writing grades for all three quarters was also tabulated for these randomly selected students.  This information was gathered to compare "blind" reading scores to grades delivered in the context of personal instruction. This information can be valuable to a program, because it can give a sense of how well grades measure performance on "final product" writing assignments and assess if grade inflation or patterns of disparity exist.

The primary reason for collecting this data was to offer comparison to scores from the blind reading, but it also had secondary uses in that it offered information about quarter-to-quarter grade deviation in individual students. Of course, there are also multiple reasons for grade deviation that do not reflect on the value of a given writing curriculum,<8> but the information was tabulated and analyzed as follows:

Conditions for Normed Reading (See AppendixC – "The Rubric")

Readers for the three-day primary trait reading were chosen on the basis of years of experience teaching composition and prior experience with normed readings like the Subject A examination. Half of the readers had experience serving at the "Big Read" for UC-wide Subject A grading that is run under the auspices of ETS. The group had an overall average of 3.4 years experience teaching composition. In order to form a more "neutral" group, none had taught the writing prompts of HCC.

Two randomly chosen readers read each essay and scored it on a fivelevel ("1" being lowest and "5" being highest), five feature (Conceptual, Thesis, Development and Support, Structuring, and Language) scale. A third reader read "splits," in which scores from the first two readers differed by more than a point on any of the features, for the Spring essays. Readings were supervised to make sure that the 310 essays were distributed so that the same student’s essays were read by as many different members of the pool as possible.

Before beginning work with each of the ten types of "live" essay, the readers participated in a group norming session with two "range-finder"essays, presented blind but each chosen from a separate Subject A and Honors sample outside the core sample of 31 students. In budgeting for the reading, administrators correctly assumed that norming would take up a significant amount (approximately 1/3) of the reading time.

In exit surveys readers were generally positive about the objectivityof the study and the procedures for norming.<11>

Overall Results of the Normed Reading (See Appendix D – "Resultsof Primary Trait Scoring")

The blind assessment of writing indicated that student prose kept pace with more and more demanding assignments that required more integration of sources (primary and secondary), more citation of evidence (direct quotation and paraphrase), progressively more complex kinds of arguments (definition, counterargument, explication, and causal analysis), more grammatical subordination, and more independent inquiry and research. The rubric was written so that scores of "1" and "2" characterize gross errors of the kind that commonlyoccur when students are faced with more complex writing tasks. Different outcomes from different student populations (Honors, Regular, Subject A) seem to support HCC practice of customizing instruction to different sectionsand limiting the teaching of Subject A and Honors students to ladder-rankedfaculty and experienced lecturers to meet their specialized curricular needs.<12>

Language Results of the Normed Reading (See Appendix D – "Results ofPrimary Trait Scoring")

Low scores in the "Language" column, a particular concern for a campus with so many second-language students, occurred in all three types of HCC section, which may indicate a need for more sentence-level instruction in the course. Because the Subject A Examination is a holistically scored entrance examination, students with deficient language abilities may not be identified by this exam.  The Writing Director of HCC strongly recommends more instruction on the conventions of standard written academic English in non-Subject-A writing classes campus-wide.

Other Results of the Normed Reading (See Appendix D – "Results of PrimaryTrait Scoring") Sentence-Level Analysis from the Writing Director: Conventions of Punctuationin Presenting Quoted Material (See AppendixE – "Summary of Sentence-Level Progress")

Progress in learning the conventions of Standard Written English at the sentence-level was probably most dramatically demonstrated in this portion of the study. The first sentence in which a quotation was integratedinto student writing, preferably with a parenthetical page citation, was analyzed using three samples: the third essay in Fall, the first essay in Spring, and the final essay in the course at the end of Spring.

These results appear to support the contention of the Directors of HCC that explicit instruction in sentence-level conventions is effective in Lower Division Writing.  Students cover this material in Winter in two week-long units: "Integrating Quotations Logically" and "Integrating Quotations Stylistically."

Sentence-Level Analysis from the Writing Director: Correct GrammaticalSubordination and Coordination in the Thesis Statement (See AppendixE – "Summary of Sentence-Level Progress")

It was expected that progress in learning the conventions of Standard Written English at the sentence-level would be less dramatic in this portion of the study. According to the ESL consultant who reviewed HCC files, Susan Earle-Carlin, students in the Humanities Core Course generally had fewer ESL constructions in the syntax of their writing than those enrolled in the Writing 39 series.  There were, however, significant signs of sentence-level progress at learning the skills of grammatical subordination and coordination with argumentative statements within the HCC sample. The last sentenceof the first paragraph, which is conventionally a thesis statement in Lower Division writing, was analyzed using three samples: the third essay in Fall, the first essay in Spring, and the final essay in the course at the end of Spring.

These results appear to support the contention of the directors of HCC that explicit instruction in sentence-level conventions is effective in Lower Division Writing. Students cover this material in Fall and Springi n two week-long units: "Thesis-Subjects / Thesis-Verbs" and "Subordination and Juxtaposition."

Summary of Recommendations to the Writing Coordinator:



Notes

1. The sequence for the 2001-2002 yearis pared to eight essays and one problem based learning assignment. The essays are as follows: Definition, Rhetorical Analysis, Counterargument, Image Analysis, Textual Explication, Comparison and Contrast, Causal Analysis,and the Documentary Research Paper.

2. Writing 39C, which was very positively reviewed in 1998, has a counterargument assignment, a causal analysis assignment,and a cumulative research paper.  Writing 39A and B have image analysis, rhetorical analysis, and application assignments as central to their courses.

3. “Improving Communication Skills, ”2001WASC Report, page 36.

4. To do such a study would require detailed entry and exit questionnaires from students, all drafts for written assignments, information from instructors to better understand the “ethnography” ofthe writing experience, and a comparable control group in HCC with a different sequence.

5. Although HCC received no funding from DUE for longitudinal study during the 2000-2001 year, samples were collected using a new web-based submission program designed to experiment with saving time otherwise spent collecting and copying student work by instructorsand staff.  (See http://e3.uci.edu/programs/humcore/study.)  Despite the great promise of web-based submission, first-year results were mixed.  Next year, when all students will submit all essays on-line, as well as in traditional form, as we use the www.plagiarism.org licensefor the course, we hope to have students better acculturated to online submission of their work.

6. The FIPSE-supported “Grammar from theGround Up” offered cash per essay as incentive in its Fall 2000 mass e-mailing.

7. Dr. Shoemaker’s initial analysis ofthe sample on July 10, 2000 was as follows:

By academic unit:
 
  Sample All HCC* New Freshmen All Undergrads
Arts 0%  [2%]   5%  6%
Bio  33%  [29%]  23% 17%
Engr 6%  [3%] 13% 10%
Hum 3% [19%] 6%  9%
ICS  0% [5%] 8%  6%
PS   0%  [2%]  3% 5%
SE  8%  [3%] 5%  10%
SS 19% [19%] 18%  26%
Unaffiliated 31% [18%]  20% 10%

Gender
 
  Sample New Freshmen All Undergrads
Female 64%  53%  52%
Male 36%  47%  47%
Unknown  0%  0% 1%

Ethnicity
 
  Sample All HCC* New Freshmen All Undergrads
Asian 42%  [49%]  56% 56%
Black  0% [2%] 2% 2%
Chicano/Lat 17% [12%]  11% 11%
Native Am.  0% [0%]  0%  0%
White 33%  [27%] 22% 22%
Other  8%  [9%] 8%  8%

Average UCI GPA

HCC sample = 3.09
All undergrads = 2.95

Other Comparisons:

Mean SAT Verbal (use highest score)
New Freshmen = 556
All undergrads = 531
[HCC Sample = 595]
[HCC All Enrolled Students = 595]*

First language -- data from F99, new freshmen only.
English only = 42%
English and another = 35%
Another = 23%
[HCC Sample = 45.95% / 27.03% / 27.03%]*

* Data on HCC enrollment derived from analysis OASIM P3107 (JCS 11-30-99).  These figures were not part of Dr. Shoemaker’ s7/10/00 analysis, but are inserted here for context.

Please note numerical differences from this initial analysisin final sample on Appendix B.

8.  Obvious reasons for deviationinclude disciplinary differences in the coursework and faculty, varying abilities in targeted skill areas in students, a course sequence aimedat successive difficulty, seasonal patterns of student attendance, etc.

9. Three of these four finished the coursewith A or A- writing grades in Spring.  The fourth received deficient grades in Fall and Spring quarters.

10. This study of 19 two-time Subject A students found that students in the LARC group averaged 2.54 writinggrades in Winter (compared to 2.00 for total non-LARC) and 2.62 in Spring (compared to 2.375 for total non-LARC).  This is particularly impressive given that students were chosen for the LARC group based on lower gradesin Fall  (1.62 for LARC and 1.67 for non-LARC).

11.  Responses from the anonymous questionnaires were as follows:
- “The norming allowed me to see other perspectives in the essays; thus, I think that it helped me to a better job for the study. Through repeated normings, I think we made the study as objective as possible.”
- “Not having taught the assignments or works in question seemed to add to the objectivity.”
- “Very well organized, systematic and insightful. Professionally done.  I think a uniform standard and criteria for evaluation were established.  The study was quite objective, especiallygiven the time and attention to norming.”
- “The factor that, to me, confirmed its objectivenessis that the readers had not taught the assignments involved.  I’m confident that we approached these papers as general writing samples, ratherthan having specific expectations about what we wanted students to produce, and so were not biased in that way.  The rubric helped to keep us focused on the general categories that define weak/strong writing.”
- “Pretty generous in comparison with other normings I have experienced.  That is, the response to our attempts at norming could have been more decisive and instructive.  If there is a wayto open the scale to ½ points, this might improve the difficultyof some decisions.  I think that letting us know the origin of some [“rangefinder” not “live”] papers (Honors/Subject A, for example) may haveaffected some scoring decisions.  Suggestions about the kinds of progressbeing evaluated and speculations about the outcomes might be considered a slight detriment to objectivity.  Seems like a well-designed and worthwhile study.”
- “Generally very helpful and productive – even fun atpoints.  Liz was good at working to establish a consensus by providing clear guidance and a rationale for norms.  Fairly objective, although when it comes to rating written work, it seems to me there is an inescapablefactor of subjective judgement that will always creep into the objective standards, but that just goes with the territory, I guess.”
- “Liz guided us into a shared criteria and judgements. I learned a lot.  Good to spend the considerable time used ‘norming’in advance.”

Brief Responses:
- Objectiveness “Generally good, I think.  Perhaps more care could be taken to eliminate student names and instructor marks”[Electronic copy should eliminate this occasional problem in future studies.]
- Objectivenes “OK”
- Objectiveness “seems fine to me”

12. Further longitudinal study of these groups should be done, however, to allow for the possibility that smaller groups sampled from Subject A and Honors magnified statistical anomalies in this study and created false differentiation from the largest group, Regular students.

13. This assignment was designed by History professor Robert Moeller, in consultation with Dr. Ray Zimmerman of the Composition program, who specializes in introducing academic discourse and principles of working with more than one text to students in Writing 39A, which also fulfills the Subject A requirement for the UCI campus.

14. Student #27 (7 sub-2 language scores)is a Chinese/Chinese-American with an “Only English” background and a 660 verbal score.  Student #1 (5 sub-2 language scores) declined to state his ethnicity; he has an “Only English” background also and a 750 verbalscore.