• View
• Edit
• History
• Print

Mike Tolinski

Effective Assessment & Feedback

How do forms of writing assessment and feedback meet student needs for placement and performance in BW composition courses?

Students placed into basic writing (BW) courses are best served by good practices of the instructors who evaluate their work and provide useful feedback about their progress. Unfortunately, the teaching of BW is sometimes complicated by less-than-reliable assessment methods used to place students. The difficulties are compounded for the many students who are sensitive about being placed into BW courses, given that some may presume they are doomed to being “bad writers” beyond help. However, these difficulties can be addressed through careful consideration of writing course placement methods, and through good BW classroom practices for providing feedback. As discussed in these FAQ components, practices that individualize student abilities and needs and put them into context are likely to be the most effective.

CompPile Search Terms: assessment, feedback, placement, testing, practice, commenting, reliability, validity, conferencing, basic, evaluation, cost, first-year

How is writing assessment used effectively to place students into first-year composition courses?

The identity of “basic writing student” is typically defined by post-secondary institutions through their use of various methods, usually involving some form of assessment. Assessment for initial placement may take many forms, but most forms reflect each institution’s need for relatively quick, inexpensive, reliable, and valid methods for placing students into specific first-year composition classrooms that would best meet the students’ (and institution’s) needs.

However, several factors determine how well a student’s needs are met, or even can be met, by a given placement method. Assessment of student abilities and placement through testing can be problematic, since it is influenced by each institution’s goals of limiting access only to students capable of doing college-level work, while upholding certain writing standards (variously defined or idealized). And ideally, assessment should allow an institution to do these things while expending a minimum of resources.

Because impromptu, timed, single-draft essay tests are often seen as fulfilling an institution’s and students’ requirements to at least some degree, these are often the assessment methods of choice for first-year writing placement. Such testing can show immediately a student’s difficulties in basic mechanics or organization that could be addressed in a BW course, which would then help the student understand fundamental requirements of later academic and professional writing. This may be why, in Richard Haswell’s 2005 overview of first-year placement methods, the first four of the ten first-year assessment/placement practices he lists (used by U.S. institutions) rely on timed/impromptu essay testing (“Post-secondary Entrance,” 2).

When they decide to rely on impromptu essays for placement, institutions are faced with many questions; below are just a few basic ones to consider:

• Is placement using impromptu essays fair for the student—can it measure essential characteristics that determine if a writing student needs preliminary coursework or extra help?
• If it does, how can essays be evaluated reliably for correct placement?
• What are some of the problems with using this kind of assessment?
• How do other methods of assessment, both direct and indirect measurements of writing ability, compare?
• How do people in the college composition community and other education interest groups compare in their views on whether timed essay assessment is a legitimate method for evaluating student abilities?
• And what can we learn from composition scholars to improve assessment for placement (or can it be improved, or is it appropriate to use)?

WORK CITED

How are impromptu/timed essays used to provide information for initial placement?

The practice of using impromptu/timed essays for first-year college composition placement varies somewhat across two- and four-year institutions, but it generally continues to be popular. Even though major questions arose during the 1990s about the legitimacy of this kind of assessment/placement method (White, 30), institutions still find many reasons for using a single unrevised block of student writing, performed during a 20–60 minute period, as the basis on which decisions can be made for placing the student. One reason is that impromptu essay testing is a direct measure of writing ability, theoretically making it more valid than indirect methods such as multiple-choice testing. And essay evaluation methods can be made, to some degree, reliable (consistently scored, and fair to the student)-- though this is not an easy task, and the reliability is not always high.

Despite the rising interest in portfolio assessment as a more accurate way of determining writing ability, Edward M. White in 1995 argued “the time has come for portfolio advocates… to recognize the important role essay testing has played in the past—and can still play—and to stop attacking essay testing as an unmitigated evil…” (31). Some might choose to look on such timed essays as an “abbreviated portfolio,” to use White’s words (32), which provides enough information for entrance placement and is clearly preferable to multiple-choice testing.

Most impromptu testing involves providing a writing prompt to incoming students, who produce essays that are then scored in some way that allows placement decisions to be made. But there are variations in how and when impromptu essay testing for placement is used by different institutions. Below are just a handful of the practices documented by Haswell (“Post-secondary Entrance,” 5–6):

• The placement essay is given in the third week of the composition course, after which “at-risk” writers are given extra instructor support in the course.
• The placement essay is run as a “class” for entering students during summer orientation, allowing time for instructor feedback and discussion.
• After the placement essay is used to place students, students are allowed to study the criteria the raters used, “turning a testing process into a learning one.”
• The institution adds a second essay to create an “abridged portfolio” that is used for placement decisions.

Such variations allow institutions to maximize the validity and reliability of the use of placement essays.

How are timed placement essays scored or rated?

Because of the inherent difficulties of using a single piece of unrevised writing for defining a student as a basic writer or not (while minimizing the costs for making this assessment), the rating or scoring methods for placement essays also varies by institution. Essays might be evaluated by English teachers who are composition/BW instructors or other English instructors, or by outside firms or agencies (“Post-secondary Entrance,” 2).

Placement teams generally seek evaluation methods which are consistent, grade-to-grader, for reliability. Holistic or “general impression” scoring allows graders to evaluate multiple attributes of writing, rather than just focusing on a few salient mechanical features. Essay scoring on a scale (typically 1 to 6 or 1 to 4) provides scoring efficiency and allows the opportunity for re-grading of an essay when graders initially disagree. The 60-minute placement essay required by the Skills Assessment Program of the City University of New York (CUNY) is one such example of a placement essay practice using six-point scoring by two “trained readers,” though some scholars doubt whether even well-known writing assessments are appropriate for placement (e.g., Bruna et al., 77).

As an alternative low-cost assessment method (of debated propriety), computer-program essay scoring has been used by some institutions. Composition specialists generally reject such methods because the software is seen as focusing too much on certain mechanical features of writing such as grammar and word-choice—rather than on other critical issues of student writing that have to do with content and argument. The 2006 book Machine Scoring of Student Essays: Truth and Consequences (Ericsson & Haswell, eds.) provides several academics’ research on the problems of computerized essay scoring. As through this collection of essays, composition scholars are now responding to claims made about programs such as e-rater, which are presented through publications typically written by the providers of the product (and even some of these writers agree that automated essay scoring does not assess all important qualities of writing and have other disadvantages (e.g., Attali & Burstein, 13, 24)).

Debates will likely continue about essay scoring, and about which how much the different methods overemphasize superficial writing standards and mechanics. Even holistic human scoring has been criticized as “forcing agreement to inappropriate standards by tyrannical methods” (White, 30).

WORKS CITED

How effective is timed-essay placement for meeting student needs?

“Where writing-placement systems work well, they protect the academic level of the course, support retention into the second year, and maintain and enrich faculty conversation about writing instruction. But they do not always work well.” (Haswell, “Post-secondary Entrance,” 2)

“Validity means honesty; the assessment is demonstrably measuring what it claims to measure. Reliability means consistency; the scores are demonstrably fair.” (White, 40)

Despite its prevalence and long use, impromptu-essay placement continues to be debated over in terms of validity, rater agreement and bias, test-design reliability, and social/political/cultural fairness. In his 2004 synopsis of placement practices (“Post-secondary Entry”), Haswell adds three more key issues crucial for placement research: “writer reliability,” or the degree to which a single piece of writing reflects a writer’s abilities; indirect measurement (such as multiple-choice tests), one answer to the high costs and low scorer reliability of direct assessment reliability; and predictability, the assessment’s correlation with later (basic) writer success in courses where they’re placed. Unfortunately, various studies have provided relatively varying degrees of clarity as to how well placement testing serves students’ needs, though they do show the many ways in which it can be ineffective.

What are some pitfalls of using timed essay assessment in assigning “basic writer” status?

Composition scholars have mapped out a minefield of potential problems institutions face when using timed essays for placement:

• Different kinds of students with different backgrounds may be poorly served by an assessment method, where “bad assessment is what gets most students labeled as ‘basic writers’” (Hilgers, 69). Even carefully considered holistic scoring methods and portfolio assessment can lead to the failure of non-traditional students because of a failure in assessment (e.g., Harley and Cannon, 72).
• ESL students invite particular problems when institutions test their writing abilities by typical methods, which might not truly determine their abilities at composing texts. Deborah Crusan notes that timed-essay testing can marginalize ESL students because many “have great difficulty producing fluent written [English] discourse, especially in timed situations” (21).
• One-shot timed essays tend to hide a student’s ability to revise his or her writing, an ability that is critical for all good writers. For example, Ann Del Principe and Janine Graziano-King find that allowing students to revise their essays in a controlled environment may present a more authentic portrait of writing ability for many students, especially in terms of essay focus and elaboration.
• In rating essay tests, a rater may have a tendency to devalue certain language as “bad writing,” Richard Haswell (“Dark Shadows”) notes that “incompetent” essays often have features that resemble “working-world” writing with features, such as a grasp of concrete language and compression of syntax, that would be favored in the workplace though perhaps not in the assessment (304, 311).

And finally, there’s a tendency in assessment practice only to compare essays, rather than to assess each student’s performance. “Diagnosis, good placement practice, looks through a placement essay in order to predict the student’s future performance, whereas pseudodiagnosis, poor placement practice, pretends to do that while actually only ranking the essay in comparison with other essays” (Haswell, “Post-secondary Entry”).

How do alternative assessment methods for first-year writing placement compare?

Since comp scholars generally agree that direct methods of assessing writing are better than indirect writing assessments (such as multiple-choice tests), impromptu essays fulfill this minimum requirement. However, some institutions continue to use indirect methods as their primary means for placement. These inexpensive, computer-mediated tests are meant to provide some picture of a student’s writing ability. Some examples include ASSET Writing Skills test put out by ACT, Inc., a 36-item test that provides students with three text samples and asks them to make choices in sentence structure, grammar, style, and organization. The Texas Higher Education Assessment (THEA) multiple-choice writing section test similarly consists of 16 passages and asks students to pick a few appropriate changes to each passage. But such tests lack opportunities for students to be evaluated on their production of texts.

Potfolio assessment of incoming students’ high-school writing has been used as means for first-year placement. This answers the need for direct writing evaluation, while also allowing a student to be judged by his or her revised work across a variety of genres and writing purposes. For example, the University of Michigan in the 1990s (Willard-Traub, et al.) attempted to replace its 50-minute placement essay with portfolio assessment. After the university started using portfolio evaluation, data showed that portfolio assessment placed about the same percentage of students into the (basic) writing practicum course as did timed-essay placement (roughly 12%) (52). Moreover, composition instructors were satisfied that about 80% of their students had been placed properly by portfolio assessment (though no data for the timed-essay period existed for comparison) (56). However, portfolio evaluation had showed itself to be more costly; thus, after a few years of development, Michigan reverted back to the lower-cost placement practice (82).

Independent of costs, an ideal assessment method would likely use multiple placement methods to get the best picture of a student’s writing ability and language practices—that is, educators “should implement multiple measures and validate with multiple measures” (Haswell, “Post-secondary Entry”). Moreover, an understanding of the context of the student-writer’s writing situation is key for making valid placement decisions. In terms of essay testing, this “validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests” (American Educational Research Association et al., quoted in Elliot, 268).

What are some current concerns about assessment for placement?

Composition community professionals and public- and private-sector groups (such as ETS) have taken positions on how or if impromptu essay tests are legitimate means of evaluating writing ability for placement or other purposes. Despite sometimes very public disagreements about the use of standardized testing (e.g., Elliot, 310), some agreement can be found between composition educators and non-teachers in their basic opposition to timed, unrevised essay tests and to multiple-choice/indirect tests, found when comparing the November 2006 CCCC’s postion statement on assessment to the business-influenced National Commission on Writing’s “Writing and School Reform” (containing the 2002 “Neglected “R”” report). Both documents do also emphasize the importance of human reader-evaluators of student writing, though the NCW’s document is open to the “exploration” of improved technologies for software that measures writing competence (69).

Groups of scholars are also concerned with how traditional writing assessment should be done in this new century. Along with the documents above, the November 2007 draft position statement on assessment of the Council of Writing Program Administrators emphasizes how assessment must account for the multiple contexts writers face when addressing the numerous needs of their 21st-century audience. The council implies that no longer can “pen-and-paper writing” be an adequate basis for assessing a student’s abilities.

From within their own field, writing program administrators (WPAs) and instructors are faced with sometimes extreme positions of prominent theorists about assessment. One example is Ira Shor’s labeling of placement testing at CUNY as “bogus” (102), since it stigmatizes non-traditional students as “basic writers” who inversely can do quite well in mainstream courses. Other scholars question whether student writing should (or can) be judged on how well it fits future employers’ notions of “good” writing, or any other single standard (Bruna et al. 74, 76).

More recently, scholars such as Brian Huot and Patricia Lynne have argued that the context-specific nature of writing presents extreme difficulties for anyone who attempts to assess a piece of writing. Huot also sees writing assessment “as social action,” shaping instruction itself, and “possibly enabling certain students while limiting others. It also helps to make clear that often assessments imposed from the outside have specific political agendas that are designed to profit certain groups” ((Re)articulating Writing Assessment, 175). This makes many standard assessment practices suspect in that they potentially shortchange students of various abilities and cultural backgrounds.

Accordingly, traditional assessment methods may not be able to claim reliability and validity. “Writing assessment procedures, as they have been traditionally constructed, are designed to produce reliable (that is, consistent) numerical scores of individual student papers from independent judges,” Huot states (“Toward a New Theory,” 549). By contrast, he and other theorists deny that good writing has “fixed,” a-contextual traits – thus, “the validity of a test must include a recognizable and supportable theoretical foundation as well as empirical data from the students’ work” (550). Thus Huot argues it is dangerous to equate reliability in testing (or statistical consistency in scoring) with “fairness” for students, since this reliability depends on the nature of the standard judgments being made in scoring (557). Rather, “Assessment practices need to be based upon the notion that we are attempting to assess a writer’s ability to communicate with a particular context and to a specific audience who needs to read this writing as part of a clearly defined communicative event” (559).

Patricia Lynne’s 2004 book on assessment calls for a “new theory” of assessment that will address issues of context and the nature of writing better than traditional “objectivist” assessment based on measurement. “Educational measurement principles, most often signified by these terms ‘validity’ and ‘reliability,’… [ignore] the complexities of written literacy valued by compositionists, including the influence and importance of communicative context, the collaborative and conversational aspects of the process of writing, and the social construction of meaning” (3). As a step in this direction, she urges compositionists to change their language about assessment, replacing the terms of “validity” and “reliability” (which are built on the idea that objective assessment of writing can legitimately be carried out) with “meaningfulness” and “ethics,” respectively (117).

Such reevaluations of writing assessment would tend to weaken arguments that favor the traditional use of impromptu essays for placement, while perhaps favoring other methods such as directed self-placement. Ultimately, whenever timed-essay testing for placement is used, alternative approaches to assessment should always be considered to balance practical concerns, such as cost-effectiveness, with the basic fairness of directing a student to the course level that will best meet his or her needs.

WORKS CITED