Student essays always seem to be
riddled with the same sorts of flaws. So sociology
professor Ed Brent decided to hand the work off -
to a computer.
Students in Brent's Introduction to Sociology
course at the University of Missouri-Columbia now
submit drafts through the SAGrader software he
designed. It counts the number of points he wanted
his students to include and analyzes how well
concepts are explained.
And within seconds, students have a score.
It used to be the students who looked for
shortcuts, shopping for papers online or pilfering
parts of an assignment with a simple Google
search. Now, teachers and professors are realizing
that they, too, can tap technology for a facet of
academia long reserved for a teacher alone with a
red pen.
Software now scores everything from routine
assignments in high school English classes to an
essay on the GMAT, the standardized test for
business school admission. (The essay section just
added to the Scholastic Aptitude Test for the
college-bound is graded by humans).
Though Brent and his two teaching assistants
still handle final papers - and grades - students
are encouraged to use SAGrader for a better shot
at an "A."
"I don't think we want to replace
humans," Brent said. "But we want to do
the fun stuff, the challenging stuff. And the
computer can do the tedious but necessary
stuff."
Developed with National Science Foundation
funding, SAGrader is so far used only in Brent's
classroom. Like other essay-grading software, it
analyzes sentences and paragraphs, looking for
keywords as well as the relationship between
terms.
Other programs compare a student's paper with a
database of already-scored papers, seeking to
assign it a score based on what other
similar-quality assignments have received.
Educational Testing Service sells Criterion,
which includes the "e-Rater" used to
score GMAT essays. Vantage Learning has
IntelliMetric, Maplesoft sells Maple T.A., and
numerous other programs are used on a smaller
scale.
Most companies are private and offer no sales
figures, but educators say use of such technology
is growing.
Consider the reach of e-Rater: 400,000 GMAT
test-takers annually, a half-million U.S. K-12
students and 46 international schools and
districts. ETS says an additional 2,000 teachers
begin using its technology each month.
But it's tough to tout a product that tinkers
with something many educators believe only a human
can do.
"That's the biggest obstacle for this
technology," said Frank Catalano, a senior
vice president for Pearson Assessments and
Testing, whose Intelligent Essay Assessor is used
in middle schools and the military alike.
"It's not its accuracy. It's not its
suitability. It's the believability that it can do
the things it already can do."
South Dakota is one of several states that has
tested essay-grading software. Officials there
decided against using it widely, saying feedback
was negative.
Not all districts had the same experience.
Watertown, S.D., students are among those who now
have their writing-assessment tests scored by
computer.
Lesli Hanson, an assistant superintendent in
Watertown, said students like taking the test by
computer and teachers are relieved to end an
annual ritual that kept two dozen people holed up
for three days to score 1,500 tests.
"It almost got to be torture," she
said.
Some 80 percent of Indiana's 60,000
11th-graders have their English assessment scored
by computer, and another 10,000 ninth-graders are
taking part in a trial in which computers assess
some routine written assignments.
Stan Jones, Indiana's commissioner of higher
education, said the technology isn't as good as a
teacher but cuts turnaround time, trims costs and
allows overworked teachers to give written
assignments without fearing the workload.
"This (allows) them to require more
essays, more writing, and have it graded very
painlessly," Jones said.
Software can also remove a degree of
subjectivity.
"It's fairly consistent. Different
teachers grade different papers differently."
- Keith Kelly, 21, of Cleveland, one of Brent's
sociology students.
The software is not flawless, even its most
ardent supporters admit.
When the University of California at Davis
tried out such technology a couple years back,
lecturer Andy Jones decided to try to trick
e-Rater.
Prompted to write on workplace injuries, Jones
instead input a letter of recommendation,
substituting "risk of personal injury"
for the student's name.
"My thinking was, 'This is ridiculous, I'm
sure it will get a zero,"' he said.
He got a five out of six.
A second time around, Jones scattered
"chimpanzee" throughout the essay,
guessing unusual words would yield him a higher
score.
He got a six.
In Brent's class, sophomore Brady Didion
submitted drafts of his papers numerous times to
ensure his final version included everything the
computer wanted.
"What you're learning, really, is how to
cheat the program," he said.
Work to automate analysis of the written word
dates back to the 1950s, when such technology was
used largely to adjust the grade level of
textbooks, said Henry Lieberman, a research
scientist at the Massachusetts Institute of
Technology.
Before long, researchers aimed to use such
applications to evaluate student writing.
SAGrader, like other programs, needs
significant prep work by teachers. For each of the
four papers Brent assigns during his semester-long
course, he must essentially enter all the
components he wants an assignment to include and
take into account the hundreds of ways a student
might say them.
Part of one assignment for Brent's class was
for students to pick a crime and explain how it
fit into sociologists' categories. Brent had to
key in dozens of words in order to ensure all
types of transgressions would be identified.
What a writer gets back is quite detailed.
A criminology paper resulted in a nuanced
evaluation offering feedback such as this:
"This paper does not do a good job of
relating white-collar crime to various concepts in
labeling theory of deviance."
Brent - who earned a postdoctoral degree in
artificial intelligence and is also an adjunct
professor in the computer science department -
said the software may have limitations, but allows
teachers to do things they weren't able to do
before.
Before Brent wrote SAGrader, a part of his
broader data-analysis program Qualrus, he only
gave students multiple-choice tests.
"Now we can focus more," he said.
"Are they making a good argument? Do they
seem to understand? Are they being creative?"
© 2005 The Associated Press
