For a person who spends as much time thinking about digital communication as I do, it is odd to go on my annual pilgrimage to the Clark Kerr campus at U.C. Berkeley to serve as a table leader at the system-wide Analytical Writing Placement Examination. (Those who know me outside of this capacity may know that Clark Kerr is a not-so distant relative, as I believe my second cousin twice-removed. Here I am with a bust of my bureaucrat ancestor.)
To date, evaluating critical thinking in college-level writing is a task that can't be done efficiently by anything other than human beings, and unlike other "mechanical Turk" operations, it is considerably more complicated than simply identifying generally agreed-upon traits, expressed by statements such as "this is a pink shoe." Furthermore, I know enough about software that can create computer-generated writing and the temptations of cut and paste plagiarism to be wary of the eventual advent of automated evaluation without live minders. It would be nice, of course, if the essays weren't hand-written in paper booklets, sometimes with scrawl less legible than gang graffiti or an x-height ant-high.