An expert on the subject writes in:
I hate that I have to partly defend Tony Bennett, but here goes. The education reform movement has sold politicians and the public what is, at best, the polite fiction of objectivity in school accountability. There’s some truth to Michael Petrilli’s flippant remark you quoted, describing these systems are “more akin to baking cookies than designing a computer,” and Bennett’s involvement is more the rule than the exception. I should know, I’ve worked in the kitchen.
The basic architectures of these systems are sketched out by elected officials, high-level education bureaucrats, and stakeholders (such as teachers’ unions or reform lobbyists). Discussions tend to focus on the traditional elementary-middle-high school model, and final designs are a compromise between what they’d like to measure and what’s actually available in the data.
Most of these systems are based on some combination of pass/fail percentage, graduation rate, attendance rate, and test score improvement or growth (Jeb Bush’s group has been particularly instrumental in pushing the latter as the solution to the snapshot-in-time “failure” of NCLB). The basic idea is that each measure has a value that’s converted to a score, and then all the scores are combined into a composite from which accountability identifications are made.
In practice, it’s not that simple.
There are well-known statistical problems with aggregated test scores from small populations, and every state sets minimum sizes for inclusion of a group. Different measures have different distributions: pass/fail rates almost always follow the classic bell curve, while attendance rates bunch at the high end with a long tail of “problem” schools. Combining measures requires making value judgments on balancing relative priorities.
Then states find schools that don’t have all the data to fit their designs, or don’t match the stereotypical model. What do they do with a K-12 school? How about a school with too few kids? Stretching the design to encompass more schools adds yet another layer of complexity and subjectivity. (This is where Indiana ran into trouble: they hadn’t accounted for charter high schools too new to have graduated any kids yet, so averaging what measures they had made this school’s score inherently worse than high schools with graduation rates, which often cluster toward the high end.)
Once states develop scores, they have to assign meaning to them. Specifying which scores fit what labels – either by defining ranges or translating scores to fit a particular rubric (like Indiana’s GPA) – is the most subjective and political part of the process. Decision-makers inevitably test designs against what they believe about particular schools. I guarantee that every state with this sort of school accountability system has an email chain like Bennett’s, with a top official questioning a favorite school’s score as too low.
Did Bennett cross an ethical line? He certainly sidled up to it, but whether or how far he actually stepped over depends on two things: First, how much Indiana’s system was modified to fit one school, as opposed to all schools with the same quirks. Second, whether there was a quid pro quo between the grade modification and campaign contributions Bennett received from the school’s founder (knowing how the reform movement operates, I suspect there was but doubt there’s any way to prove it).
So what does this mean? Major factions within both parties have bought into these accountability systems, even though most who champion them – including and especially Arne Duncan and his staff – don’t understand their complexity. Politicians and special interests misuse information that ought to identify trends and target improvements as ideological weapons. The media’s default position of blind credulity occasionally gives way to scandal-mongering, without any real attempt to understand the issue.
How’s that for a cheerful perspective on the future of American education?
Updates from readers originally posted in this post, in addition to an email from an Indiana school official, can be found here.
The Dish 