The Good, Bad & Ugly of Value-Added Analysis

Assessment, Learning, Teacher Quality

I’m on the road all week — from DC to Oregon to Philadelphia to Oklahoma City — and everywhere I go people seem to be talking about the L.A. Times’ recent expose into the city’s school teachers, and the extent to which individual teachers are either helping students learn — or holding them back.
The conversations are based on the Times’ decision to use value-added analysis, which rates teachers based on their students’ progress on standardized tests from year to year. Thickening the plot, the Times produced this report using seven years of data the school district had — but had never analyzed. As the paper explains: “Value-added analysis offers a rigorous approach. In essence, a student’s past performance on tests is used to project his or her future results. The difference between the prediction and the student’s actual performance after a year is the ‘value’ that the teacher has added or subtracted.”
Because the idea of value-added analysis, or VAA, seems to be everywhere in K-12 education discussions (it has been embraced by the Obama administration, and many of the field’s leading philanthropic entities, from Gates to Walton to Broad, are intrigued by the approach), I want to offer what I see as the good, the bad and the ugly of VAA — and of the Times’ decision to use VAA as the foundation of its landmark report:
The Good — As the Times rightly reports, “though the government spends billions of dollars every year on education, relatively little of the money has gone to figuring out which teachers are effective and why.” This has been a catastrophic failure by all of us to this point, since all sides agree the effectiveness of the teacher is the single most important in-school factor toward determining the extent to which young people will learn. Children should not have their minds subjected to a roulette wheel of opportunity; every child deserves a highly-effective teacher. And although the Times article is primarily about the VAA scores, it identifies other core conditions the most effective teachers shared, including the encouragement of critical thinking and “the surest sign of a teacher’s effectiveness — the engagement of his or her students.” In this way, the Times demonstrates its seriousness in trying to unpack the mystery of what makes some teachers more effective than others. And we need as much of that as we can get.
The Bad — Unfortunately, despite some caveats throughout about how VAA would only make up a percentage of a teacher’s future evaluation, the reality is that VAA is assuming a disproportionate share of the emerging analyses. The Times says as much when it admits that VAA “offers the closest thing available to an objective assessment of teachers.”
But what if the closest thing available isn’t actually the closest thing to the truth? Based on this logic, the NFL should only draft college players based on the things it can observe objectively — like their 40-yard dash times, or the number of times they can bench press 225 pounds. Yet as any fan knows, this isn’t the best path toward finding the best players (but don’t tell Al Davis). And a similar approach in education is also not the best path, precisely because it simplifies an extremely complex undertaking. After all, if the only thing I’m going to be evaluated on is my speed, why bother working on my pass-catching ability? And if in reality all that matters is my VAA score (despite what people say), why do anything other than focus on preparing children for the tests? So the “bad” is less about the VAA scores, and more about their being used in relative isolation. When we do that, we get Campbell’s Law.
The Ugly — I see the ugly aspects of this unfolding on both sides of the debate. For the Times, I think their article reflects their own limited understanding of education, and teaching, and the core conditions of a powerful learning environment. Journalism exists to educate the general public about core issues that are essential to our civic health and well-being. The Times says it wrote the article to help parents stop feeling like they’re in the dark when it comes to their children’s schooling. Yet the main thing I notice when I speak to my friends who are parents (and non-educators) is how much they feel they must rely on test scores to gauge a school’s overall health — even though any good educator knows it provides no more than a partial sliver of the picture a parent needs in order to make a sound decision about where they should send their child. Does the Times’ study help paint a fuller picture for parents? Or is it merely painting over the same narrow corner of a canvas that is in fact much larger, richer, and more opaque than we’d like it to be?
On the other side, I see that the L.A. Teachers Union has already called for a massive boycott of the Times, and is looking into legal action.
Can I please apply to be the Union’s communications director?
Why wouldn’t their lead story be a public statement that uses the article to amplify the shared need for better information about how students learn and what teachers can do to be more effective — and then restate that test scores from year to year represent a single piece of the picture that is both valuable and overvalued? In effect, use the Times article to focus attention on the need for better, more balanced information about student learning and teacher effectiveness, not simply to excoriate a major newspaper and deny any validity of VAA whatsoever.
But that’s not what they did, of course, which will only contribute to the growing national sense that teachers’ unions are the most convenient villains in an evolving script that is much more complex than good guys v. bad guys (or reformers v. the status quo).
Once again, I’m left with the same simple thought: We can do better.