158: That's How We Do It In Government

This podcast is adapted from an entry in my education blog, available at seligman4schools.blogspot.com .  That blog occasionally contains comments a bit too controversial for Math Mutation, though I would always be interested in feedback from podcast listeners.  I hope to see some of you there!  Now, let's get started.

At a recent meeting of the curriculum committee discussing progress in English Language Learner education, a Hillsboro, Oregon School District official presented a statistic that looked something like this:
Mean test score: 73%
Margin of error: 9%
Adjusted score : 82%

Now, for any of us in engineering or other professions that use margins of error, this looked distinctly odd. A "margin of error" represents the imprecision in a measurement, and inherently can show uncertainty in either direction, at some specified level of confidence.   So a rating 73% with a 9% margin of error might mean we are 95% confident that the true score has a range of 64%-82%.   It's equally likely that the valid result is at either end of the range.   Why does it make sense to add the margin of error to the mean score, calculating the maximum score at the high end of the confidence interval, when reporting the official result? The answer, when I raised the question: "That's how we do it in government."

This is a nice trick: it enables every statistic to be presented in the best possible light for promoting the success of current public officials. It also is inherently insane, in my opinion, granting bonus points for the imprecision of the measurement.  Think about it: normally, measurements with lower margins of error are seen as more valuable, as they give a clearer and more precise picture. But look at the scores above: if they worked hard on developing a better test and lowered their margin of error, the "adjusted score" would actually be penalized! And if they know the true scores are going down, they can game the system by lowering the quality of the tests or sampling, aiming to increase the margin of error rather than improving student knowledge.   Is this the right way measurements should be done in our education system?

I can see how this would become the custom in government: once one official does it, everyone else has to follow suit, or else their statistics would appear inferior. Imagine if the district suddenly stopped "adjusting" these scores. "Look, in Hillsboro the scores went down 9% this year!" Any elected officials involved would see their opponents demagogue the issue, and the employees who stopped the adjustments would suffer for it.

Don't take this post as a criticism of the particular official (Travis) who made this presentation though: in fact, I am commending him. In a regime where this silly "adjusted score" must be produced, the most intellectually honest policy is to do what he did: present the actual source numbers in addition to the final adjusted score, and let the viewers see the full story. I'm happy to see our district doing this.

The big lesson: any time a government body reports an "adjusted" statistic, look very closely at the adjustment, and demand to see the raw scores if not shown.

And this has been your math mutation for today.

Margin of Error at Wikipedia

My blog