Grading Teachers

This post is a follow up to my earlier post criticizing a paper, "The Economic Effect of Higher Teacher Quality", by Hanushek in which I address an issue raised in the comments.

Evaluating teachers based on how well their students do academically is difficult because teachers aren't (currently in the US) the only or even the most important factor in student success. So to properly measure the effect of teachers you first have to account as best you can for the other important factors. This is typically done by constructing a model which predicts how well a student will do based on everything besides their current teacher and then comparing this to their actual performance. This prediction will assume in effect that their current teacher is average. Any difference in actual performance is then attributed to the teacher being above or below average (depending on whether the student did better or worse than predicted). Clearly this will be very unreliable for a single student but the hope is that when averaged over many students the errors will tend to cancel out allowing differences in teacher quality to be detected.

What do these models look like. The most important factor in how well a student does is the characteristics of the student themself. This means things like how smart they are (in terms of IQ), how well educated their parents are, what their household income is etc. Such differences among students are far more important in predicting how well they will do academically than differences among their teachers. Since we are generally interested in evaluating the effect of a teacher over a school year another important factor is how well the student has done previously. If you are evaluating a third grade teacher and at the beginning of third grade a particular student is 4 months above their predicted grade level this must be accounted for when predicting where that student will be at the end of third grade (assuming an average teacher). Empirically students doing better or worse than expected at the beginning of a school year will still be doing better or worse than otherwise expected at the end of the school year but not by as much. So for example the student 4 months ahead of their predicted grade level might be expected to be 2 months ahead of their predicted grade level a year later. Similarly a student 4 months behind might be expected to be 2 months behind a year later. Models typically account for this by including a decay factor r so a student x months ahead of their expected grade level at the beginning of a school year will be predicted to be r*x months ahead of their otherwise expected grade level at the end of the school year. Note such models predict the difference n years later will (r**n)*x, this is a consequence of iterating the model predictions.

Hanushek uses such a model in his paper but does not appear to understand the implication noted above that any good or bad effects of for example 1-3 grade teachers will have largely decayed away by the time their students leave school and enter the work force. Hence as I noted before he does not appear to be correctly computing the predicted economic effects based on his own model assumptions.