As the prevalence of teacher accountability measures has risen over the past decade so too has the evidence base about their potential bias against teachers of color. In this paper, we use item response theory (IRT) to build on this base of evidence, finding that teachers of different identities with comparable teaching expertise receive systematically biased observation ratings on specific items across the continuum of instructional effectiveness. Given the use of these measures in high-stakes decisions around educators’ careers, the repercussions of structural bias in teacher evaluations not only are of direct harm to teachers (and students) of marginalized identities but also reinforce the dominant cultural hegemony in our educational system.