National Council on Teacher Quality rankings: An outsider’s view

― Jeanne M. Slattery

Clarion’s teacher training programs were recently ranked by the National Council on Teacher Quality (NCTQ) in the bottom half of all schools evaluated. As an outsider to teacher training programs, I’ve been struggling to identify what their data really mean, as this ranking has a number of consequences for the College of Education, my department (which is not directly affiliated with the College of Education), and our university.

The NCTQ rank orders programs using a series of criteria that they review both from materials freely available on the internet and those provided by the Education programs and the university. They never make site visits, observe faculty teaching, nor interview students/principals, or superintendents. As Ray Puller, acting dean of the College of Education observed, “In a nutshell it is like evaluating a restaurant only by its menu” without ever going to the restaurant to taste the food, see how clean it is, or observe the service. This is an important point, as Clarion is accredited both by the Pennsylvania Department of Education (PDE) and National Council for Accreditation of Teacher Education (NCATE), which do these things regularly. In fact, Chris McCarrick, a faculty member in English, observed, “CAEP standards/assessments [which we do well on] are far more rigorous than NCTQ. My area (National Council of Teachers of English [NCTE]) demands even more rigor.”

Confused by this alphabet soup? Each of Clarion’s teacher education programs is accredited by the Council for the Accreditation of Educator Preparation (CAEP). Each program develops assessment instruments and rubrics aligned with CAEP’s standards. The instruments, rubrics and data collected undergo internal and external review, culminating in a comprehensive site visit. In contrast with the NCTQ’s input-based approach, CAEP-mandated assessments provide data that show what teacher-candidates know and are able to do: they are outcome-based. Clarion uses the results of these assessments to continuously improve its teacher education programs.

Although the NCTQ indicates that they received sufficient information to evaluate and rank Clarion University’s Education programs, it is clear that “people in the know” do not believe materials have been submitted, at least not since 2011. It is unclear why this may have happened but, fundamentally, no information only means no information, not that the programs are in some way deficient. Perhaps one of the things we could learn from this report is that we should spend more time communicating about the quality and delivery of our teacher training programs.

Figure 1. Imagine a distribution where everyone falling above the red line passed, everyone below the red line failed.

Figure 1. Imagine a distribution where everyone falling above the red line passed, everyone below the red line failed.

As a psychologist, I am interested in the methodology used to rank and evaluate these programs. What we know is that the upper 50% of the schools (presumably schools that provided information) were rank ordered in terms of quality. There are at least three interesting implications to this process:

First, regardless of quality, 50% of programs were guaranteed to fail. See Figure 1. I can guarantee that my students would be unhappy if I were to use a similar grading strategy next Fall!

Figure 2. Two different possible distributions, with very different implications. In Group A, even "failing" programs did very well, while in Group B, even "passing" programs performed poorly.

Figure 2. Two different possible distributions, with very different implications. In column A, even “failing” programs did very well, while in column B, even “passing” programs performed poorly.

Second, because programs were rank ordered, we do not know whether the programs in the top half were in any meaningful way different from those in the bottom half. See Figure 2. My students would be unhappy with this sort of distribution ― and so would I! For example, I could imagine that all or almost all of my students performed very strongly, yet half “failed” (as in column A, Figure 2). On the other hand, my students and I would be equally unhappy if our grade distribution looked like column B, where people “passed,” despite poor overall performance. We don’t know whether NCTQ’s data look more like column A or column B (or any of the other possibilities), and thus have difficulty interpreting their numbers and taking their conclusions seriously.

Third, because their data are only presented ordinally rather than with some sort of interpretative label (e.g., Exceeds All Criteria), programs falling below the cut-off could be performing quite well ― or not. They could be performing in the 49th percentile (about average) or the 1st (poorer than almost all programs). We just cannot tell! The 50% cut-off is an arbitrary one, which says little about a program’s actual performance.

* * *

Climb+that+Tree!

We love this cartoon and want to credit the artist. Please let us know if you can identify the artist.

Concerns about the NCTQ rankings are even more serious than what I have described so far. Pennsylvania State System of Higher Education (PASSHE) described their concerns in this manner, “There will always be disagreements about what are the best methods to use; that is to be expected. Because of the differences in how the states collect and analyze data, it is extremely difficult to make true comparisons.” The inherent assumption of rankings such as NCTQ’s is that there is one and only one way of teaching and of demonstrating success, just as in this classic cartoon.

Sawchuck (2014), writing in Education Week, was more pointed in his criticisms of the validity of the NCTQ data:

Criticism focused on the NCTQ’s tack of reviewing syllabi and other course materials rather than visiting institutions; its use of open-records requests and current students to obtain documents; the complaint that its standards weren’t agreed to by the profession; and the fact that its research products aren’t peer reviewed. Additionally, critics have claimed that the project is ideologically driven, given NCTQ’s role as incubator of an alternative-certification group, the American Board for Certification of Teacher Excellence (ABCTE), which received federal funding from the George W. Bush administration. (para. 9)

Sawchuck’s (2014) last point is an interesting one: science is rarely value-free. What assumptions does NCTQ hold about teacher training? Ravitch (2012) describes these assumptions:

NCTQ was created by the Thomas B. Fordham Foundation in 2000. I was on the board of TBF at the time. Conservatives, and I was one, did not like teacher training institutions. We thought they were too touchy-feely, too concerned about self-esteem and social justice and not concerned enough with basic skills and academics. In 1997, we had commissioned a Public Agenda study called “Different Drummers”; this study chided professors of education because they didn’t care much about discipline and safety and were more concerned with how children learn rather than what they learned. TBF established NCTQ as a new entity to promote alternative certification and to break the power of the hated ed schools. (para. 11)

While any program ― teacher training or otherwise ― should consider whether or not it is doing a good job, there are sufficient grounds to question whether NCTQ has helped us answer that question. As Sawchuck (2014) observed, NCTQ’s conclusions should be questioned, given that there is no national consensus about standards, goals, and assessment methods. NCTQ’s data appear to be weak, indirect, and incomplete, and seem focused more on freely-available products on the internet rather than assessed outcomes (which are generally the gold standard). Finally, ranking programs against each other provides little information as to whether individual programs provide strong teacher training, at least if you agree with PDE’s and NCATE’s methods and conclusions.

Faculty, students, and the general public should want to know whether programs are doing their job well. This is an important question for all constituencies. Unfortunately, NCTQ, rather than having resolved this important question, seems to have muddied the water and confused the public.

References

National Council on Teacher Quality. (2014, June). Teacher prep review 2014 report. Retrieved from http://www.nctq.org/dmsStage/Teacher_Prep_Review_2014_Report

Ravitch, D. (2012, May 24). What is NCTQ? (and why you should know). Washington Post. Retrieved from http://www.washingtonpost.com/blogs/answer-sheet/post/ravitch-what-is-nctq-and-why-you-should-know/2012/05/23/gJQAg7CrlU_blog.html

Sawchuck, S. (2014, June 17). Alternative certification deemed weak by NCTQ in new teacher-prep report. Education Week. Retrieved from http://blogs.edweek.org/edweek/teacherbeat/2014/06/alternative-certification_deemed_weak.html

Thanks to Melissa K. Downes, Christopher J. McCarrick, Shannon Nix, and Randy Potter for their thoughtful and insightful comments on earlier drafts of this piece.


Jeanne M. Slattery is a professor of psychology at Clarion University. She is interested in thinking about what makes teaching and learning successful, and generally describes herself as a learner-centered teacher. She has written two books, Counseling diverse clients: Bringing context into therapy, and Empathic counseling: Meaning, context, ethics, and skill (with C. Park), and is writing Trauma, meaning, and spirituality: Research and clinical perspectives. She can be contacted at jslattery@clarion.edu

Advertisements
This entry was posted in Teaching and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s