Mikhail Zinshteyn has a piece on 538 about the failures of the SAT to predict college success:

The College Board argues that college readiness can be measured by how well a student scores on the SAT, one of the many standardized tests it produces. A student who earns a 1550 on the SAT out of a possible 2400, the College Board says, has a 65 percent chance of achieving a B- average in her first year of college. Students who clear this threshold graduate from college after six years 69 percent of time, while those who score below 1550 graduate in six years just 45 percent of the time, according to the College Board. In 2014, more than half of SAT test-takers earned scores lower than 1550, a sign to the College Board that they’re unlikely to be college-ready. {…}

In the study, Hiss and his co-author broke down the high school transcripts and college performances of 123,000 students in 33 colleges and universities of various sizes and statures that did not require test scores as part of the admissions process. The authors compared students who did submit ACT or SAT scores to those who did not, granting the institutions anonymity in exchange for access to student admissions data.

Overall, students who didn’t submit their ACT or SAT scores posted high school GPAs that were similar to students who did. The report also found that among the accepted students, those with strong GPAs in high school performed reasonably well in college, while students with relatively strong ACT or SAT scores but lower high school GPAs finished with slightly lower college GPAs and graduated less frequently.

Interesting stuff. Using the SAT (or ACT or any standardized test) on its own does seem insufficient. It is nonetheless the standard we often use when comparing schools. But is it really used, to the exclusion of other factors like GPA, to determine collegiate readiness and admissions? I’d agree that where this happens it is a mistake. But I usually see some combination of test scores plus GPA plus class rank. The only question is how we’re allocating the percentages. Hiss wants to see us use the SAT mostly as a compliment, given that different schools have different GPA metrics.

That last part seems important to me, though. To the extent that we start using GPA more, the measure will taint itself. From Unfogged:

Admissions director: who do you want to send the letters out to?
Me: Everyone with at least a 3.5 GPA is eligible.
Him: That’s 90% of our applicants.
Me: What! How?!
Him: grade inflation. Some of the richest schools in Dallas and Houston will have their entire student body with GPAs upwards of 3.2.
Me: How are colleges supposed to make sense of that?
Him: First, we recalculate their GPAs and throw out all the non-academic courses. But probably 80% of our applicants are still going to be over a 3.5, recalculated. What we do is know the high schools individually, and you don’t compare individuals from different types of high schools. A 3.5 from [poor school in San Antonio] means something different than a 3.5 from [rich school in Dallas]. So you have to understand each school.

If this is already a problem in Texas, it seems likely to spread exponentially as teachers and schools realize they can get more kids into college with a more generous grading system. Which is far, far easier than a school teaching to or otherwise gaming a test. So the more you rely on it, the less reliable it becomes.

When I am comparing schools, the two metrics I look at are (a) Standardized tests scores (both the ACT and SAT) and (b) class ranking. I sort of assume that the grade inflation Heebie Geebie refers to is pretty universal (though if Hiss’s study is any indication, maybe I am overestimating it). Of course, class ranking is also fraught with danger. All schools aren’t created equal. I went to a particularly good school, so my class rank was relatively low (I missed the top quarter, barely making the top third, with a 3.6 GPA). The state schools back home rely heavily on class ranking, and this limited my options. (While schools in other states, aware of how competitive my school is, were talking scholarship.)

My wife’s high school, which was specifically geared towards the gifted and talented, refused to even provide a class ranking. Which was no matter, as she could have gotten into almost any school in the country and chose an out-of-state flagship school largely because it was a full ride scholarship.

Unlike GPA, you can’t fudge class ranking easily, but it has those holes.

The 69%/45% difference in the SAT doesn’t breed confidence, however. It suggests either that the false negatives are indeed high, or our universities (and/or support system) is dramatically failing almost a third of high-scorers (and perhaps some of the lower-scorers as well). Alternately, the high SAT score could be indicative of going into more competitive, higher-fail-rate schools? It’s really hard to say.

My own view (and I’m completely spitballing here) is that if push came to shove, you could probably get around 1/2 to 2/3 of young people to graduate, under ideal circumstances. The “ideal circumstances” does a lot of heavy lifting there, however. And I question the extent to which the added value matches or exceeds the cost of providing both the education and the ideal circumstances most conducive to graduation. This is one of the reasons why the cost of educating people in college (whether born by the student, the student’s family, or the state) is so important. The higher the cost, the more justification is required to send people to college.

Which always, always, always brings us back to the question that those of us who believe “Universal education isn’t the answer” have difficulty confronting. How do you decide? Every metric we have is flawed. Which brings us back to the cost of education, because with every increase, the consequences are that much greater.