Physics Subject GRE, Part 2

So, what do the Physics Subject GRE scores mean? I would love to see a good study evaluating correlations with PhD completion and assorted other measures of academic success. If anyone knows of one, please leave a comment. In the meantime, I wanted to brainstorm some scenarios that would result in high or low scores.

Qualities required to score highly on the Physics Subject GRE
1. The tester has encountered all or almost all of the material on the exam.
2. The tester has excellent recall of equations.
(I'd say successfully completing the test had less to do with recalling basic concepts and basic equations, and more to do with remembering specific equations that exactly equate the variables given.)
3. The tester is pretty darn quick.
(The test is 100 questions in 170 minutes.)
4. The tester can solve word problems.
5. The tester can afford to take the test.
($140-160, plus fees for score distribution)
6. The tester isn't suffering some disabling condition while taking test.

Scenarios that yield a lower score on the Physics Subject GRE
Basically, the opposite of any of the items listed that would yield a high score.

So, let's evaluate some of the list items, considering how the items might predict future grad school success, and how the items might be artificially skewed. Obviously, items 1-4 should be highly correlated with grad school success. I certainly would predict a grad student who has encountered all relevant material, has excellent recall, is quick, and can problem solve would have 4 qualities that would help with successful PhD completion. Are these 4 qualities enough by themselves to predict success? Absolutely not. But, they all help.

Conversely, would missing one or more of these qualities (1-4) ensure failure? I'd say not having encountered the relevant material (1) or not being able to problem solve (4) would put a person at significant disadvantage. To me, the excellent equation recall (2) and the quickness (3) required for this exam are disproportionate to how much those qualities affect PhD course work and research. Basic concepts matter so much more (and knowing how/where to look up specifics), and the ability to think through a long, multi-step problem matters so much more. Being able to do each step quickly and accurately without having to look anything up certainly would help, though. And maybe one would argue that excellent recall without understanding basic concepts would result in bad test scores anyway, so maybe the test does test depth of understanding more than I'm giving credit.

One might think items 5 and 6 don't matter, and maybe they don't matter much. But the test is expensive, even for a college student in the US. The expense is a major deterrent to taking it more than once, for sure, as are scheduling issues. Since the test is covering what's being covered in classes, people try to take it as late as possible in hopes that they will have actually covered the relevant material in class by then. The expense and the scheduling issues mean many people have one shot, so I'm sure a number of people take it even if they're sick as a dog or their Grandpa just died or they have a concussion or whatever. I'm sure this number is statistically insignificant, but I think grad committees would do well to remember this possibility and take a score with a grain of salt if it looks like an outlier compared to other application components.

Finally, what could artificially skew items 1-4 for a person or a segment of the population? Item 1 is the easiest for me to rip on. A good portion of the material covered on the exam is not covered until the senior year at many universities in the US. The exam must be taken near the beginning of the senior year for the applicant that wants to go straight from undergrad to grad, which is the vast majority of students. Meaning students in the US are getting killed on one of the most important requirements for getting a good score on the exam; we've never even encountered a lot of the material we're being tested on. (Instead we've been taking the required English and history and psychology credits and so forth, but I digress...) But these same students would likely score much higher by the time they actually enter grad school, which is when grad schools actually want them to have covered the material. Luckily, many graduate committees recognize this issue and evaluate domestic and international students separately for GRE scoring.

Another example of a subset of population that gets killed by a skew in item 1: students from institutes with a small physics department. Why? I'm sure there are multiple reasons a small department would have some disadvantages, but in several cases I know, a big disadvantage comes from the frequency of basic class offerings. In a good, but small department, students may take all the relevant courses, but they may have to take the courses very much out of the usual order. For example, the student may have to take their first real Mechanics and E&M (beyond intro level) as a senior. Meaning the student would take the exam without having any mechanics or E&M other than intro level. Mechanics and E&M together make up 38% of the exam. That student is screwed.

Ok, I'm really almost done here. I have a couple other skews to discuss.

One skew people talk about a lot is the test taking traditions in various countries. In the US, most students take tests in their physics classes that take 1-3 hours to solve a few (2-6?) problems. The physics GRE is *100* problems in less than 3 hours. This shift in test taking style is enormous, and I can't imagine that it wouldn't artificially affect scores. I've been told that in Asia and India, the test-taking culture is more similar to the GRE, and that continuity helps when taking the GRE.

The final skew I'd like to discuss: studying. Normally, I'd say studying is not an artificial skew, and that a person that studies is a person willing to work hard is a person who is likely to succeed. But in the case of the GRE, I feel like the studying correlation gets messed up. Why? From what I gather, studying for the GRE breaks into 3-4 categories: 1) studying 1 week or less, 2) studying a few months in the evenings while working or taking classes, generally on your own, 3) taking test prep classes, 4) taking months off of work/school to study, very systematically, probably with lots of commercial test aids.

For students in category 1, maybe they don't study and score badly because they are overly-confidant, or lazy, or don't care, or think it doesn't matter much. If they are any of the first 3, it's a good indication they won't perform well in grad school. If it's the 4th reason, possibly having been told this information by professors they trust, then damn, that sucks for them.

For students in category 2 that still score poorly, maybe they don't have the money or resources to take a test prep class, or one is not available in their area. Or they suck at studying or are lacking in a key quality. First few reasons, it's hard to say it would correlate with poor grad school performance. Last 2 reasons, they'd probably suck at grad school.

For category 3 students, I think they learn a lot of good tricks. But they have to be able to afford those classes, and those classes have to be available in their area.

For category 4 students, I've mostly heard that this type of studying is something some international students do because they know/think they have no chance without practically perfect GREs. So I certainly think high scores achieved this way indicate a large amount of drive, which is a huge must in a PhD program. I don't know how much the scores still depend on innate ability at this level of studying, so I don't know how much correlation is left with other abilities. If anyone else knows, pipe up.

So, in conclusion? Excellent scores on the physics GRE's indicate some very desirable qualities for grad school success. The testable qualities, by themselves, don't ensure grad school success, but would certainly help. Some of the testable qualities can be skewed for individuals or segments of the population, so be careful when interpreting results, and know that a poor score definitely does not perfectly correlate with poor performance in grad school.