April 2009

If you don’t understand randomness, you can have a hard time telling a real problem from chance fluctuations. This is especially true when it comes to tests of any kind.

Your grade on any test is a combination of luck and skill. I am not going to argue how much of each goes into the mix. Just accept for the moment that to the degree that luck is involved, your test score will vary.

Let’s say you get two A’s in a row. What could explain that?

• You were lucky –that is your true abilities fall below the A range
• You are an A student–that is, your average score is in the A range

Time, and a few more tests,  will tell which answer is correct.  But almost nobody waits around that long.

Apparently the concept of “random variation” didn’t have much survival back when we were evolving those big brains of ours because random chance isn’t the first explanation we come up with. Instead, we’ve evolved to be superb pattern recognizers and inventors. We are emotionally tuned to find patterns and solutions. We comfortable and feel in control when we think we have “explained” things.

“Uses time wisely” and “Works to potential” were my two worst conduct grades in grade school and high school. Lots of A’s, lots of F’s and everything in between.

First semester of my freshman year, I faced two very difficult final exams. Based on past experience I expected to get “bad” grades. Instead, I aced both tests. I cast about for an explanation.  I recalled that I was wearing the same shirt both times.  Maybe the shirt was lucky.  My prefrontal cortex told the rest of my brain that this was superstition. But I could not shake the association. I remember thinking; “This is nonsense. The shirt had nothing to do with it. But what if I am wrong? Who cares, nobody has to know. What harm can it do? “ I wore that shirt to every test, for the next 4 years and couldn’t help myself—I wore to my oral exams for my PhD.

Between High School and College I had become an A student—I’m not exactly sure how. But my first semester I didn’t know that, and so I came to believe that I had a lucky shirt. This particular lack of logic didn’t hurt me much, but similar reasoning can mean unnecessary criticism of the struggling student and it can prevent a mediocre student from taking actions that really will lead to improvement.

Every semester I have conversations like this with at least one of my students:

Student: Dr. Argiros, I failed Sociology this month what should I do?

Me: What did you get?

Student: 71

Me: What grade did you get last month?

Student: 77

Me: And the month before?

Student: 75

Me: So, you have been passing so far, but just barely, and this month your grade is a little below the bar. Is that right?

Student: Yea, I’m not sure what happened.

Me: The difference is just a few points; do you think it might have been bad luck?

Student: Surprised at my response, Maybe, but I really didn’t understand all that stuff about capitalism we had to learn this month. I liked the stuff we were doing before that better.

Me: O.K. I can buy that. It’s easier to learn something if you like it. But I still don’t see a big difference between a 71 and a 77. I don’t think you should focus on what happened this month only. Whatever you are doing now is good enough to get you grades in the 70s. If you don’t want to get a grade lower than a 75 then you need to figure out what to do so that your average scores are higher—in the 80’s. That way if you are unlucky one month you will probably still pass. Does that make sense? If you like, I’d be happy to look at your notes and we could talk about what you are focusing on …

Sometimes this works. But more often than not, the student never comes to talk to me about study skills.

Either in the conversation with me, or later on when he is discussing the situation with parents and friends, almost invariably the focus shifts back to a comparison between this month’s failing grade and last month’s passing one. Having decided that he only passes when he is interested. He may decide there is nothing he can do except hope that he likes the rest of the semester better.

Even if he decides he wants to do better, unless he changes the way he has framed the problem, whatever he picks as the “reason” is not likely to do much good. It will have little more real impact than my “lucky” shirt. Here is the kicker. Because of regression toward the mean, there is a very good chance that next month’s test scores will be higher anyway. This will confirm the change in his mind. And he will continue another month and another—at least until his luck changes. Score: Superstition:1 Improvement:0.

I’ve been reading The Drunkard’s Walk: How Randomness Rules Our Lives, by Leonard Mlodinow.  No, no, don’t click away yet.  The ideas in this book are important.  For anyone who has ever tried to change herself, or to teach something to somebody else, the most important is on page 9. Its called “regression toward the mean.”

Here’s the short version.

When we do something well its usually a combination of skill and luck. Ditto the reverse. When we perform badly its lack-of-skill and/or lack-of-luck.  Examples–SATs, golf shots, investments, job interviews, sales calls, first dates (do people still date?), Yorkshire pudding.  Whenever luck is involved there will be variations in performance–we will do better sometimes than others. These variations will tend to cluster around their true value, or average value.  If you have a really good day on the golf course, chances are your next day will be worse.   Conversely, if you have a really bad day, then the next game you play will most likely be an improvement.   In both cases, the scores are tending to move in the direction of your true or average score.   This means you can’t really tell very much from one or two observations.  Its only by watching and recording outcomes consistently, time after time that you can discern whether your golf game, investments, or cooking is getting worse, getting better or staying about the same.

The problem is that our brains have difficulty holding on to this concept.  We are hard-wired to make associations (to find patterns).  We naturally tend to discount the role of chance and to see patterns where they don’t actually exist.

Mlodinow tells this story to illustrate.  In the 1960’s Daniel Kahneman was teaching behavioral psychology to Israeli flight instructors to help them improve their teaching.  A basic principle is that rewards(praise) work better than punishment (scolding).  A flight instructor  disagreed with this, saying when he yells at a student,they do better next time but when he praises a student that student does worse the next time.  Therefore, the flight instructor concluded,  scolding was more effective than praise.

What was going on?  Chances are that a performance worthy of a good verbal thrashing is way below average and, because of regression toward the mean, it is highly likely that a better performance will follow.   The student does better next time but its because of random variation, not because of the the instructor’s tirade.  Conversely, that excellent performance that the instructor praised was also partly a matter of luck and, statistically speaking, the most likely outcome next time will be a  worse performance but that too had nothing at all to do with the instructor’s praise.  (By the way, Kahneman devoted much of his career to studying how humans consistently misinterpret random events.  In 2002 he received the Nobel prize in economics along with Tversky for his work.)

Still seem a little “out there?”  I’m going to follow up with a few examples in my next post.