In Which Direction?

According to a psychological study, students (in a sample of 93) were classified as high working-memory or low working-memory. They were then given a math test and the HWM students did better. After that they were given a test in what was termed a “higher, real world pressure situation” and the HWM scores dropped to the LWM levels. The conclusion:

Since working memory is known to predict many higher-level brain functions, the research calls into question the ability of high-pressure tests such as the SAT, GRE, LSAT, and MCAT to accurately gauge who will succeed in future academic endeavors.

In which direction? Are they claiming that those tests measure things other than working memory and are thus unfair or are they claiming that they only test working memory and are thus unrealistic? Or is this an attempt by reporters to justify flunking algebra?

Notable and Quotable

There's a political activist who's more numerate than many social scientists. Technology Review reports:

“You’re making significant resource decisions based on 160 people?” asks Mitch Stewart, director of the Democratic campaign group Organizing for America. “Isn’t that nuts? And people have been doing that for decades!”

Sample Sizes 41 and 67

According to a recent study:

Researchers from the University of California quizzed 41 women in long-term relationships about how attractive they found their husbands or boyfriends.

And, twice over the course of a month, they were asked how committed they were to their relationship and how close they felt to their partner.

In a second, similar experiment, 67 women were questioned twice about how likely they were to find fault with their other half.

The results showed that the women’s attitudes changed depending on how fertile they were and how attractive their man was.

A propensity to rush into print based on minimal evidence in the field of gender relations is not limited to feminists.

Rosin v. Hymowitz

Hanna Rosin's article on the “hookup culture” and how it empowers women disagees with Kay S. Hymowitz, who found that the least empowered women were least likely to marry. The only part of the study that came close to looking at control groups had a sample size of 53:

The researchers found their first day of interviewing so enlightening that they decided to ask the administration if they could stay on campus for four years and track the 53 women’s romantic lives.

I won't more than mention that the study might have indicated the opposite of Ms. Rosin's claim.

At Long Last …

There's some actual data to support one of Freud's theories, based on a study with a sample size of 11:

Researchers took 11 people with anxiety disorders who all received psychoanalysis as part of the study, to see what underlying unconscious conflict might be causing the anxiety.

Words capturing the nature of the unconscious conflict were then selected from the interviews, as well as words that related to each patient's experience of anxiety disorder symptoms.

A control category with words that had no relationship to the unconscious conflict or anxiety symptoms were presented to three patients wearing scalp electrodes to measure brain responses.

Researchers found a new measurement of the brain's alpha wave frequency, linked to communication, was made when the unconscious conflict words were presented directly before the conscious words.

Results showed highly significant correlations, suggesting an inhibitory effect, were obtained when the amount of alpha generated by the unconscious words were grouped with the conscious words.

Wow. Is that the best they can do?

The Conclusion Makes Sense but …

According to a recent study, organic food can turn people into obnoxious snobs:

To find out, Eskine and his team divided 60 people into three groups. One group was shown pictures of clearly labeled organic food, like apples and spinach. Another group was shown comfort foods such as brownies and cookies. And a third group -- the controls -- were shown non-organic, non-comfort foods like rice, mustard and oatmeal. After viewing the pictures, each person was then asked to read a series of vignettes describing moral transgressions.

………

"We found that the organic people judged much harder compared to the control or comfort food groups," says Eskine. "On a scale of 1 to 7, the organic people were like 5.5 while the controls were about a 5 and the comfort food people were like a 4.89."

When it came to helping out a needy stranger, the organic people also proved to be more selfish, volunteering only 13 minutes as compared to 19 minutes (for controls) and 24 minutes (for comfort food folks).

I don't think a study with a sample size of 60 should be taken seriously even if its conclusion (that “organic people” act like jerks) is plausible.

I won't more than mention that it was the sight of the organic food that had the effect, which might have been due to us anti-environmentalists getting p___ed off.

A Rival Blog

There's another blog on roughly the same topic but somewhat more active.

Four Small Studies

According to four studies (sample sizes 85, 38, 36, and 34), people give more allegedly right-wing answers to survey questions when they are less able to think. The only one that even came close to a decent size was also the one most subject to selection effects. (It found that drunker people in bars gave more right-wing answers. The lesson we might learn is that high-IQ conservatives don't go to bars.)

Maybe they should come back when they have real data.

Is There a Slant to Small Samples?

Maybe I should start checking to see if studies with a sample size of 17 are likely to bolster ridiculous feminist cliches. I suspect I can find at least 17 of them.

Are Drivers of Expensive Cars @ssholes?

On the one hand, it seems intuitively obvious and there's even some empirical data to back up this conclusion. On the other hand, the data for car classes 2–5 are well within random variation, which means the conclusion is ultimately drawn from a sample of the five cars in class 1. (I also suspect that the lines between the classifications were drawn in order to derive a predetermined conclusion.)

Besides, the intuitively-obvious result would have class 5 be the outlier instead of class 1.

20 Subjects vs. 428 Subjects

Wired is reporting that a study with 428 subjects indicated that being sleepy might aid creativity and that a study with 20 subjects (I checked the ScienceNews report) purportedly indicated that being drunk can aid creativity. I have a question: Why is a study with 20 subjects reported as though it were as sound as a study with 428 subjects? It looks like there is some actual evidence that being sleepy can aid creativity. The claim that being drunk can aid creativity is still as anecdotal as ever.

An Unforgettable Small Sample

The most unforgettable scientific lecture ever was based on a sample size of 1. Professor G. S. Brindley tested an experimental treatment for erectile dysfunction on himself and, to prove it worked, dropped his pants during a presentation of the paper at a scientific conference. After that it got weird…

More Brains!

Studying to be a London taxi driver can increase the brain size of at least 39 people.

A few years ago, I would have taken a small study like this more seriously. Now it is, at best, an indication that a real study might be worthwhile.

This Would Be Creepier …

… if it were less dubious.

According to a study with 10 subjects (seen via io9)

"The most surprising thing in this study is that mere inductions of neural activation patterns...led to visual performance improvement...without presenting the feature or subjects' awareness of what was to be learned," said lead researcher Takeo Watanabe. He continues:

We found that subjects were not aware of what was to be learned while behavioral data obtained before and after the neurofeedback training showed that subjects' visual performance improved specifically for the target orientation, which was used in the neurofeedback training.

Is this research mind-blowing and exciting? Absolutely. I mean come on — automated learning? Yes. Sign me up. But according to research co-author Mitsuo Kawato, the neurofeedback mechanism could just as soon be used for purposes of hypnosis or covert mind control. And that... I'm not so keen on.

"We have to be careful," he explains, "so that this method is not used in an unethical way."

I think taking a study with a sample size of 10 seriously is unethical enough.

Besides, earlier mind-control research went nowhere … or at least that's what THEY want us to think …

Don't Tell the Tobacco Companies

According to a study with 36 subjects, carbon monoxide makes people better able to tolerate noise pollution.

On the other hand, maybe it makes them deaf.

They're Starting to Take Care of It

I'm sure many of my fellow nerds have seen an article purporting to claim that observers can see evidence of a possible ”love gene” in brief video clips (original article here). This far-reaching conclusion was based on a sample size of 23. The good news is that experts are starting to react:

But the paper has drawn harsh criticism from geneticists because of its small sample size. Daniel MacArthur, who blogs at Genetic Future, said, “[A simple size of] 23 for genetics means the paper might as well not exist. It carries no useful info. Without a larger sample and independent replication, it’s safest to simply assume these results are false.” Joe Pickrell from Harvard Medical School, agreed: “If the sample size is 23… there’s no way that’s a real association.”And Chris Gunter from the HudsonAlpha Institute for Biotechnology added, “The literature is full of behavioural genetics studies which don’t replicate, with similarly small numbers.”

Thin Evidence about Thin People Believing in Thin Ectoplasm

There is some evidence that skinny people are afraid of ghosts:

There are indications that subjects with schizotypal personality have a lower Body Mass Index. Also schizotypal personality is linked to a higher incidence of paranormal belief. In this study we examined whether low Body Mass Index is also linked to paranormal belief. In a pilot study 48 students of psychology (85.4% women) between the ages of 20 and 27 years were administered a questionnaire assessing weight, height, and paranormal belief. Analysis suggested an association between belief in paranormal phenomena and low Body Mass Index.

On the other hand, the evidence disappeared once they did a real study:

In a follow-up study with 300 subjects and equal sex distribution, the relationship was examined under control of schizotypy. The results for Body Mass Index could not be confirmed; however, paranormal belief was heavily associated with the cognitive-perceptual component of schizotypy.

Aren't ghosts also supposed to vanish in the light?

It's Been Done

I'm not the first to point out the absurdly-small sample sizes of social-science research. Tversky and Kahneman (seen via The Wall Street Journal) beat me to it by several decades.

On the other hand, their sample size was only 84. On the gripping hand, that might be a dividing line between social-science research that should be taken seriously and social-science research that should be regarded as tentative.

One more point: My usual reaction to reports of research on cognitive bias is to wonder why we don't see more research into the cognitive biases of psychologists. It turns out they've done such research, at least as far as the cognitive bias of believing in small samples is concerned. The problem is that it isn't as widely reported as first-order cognitive-bias research.

Has there been any research done on the cognitive biases of science reporters? (I wouldn't be surprised to find there has been such research and the reporters don't mention it … for obvious reasons.)

Life, the Universe, and Everything

The great question of Life, the Universe, and Everything turns out to be “What was the sample size for the paper ‘Subcortical processing of speech regularities underlies reading and music aptitude in children’?”

Digression: The problem with small-sample research isn't that it is necessarily invalid. A small-sample paper might be about something real. One problem is that if it were about a real phenomenon we should expect it to be backed up by large samples and frequently it isn't. Another problem is that it is usually reported as something definitive and small-sample research is always tentative.

Do Neuroscientists Have This Fold?

According to an article in Discover, there's a fold in the brain that helps keep reality and imagination separate. After seeing the sample size involved:

Of the 53 people selected for the study, some had this fold on both sides of their brain, some had it on one side, and some had no fold.

I wondered what part of the brain is responsible for taking studies with small sample sizes seriously.