nothing up my sleeve (p < .05)

My ability to figure out magic tricks increased greatly the day I realized that magic tricks are optimal in an important sense. A trick is basically as impressive as it can be without giving away its secret. When the magician saws the woman in half, the parts you can see are all you can see. If the trick would be even more impressive if the woman wiggled her toes but she doesn't wiggle her toes, there's a reason there's no toe-wiggling.

It's distressing the extent to which I have been able to export this epiphany to the evaluation of quantitative social science.

I was recently talking to a colleague who does not do quantitative research about a paper we had both, in very different contexts, read. The paper addressed its substantive question using Approach A. It could have used Approach B instead. Approach A is good enough for the standards of where the paper was published, but Approach B would be the approach preferred by more quantitatively-discerning types. The paper acknowledged the existence of Approach B but made substantive and statistical arguments for why Approach A was superior. In talking to my colleague, I explained that these arguments were really not very good arguments, and that, indeed, people who understand the technical issues are not going to very persuaded by results from Approach A because you really need results from Approach B to be able to assert the conclusions of this paper with any real confidence.

The thing, though, was that I went on to take for granted that analysis using Approach B wouldn't yield statistically significant (i.e., publishable) results. My colleague asked how I could be so certain of this, since no results from Approach B were reported in the paper.

I replied that, if Approach B would have yielded the same results as Approach A, the author would have announced this fact to assuage the concerns of people like me. Especially because the author clearly understands how to do Approach B and it would have only taken, say, five minutes to check. So when I saw that the paper contained these weak arguments for the superiority of Approach A over Approach B, and made no mention of what the results would have looked like using Approach B, I read this as basically equivalent to the paper containing a giant invisible footnote that said "We tried Approach B and it doesn't work."

I hate this.