Confounding science with science

Scott Alexander is resplendent in this blog post about science, statistics, confirmation bias, control groups, and the study of whether psychic effects are real. If that list of keywords is not enough to hook you, I really knew the article attained a blazing level of reading delight when I reached this paragraph:

Then there’s Munder (2013), which is a meta-meta-analysis on whether meta-analyses of confounding by researcher allegiance effect were themselves meta-confounded by meta-researcher allegiance effect. He found that indeed, meta-researchers who believed in researcher allegiance effect were more likely to turn up positive results in their studies of researcher allegiance effect (p < .002).

Everything about it is a delight. The layers of meta-analysis. The English noun-phrase-constructing rules that permit the construction of a sentence in which the prefix "meta-" appears five times, variously modifying words which themselves are modifying other "meta-"-modified words.

I wonder if the same researcher bias/confounding exists in fields where the experiments are entirely done on computers. Can researchers' belief in the effectiveness of certain machine learning techniques affect their experiments? What about physics simulations? I don't see how, but of course I deeply believe in the inviolable sanctity of mathematics. This is an opinion founded in my acknowledged bias. Maybe coders would self-sabotage by writing bad code, so that experiments run slower? ... but in the end this wouldn't affect the actual outcome, just the agony and feasibility of running the experiment many times.

On a larger scale, I am supremely happy that scientists are using their scientific reasoning to criticize the very practice of science itself. In the same way that I frequently remind myself that the basis of the field studying privacy is "trust no one"*, it would be nice to have big science conferences where we all get together and just shake our heads at how unreliable the current practice of science is. Apparently. I mean, check out this conclusion:

But rather than speculate, I prefer to take it as a brute fact. Studies are going to be confounded by the allegiance of the researcher. When researchers who don’t believe something discover it, that’s when it’s worth looking into.

... which sounds convincing. 

But.

You know what?

I'm skeptical. 

This post's theme word is obverse, "the more conspicuous of two alternatives or cases or sides." The skeptic and his obverse performed a coordinated, randomized, double-blind study.

*Or, as I memorably put it during a job interview, "We've known for a long time that almost everything is impossible."

Maximum bagel/person/day rate

Testing the edge cases is important. This edge case is ridiculous.

Who would ever need more than 500 bagels per person, per day? The imagination exercise is left to the reader.

This post's theme word is schmeer, "the entire set (as in the whole schmeer)," or "bribe or flattery," or "spread or paste," or "to butter up: to flatter or bribe." Even if you schmeer the owner with a schmeer of money or positive media exposure, the whole schmeer of bagels is not available for today's cream cheese schmeering party.

*that I've found so far