Going from a p-value of 10^-16 to 0.027 is --- painful. IFrom the lack of a response, I tend to infer that there's no arguing back...
Prediction: the original association will continue to be cited without correction.

Have I really not bookmarked this before?

"Due to its simplicity and familiarity, the Gaussian copula is popular in calculating risk in collaterized debt obligations, but it imposes asymptotic independence such that extreme events appear to be unrelated. This restriction might be innocuous in normal times, but during extreme events, such as the housing crisis, the Gaussian copula might be inappropriate. This paper explores various copula specifications and finds that the degree to which housing prices are related based on the Gaussian copula is too small compared with real housing price data."

OH KARL FRISTON NO.

ETA: From Tal's follow-up post, Friston is quite gracious about this.

"We here report for the first time, to the best of our knowledge, rat motor cortex neurons predicting the behavior of the American stock market. We implanted the motor cortex of the brains of rats with silicon electrodes. Using the correlation technique, we monitored the activity of neurons in our rats while simultaneously tracking the activity of stocks in the U.S. stock market."

This actually looks more like a job for nonparametric regression, or even relative distribution comparisons, but still...

It's the comparison of the same teacher in the same year on the same subject but in different grades which clinches the model being an EPIC FAIL.

To be clear, the bad data analysis is on the part of whatever hacks came p with the value added model being used here. These results are insane.

"Just as the financial system has to be changed to serve the needs of the people before the needs of the bankers, the drug trial system has to be changed to lower the incentives for cheating (and massive death tolls) just for a quick buck. As I mentioned before, it’s still not clear that they would have made less money, even including the penalties, if they had come clean in 2000. They made a bet that the fines they’d need to eventually pay would be smaller than the profits they’d make in the meantime. That sounds familiar to anyone who has been following the fallout from the credit crisis.
"One thing that should be changed immediately: the clinical trials for drugs should not be run or reported on by the drug companies themselves. There has to be a third party which is in charge of testing the drugs and has the power to take the drugs off the market immediately if adverse effects (like CVT events) are found. Hopefully they will be given more power than risk firms are currently given in finance (which is none)- in other words, it needs to be more than reporting, it needs to be an active regulatory power, with smart people who understand statistics and do their own state-of-the-art analyses – although as we’ve seen above even just Stats 101 would sometimes do the trick."

"I also worry that it is too easy to find correlations of this kind, and we don't have any idea just how easy until a concerted effort has been made to show that the spurious ones are not supportable. For example, if we took "has (vs. does not have) pharyngeal consonants", or "uses (vs. does not use) close front rounded vowels", would we find correlations there too? I have some colleagues here at the University of Edinburgh, within Simon Kirby's research group, who have run some informal experiments on the data Chen uses to see if dredging up spurious correlations of this kind is easy or hard, and so far they have found it jaw-droppingly easy. (I won't say any more, because I am in the weird position of producing unrefereed telegraphing of unrefereed and informal objections to an unrefereed and unpublished working paper, and it's all getting a bit too weird for me.)"

How many languages are there in Europe? Order of 10^2. How many variables can an economist get cross-country data on? Again, order of 10^2. How many discriminable syntactic features do languages have? Easily order of 10^3 if not come. Conclusion: this is not what I mean when I say that economists should do more data-mining.

"There's a special place in hell for the perpetrators of this, where I hope the gods of mathematics and reason are devising some exquisite tortures for them—perhaps in the form of endlessly reading Introduction to Statistics and doing the same problem sets over and over through eternity . . ."

In a word: no. In more words, http://bactra.org/weblog/857.html

How does anyone _not_ read this paper and think that they were correlating everything they could until they got a "significant" effect?
--- I am very tempted right now to make this a problem set in ADA, but that's just asking for trouble, yes?

To be clear, "bad_data_analysis" applies to what Yarkoni is talking about, not to Yarkoni.

See also Harris's summary paper in _Science_, http://www.sciencemag.org/content/333/6044/826.full

Sadly, in this case fitting crappy power laws to the works of Gene Stanley and Laszlo Barabasi is not an_intentional_ joke.

I am a bit surprised that he doesn't point out that this year's teacher predicts last year's "value added".

Shorter: many people taking pointless tests are not actually motivated to try very hard.  Those who are motivated to try hard on pointless tests do better, and are different people in many ways.  In other breaking news, snow is cold and water is wet.  (To be clear, my "bad data analysis" tag here refers to the IQ-mongers, and not to this paper.)

Somebody (hopefully not me) really needs to write a book about vampirical hypotheses testing.

We call this memorizing the training data, yes.

Exercise for the student: Devise at least two reasons why the causality might run from high income to frequent social drinking, rather than vice versa.  (This is I think too elementary to make a good problem for ADA.)

I'm tempted to make this into a problem set, but it's probably not challenging enough.

Drum nails this one; thinking that this tells us anything about the effects of the over-all level of union strength (or pro-union laws) is a simple fallacy of composition.  Which economists  are shockingly ready to commit whenever they see a "natural experiment".

Now that is how you give these idiots the business...  The last paragraph is a lovely encapsulation of just how foolish the whole enterprise really is.

The silly ESP paper is not actually a methodological outlier --- and that's a problem.

"The bit that is most striking is the 'papers with 1,328 different co-authors.' My inner Diego Gambetta ... suggests that this may in part be a signalling phenomenon. If you are doing research in a field where someone has convincingly demonstrated that the emperor has no clothes, you want to signal as convincingly as possible that you (in contrast to those losers down the hall/in the department across the river/the country/wherever) have results that really stand up. One obvious way to do this is to get the little boy who pointed out the emperor's sartorial shortcomings to co-author a paper with you. Hence, I suspect, lots and lots of people who are eager to co-author something with Ioannidis (and plausibly, many, many more than he might be comfortable co-authoring with).... (NB that I am not suggesting that Iannidis was motivated by such an opportunity; I would be startled and amazed if he had been)."

Oh no no no no. Rehabilitation FAIL.

"To be frank, when I dug into the Alesina and Ardagna paper and finally understood the work their 1.5% primary deficit reduction was doing I wandered around stunned for a day or two. I called a bunch of people I trusted on macroeconomics and tried to see if I was missing something; was our elite discourse, the Sensible People Stuff, really being driven by this?"

Y'all can stop sending this to me now.

"consequences of students ... not being randomly assigned to teachers within a school. It uses a [Value-Added Measure] to assign effects to teachers after controlling for other factors, but applies the model backwards.... [S]tudents’ fifth grade teachers appear to be good predictors of students’ fourth grade test scores. Inasmuch as a student’s later fifth grade teacher cannot possibly have influenced that student’s fourth grade performance, this curious result can only mean that students are systematically grouped into fifth grade classrooms based on their fourth grade performance. ... The usefulness of value-added modeling requires the assumption that teachers whose performance is being compared have classrooms with students of similar ability (or that the analyst has been able to control statistically for all the relevant characteristics of students that differ across classrooms). ..."

"I went to a physics conference, and all I got was a lousy power law"

"So I stand corrected. Credit reporting agencies don't pass along your credit scores to prospective employers. They do pass along your entire credit history and specifically promote it as a way of weeding out problem candidates, but there's no credit score. Just your entire credit history."

Sampling to apparent significance.

Yet another methodological EPIC FAIL.

That is NOT how you bring a mathematical model to empirical data.

... in which Ta-Nehisi Coates schools the Freakonomics crowd in the concept of "sample selection bias".

They're not actually doing anything _flagrantly stupid_, except that they persist in talking about discrete generations when all their distributions massively overlap. Model specification FAIL.

The judge in the case seems to be an innumerate idiot.

Back-up for the Hough review. Also: might make a good mini-project for the data-mining class, though I'd have to teach about spatio-temporal methods (which I should anyway [but where would the time come?]).

In which Denny Borsboom and friends take their turn bashing their heads against the wall.

Now this is what I call an epic take-down.

"The current financial crisis has brought out a fatal flaw in the foundations of the economic theories that guided economic agents and regulators: the unwarranted claim to precision and robustness. In this article I try to diagnose this flaw and discuss possible remedies. I argue that actual agents are intrinsically less sophisticated than the models assume they are, and that the various proposals to sustain the models by appealing to "as-if rationality" all fail. I next consider behavioral economics as an alternative to the standard models, claiming that while they may allow for successful retrodiction, they do not hold out much promise for prediction. I also discuss the use of statistical models, arguing that they are subject to so many traps and pitfalls that only a handful of elite practitioners can be trusted to use them well. Finally, I offer some speculations to explain the persistence in the economic profession and elsewhere of these useless or harmful models."

Jesus fucking Christ. Levitt actually put his name to this?!? This is the kind of thing I cobble together as a "what's gone wrong here?" assignment for my classes; maybe I should give 'em this.

To be explicit: in addition to all the problems Klein notes, the reason we frown on drunk driving more than on drunk walking is that one puts an incompetent in control of several thousand pounds of high-speed machinery, and the other a few hundred pounds of shambling biped. It's the DANGER TO OTHER PEOPLE that matters.

Making of the poster.

The key part is the observation that there is NO POWER to detect reasonably-sized deviations from random walks in only forty years' of data.

Institutionalizing BMI, despite its ineffectiveness and the existence of superior alternatives. (Which, errr, make it even more obvious that I'm way over-weight, so this isn't rationalization on my part.) Lots of issues here for a data-mining class.

Aaaaaaaaaaarrrrrrrrrrrrrrrrrrrrgh.

Having read the papers and re-implemented the method, I now concur with Mark: this is an methodological EPIC FAIL.

Mark Penn, flaming innumerate.

Cultural traits are correlated across space and time, and ignoring this in regressions gives you nonsense. (Of course if you want to see really flagrant examples of this fallacy, look at macroeconomists doing cross-country regressions of growth rates.)

There is a serious selection bias issue here (disproportionately looking at the badly-performing loans), but wow does everyone come out of this looking like a dangerous idiot. (Use this as a negative example in the next incarnation of the data-mining class?)

Good to see some push-back on this. (I was once able to reduce a statistical geneticist to hysterics by reading aloud from a table of heritabilities from the paper being critiqued here. I am not sure if they have quite forgiven me yet.)

You guys are not making it any easier for me to maintain my favorable stance towards evolutionary psychology, y'know? (Among everything else, "altruistic" != "egalitarian".)

In a logistic regression to predict voting, being a Virgo has a positive and significant coefficient...

Wow, that is some really bad data analysis... A pure instance of the effect Kieran once described as wondering why all the closets you open are so full of your clothes.

File under: people who do not grasp the meaning of the term "controlled experiment". See also: "ecological fallacy", "massive confounding".

Update: I gave in: http://bactra.org/weblog/592.html

Hallucinating power-laws in the interactome. Complete with a sound re-analysis of the data!

"[E]ither [collateralized debt obligations] are being used to arbitrage a substantial price discrepancy in the fixed income markets or CDOs are being used to convert existing fixed income instruments that are accurately priced into new fixed income instruments that are overvalued. The first possibility assumes the existence of a substantial market inefficiency, perhaps the most substantial inefficiency ever found in the finance literature. The second possibility seems more likely. In other words, either CDOs are evidence of a substantial and pervasive market imperfection, or they are being used to create one."

Interaction effects that "scream WILL NOT REPLICATE"

"If this book were turned in as the final to an entry-level statistics class, Penn would not only be failed, but the professor might well retire in shame." Hmm, how to design a class like that?

Obviously, Brumm is acting as a propaganda agent for the Mole People.

"We find that gradients and waves observed in ... maps resemble sinusoidal mathematical artifacts that arise generally when PCA is applied to spatial data, implying that the patterns do not necessarily reflect specific migration events."

"Brooks knows one big thing: that the world can be easily divided into groups (preferably two) and these groups are really, really different for each other. ... Brooks also desperately wants to infer political divisions from sociological divisions."

Shorter Simon & Levy (1963): I am sickened by the weakness of your model's goodness-of-fit test. (Does make me reconsider the many papers I still see using Cobb-Douglas...)

For MY's "1.1 percent", read "2.1 percent". The point, however, stands.