🎙️Wrong numbers and why they survive | Complex System Podcast
Patrick McKenzie interviews famous Wall Street quant and author Aaron Brown(Poker Face of Wall Street, Red-Blooded Risk) about his new book Wrong Number, which tackles a stubborn, nagging mule of a question:
“Why do institutions that produce bad statistics face so few consequences?”
I really enjoyed the interview and if Aaron’s other writing is any indication, the book will be outstanding. But I just want to excerpt a section from below that I appreciated.
Timestamps
(01:12) The agricultural demand curve discrepancy
(04:06) Why experts prioritize teaching over learning
(05:17) Institutional indifference to error
(06:26) The brand halo of high-status institutions
(08:34) Lessons from COVID-era decision-making
(10:19) Financial statements versus scientific rigor
(18:19) The difficulty of auditing and replicating research
(22:12) The CDC eviction moratorium and its justification
(23:34) The NTSB curbside carrier safety study
(26:41) Conspiracy versus incompetence in data manipulation
(30:05) Error correction in financial markets
(32:52) The culture of the advantage gambler versus the academic
(35:28) Betting as a tax on bullshit
(38:44) Using market pricing to evaluate risks
(41:04) The track record of scary predictions
(43:34) Environmental success stories and technological optimism
(48:21) Energy efficiency and the path to global wealth
Betting as a tax on bullshit (emphasis mine)
Patrick: I think Nate Silver calls this the “River” versus the “Village.”
Aaron, agreeing: As somebody said about Nate Silver, betting is a tax on bullshit. [Patrick notes: I associate that line with Marginal Revolution. The post coining it was, fittingly, about Nate Silver.]
Patrick continuing: In some fields, it seems viscerally distasteful that someone could be keeping a record of someone being wrong. That person is a threat to social harmony. When folks from the advantage gambler camp say, “You’ve expressed 99% credence that X is true; would you bet $50,000 at even odds?” it functions as a tax on bullshit. Some people find it extremely negative to be seen publicly responding to that in a repeated fashion.
Aaron: My friend Philip Tetlock did a book, Expert Political Judgment, which showed that experts in a field have less than random—or worse than random—predictions. The more prominent the expert, the worse the performance.
Here’s a good trader question. When somebody says, “Gold is overpriced, it’s going to fall to a thousand dollars,” you ask them: “How much would the price of gold have to go up before you admitted you were wrong?” For most people, it’s a blank look. They haven’t thought about it. A trader will tell you, “I think it’s going to a thousand, but my stop is six thousand. If it hits six, I’m getting out; I was wrong.” If you haven’t thought about that, you haven’t taken the first step toward forming a bet. If no evidence will convince you, then it’s an article of faith.
While humans use git to version defenseless letters and numbers into knowledge with traceable lineages, they themselves resist self-audit. To torture the analogy, betting is like a merge you can’t roll back for free.
I really just love the way Patrick put this:
In some fields, it seems viscerally distasteful that someone could be keeping a record of someone being wrong. That person is a threat to social harmony. When folks from the advantage gambler camp say, “You’ve expressed 99% credence that X is true; would you bet $50,000 at even odds?” it functions as a tax on bullshit. Some people find it extremely negative to be seen publicly responding to that in a repeated fashion.
These talkers want an infinite Sharpe. Return for zero risk. The bettor/trader/investor finds THAT “viscerally distasteful”. How dare you lay claim to the Holy Grail without so much as a dent in your armor?
It’s quite predictable that I’d enjoy such a podcast. I’m partial to idea of “taxing” lies and laziness, but the episode is also a welcome reminder that metrics are tabulated, presented, and interpreted by humans. There’s an irreducible amount of subjective cradling the objective.
Brown is the most recent messenger in a parade of writers I’ve been sharing here. Zvi Mowshowitz, Ben Recht, C .Thi Nguyen, Dan Davies, James C. Scott. Each one is touching a different part of the corruption-of-metrics elephant, whether it’s malice or incompetence or coordination failure.
Scott’s critique of modernism and its high-minded faith in optimization bridges the hubris of retro-futuristism to these voices warning us today. But the interview with Brown was alarming not because the examples he presents were defined, not by hubris, but by feckless apathy.
Retro-futurism was at least optimistic. But today’s failures are like forfeits because nobody felt like waking up for the morning game. Meh, there’ll be another one next week, and nobody will care who wins that one either.
“Hey, we’re going to Mars!”
“What’s the difference if I have to go with you?”
Despite our ascendant progress in science and mechanics, the human psyche starts at zero with every birth. An interminable game of Trouble with the pop-o-matic® bubble bonking every generation back to the home base.
The tension you feel “in the room” is that we have come so far and yet remain in the same place. The battle for hearts and minds on both sides of the argument will be fought with McNamara-esque precision. Countable, listable, sortable.
You’ll be a little more awake for it if you consider what Brown saw when he stopped to get a better look.