By Cathy O’Neil, a data scientist. Cross posted from mathbabe
I just finished reading Nate Silver’s newish book, The Signal and the Noise: Why so many predictions fail – but some don’t.
The Good News
First off, let me say this: I’m very happy that people are reading a book on modeling in such huge numbers – it’s currently eighth on the New York Times best seller list and it’s been on the list for six weeks. This means people are starting to really care about modeling, both how it can help us remove biases to clarify reality and how it can institutionalize those same biases and go bad.
As a modeler myself, I am extremely concerned about how models affect the public, so the book’s success is wonderful news. The first step to get people to think critically about something is to get them to think about it at all.
Moreover, the book serves as a soft introduction to some of the issues surrounding modeling. Silver has a knack for explaining things in plain English. While he only goes so far, this is reasonable considering his audience. And he doesn’t dumb the math down.
In particular, Silver does a nice job of explaining Bayes’ Theorem. (If you don’t know what Bayes’ Theorem is, just focus on how Silver uses it in his version of Bayesian modeling: namely, as a way of adjusting your estimate of the probability of an event as you collect more information. You might think infidelity is rare, for example, but after a quick poll of your friends and a quick Google search you might have collected enough information to reexamine and revise your estimates.)
The Bad News
Having said all that, I have major problems with this book and what it claims to explain. In fact, I’m angry.
It would be reasonable for Silver to tell us about his baseball models, which he does. It would be reasonable for him to tell us about political polling and how he uses weights on different polls to combine them to get a better overall poll. He does this as well. He also interviews a bunch of people who model in other fields, like meteorology and earthquake prediction, which is fine, albeit superficial.
What is not reasonable, however, is for Silver to claim to understand how the financial crisis was a result of a few inaccurate models, and how medical research need only switch from being frequentist to being Bayesian to become more accurate.
Let me give you some concrete examples from his book.
Easy First Example: Credit Rating Agencies
The ratings agencies, which famously put AAA ratings on terrible loans, and spoke among themselves as being willing to rate things that were structured by cows, did not accidentally have bad underlying models. The bankers packaging and selling these deals, which amongst themselves they called sacks of shit, did not blithely believe in their safety because of those ratings.
Rather, the entire industry crucially depended on the false models. Indeed they changed the data to conform with the models, which is to say it was an intentional combination of using flawed models and using irrelevant historical data (see points 64-69 here for more).
In baseball, a team can’t create bad or misleading data to game the models of other teams in order to get an edge. But in the financial markets, parties to a model can and do.
In Fact, Every Failed Model is Actually a Success
Silver gives four examples what he considers to be failed models at the end of his first chapter, all related to economics and finance. But each example is actually a success (for the insiders) if you look at a slightly larger picture and understand the incentives inside the system. Here are the models:
- The housing bubble.
- The credit rating agencies selling AAA ratings on mortgage securities.
- The financial melt-down caused by high leverage in the banking sector.
- The economists’ predictions after the financial crisis of a fast recovery.
Here’s how each of these models worked out rather well for those inside the system:
- Everyone involved in the mortgage industry made a killing. Who’s going to stop the music and tell people to worry about home values? Homeowners and taxpayers made money (on paper at least) in the short term but lost in the long term, but the bankers took home bonuses that they still have.
- As we discussed, this was a system-wide tool for building a money machine.
- The financial melt-down was incidental, but the leverage was intentional. It bumped up the risk and thus, in good times, the bonuses. This is a great example of the modeling feedback loop: nobody cares about the wider consequences if they’re getting bonuses in the meantime.
- Economists are only putatively trying to predict the recovery. Actually they’re trying to affect the recovery. They get paid the big bucks, and they are granted authority and power in part to give consumers confidence, which they presumably hope will lead to a robust economy.
Cause and Effect get Confused
Silver confuses cause and effect. We didn’t have a financial crisis because of a bad model or a few bad models. We had bad models because of a corrupt and criminally fraudulent financial system.
That’s an important distinction, because we could fix a few bad models with a few good mathematicians, but we can’t fix the entire system so easily. There’s no math band-aid that will cure these boo-boos.
I can’t emphasize this too strongly: this is not just wrong, it’s maliciously wrong. If people believe in the math band-aid, then we won’t fix the problems in the system that so desperately need fixing.
Why Does He Make This Mistake?
Silver has an unswerving assumption, which he repeats several times, that the only goal of a modeler is to produce an accurate model. (Actually, he made an exception for stock analysts.)
This assumption generally holds in his experience: poker, baseball, and polling are all arenas in which one’s incentive is to be as accurate as possible. But he falls prey to some of the very mistakes he warns about in his book, namely over-confidence and over-generalization. He assumes that, since he’s an expert in those arenas, he can generalize to the field of finance, where he is not an expert.
The logical result of this assumption is his definition of failure as something where the underlying mathematical model is inaccurate. But that’s not how most people would define failure, and it is dangerously naive.
Medical Research
Silver discusses both in the Introduction and in Chapter 8 to John Ioannadis’s work which reveals that most medical research is wrong. Silver explains his point of view in the following way:
I’m glad he mentions incentives here, but again he confuses cause and effect.
As I learned when I attended David Madigan’s lecture on Merck’s representation of Vioxx research to the FDA as well as his recent research on the methods in epidemiology research, the flaws in these medical models will be hard to combat, because they advance the interests of the insiders: competition among academic researchers to publish and get tenure is fierce, and there are enormous financial incentives for pharmaceutical companies.
Everyone in this system benefits from methods that allow one to claim statistically significant results, whether or not that’s valid science, and even though there are lives on the line.
In other words, it’s not that there are bad statistical approaches which lead to vastly over-reported statistically significant results and published papers (which could just as easily happen if the researchers were employing Bayesian techniques, by the way). It’s that there’s massive incentive to claim statistically significant findings, and not much push-back when that’s done erroneously, so the field never self-examines and improves their methodology. The bad models are a consequence of misaligned incentives.
I’m not accusing people in these fields of intentionally putting people’s lives on the line for the sake of their publication records. Most of the people in the field are honestly trying their best. But their intentions are kind of irrelevant.
Silver Ignores Politics and Loves Experts
Silver chooses to focus on individuals working in a tight competition and their motives and individual biases, which he understands and explains well. For him, modeling is a man versus wild type thing, working with your wits in a finite universe to win the chess game.
He spends very little time on the question of how people act inside larger systems, where a given modeler might be more interested in keeping their job or getting a big bonus than in making their model as accurate as possible.
In other words, Silver crafts an argument which ignores politics. This is Silver’s blind spot: in the real world politics often trump accuracy, and accurate mathematical models don’t matter as much as he hopes they would.
As an example of politics getting in the way, let’s go back to the culture of the credit rating agency Moody’s. William Harrington, an ex-Moody’s analyst, describes the politics of his work as follows:
In 2004 you could still talk back and stop a deal. That was gone by 2006. It became: work your tail off, and at some point management would say, ‘Time’s up, let’s convene in a committee and we’ll all vote “yes”‘.
To be fair, there have been moments in his past when Silver delves into politics directly, like this post from the beginning of Obama’s first administration, where he starts with this (emphasis mine):
To suggest that Obama or Geithner are tools of Wall Street and are looking out for something other than the country’s best interest is freaking asinine.
and he ends with:
This is neither the time nor the place for mass movements — this is the time for expert opinion. Once the experts (and I’m not one of them) have reached some kind of a consensus about what the best course of action is (and they haven’t yet), then figure out who is impeding that action for political or other disingenuous reasons and tackle them — do whatever you can to remove them from the playing field. But we’re not at that stage yet.
My conclusion: Nate Silver is a man who deeply believes in experts, even when the evidence is not good that they have aligned incentives with the public.
Distrust the Experts
Call me “asinine,” but I have less faith in the experts than Nate Silver: I don’t want to trust the very people who got us into this mess, while benefitting from it, to also be in charge of cleaning it up. And, being part of the Occupy movement, I obviously think that this is the time for mass movements.
From my experience working first in finance at the hedge fund D.E. Shaw during the credit crisis and afterwards at the risk firm Riskmetrics, and my subsequent experience working in the internet advertising space (a wild west of unregulated personal information warehousing and sales) my conclusion is simple: Distrust the experts.
Why? Because you don’t know their incentives, and they can make the models (including Bayesian models) say whatever is politically useful to them. This is a manipulation of the public’s trust of mathematics, but it is the norm rather than the exception. And modelers rarely if ever consider the feedback loop and the ramifications of their predatory models on our culture.
Why do People like Nate Silver so Much?<
To be crystal clear: my big complaint about Silver is naivete, and to a lesser extent, authority-worship.
I’m not criticizing Silver for not understanding the financial system. Indeed one of the most crucial problems with the current system is its complexity, and as I’ve said before, most people inside finance don’t really understand it. But at the very least he should know that he is not an authority and should not act like one.
I’m also not accusing him of knowingly helping cover up the financial industry. But covering for the financial industry is an unfortunate side-effect of his naivete and presumed authority, and a very unwelcome source of noise at this moment when so much needs to be done.
I’m writing a book myself on modeling. When I began reading Silver’s book I was a bit worried that he’d already said everything I’d wanted to say. Instead, I feel like he’s written a book which has the potential to dangerously mislead people – if it hasn’t already – because of its lack of consideration of the surrounding political landscape.
Silver has gone to great lengths to make his message simple, and positive, and to make people feel smart and smug, especially Obama’s supporters.
He gets well-paid for his political consulting work and speaker appearances at hedge funds like D.E. Shaw and Jane Street, and, in order to maintain this income, it’s critical that he perfects a patina of modeling genius combined with an easily digested message for his financial and political clients.
Silver is selling a story we all want to hear, and a story we all want to be true. Unfortunately for us and for the world, it’s not.
How to Push Back Against the Celebrity-ization of Data Science
The truth is somewhat harder to understand, a lot less palatable, and much more important than Silver’s gloss. But when independent people like myself step up to denounce a given statement or theory, it’s not clear to the public who is the expert and who isn’t. From this vantage point, the happier, shorter message will win every time.
This raises a larger question: how can the public possibly sort through all the noise that celebrity-minded data people like Nate Silver hand to them on a silver platter? Whose job is it to push back against rubbish disguised as authoritative scientific theory?
It’s not a new question, since PR men disguising themselves as scientists have been around for decades. But I’d argue it’s a question that is increasingly urgent considering how much of our lives are becoming modeled. It would be great if substantive data scientists had a way of getting together to defend the subject against sensationalist celebrity-fueled noise.
One hope I nurture is that, with the opening of the various data science institutes such as the one at Columbia which was a announced a few months ago, there will be a way to form exactly such a committee. Can we get a little peer review here, people?
Conclusion
There’s an easy test here to determine whether to be worried. If you see someone using a model to make predictions that directly benefit them or lose them money – like a day trader, or a chess player, or someone who literally places a bet on an outcome (unless they place another hidden bet on the opposite outcome) – then you can be sure they are optimizing their model for accuracy as best they can. And in this case Silver’s advice on how to avoid one’s own biases are excellent and useful.
But if you are witnessing someone creating a model which predicts outcomes that are irrelevant to their immediate bottom-line, then you might want to look into the model yourself.
120 comments
Comments are closed.
Amen
Moral Hazard
Individuals cause the problems but are not held accountable.
There should be a lot of financial guys poor and out of work … if not in jail.
Nate Silver is just a small part of the problem when it comes to the financial crisis – economists and modeling, the idea that the finacial crisis is fundamentally due to corruption just never seems to occur to them:
http://economistsview.typepad.com/economistsview/2012/12/missing-the-point-in-the-economists-debate.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+EconomistsView+%28Economist%27s+View%29
Models about rational or irrational behavior miss the point – the frauds and thieves who ran the system were perfectly rational, (I’m using the economic man defintion – they considered risk versus reward, the bribes…um, campaign contributions they made, the shills…I mean the academic researchers who vouched for their models, and concluded they could enrich themselves at the expense of everyone else with no risk – they were right!)
So Nate is just looking at the meme that most of the media choose to accept – it was those complicated models! (a few bad apples, but shirley not enough to account for the entire crisis) damn math…
Oh, many of us are fairly certain it occurs to them, but they are the select stooges who receive those political and business apointments which net them large sums of money.
Which was why Larry Summers, prior to being re-appointed to a “democractic” administrion (in this case Obama’s), while president of Harvard, fired the first Black-American woman to receive a PhD in mathematics from them.
(She had questioned their derivatives investment strategy, which ended up predictably costing them billions in their endowment fund.)
http://www.globalresearch.ca/the-fiscal-cliff-is-a-diversion-the-derivatives-tsunami-and-the-dollar-bubble/5316082
http://petras.lahaine.org/?p=1689
Shirley, that’s enough!
Thanks for writing this post. I’ll admit my distaste for Nate Silver and Nate Silver worship was more related to the reasons you outlined under “Why do People like Nate Silver so Much”, and I have zero interest in reading his book. I can see why the book angered you so much. And, I can see why it is so loved by some. I cannot believe that he presented misaligned incentives (kind of a namby-pamby way of putting it, btw) as just one among many problems affliciting medical research, but not the root cause.
Yes, he predicts elections correctly but looks like he’s bigtime into soothing the conscience of the (neo)liberal elite. That is quite a lucrative gig.
Time to re-read the Financial Modeler’s Manifesto:
http://www.wilmott.com/blogs/eman/index.cfm/2009/1/8/The-Financial-Modelers-Manifesto
The worship of “complexity” is flushing this whole world down the drain. Look at my model, it’s got Hyperbolic PDEs.
Rediculous solution. PDEs are deterministic, time reversible and produce no entropy. Yves has heard me on this soapbox before. Nearly all models for economic behavior assume independent trials and a stationary random process, an assumption that works on the average but seldom in the extreme. Well behaved hurricanes have predictable tracks assuming the jet stream behaves as a stationary random process. Hurricane Mitch turned out to have a totally unpredictable track. To quote from Chapter 12 of Bendat and Piersol’s Random Data, Analysis and Measurement Procedures, “Examples include nonstationary ocean waves, atmospheric turbulence and economic time-series data. The basic factors producing such data are too complex to allow ther performance of repeated experiments under similar conditions.
Most extreme events are the result of a sequence of events that has never occured before and therefore have unknowable probabilities.
Rediculous is a ridiculous spelling, Sorry about that.
Just undiculous. That gets you back to diculous. If you change your mind, you can just rediculous.
Waaaait a minute – if you undiculous doesn’t that mean you NOT diculous” And if you NOT diculous how does that get you back TO diculous , unless there’s a wormhole involved here somewhere, mumble. mumble ….
I believe you meant undiculate.
One thing you can say with 100% certainty…If a dollar is spent into the economy (non-government) by the government the number of net dollars in the non-government will increase by a dollar, and the new level will remain that way until the government chooses to change it either by more spending or taxation.
If a bank lends 1$ of debt into the same economy, the number of net dollars existing in the economy remains unchanged…no new money is added unless the negative money (liability) is ignored.
Both actions increase GDP by 1 dollar (unless the recipient immediately puts it in savings).
“If a bank lends 1$ of debt into the same economy, the number of net dollars existing in the economy remains unchanged…no new money is added unless the negative money (liability) is ignored.”
First, if the bank was sitting that dollar, it is now being used. Second, even given the liability, the fact that the dollar is available now, while the liability is later could be important.
Good, informative piece with fine advice.
“I’m also not accusing him of knowingly helping cover up the financial industry. But covering for the financial industry is an unfortunate side-effect of his naivete and presumed authority, and a very unwelcome source of noise at this moment when so much needs to be done.”
You may not accuse him of helping cover for the financial industry but I will.
The one reliable predictor among Obama devotees (and I would count Silver as one of those) is they never dare contradict fearless leader, no matter how absurd the contention. I have failed to get one Obama supporter to admit that O has shielded the banking industry in any way. They point to Dodd-Frank which they know nothing about but believe it somehow effectively regulates the industry. If you press them they will become irrational and accuse you, as one did me, of wanting to burn down the banks! Which I definitely do not want to do.
The FIRE sector has become a third rail for Democratic partisans. You just don’t go there.
Like Obama, I do not believe Silver is naive. He wants to continue to be published and have access to powerful circles. He does that the same way everyone else does: by keeping his mouth shut.
What’s the point of having a Leader if you can contradict Him? Let’s be reasonable, here.
I have no idea WTF Nate Silver is, have never heard of his book, can’t imagine ever reading anything popular, and am not a mathematician. I have read Keynes’ Treatise on Probability, which is the only book anyone interested in modeling the economic future ever has to read. Keynes explains in words any literate person can understand why the economic future is UNKNOWABLE. Those trying to model it are either charlatans or fools. Either way, their models are just nonsense wrapped in calculus. They might as well be multiplying by zero.
Excellent comments, jc, and thanx for mentioning Keynes factually, as opposed to sooooo many today who routinely misuse his name regarding specific economic ideas, most of Keynes excellent ideas WERE NOT accepted at Bretton Woods, instead the vast majority of those economic routines adopted were American, from Harry Dexter White.
“Keynes explains in words any literate person can understand why the economic future is UNKNOWABLE. Those trying to model it are either charlatans or fools. Either way, their models are just nonsense wrapped in calculus. They might as well be multiplying by zero.”
A-men, brother.
I agree.
I was liking Cathy’s rant until she said that complexity was the crucial problem with finance……not the global inherited rich that make all the MAJOR capital decisions and continue on with their control because of ongoing inheritance and accumulating private ownership of property.
Economists and the models they create are cover for the aforementioned reality of our class system. I gave up trying to become an economist because none would ever start with the proper facts to build their BS from……computers systems at least have to start from reality and build on it so I felt much more comfortable using my skills in that environment.
Um, the 0.1% are for the most part not global inherited wealth. They are mainly hedgies, PE guys, and CEOs. They are the new robber barons.
Yves,
I expect you to know more than I about this but please explain who owns the Fed. I don’t believe it is the hedgies, PE guys and CEOs.
Who are the owners of the City of London, etc.?
It is my understanding, which I have expressed here before, that the Fortune 500 are mostly the wannabees and the folks with REAL money and property are well hidden behind trust and interlocking directorships….and have been in control of money and finance longer than the hedgies, PE guys and CEOs have been alive…..family money that keeps accumulating.
Please clue me in.
Yves,
Could you elaborate on the real make-up of the 0.1%? Who they are? What they (think they) want?
Are they rentiers? Or the Golden Horde?
To build a strong movement to check their assault, it is critical to give the widest possible audience a consistent, shared, view (model) of their opponents.
Thanks for all you do!
“they will become irrational and accuse you, as one did me, of wanting to burn down the banks! Which I definitely do not want to do.”
Tell them to go ahead and accuse ME of it, because I do!
Ha, ha – they accuse you of wanting to burn down the banks and they’re the ones playing with FIRE ..
“make his message simple, and positive, and to make people feel smart and smug, especially Obama’s supporters.”
This pretty much summarizes the problems with “the professional left,” progressives or whatever they call themselves—the establishment Democrats feed the liberal-fanboy crowd a sense of smart smugness with the right hand, while throwing them under the bus on actual policy issues with the left.
“Smug” nails it. Rachel? Oh, Rachel!
There is a very fundemental question to ask them: do you want to “be right” (by which I mean be smug, have your biases constantly reinforced) or do you want to win (suceed)? Because the left is not winning, except for the lifestyle left, they are losing on every front.
Do you want to be right or do you want to win/suceed? Suceeding might require tactical alliances even with those who might not agree wtih you on everything. I don’t mean Obamaish compromise, that is completely worthless, I mean tactical alliances on specific issues (with libertarians on civil liberties for instance). Suceeding might require alliances on those with different lifestyles from you. In terms of interests the whole working class, if not necessarily the middle class (though I urge that they should sympathize as well), should be left sympathetic on economics, but most have been lost possibly permanently. Suceeding at the very least requires you TO SEE THAT YOU ARE LOSING. Not blind smugness and pretending you are winning because it makes you feel good. The left is losing on near everything, on the safety net, on unions and their right to organize, on civil liberties, on the environment and climate change. I don’t know what the answer is but does anyone even really want to suceed on any of these things? Not those who are mostly touring the pundit circle for their own enrichment, what about the rest of us?
“lifestyle left”?
nehru jackets
Case in point: Alan Grayson teamed up with Ron Paul on Audit the Fed. It was Bernie Sanders who was responsible for the procedural move that led it to be watered down.
Yet when Matt Stoller and Glenn Greenwald pointed out in the runup to the election that Ron Paul was to the left of Obama on key issues, certain folks on the left (Charles Pierce springs to mind) accuse him of derangement and bad faith, as opposed to making legitimate observations.
Agree completely. The problem is that too many of these folks are doing fine whether progressive values win the day or not. I had been writing a series of articles on this called “Progressive Idiocy”, but people just don’t seem to be interested. I think the truth is that if they can’t depend on the Democratic Party and all these supposedly progressive institutions, what do they do? The answers to that question aren’t easy.
I guess this is not too surprising–Michael Mann, for instance, has already taken Silver to task for misunderstanding climate science (http://www.huffingtonpost.com/michael-e-mann/nate-silver-climate-change_b_1909482.html), but I kept waiting for the part where Silver was being malicious–acting with evil intent. He clearly falls victim to the “everything looks like a nail when all you have is a hammer” way of thinking, common among those trained in economics. But I do not think that you have shown that he has deliberately deceived.
I agree; perhaps ‘perniciously wrong’ would have been a better choice than ‘maliciously wrong’?
I don’t think that word means what she thinks it means.
Amen!
This isn’t just Silver’s blind spot, it’s the blind spot of nearly every financial commentator who blames the meltdown on math.
It is very easy to trace meltdowns and errors to bad models, but those bad models were deliberately allowed to flourish by the real decision makers.
There’s very high selection pressure on financial models to be profitable, not to be accurate. Often those two things align, but you shouldn’t assume they do.
And sometimes the models are no more than window dressing for the customers.
All of the areas where Nate says that the models were bad, in fact had their own Nate Silvers that were ignored by the Fox News types.
I was only waiting for a review like this. Nate Silver is as pretentious as absurd when it comes to more than simply putting down data. When understanding is necessary, there is no there there as with climate change and dismissing Michael Mann or routinely dismissing political analysts who actually try to understand policy.
A-fucking-men
Having spent time in grad school has made me very skeptical of scientific establishments. The problem is not with the models but with the people who make the models. What I witnessed is at least 80% of people in academia taking shortcuts and ignoring biases to make their data look better than it is.
The problem is amplified in fields where there is greater financial implications to research.
I’m truly disgusted by the whole thing.
Tell it, brother!
Nate Silver is the poster boi of the moment for the “creative class” — a hip yet serious, geeky yet easy-listening technocrat with the following job description: Holding up a huge honkin’ sign with “LOOK! OVER THERE!” spelled out in big blinking lights.
Sure, in his narrow speciality of election polling, Silver is the best of a very bad lot, but how hard can it be to call a rigged game the right way? In any case, since the entire election was an exercise in “Look! Over there!” Silver was very well equipped to deal with it.
Richard Charnin’s open modeling versus Nate.
http://richardcharnin.wordpress.com/
http://richardcharnin.wordpress.com/2012/11/26/late-votes-and-the-true-vote-model-indicate-that-obama-may-have-won-by-16-million-votes/
Late Votes and the True Vote Model indicate that Obama may have won by 16 million votes
Richard Charnin
……..
http://bigdanblogger.blogspot.com/2012/11/an-open-letter-to-big-dan-from-guy-whos.html
An Open Letter To Big Dan From The Guy Who’s BETTER Than Nate Silver: RICHARD CHARNIN – TRUTH IS ALL
I met Richard Charnin on THE BRAD BLOG. His comments under the username “TRUTH IS ALL” about his election and voting analysis were and are impressive to everyone. I called him THE ULTIMATE NUMBERS CRUNCHER. Anyone who researches and is hip to electronic voting machine fraud, election fraud, follows Bev Harris’ BlackBoxVoting.org, etc… knows of Richard Charnin. He has written several BOOKS on these subjects. He wrote me the following on Facebook, and I feel fortunate to have friends like him write to me, and I’d like to share this with everyone. Here is Richard Charnin in his own words.
RICHARD CHARNIN: LET ME TELL YOU ABOUT NATE –
I saw your POST about Nate Silver. Let me tell you about Nate.
Nate Silver DID NOT GET IT RIGHT. He predicted 313 EV for Obama.
http://www.truthdig.com/eartotheground/item/nate_silvers_final_prediction_20121106/
My final prediction had 332 EV for Obama. That was exactly right.
https://docs.google.com/spreadsheet/ccc?key=0AjAk1JUWDMyRdDQzLWJTdlppakNRNDlMakhhMGdGa0E#gid=8
Getting it right is not that big a deal. There were 10 states that were “competitive” if you believe the Likely Voter Polls. I got them all. I WAS JUST LUCKY. I used the LV polls just like everyone else. Obama’s expected EV was 321:the sum of the expected values (state win probability * electoral vote). The snapshot was 332 (the actual summation of the winning electoral votes, not the expected value). The LV polls understate Obama’s true vote, which I calculated to be 371 expected and 380 snapshot.
http://richardcharnin.wordpress.com/2012/11/05/final-forecast-the-2012-true-vote-election-fraud-model/
In 2008, I also had Obama’s bogus 365 EV exactly right in 2008 (the expected value, not the snapshot).
I produce several measures for calculating EV: theoretical, snapshot, and simulation mean value. Obama had 420 True EV.
My model is an open book. Can’t say the same about Nate’s. But I can understand why. His model is junk.
The purpose of a Monte Carlo Electoral Vote Simulation is to calculate the PROBABILITY OF winning the electoral vote. The probability is the number of winning election trials divided by the total number of trials. I used 500 election trials. That is all that is necessary. The Law of Large Numbers takes effect quickly in the rapid convergence to the mean. Nate does 100,000 trials – overkill. Prof. Sam Wang does nearly a million combinations – laughable.
…..
And, this has been going on for some time. See:
http://richardcharnin.wordpress.com/2012/11/17/a-reply-to-nate-silvers-ten-reasons-why-you-should-ignore-exit-polls/
A Reply to Nate Silver’s “Ten Reasons Why You Should Ignore Exit Polls”
17 Nov
A Reply to Nate Silver’s “Ten Reasons Why You Should Ignore Exit Polls”
Richard Charnin (TruthIsAll)
Oct. 29, 2010
Nate, this is a reply to your November 2008 post. I realize it is two years after the fact, but with the midterm elections next week, I thought it would be instructive to review what you said about exit polls. I for one would like to know if you feel the same way about them.
By the way, I’m still waiting for your response to these questions I posed back in July. But after reading your “ten reasons”, I can come up with ten reasons why you have never responded. The “experts” whom you cite are anything but.
…….
http://richardcharnin.com/
Election Fraud (1968-2012)
Quantitative Analysis and True Vote Models
I am just beginning to read this “creatives class” article at Open Left and I see a possibly questionable assumption passed off as fact in the first paragraph. Did Obama really “defeat Clinton in a grueling 50 state campaign”? Or did Clinton win the 50 state grueling part of
the campaign, and then Obama won the High Noon Theft part of it at the DemCon itself? Riverdaughter has written posts at The Confluence which would suggest that Clinton won the “grueling 50 state” part of the excercise. It would be nice if someone were to pull all those posts together into one place.
And that “creatives class” article leads to another article, almost as if Mr. Strether hopes we will follow the trail of smoking bread crumbs . . . an article by Matt Stoller. I will just cutpaste what seems to me to be a deeply incriminating couple paragraphs from that article.
“All I’ll add is that it’s time to think through the consequences of a party where there is a new chief with massive amounts of power. I’ve been in the wilderness all my political life, as have most of us. The Clintonistas haven’t, and they know what it’s like to be part of the inside crew. We have a leader, and he’s not a partisan and he can now end fractious intraparty fights with a word and/or a nod. His opinion really matters in a way that even Nancy Pelosi’s just did not. He has control of the party apparatus, the grassroots, the money, and the messaging environment. He is also, and this is fundamental, someone that millions of people believe in as a moral force. When you disagree with Obama, you are saying to these people ‘your favorite band sucks’.
Like many of us, I endorsed Obama, gave him money, and I intend to work to get him elected. He is attempting to completely rewrite the rules of politics, and we should try to figure out what that means for where we take our meager work. Obama is now the party leader. And he has ensured and we have given him the mandate that when he speaks, he speaks for all of us. I hope he’s a vibrant progressive when he gets into office, and we should begin figuring out how to put ourselves in a position to help him take the country in a progressive direction.”
Has Matt Stoller written any articles about how Obama actually LOST the 50 state campaign for pledged delegates and only got the DemCon 2008 nomination by open theft and cheating on his behalf by the pro-upper-class party leaders?(If Riverdaughter’s analysis is indeed correct).
Has Matt Stoller ever explained why he thought Obama was more “progressive” than Senator Clinton was to begin with?
Why was Stoller (as well as the other creatives) such a mark
for a slick “technically black” carnie? The same question could be asked of Al Giordano of course, and of many others.
Same thing applies to groundwater modeling. The model says one thing, while reality says another – yet the model is taken as gospel truth. Reality must be wrong. The result (in my neck of the woods) is more funds/jobs get devoted to modeling, the empirical people get pink slips, and the water supply dwindles.
All models are wrong, but some are useful.
Russ Roberts at Cafe Hayek linking to, quoting this post…
“there is a convergence of views between some folks in the Occupy movement and some of us with classical liberal views especially with respect to incentives and public choice. I suspect our solutions are different but the diagnosis overlap is interesting in and of itself. Read the whole thing”
Great piece, but while the title seems to accuse Silver of malice, in the body you go out of your way to accuse him of nothing more than naivete. As far as I can tell Silver is a pretty bright guy. The level of naivete needed not to see the clear incentives for, say, ratings agencies to use models that allow them to rate mortgage backed securities AAA, that is to think they tried their hardest to map reality and just failed spectacularly…that would mean that Silver has the mentality of a small child. Which could be, but I doubt it.
We say a lot about ourselves with the things we buy into. Just look at the quote from Silver about Obama and Gietner:
“To suggest that Obama or Geithner are tools of Wall Street and are looking out for something other than the country’s best interest is freaking asinine.”
What does this suggest about Silver? Naivete maybe…but it suggests to me something a bit more malevolent.
For a modeler to be so uninquisitive about the causative factors and make such confident pronouncements about why we had a crisis is not just naive, it’s reckless. This is someone who is involved in politics and therefore reads the newspapers, and to do the rating agency section of his book had to read about them, at a minimum. To airbrush out the larger context isn’t naive, it’s willful blindness.
In financial documents, parties often indemnify the person providing the service from liability. The standard “outs” are “bad faith and gross negligence”. Silver is so grossly negligent on the question of cause and effect as to be tantamount to malice. Or so is my take on Cathy’s logic.
Yes, I think you are correct about the malice logic…but it’s the ‘willfull’ part of the blindness that I find especially troublesome, and that I think may be downplayed by using words like naive. Though ultimately it may be a bit niggling on my part, and a little too unsympathetic. Again, great piece, and I look forward to Ms. O’Niel’s book (I think I may skip Mr. Silver’s).
Cathy goes overboard on being fair, and you are right, calling him naive undermines the “malice” claim.
Nate Silver, Malcolm Gladwell, Andrew Ross Sorkin.
Non, Non, Non.
“We didn’t have a financial crisis because of a bad model or a few bad models. We had bad models because of a corrupt and criminally fraudulent financial system.”
True.
But the overarching truth of far greater significance is that the “corrupt and criminally fraudulent financial system” is an entirely unavoidable and, indeed, structurally indispensable part of late capitalism itself, which cannot function at any level without at least 3% annual compound growth, which in turn ultimately runs up against the twin barriers of physical impossibility and conceptual implausibility.
“…late capitalism itself, … cannot function at any level without at least 3% annual compound growth…”
Do you have any citations for this? It sounds interesting.
Citations as follows (and these three should explain EVERYTHING):
http://www.webofdebt.com/
http://www.jayhanson.us/page134.htm
http://petras.lahaine.org/?p=1689
The fact of the matter, dutch, is that about 98% of everything we are “taught” or indoctrinated in, when we are in school is pure fantasy; e.g., the impetus and motivation behind all those wars to expand empire, etc.
The historic expansion of empire has always been to increase wealth accumulation and extraction from the majority, until said empire is bled dry and thus crumbles.
The prime culprit is the duality of money creation and interest: a royal or religionist or predatory capitalist has the divine right to create the money, which is then lent out with mandatory interest. The forever increasing debt spiral.
I tried reading the book, but couldn’t make it through all the stuff about finance and economics at the beginning. Reading NC and NEP on a regular basis has completely ruined my ability to listen to mainstream commentators for any length of time without getting pissed off.
Welcome
Thanks for this piece Ms. O’Neil. The Silver phenomenon has been a huge puzzle to me – and even a shock.
I have to confess that until about three weeks before the election, I had only the vaguest sense that some guy named “Nate Silver” existed. I had never read his column, I guess because I find the horse race dimension of politics to be intensely boring. And I still haven’t read one of his columns.
But then there was an explosion when a few conservative pundits made a few criticisms of Silver, and where Silver himself, and criticisms of him, became a massive election issue. I learned that not only was there this guy named “Nate Silver”, but that a whole world of liberal pundits and opinion leaders were intensely devoted to him and emotionally invested in whatever it is he did to a staggering degree. Krugman wrote three columns on the fracas I believe. WTF?! I have no problem with people wanting to argue with conservatives and show that they are wrong. But it stunned be that so many people cared so much about Nate Silver, and whether his polls were or were not accurate, that they were willing to take time out from the closing days of the campaigns to spend lots of time arguing about polls – as though the polls themselves were one of the really big issues we were deciding in the election. They responded to the criticisms of Silver with a hysterical and angry outpouring of geeky solidarity, as though Mitt Romney had just punched a nun.
@dan kervick – thanks for your comment. I entirely agree with you. One thing you didn’t mention though was the plug he received from The Daily Show. I don’t know how much of a Jon Stewart “bump” there is for anyone who appears on his show but I’m willing to bet a few dollars that it’s welcome exposure…
– sorry for the adding on but isn’t what Nate Silver does in a sense a form of “nombrilism”? Especially regarding elections? I simply don’t see the value of it.
Thank you, thank you, thank you for writing this piece.
To date, my only way of explaining this phenomenon is by simultaneously clenching my jaw, making a gutteral “errrrrrrrr” sound, and then raising my opened palms to the heavens, slamming them down on my hips, turning on my heel , and stomping away.
Words–when you have them–are always better. So–thank you for furnishing actual words to my emotion.
For those of us who are not only not math-inclined, but for whom language gets all smooshed up in our brains when attempting to explain something that is mathematical at its base–could we develop some sort of “shorthand” for getting to the point that you just made in your post?
For example, perhaps, we could throw a silver-colored card on to the proverbial playing field while declaring, with authority, (preferably–over a loudspeaker or public address system) “Silvering,” and then following up with a reasonable penalty for said “silvering.” Like, “Silvering. 5 Yards.”
This would make my life soooo much easier, and it would make me seem soooo much more reasonable than the clenchy-jawed-turning-on-heel exasperated thing that I am in the habit of doing.
Now. In defense of Nate Silver–he reminds me a bit of my husband. Mr. Silver (as I know for a fact that my husband is) is probably a super nice guy. Such folks are so nice, in fact, that they just aren’t instinctively wired to factor in “the angle” from which {less-nice guys} operate dishonestly and/or exploitatively. Some have characterized this as “naivete,” but that word isn’t quite right. “Blind spot” is probably a little bit better. But while I’m here and inventing words anyway,–how about this one: “Nicete.”
Nicete- (noun) the quality or state of being that is characterized by a chronic blind spot brought about by the inability to instinctively factor in others’ mal-intention and/or tendency to exploit a situation for personal gain. One who suffers from “nicete” comes to the table with an instinctive and unwavering expectation that everyone else at the table has come in comparable good faith and with a comparable expectation of fair dealing.
Silvering- (verb) the act of analyzing a situation with an instinctively unwavering expectation that everyone else at the table has come in comparable good faith and with a comparable expectation of fair dealing. “Silvering” is generally brought about by persons who exhibit the quality known as “nicete.”
Oh. And just in case you, Nate Silver, should stumble upon my comment here–let it be known that said comment has not been written with an unkind or unflattering intention. You are really quite adorable, and the world would be a nicer place if there more folks wired to be as nice as you probably are. But you just might to consider partnering with a worldly, skeptical person/mathbabe who could, from time-to-time, remind you to factor the bad guys into your math. :-)
I like both your coinages, but I have to disagree with seeing Silver as an example of “nicete”. I think you can have genuine nicete (think of the stereotypic absentminded professor, someone who both leads a cloistered life and has narrow interests) versus someone like Silver, who may indeed be geekily narrow in his focus, but has been out in the real world too long not to have an idea what the real deal is. So people like him can use “nicete” as a pose, a form of protective coloring. That in fact makes them hugely valuable to their patrons, since they are signaling that they won’t ask uncomfortable questions, but will go about doing their technical work in a complaint manner.
damn those stereotypic-absentminded-professor imposters!
Yves,
I think your description fits Mr Krugman to a T as well.
But is our Krugmans learning? YES!
Pearl, I loved your post. Great imagery and language both.
As a math guy — at times a bit Silvery and Niceive — I had one other reaction that might help you: Niceive people don’t play poker well. Every financial model lives inside a poker game. The Big Swinging Dicks are not looking at data or models — they’re scanning the table for Tells and Marks. The Niceive guys they hire just add “geek glamor”.
Big Swinging Dick takes a mark (potential investor) around his Desk/trading floor and points out geeks: that one went to CalTech, this one went to MIT. Underlying message: you really think you’re smart enough to beat me? Foggettaboudit: don’t try to beat me, join me (invest in my hedge fund, etc)
Keep up the good work!
Please correct me if I am wrong but Nate Silver’s presidential predictions were no better (and in 2008 worse) than what you got by tallying up the numbers compiled at RCP in the obvious way. I know he applied his method to RCP-provided numbers, but his secret recipe appears to have moved the dial just epsilon. I don’t even think he denies that. So what exactly is the deal about Silver when anyone with access to the Internet could have drawn almost identical conclusions in 5 minutes. Is it just because Joe Scarborough and Dick Morris are dickheads that Silver looks like a genius?
“Is it just because Joe Scarborough and Dick Morris are dickheads that Silver looks like a genius?”
Yes, in a nutshell. The Republicans had deluded themselves, but there were others doing the same thing as him just as accuratelly. He had a bigger circulation is the difference.
PS, i’m loving “nicete”.
JoeScar may be a dick, but he is also innumerate. Disinterested observers lacking some actual knowledge would NOT have been able too get the score in 5 minutes.
Innumerates saw One Big Number: the national voter preference number. Obama 47% v Romney 47% or whatever.
Numerates: hey there’s an electoral college, the math is a little more complicated than One Big Number.
Of course people Inside the game with money at stake may have ‘played innumerate’ for personal gain.
Propaganda is never naïve. Silver is just deflecting attention away from the looters and the looting and laying off the blame on to technical issues, like modeling. It’s just a riff on the agentless explanations for all the looting crises we are seeing. It’s a historical process, or an economic one, or modeling. It is never the responsibility of the perpetrators, the elites, and the beneficiaries, the rich.
Yeah, or like the insistence that people like Obama are just mis-educated by “theoclassical economists” into thinking that US Fedgov is just like a household.** Bill Black claims just this–yet again!!– on his other thread here today, 12/21.
I’m sorry but, even the stupidest American is not that stupid. I don’t know who Black et al think they’re talking to, but I’m getting tired of this pretension.
**I do believe Margaret Thatcher might have believed this, but she’s the only one.
It’s a lot like the War in Vietnam. Some people thought that the problem was not enough technocratic counter-insurgency experts with the right models and methods. Nope. After the Pentagon Papers came out, we discovered that there were lots of experts saying it was a bad idea. The problem was the decision makers.
“Why? Because you don’t know their incentives, and they can make the models (including Bayesian models) say whatever is politically useful to them.”
Not only can Bayesian inference be used to tweak the models to say something convenient for the modeler, Bayesian inference is _especially_ useful for that purpose, because it introduces “prior belief”, which generally is a personal input from the modeler.
“We didn’t have a financial crisis because of a bad model or a few bad models. We had bad models because of a corrupt and criminally fraudulent financial system.”
(puts face in palms)
And Nate Silver is the enabler, the naive one?
Who knows, with posts like this MB might someday make a name for herself, get her own book deal, etc.
huh
Tony,
Are you saying that Nate Silver is corrupt and criminally fraudulent?
then say it loud
“Expert Opinion”?
some dude writes a book where he talks about financial math models and actually says we have to trust expert opinion?
I’m laughing so hard I can’t type straight. I applaud Ms. O’Neil’s ability to summon up the requisite critical rage to complete a post without hysterically laughing and passin out on the floor,.
Fortunately I live a life of ignorance so have never heard of this dude, except here. Just like the dude Ezra Klein. Never heard of him, until they started biching about him here.
There’s a lof of us out here that don’t pay attention to any of these bozos. It’s a comforting thought for folks who lose hope in humanity. And I’m not out buyin a Bushmaster either. I have two 0.22 rifles but both are single-shot for competitive shooting and two BB guns. I just mind my own business, like most normal people, and tune out nearly everything. If the world falls apart, I’ll tune that out too, cracking up at how all he models said it was a 9 sigma even.
“Fortunately I live a life of ignorance so have never heard of this dude, except here. Just like the dude Ezra Klein. Never heard of him, until they started biching about him here.”
Same here, I chalk it up as the price of reading NC. And I did read Silver’s intro to The Signal and The Noise… Unbelievable! Clearly he is trolling for his paymasters, the “big lies” he so straight-faced says there prove to me the book is an exercise in political propaganda, not any intelligent review and exposition of modeling, financial or mathematical.
To amplify your point on medical research, specifically pharma, try this quick Ted Talk.
http://www.ted.com/talks/ben_goldacre_what_doctors_don_t_know_about_the_drugs_they_prescribe.html
Good short course on manufacturing success bias.
Re: the wavy line between expert and useful idiot
“It is difficult to get a man to understand something, when his salary depends on his not understanding it.”
― Upton Sinclair, I, Candidate for Governor: And How I Got Licked
QED
“RAWR.. IT’S POPULAR LITERATURE AND CONDENSES DIFFICULT SUBJECTS, BUT I KNOW MATHS SO I CAN DISMISS IT CONDESCENDINGLY! FEEL MY POWER!”
Seriously – If you haven’t read the book, don’t leave a comment bashing it based on the site’s review, think for yourself.
As for the review – I felt he addressed those issues. You quoted him yourself writing that “agencies were willing to rate instruments structured by cows.” He also talked about the ratings agencies sharing their models with firms to make them easier to manipulate, as well as their unwillingness to adjust models despite seeing that the data was off. How much clearer can you get than that?
Do you really need him to come out and scream “THEM KNOWINGLY RATING BAD INSTRUMENTS + SHARING RATING MODELS FOR EASY MANIPULATION = FOUL PLAY IN THE INDUSTRY?” I’d assume anyone with a head on their shoulders could figure that out. Yes, he does simplify his main points down to problems with the model, but he doesn’t falsely claim that everyone was trying hard to fix them. Readers can easily deduce that the agencies weren’t playing fair.
His whole chapter on political forecasting can be summarized as “things don’t have a simple explanation, be willing to listen to nuanced answers and piece together clues from different sources.” I’d assume he thinks his readers are smart enough to heed that message and read his book as an introductory text with some explanations, not as the definitive answer of all the world’s problems.
If by ‘popular’ you dismiss any notion of accuracy and fairness, think again. Simplifying and amplifying shorthand concepts from specialist communication to generalist communication isn’t always lossy or distorting. And it often blows up the claims of specialist knowledge/thinking. Oz the Great and Powerful is behind the curtain.
Ms. babe is calling out a serious contradiction. You can’t say both the ratings agencies built flawed models and were the victims of flawed models. Well, you can if you’re writing theology. Any other field, you’ve got to recognize agency. If you build flaws into the model, you’re not the innocent victim of a flawed model.
As to anyone with a head on his shoulders, it’s been a long hard fight against the ‘honest but mistaken’ smokescreen. Sincere but overwhelmed. Decent but deceived. You’ve heard it all before, I’m sure.
The right wing has built a global noise machine of originators, promoters, and repeaters. Sounds like Ms. babe has found one of its cogs.
Look, have you actually read the chapter in question? He never portrays ratings agencies, or anyone complicit in pushing these faulty models as “victims.” Rather, he explicitly reminds the reader again and again that out of self-interest – “keeping the party going” – that they stuck by these faulty models. The reader is left to decide whether it was because of self-delusion or voluntary fraud. You’re making it seem like Silver implies it was only the former, and he does no such thing.
*that it was out of self-interest
Awww, did you get your widdle technocrat fweelings hurt? LOL, ya’ll are pathetic, anti-knowledge schmucks who don’t understand basic things like cause and effect.
As Mark Twain would say:
Lies, damn lies and ststistics …
Figures don’t lie but liars figure …
When i first ran across the Fallacy of Misplaced Concreteness lo those many years ago, little did i know that it would be almost as suitable for any occasion as “And this too shall pass away” ….
EXCELLENT post, with lots of good quotable insights. The misleading — and, frankly, offensive title — was added by NC. The original title of the post was
“Nate Silver confuses cause and effect, ends up defending corruption”.
That was perfect. Why was it “improved”?
She gave me the HTML yesterday and didn’t have that title.
And I’m sorry, I like my title better. Anything an author says in a post is fair game for a title, and she did say “maliciously wrong”, so the title as it stands is legitimate.
If you are offended by this sort of thing, you need to stop visiting this site. Seriously. I say far worse things on a daily basis.
See Hugh’s comment above:
http://www.nakedcapitalism.com/2012/12/cathy-oneil-why-nate-silver-is-not-just-wrong-but-maliciously-wrong.html#comment-972457
[sorry – posted in the wrong place]
okay, but Cathy uses the word “naive” three times to describe SIlver, so that seems to be her primary characterization of him.
Arguing that naiveté is not a defense for those who enable corruption is another good point.
But doesn’t the attribution of “malice” go beyond challenging naiveté as a defense?
I have the advantage of knowing Cathy personally. She pulled her punches in her piece. She was really outraged by his book.
Yves, my concern was having the great work of your site undercut by inadvertent choices of words that sew division among people who should find common cause. Humble apologies for raising your blood pressure!
But I see your point now. Cathy did say
“There’s no math band-aid that will cure these boo-boos. I can’t emphasize this too strongly: this is not just wrong, it’s maliciously wrong. If people believe in the math band-aid, then we won’t fix the problems in the system that so desperately need fixing.”
I’ll need to ask Cathy about her choice of the phrase “maliciously wrong”, which seems a bit out of sync with the very powerful point. Maybe “mendaciously wrong” would be better …?
I guess I am not a big believer in common cause. While in theory that is a good aim, in practice that is too often used as an excuse for sellouts.
I know a very accomplished scholar who is trying to update Marx’s critique to the financialized economy. He is convinced that progress for the majority lies in increasing access to the products based on the mathematics of derivatives as a way to hedge risks (of greater income inequality, effects of global warming, etc.)
He asked me to explain the mathematics to his classes of social science and humanities graduate students, and I have done some of that though I always stress the fact that the problems with the system do not lie in improving, or increasing access to, the models.
Cathy’s above critique provides some helpful phrases I can use. But I think many well-intentioned people confound the problems with the mathematics, politics, and legal issues in the current system.
http://www.jayhanson.us/page137.htm
okay, but Cathy uses the word “naive” three times to describe SIlver, so that seems to be her primary characterization of him.
Arguing that naiveté is not a defense for those who enable corruption is another good point.
But doesn’t the attribution of “malice” go beyond challenging naiveté as a defense?
I think that Dan Moran’s explanation (“I don’t think that word means what she thinks it means”) is the simplest and most likely, and that what “malicious” meant to Cathy is not the same as what it means to most users of the word, or the dictionary definitions. It should be easy to clear up.
I had the same thoughts on reading the piece, and before seeing any comments about it.
And if she was operating with the commonly accepted definition of “malicious” in mind, then other parts of the piece should have been worded somewhat differently to avoid the impression of a contradiction — which several of us registered, it seems.
See my comment earlier:
http://www.nakedcapitalism.com/wp-admin/comment.php?action=editcomment&c=972714
I did see your comment earlier — I’m focusing on the effect on the reader, and it seems that several of us got the same impression: that the repeated use of “naive” and a couple of other features of Cathy’s diction contradicted what we understand “maliciously” to mean (specifically WRT conscious, deliberate intent).
It’s a great critique and it’s really just a quibble, but I think a slight bit of editing would strengthen it: either by replacing “malicious” with something like “pernicious” or “mendacious,” or altering the other bits that seem incompatible with “malicious.”
She should have simply said “naive at best.”
That captures it.
I take great pride in the fact that I’ve never let anyone be bullied within my zone of influence. Never. Not once in 52 years of living (marked by approximately 48 years of noble scrapping).
So, dweezles, nerds, bookworms, all bully targets, rally round! For I say welcome, and I also say, you will always be safe within my aura.
Unless we’re outnumbered by say, 5 to 1. If we’re looking at 5 to 1, or God forbid more, we are, of course, in desperate trouble. But regardless of the odds, no matter how hopeless our situation, I give you my word, I will not hesitate to throw verbal ripostes –or fists of iron, should it come to that– to defend you.
This is the code I live by.
Note: There are two exceptions to my code, it does not include Nate Silver or Megan McCardle.
I admit, I would like to see them him bullied about. Not physically. There’s no need for that. But certainly, I would like to see them corned, like rats, and intellectually harried.
No quarter shown. No mercy! This post –plus comments!– seems like a good start.
Note II: I retire now, to my Buddhist cave. I need to imagine a hard journey to tranquil Tibetan lands, where I will ask stoic monks scheduled for self-immolation* for imaginary guidance.
Tell me again, my serene Master, who is worthy of defending, and who is not?
*China, you bully.
Merry Christmas, Max424. You will enjoy the back-and-forth chronicled here, in which Kurt Harris administers a heavy dose of truth to Megan McArdle: http://www.archevore.com/panu-weblog/2011/4/9/jousting-with-the-atlantic.html.
Mathematical models can only work if there is an underlying reality being modeled. In physics there is mass, duration, length, and a few others – defined and measurable quantities that interact with each other in specific ways. “F = ma” models a real physical relationship that must hold exactly for all times and places. The model works because the underlying reality ensures that it does.
Econometrics models perceptions and behaviors – things which have no fixed relationships to each other. They vary from person to person and from one instant to the next. Economic models can only describe things that have already happened. How things will happen in the future may be very different and the models made earlier will be useless. Econometrics has no subject matter.
“Econometrics has no subject matter.”
that’s another keeper!
I’m always skeptical of models. What I always try to figure out when I’m confronted with the results of one is to find out how that model has been validated. As dutch says in a previous comment, that’s a dicey question with economic models – there doesn’t seem to be any way of validating.
The one way is the one that Cathy O’Neil mentions – is the model builder (or model salesman) depending on the accuracy in some way? If so, I suppose there’s some faint hope it’s accurate.
As for Nate Silver, I trust the accuracy of his political polling analysis. Beyond that, though, I see no reason to believe him about much of anything. Whoever mentioned he seems to have a streak of authority worship in him is about right.
I haven’t read The Signal and the Noise, but I read an excerpt published by the NYTimes, and I was somewhat disappointed – the arguments in the excerpt struck me a somewhat sloppy. Superficially it sounds fine (good narrative), but doesn’t entirely hold up when examined closely (less there than meets the eye). Silver seems prone to munge distinct things together (e.g. chaos theory and probabilistic modeling) and make somewhat sloppy intellectual leaps – the writing has the feel of someone who is overreaching to create a simple compelling story. And though overreaching good-sounding narrative can facilitate career ascendency/celebrity, I like clean, incisive, analytical thinking, so it slightly bugs me.
E.g. what is this really saying?: “Why are weather forecasters succeeding when other predictors fail? It’s because long ago they came to accept the imperfections in their knowledge. That helped them understand that even the most sophisticated computers, combing through seemingly limitless data, are painfully ill equipped to predict something as dynamic as weather all by themselves. So as fields like economics began relying more on Big Data, meteorologists recognized that data on its own isn’t enough.”
This review of his book, by Michael Mann, is worth a read. Some of the points Mann makes rather echo some of Mathbabe’s comments. E.g.
Mann: When it came to areas like climate change well outside his own expertise, he to some extent fell into the same ‘one trick pony’ trap that was the downfall of Levitt (and arguably others like Malcolm Gladwell in The Tipping Point). That is, he repeatedly invokes the alluring, but fundamentally unsound, principle that simple ideas about forecasting and prediction from one field, like economics, can readily be appropriated and applied to completely different fields, without a solid grounding in the principles, assumptions, and methods of those fields. It just doesn’t work that way (though Nate, to his credit, does at least allude to that in his discussion of Armstrong’s evaluation of climate forecasts).
But I am glad that the “Nate Silver phenomenon” is causing people to reorient somewhat toward the value of math and factual reality. A good Paul Rosenberg column on this: Math wins huge upset over GOP – are Democrats paying attention? Comforting stories that don’t add up lead Republicans to reject what the polls were telling them before election day.
I have to give CWCD on Silver’s overall outstanding performance in the elections, and found it hard to resist watching some of the most loathsome in the belly of the beltway beast make fools of themselves trying to take him down.
That said, I have never been much of a fan of Silver otherwise, and Cathy O’Neil’s fine piece hits squarely upon the problem I have found in Silver’s approach: that he utterly fails to take into account the politics. Political and economic power, who has it, how they will use it and how it will shift as a result has to be part of the analyses. I haven’t read Silver’s book, but based on O’Neil’s account, it doesn’t sound like there is much daylight between Silver and those who blamed the Iraq War on bad intelligence.
My problem with Silver began with the healthcare debate when he displayed many of the same attributes that O’Neil discusses in a little ditty titled, “Why Progressives Are B–sh– Crazy to Oppose the Senate Bill” on 12/15/09:
http://www.fivethirtyeight.com/2009/12/why-progressives-are-batshit-crazy-to.html
where he had all these nice mathematical models showing how much better off those in certain scenarios would be under the plan while completely ignoring the power (and steadily increasing power at that) of the insurance and pharmaceutical companies to negatively affect, for example, his assumed rates of inflation. Or as Richard Eskew put it so well in his response titled, “Why Progressives Would Be ‘B–sh– Crazy’ to listen to Nate Silver on Health Reform” on 12/16/09:
http://crooksandliars.com/rj-eskow/why-progressives-would-be-batshit-crazy-l
“Silver’s heart may be in the right place, and his math is right, but many of his assumptions are flat-out wrong. More importantly, he fails to place his work in the proper human and political context. It’s like this: You can build the best model in the world for predicting the outcome of hockey games. But if you knew that sometime during the third period Rahm Emanuel was going to drive out on the ice in a Zamboni and flatten your team’s entire defense, wouldn’t that change your model a little?”
As for the old stupid vs. evil question in regards to Silver, even if it really matters, I just think it is difficult to discern the willfulness of his blind spots. Actually, the equation should be triangular, with ego playing just as important a role. Ego can dissuade even the most intelligent and plugged-in from doubt and self-analysis, and everything I’ve seen about Silver points to a large ego.
I think many may be being too hard on Mr Silver. I read the book and loved it and Im not sure I would reach the same conclusions that the author of this post has. Not ALL the same conclusions anyway.
A point to make regarding models. SIlver thinks we need MORE of them not less of them. Not only that he thinks those making the models should put some of their OWN skin in the game and BET on them ……. personally! Let the best model win!! I think he makes excellent points here. I also dont think he has a hero or expert worship issue, but he does think that people who ARE making models that make predictions, and the predictions ARE turning out better than others, those people should be noted and their models analyzed.
I agree that Mr Silver does not fully grasp economics and financial markets but who does? I think much of what he talked about in this book IS a step in the right direction.
Haters gonna hate.
Silver’s analysis of baseball is a joke, too. No mention of performance enhancing drugs, which is like analyzing the economy without saying anything about subprime mortgages. But I’m not sure why anyone is angry. What did you expect from a guy whose predictions lag behind Paul the Octopus?
mal·ice
noun
1.
desire to inflict injury, harm, or suffering on another, either because of a hostile impulse or out of deep-seated meanness: the malice and spite of a lifelong enemy.
“dangerously naive”
these two things seem to be mutually excusive
Models are composed of the rabbits you put into it…whatever rabbit you put in, is the rabbit you pull out. The emphasis on politics (and even ideology if she felt to do so) is completely appropriate. Economic models are of course the same thing, the only question is whether that model has anything to do with reality.
http://richardcharnin.com/OpenLettertoNateSilver.htm
An Open Letter to Nate Silver from Richard Charnin
Richard Charnin (TruthIsAll)
Updated: Aug. 2, 2010
Nate, since your recent hiring by the NY Times, the R2K flap and your exchanges with Zogby, you have been getting lots of publicity from blogs such as vanity fair and motherjones.com. Your characterization of Zogby’s expertise (that he is the “world’s worst pollster”) says more about you then it does about him. Zogby ranked #1 in 1996 and 2000 (yes, Gore won Florida, despite what the NY Times said), and came close in the 2004 and 2008 elections, yet you fail to give him credit and rank him dead last. Why? Because you go along with the media-perpetuated myth that the recorded vote is sacrosanct. In other words, you discount the fraud factor and fail to distinguish between the True Vote and the recorded vote.
Below, you will see why Gore won by perhaps three million more than his recorded 540,000 vote margin; why Kerry won the True Vote by 10 million; why the Democratic Tsunami was denied in the 2006 midterms; and why Obama won by nearly 22 million votes in 2008, not the 9.5 million recorded.
I hereby challenge you to try and debunk the data, logic and mathematics used in the True Vote Model. If you cannot do so, then the underlying premise of your ranking system (that the recorded vote is an appropriate baseline to measure pollster performance) is invalid.
As an Internet blogger who has been posting pre-election and exit poll analyses to prove election fraud since 2004, I have occasionally looked at your postings on fivethirtyeight.com. I will say right here that unlike the bloggers and mainstream media (MSNBC, the NY Times, etc.) who extol your forecasting “expertise”, I do not believe you are quite the polling guru that they claim you are.
I say this as one who has been building quantitative models since 1965 for defense/aerospace manufacturers, Wall Street investment banks and has consulted for many financial and corporate enterprises. I have three degrees in Mathematics, including an MS in Applied Mathematics and an MS in Operations Research.
Your 2008 simulation model win probabilities did not sync with the projected vote shares. The major flaw in your model was to conflate it with your pollster rankings, an ill-conceived methodology. The first rule of model building is KISS (keep it simple stupid). You not only introduced an extraneous variable into your model, but the rankings were incorrect – a double whammy. Now, what do I mean by this, you ask?
You fail to distinguish the True Vote from the Recorded vote by ignoring vote miscounts. The premise on which your models are based (that fraud does not exist) is incorrect from the get-go. In your ranking system, pollsters who come close to the recorded vote (i.e. Rasmussen in 2004) are ranked high, but pollsters who come close to the True Vote (i.e. Zogby) are ranked low. The fact that Zogby is ranked at the bottom is a clear indictment of your approach. Ranking pollsters based on their performance against the recorded vote is a waste of time. Fortunately for you, your fans are unaware of the distinction between the recorded vote and the True Vote. In fact, most are unaware of the extent in which their votes have been compromised by fraud. In your models, election fraud is never a factor.
This is the simple, yet fundamental equation that you seem to be blissfully unaware of: Recorded Vote = True Vote + Fraud.
In every election since 1968, the recorded vote has deviated widely from the True Vote. In the eleven elections, the Republicans won the recorded vote by 49-45%; the Democrats won the True Vote by the reverse: 49-45%.
The very conservative 3% exit poll margin of error was exceeded in 66 of 238 state exit polls conducted for the NEP in the five presidential elections from 1988 to 2004 – and 65 “red-shifted” in favor of the Republican. Approximately six (0.025*238) should have been exceeded assuming the elections were fair.
The probability that the margin of error would be exceeded in 65 of the 238 state exit polls for the Republican is calculated using the Excel function
= BINOMDIST (65,238,0.025,FALSE) = 1 in 43,729,463,568,632,100,000,000,000,000,000,000,000,000,000,000!
In 2004, Zogby’s final polling in nine battleground states was within 0.5% of the unadjusted exit poll average (after allocating undecided voters).
Kerry led in 8 states by 50.2-44.8%. The base case assumption was that he would capture 75% of the undecided (UVA) vote and win all 9 states by 53.7-45.9%. Assuming a conservative 55% UVA scenario, he would still win 8 states by 52.7-46.8%. Kerry officially won 4 of the 9 states by 50.1-49.4%. The margin of error was exceeded in 7 states, a 1 in 4.7 billion probability.
much more at the link…
Rep,ly to Richard Charnin:
I am an open-minded person, but after a minute or so I stop reading your stuff because it fails to link to documentation for each one of your (often hard-to-believe) claims, as it would need to do in order to be able to convince any open-minded person who is reading such unorthodox assertions. You just expect readers to take on faith what you say. If you weren’t contradicting what’s widely believed, you could do that and still (unfortunately) have an audience. But not with the sorts of thing that you are alleging.
http://richardcharnin.com/Silver2.htm
Open Letter to Nate Silver (part 2): Zogby/Rasmussen, RV/LV, Simulation Mean/Expected EV, Recorded/True Vote
Richard Charnin (TruthIsAll)
July 17, 2010
This is an update to my open letter to you of a few days ago. The purpose is to further discuss a) the difference between 2008 pre-election registered voter (RV) polls and likely voter (LV) polls for forecasting, b) Zogby’s poll vs. selected GOP-leaning polls and c) the close match between the RV polls and the post-election True Vote Model and d) Pollster rankings measured by how close their projections were to the True Vote.
I have added three new tables.
Table 5 is a 2008 Pollster True Vote Ranking Chart (15 polls)
The Gallup (RV) tracking poll ranks #1 with a 57.1% Obama projection (after UVA)
CBS (LV) and ABC/WP (RV) are tied at #2 with a 56.6% projected share.
Zogby is ranked #4 with a 55.1% share.
Pollsters with a GOP bias brought up the rear:
Battleground ranked #14 with a 52.4% share.
Rasmussen ranked last at #15 with a 52.1% share.
Table 6 is a comparison of final RV and LV polls
The average LV poll had Obama winning by 50.3-44.0 before allocating undecided voters (UVA) and 53.4-45.1 after UVA.
The average RV poll had Obama winning by 53.3-39.5 before UVA and 57.6-40.9 after UVA.
Zogby’s LV poll had Obama winning by 54-43 before UVA and 55.1-43.4 after UVA.
Consider the final ABC and Gallup RV Polls (combined 5293 sample, 1.8% MoE).
They had Obama winning by 53.5-40.5 before UVA and 56.9-41.6 after UVA.
You rank Zogby dead last, yet his LV poll numbers are right in the middle of the RV and LV groups.
He is closer to ABC and Gallup than Rasmussen, Hotline and FOX.
You have lowered Rasmussen’s ranking but you still rank him much higher than Zogby.
Rasmussen has a strong GOP bias. Hotline, FOX and Battleground also lean to the GOP.
Do you have any evidence that Zogby’s polls are biased?
Do you still feel that you are justified in ranking Zogby last?
Table 7 displays the post-election True Vote Model.
It closely matches the RV projections.
It proves that the NEP returning voter mix is bogus.
My final 2008 Election Model gave Obama 53.1%. The 5000 election- trial Monte Carlo EV simulation produced a 365.8 mean EV.
The convergence of the mean to the expected value illustrates the Law of Large Numbers.
The expected value is given by the simple summation: EV = å Win probability (i) * EV (i), where i=1,51 states
Obama had an expected 365.3 electoral votes, matching his recorded total.
The Election Model exactly matched the recorded EV and was within 0.2% of the popular vote.
But it was wrong. It understated Obama’s True Vote.
The final state polls were LVs, not RVs.
Do you still believe that Obama’s 52.9% recorded share reflects the True Vote?
Do you still think that Obama had just 365 electoral votes?
The True Vote model indicates that Obama had close to 58% and over 420 EV!
More at thelink…
http://richardcharnin.com/SilverRankings.htm
John Zogby vs. Nate Silver: 1996-2008 True and Recorded Vote Rankings
Richard Charnin (TruthIsAll)
Updated: July 24, 2010
As discussed in my open letter to Nate Silver, his methodology for ranking pollsters is based on an invalid premise: that the recorded vote is an appropriate basis for measuring performance. Due to systemic election fraud, the recorded vote is not justified. The best measure is the True Vote, which is derived from total votes cast, rather than votes recorded. Using the Census value for total votes cast in the prior and current elections, we deduct four-year voter mortality and, combined with a best estimate turnout of living voters in the current election, we utilize National Exit Poll vote shares to calculate the True Vote.
Given the True Vote for the 2000, 2004, 2006 and 2008 elections, we can measure pollster performance in predicting the vote. Good pollsters such as John Zogby should not be penalized in the rankings because of election fraud. Conversely, biased pollsters such as Rasmussen should not have been rewarded in Silver’s rankings for predicting a fraudulent recorded vote.
Reputable election analysts who have crunched the numbers agree that the 2000 and 2004 elections were stolen and Democratic Landslides were denied in the 2006 midterms and the 2008 presidential election.
The following tables illustrate pollster performance for the four elections against both the True Vote and the recorded vote. The rankings are straightforward; they are based on the deviation between the final poll (adjusted for undecided voters) and the True and recorded votes.
The projections allocated 75% of undecided votes to the Democrats, who were the challengers in 2004, 2006 and 2008.
In 2000, Clinton was the incumbent who had high approval and a strong economic record, therefore a 50/50 split was assumed in the undecided vote.
In 2004, Bush had a 48% approval rating which declined to 25% in 2008. Obama was the de-facto challenger; McCain represented the incumbent.
Nate Silver ranks Zogby DEAD LAST. The historical record proves that Silver is DEAD WRONG.
This is what Zogby had to say just before Election Day 2004:
The key reason why I still think that Kerry will win… traditionally, the undecideds break for the challenger against the incumbent on the basis of the fact, simply, that the voters already know the incumbent, and it’s a referendum on the incumbent.
And if the incumbent is polling, generally, under 50 percent and leading by less than 10, historically, incumbents have lost 7 out of 10 times. In this instance you have a tie, a President who is not going over 48, undecideds who tell us by small percentages that the President deserves to be reelected. And in essence, it gives all the appearances that the undecideds — the most important people in the world today — have made up their minds about Presiddent Bush.
The only question left is: Can they vote for John Kerry? If it’s a good turnout, look for a Kerry victory. If it’s a lower turnout, it means that the President has succeeded in raising questions about John Kerry’s fitness.
There was a very heavy turnout of 22 million first-time voters and others who did not vote in 2000. In his Election Day polling, Zogby had Kerry winning by 50-47% with 311 electoral votes, indicating that 75% of undecided voters broke for Kerry. This was a virtual match to the 52-47% unadjusted state exit poll aggregate data later released in the Edison Mitofsky 2004 Evaluation Report.
In 2004, Zogby’s final polling in nine battleground states was within 0.5% of the unadjusted exit poll average (after allocating undecided voters).
Kerry led in 8 states by 50.2-44.8%. The base case assumption was that he would capture 75% of the undecided (UVA) vote and win all 9 states by 53.7-45.9%. Assuming a conservative 55% UVA scenario, he would still win 8 states by 52.7-46.8%. Kerry officially won 4 of the 9 states by 50.1-49.4%. The margin of error was exceeded in 7 states, a 1 in 4.7 billion probability.
—-
In 1996, Zogby was within 0.3% of the recorded vote.
He ranked # 1.
In 2000, Zogby was within 0.1% of the recorded vote.
He ranked #1
But there were 6 million uncounted votes.
Gore won by at least 3 million votes.
The election was stolen.
In 2004, Zogby was within 1.2% of the recorded vote.
His Election Day polling had Kerry by 50-47%.
Kerry’s True Vote was 53.2% – a 10 million margin.
The election was stolen.
In 2006, Zogby ranked #7.
The pre-election Generic Poll Trend Model forecast a 56.4% Democratic Landslide.
The unadjusted National Exit Poll had 56.4%.
The landslide was denied.
In 2008, Zogby was within 2.2% of the recorded vote.
He ranked # 4.
Obama had a 58% True Vote share and won by 22 million votes.
The landslide was denied.
So why is Zogby at the very bottom of Silver’s pollster rankings?
more at the link…
http://richardcharnin.com/SilverTimesTruth.htm
The Truth about Elections: Not “Fit to Print”
Richard Charnin (TruthIsAll)
Sept. 30, 2010
The NY Times and/or Nate Silver decided that my reply to Silver’s blog was news that was not fit to print. They deleted it 15 minutes after it was posted in response to:
http://fivethirtyeight.blogs.nytimes.com/2010/09/30/the-uncanny-accuracy-of-polling-averages-part-2-what-the-numbers-say/#preview
This was my post:
Nate,
You gauge polling accuracy by comparing the average results to the recorded vote. You should at some point acknowledge that final LV poll projections have been accurate in matching fraudulent vote shares and that the recorded vote is never equal to the True Vote due to systemic Election Fraud.
In fact, final RV poll projections (adjusted for allocation of undecided voters) have closely matched the unadjusted and preliminary exit polls that exposed the fraudulent 2004, 2006 and 2008 elections. In each election, the Final National Exit Poll (which is always FORCED to match the recorded vote) required more returning Bush voters from the prior election than were alive in 2004 and 2008.
But you never mention the F-word. The fraud component is included in my 2010 Midterms House and Senate Forecast Model:
http://richardcharnin.com/2010ElectionForecastModels.htm
For the record, my 2004 Pre-Election Model Monte Carlo simulation gave Kerry 52% and 337 electoral votes, matching the 114,000 sample aggregate of the unadjusted state exit polls. The Final 2004 National Exit Poll required 6 million more returning Bush 2000 voters than were living in 2004.
Proof:
Bush had 50.45 million votes in 2000; the Final 2004 NEP indicated that there were 52.6 million returning Bush voters. Given 1.25% annual voter mortality and assuming 97% turnout of living Bush 2000 voters, there could have been no more than 46 million returning Bush voters. So we have over six million Bush phantoms.
Kerry won the 2004 True Vote by nearly 10 million. Yet Rasmussen had a high ranking from you a few years ago based on the close match of his LV poll to the bogus recorded vote.
In 2008, the Election Model predicted 53.1% for Obama (he had 52.9% recorded) and exactly matched his 365 EV. But it was wrong. The post-election True Vote model (based on the Final 2008 National Exit poll with a feasible returning voter mix) indicated that he had 57-58% and nearly 420 EV.
Proof:
The Final 2008 NEP required an impossible 12 million more returning Bush than Kerry voters to match the recorded vote. Bush won the bogus 2004 recorded vote by 3 million. He had 22% approval on Election Day 2008. So where did all those millions of phantom Bush voters come from? Based on the True 2004 vote, there were 10 million MORE returning Kerry voters than Bush voters.
View the full proof here: http://richardcharnin.com/ObamaProof.htm
Richard Charnin
You’ve made a superb case for a book you will write ripping to shreds Nate Silver and other “experts” who share his style of deception.
That style of deception is enormously rewarded in academia. Just give examples: Milton Friedman and Steven Levitt come immediately to mind.
Eric,
I have already written the book – and devote a few pages to debunking Nate Silver in it:
http://www.amazon.com/Matrix-Deceit-Forcing-Pre-Election-Fraudulent/dp/1480077038/ref=sr_1_1?s=books&ie=UTF8&qid=1351586974&sr=1-1&keywords=matrix+of+deceit
Great post.
excellent post. Nate Silver aint nuthin but a commie baby.
Thanks for this! (And thanks to Massimo P for linking it.) I had not read this book. And I had no idea that this guy bought into that innocent mistakes in “failed models” B.S. regarding the financial crisis. I really thought it was generally accepted that the reasons the rating agency models failed was because they deliberately set them up to deceive! That much is obvious. What’s wrong with this guy? Sounds like a case of not seeing the forest for the trees. (*eyes rolling*)