THE SIGNAL AND THE NOISE

Nate Silver has developed a reputation of being able to predict events in the fields of baseball and politics better than most other people.  He is obsessed with numbers and what they mean.  Near as I can interpret what the title means is that in this age of computers there is so much information (noise) that we too often miss the important indicators (signal).

We have been told that the good thing about computers and the internet is all the information coupled with the fact that the bad thing about computers and the internet is all the information. Information is supposed to help you make better decisions.  Some decisions are relatively trivial, but others are life changing and all of them accumulate.  Silver advises on separating the noise from the signal.

The noise in all this information can be very distracting, even misleading.  Scientific studies can involve pages and pages of small details.  They can be very important, but what do they really mean.  One phrase becoming more common is that "correlation is not causation."  There is a lot of correlation in information, but it is more critical to determine the causation factors.

Thomas Bayes provided the underlying philosophy for Nate Silver. Thomas's birthdate is uncertain, but assumed to be about 1701.  He was born in England, but because he was of a non conformist background went to the University of Edinburgh and eventually became a Presbyterian minister.  He had an interest in mathematics, but only published one book before his death.  His fame rests on a book that was published two years after his death.  We all deal with imperfect information.  Bayes recognizes all we can do is proceed to a better decision.  No human will ever have all the facts.  Each event allows us to recalculate.  Probabilities are constantly refined as we get closer to the truth, with-out ever fully arriving.

Baseball statistics helped relieve some boredom Nate felt at one of his jobs.  He became more and more obsessed.

Poker was another obsession.  Poker depends upon understanding probabilities and masking your confidence.  Nate got involved at a fairly high level, but has since greatly lowered his involvement.

Climate change is a topic of much predictions.  Nate points out is that there is a definite cause that has been identified and accepted.  Weather is actually an area of prediction that has improved with more knowledge.  Most commercial forecasters predict more rain than actually occurs which Nate attributes to the idea that rain is more upsetting when not predicted.  Climate has many variables that make prediction difficult.  For example solar cycles, the El nino currents, volcanoes,  pollution, etc.

In the most recent American presidential election Nate was accused of slanting his statistics to boost his ideological preferences.  Looking back it seems conservatives were not only guilty of wishful thinking, but also of trying to create the impression that Nate's figures were not as accurate as theirs. Since the election there has been a lot of analysis of what it all means and to me it seems they are still guilty of wishful thinking.  Misreading what the public really wants is not only harmful to politicians, but to everyone.