She Won’t Survive

I will survive.
—Gloria Gaynor.

I had no idea that it was that easy to get the attention of, much less—apparently—annoy the hell out of a national talking head for a semi-big-time news network like MSNBC, but apparently in the brand-new world of social media such things are easily possible. Such, at least, is what I learned when I happened to object to that network’s Joan Walsh’s cheerleading for Hillary Clinton on Twitter the weekend before the New Hampshire primary. I won’t get into the particulars—the lowlight was probably when she got taken to task by a city councilman from New Rochelle, New York for attempting to use race as a bludgeon (the councilman is black, seems like a decent guy)—but suffice it to say that many supporters of Hillary Clinton seem to think that she deserves the Democratic nomination on the basis that she has climbed through all sorts of slime to get to the position she is in now. From one perspective, of course, that might be a good reason to think she should not be elected—crawling through slime tends to get dirty—but as Glenn Greenwald, the journalist who broke the Edward Snowden story, pointed out the other day, logic does not appear to be a strong suit in Hillaryland. What Greenwald’s story suggests is that the difference between Clinton supporters and Sanders’ supporters is that the latter understand the logical error known as “survivorship bias,” and the former don’t. The trouble for Hillary Clinton’s campaign is that without such an understanding, there seems little reason to vote Democratic at all.

That then would seem to make “survivorship bias” a significant concept—but what it is it? Essentially, survivorship bias is the magical belief that something successful possesses a special quality that caused that success, instead of considering that it may simply be the result of coincidence. Nicolas Taleb advances an example of how survivorship bias can skew our assessments of the world in his book, Fooled By Randomness: imagine, he writes there, 10,000 money managers whose annual results are decided by a coin flip. If the flips are conducted for five years it could be expected, simply out “of pure luck,” that 313 of those managers would have “winning” records—that is, for every year for five years running, those 300-odd managers would have won their coin flip. One can only imagine how they might feel about themselves; one suspects that at least a few of them would write books describing their “successful methods” for “beating Wall Street.” (And perhaps one or two of those books would themselves be successful, increasing the self-esteem of those people even more.) In other words, imagine Donald Trump.

It’s the notion of survivorship bias that is the very basis for science—the thought that maybe the eye of newt wasn’t what made little Timmy well, but instead that he happened to get well on his own. And it’s also something that, according to Glenn Greenwald, Hillary Clinton’s supporters in the U.S. media simply don’t understand—which is how we have gotten the narrative known by the name “Bernie Bros.” Greenwald explained the point recently in a piece for The Intercept, the magazine he started after being one of the first journalists to meet Edward Snowden, the former federal employee who blew the whistle on the National Security Agency’s spying on Americans.

What Greenwald calls the “‘Bernie Bros’ narrative” has, he says, two components: the first the conviction that Hillary Clinton has not received universal acclaim because of sexism, and the second that “Sanders supporters are uniquely abusive and misogynistic in their online behavior.” The goal of this game, Greenwald goes on to say, is to “delegitimize all critics of Hillary Clinton by accusing them of … sexism, thus distracting attention away from Clinton’s policy views, funding, and political history.” Greenwald’s insight is that, while many in the mainstream media have taken the idea seriously (or at least claimed to), in fact being subjected to “a torrent of intense anger and vile abuse” is simply a function of being on the Internet. “There are,” as Greenwald points out, “literally no polarizing views one can advocate online … that will not subject” a person to such screeds. In other words, pro-Clinton journalists are attracting hateful messages from supposed Sanders supporters because they are on the Internet, not because Sanders’ supporters are somehow less polite than partisans of other candidates: “If you spend your time praising Clinton and/or criticizing Sanders,” Greenwald observes, “of course you personally will experience more anger and vitriol from Sanders supporters than Clinton supporters.” As Greenwald points out, Sanders’ women supporters—and boy, there seem to be a lot of them—also have unpleasant experiences online. But because—surprise surprise—Hillary is the “establishment” candidate, very few of them have the pulpit of the national media from which to parade their hurt feelings.

What the whole episode I think demonstrates—though Greenwald does not draw this out—is precisely what this primary season is about: it conclusively demonstrates that Clinton’s version of the Democratic Party has very little interest in considering the role of chance in how our lives turn out. That’s a pretty stunning renunciation for a party that once denounced a Republican candidate (as Jim Hightower said about George H. W. Bush during the 1988 Democratic Convention) for being “born on third base and think[ing] he hit a triple.” Survivorship bias, in other words, has been the intellectual link between the Democratic Party’s reliance on science and its interest in society’s less fortunates: it’s not only what makes the Democratic Party the party whose members are far more concerned about the welfare of their fellow citizens, but also far more likely to believe the word of climate change scientists. To either misunderstand—or worse, deliberately misunderstand—the concept of survivorship bias is a far stronger argument against a Clinton presidency than virtually any listing of the campaign contributions she has accepted from various dubious sources. Which is something, because Clinton’s financial dealings with such charming fellows as the gentlemen at Goldman Sachs and the sheiks of Saudi Arabia are pretty alarming—and alarmingly plentiful.

Yet, maybe it’s a sign of hope that the American electorate is rejecting Hillary Clinton because for all Hillary Clinton claims to be a “survivor,” she doesn’t really understand what it means.

Advertisements

This Pitiless Storm

Poor naked wretches, whereso’er you are,
That bide the pelting of this pitiless storm,
How shall your houseless heads and unfed sides,
Your loop’d and window’d raggedness, defend you,
From seasons such as these?
The Tragedy of King Lear Act III, Scene 4

“Whenever people talk to me about the weather,” the Irish writer Oscar Wilde once remarked, “I always feel quite certain that they mean something else.” As it happens, the weather at this year’s British Open has been delayed by high winds and will not be finished with the regulation 72 holes until Monday at the earliest. Which raises a question: why does the Open need to finish all 72 holes? The answer concerns something called a “Simpson’s Paradox”—an answer that also demonstrates just how talk about the weather at the British Open is in fact talk about something else. Namely, the 2016 American presidential election.

To see how, it’s first necessary to see the difference between the British Open and other professional golf tournaments, which are perfectly fine with shortening themselves. Take for instance the 2005 Northern Trust Open in Los Angeles: Adam Scott won in a playoff against Chad Campbell after the tournament was shortened to 36 holes due to weather. In 2013, the Tournament of Champions at Kapalua in Hawaii was “first cut to 54 holes because of unplayable conditions over the first two days,” according to Reuters, and was under threat of “being further trimmed to 36 holes.” The same story also quoted tour officials as saying “the eventual champion would wind up with an ‘unofficial win’” were the tournament to be shortened to 36 holes. (As things shook out they did end up completing 54 holes, and so Dustin Johnson’s win officially counted.) In a standard PGA tournament then, the “magic number” for an “official” tournament is 54 holes. But if so, then why does the Open need 72?

To answer that, let’s take a closer look at the standard professional golf tournament. Most such tournaments are conducted according to what the Rules of Golf calls “stroke play”: four rounds of golf, or 72 holes, at the end of which the players who have made it that far add up their scores—their number of strokes. The player with the lowest score, it may seem like it goes without saying, wins. But it does need to be said—because that isn’t the only option.

Many amateur tournaments after all, such as the United States Amateur, use the rules format known as “match play.” Under this format, the winner of the contest is not necessarily the player who shoots the lowest overall score, as in stroke play. Instead, as John Van der Borght has put the matter on the website of the United States Golf Association, in match play the “winner is the player who wins the most holes.” It’s a seemingly minor difference—but in fact it creates such a difference that match play is virtually a different sport than stroke play.

Consider, for instance, the Accenture Match Play tournament—the only tournament on the PGA Tour to be held under match play rules. The 2014 edition (held at the Dove Mountain course near Tucson, Arizona), had some results that demonstrate just how different match play is than stroke play, as Doug Ferguson of the Associated Press observed. “Pablo Larrazabal shot a 68 and was on his way back to Spain,” Ferguson noted about the first day’s results, while “Ernie Els shot 75 and has a tee time at Dove Mountain on Thursday.” In other words, Larrazabal lost his match and Els won his, even though Larrazabal was arguably the better player at this tournament—at least, if you consider the “better player” to be the one who puts his ball in the hole most efficiently.

Such a result might seem unfair—but why? It could be argued that while shooting a lower number might be what stroke play golf is, that isn’t what match play golf is. In other words, Larrazabal obviously wasn’t better at whatever it was that this tournament measured: if Larrazabal couldn’t beat his opponent, while Els could, then clearly Els deserved to continue to play while Larrazabal did not. While you might feel that, somehow or other, Larrazabal got jobbed, that’s merely a sentimental reaction to what ought to be a hardhearted calculation: maybe it’s true that under stroke play rules Larrazabal would have won, but that wasn’t the rules of the contest at Dove Mountain. In other words, you could say that golfing ability was, in a sense, socially constructed: what matters isn’t some “ahistorical” ability to golf, but instead how it is measured.

Here’s the $64,000 question a guy named Bill James might ask in response to such an argument, however (couched in terms of baseball players): “If you were trying to win a pennant, how badly would you want this guy?” In other words, based on the evidence presented, what would you conclude about the respective golf ability of Els and Larrazabal? Wouldn’t you conclude that Larrazabal is better at the task of putting his ball in the hole, and that the various rule systems that could be constructed around that task are merely different ways of measuring that ability—an ability that pre-existed those systems of measurement?

“We’re not trying to embarrass the best players in the game,” said Sandy Tatum at the 1974 U.S. Open, the so-called Massacre at Winged Foot: “We’re trying to identify them.” Scoring systems in short should be aimed at revealing, not concealing, ability. I choose Bill James to make the point not just because the question he asks is so pithy, but because he invented an equation that is designed to discover underlying ability: an equation called the Pythagorean Expectation. That equation, in turn, demonstrates just why it is so that match play and stroke play are not just different—yet equally valid—measures of playing ability. In so doing, James also demonstrates just why it is that the Open Championship requires that all 72 holes be played.

So named because it resembles so closely that formula, fundamental to mathematics, called the Pythagorean Theorem, what the Pythagorean Expectation says is that the ratio of a team’s (or player’s) points scored to that team’s (or player’s) points allowed is a better predictor of future success than the team’s (or player’s) ratio of wins to losses. (James used “runs” because he was dealing with baseball.) More or less it works: as Graham MacAree puts it on the website FanGraphs, using James’ formula makes it “relatively easy to predict a team’s win-loss record”—even in sports other than baseball. Yet why is this so—how can a single formula predict future success at any sport? It might be thought, after all, that different sports exercise different muscles, or use different strategies: how can one formula describe underlying value in many different venues—and thus, incidentally, demonstrate that ability can be differentiated from the tools we use to measure it?

The answer to these questions is that adding up the total points scored, rather than the total games won, gives us a better notion of the relative value of a player or a team because it avoids something called the “Simpson’s Paradox”—which is what happens when, according to Wikipedia, it “appears that two sets of data separately support a certain hypothesis, but, when considered together, they support the opposite hypothesis.” Consider what happens for example when we match Ernie Els’ 75 to Pablo Larrazabal’s 68: if we match them according to who won each hole, Els comes out the winner—but if we just compared raw scores, then Larrazabal would. Simpson’s Paradoxes appear, in short, when we draw the boundaries around the raw data differently: the same score looks different depending on what lens is used to view it—an answer that might seem to validate those who think that underlying ability doesn’t exist, but only the means used to measure it. But what Simpson’s Paradox shows isn’t that all boundaries around the data are equal—in fact, it shows just the opposite.

What Simpson’s Paradox shows, in other words, is that drawing boundaries around the data can produce illusions of value if that drawing isn’t done carefully—and most specifically, if the boundaries don’t capture all of the data. That’s why the response golf fans might have to the assertion that Pablo Larrazabal is better than Ernie Els proves, rather than invalidates, the argument so far: people highly familiar with golf might respond, “well, you haven’t considered the total picture—Els, for instance, has won two U.S. Opens, widely considered to be the hardest tournament in the world, and Larrazabal hasn’t won any.” But then consider that what you have done just demonstrates the point made by Simpson’s Paradox: in order to say that Els is better, you have opened up the data set; you have redrawn the boundaries of the data in order to include more information. So what you would have conceded, were you to object to the characterization of Larrazabal as a better golfer than Els on the grounds that Els has a better overall record than Larrazabal, is that the way to determine the better golfer is to cast the net as wide as possible. You have demanded that the sample size be increased.

That then is why a tournament contested over only 36 holes isn’t considered an “official” PGA tournament, while 54 holes isn’t enough to crown the winner of a major tournament like the Open Championship (which is what the British Open is called when it’s at home). It’s all right if a run-of-the-mill tournament be cut to 54 holes, or even 36 (though in that case we don’t want the win to be official). But in the case of a major championship, we want there to be no misunderstandings, no “fluky” situations like the one in which Els wins and Larrazabal doesn’t. The way to do that, we understand, is to maximize chances, to make the data set as wide as possible: in sum, to make a large sample size. We all, I think, understand this intuitively: it’s why baseball has a World Series rather than a World Championship Game. So that is why, in a major championship, it doesn’t matter how long it takes—all the players qualified are going to play all 72 holes.

Here I will, as they say in both golf and baseball, turn for home. What all of this about Simpson’s Paradoxes means, at the end of the day, is that a tournament like the Open Championship is important—as opposed to, say, an American presidential election. In a presidential election as everyone knows, what matters isn’t the total numbers of votes a candidate wins, but how many states. In that sense, American presidential elections are conducted according to what, in golf, would be considered match play instead of stroke play. Now, as Bill James might acknowledge, that begs the question: does that process result in better candidates being elected?

As James might ask in response: would you like to bet?