Uh oh.  No family reunion this spring—but still, Uncle Ron managed to get you on the phone for a chat.  It started pleasantly enough, but now the topic has moved to politics.

He says he’s worried that Biden isn’t actually running things as president, and that instead more radical folks are running the show.  He points to some video anomalies (“hand floating over mic”) that led to the theory that many apparent videos of Biden are fake.

You point out that the basis for that theory has been debunked—the relevant anomaly was due to a video compression error, and several other videos taken from the same angle show the same scene.

He reluctantly seems to accept this. The conversation moves on.

But a few days later, you see him posting on social media about how that same video of Biden may have been fake!

Sigh. Didn’t you just correct that mistake? Why do people cling on to their beliefs even after they’ve been debunked?  This is the problem; another instance of people’s irrationality leading to our polarized politics—right?

​Well, it is a problem. But it may be all the thornier because it’s often rational.

Real-world political cases are messy, so let’s turn to the lab.  The phenomenon is what’s known as belief perseverance: after being provided with evidence to induce a particular belief, and then later being told that the evidence was actually bunk, people tend to still maintain the induced belief to some extent (Ross et al. 1975; McFarland et al. 2007).

​A classic example: the debriefing paradigm.  Under the cover story of testing subjects’ powers of social perception, researchers gave them 15 pairs of personal letters, and asked them to identify which was written by a real person and which was written by the experimenters.

Afterwards, they received feedback on how well they did. Unbeknownst to the subjects, that feedback was bogus—they were (at random) told that they got either 14 of 15 notes correct (positive feedback), or 4 of 15 correct (negative feedback). Both groups were told the average person got 9 of 15 correct.

As a result of this feedback, they come to believe that they are above (or below) average on this sort of task.

Some time later, subjects are “debriefed”: told that the feedback that they received was bunk, and had nothing to do with their actual performance. They’re then asked to estimate how well they would do on the task if they were to perform it on a new set of letters.

The belief perseverance effect is that, on average, those who received the positive feedback expected to do better on a new set of letters than those who received the negative feedback—despite the fact that they had both been told that the feedback they received was bunk!

So even after the evidence that induced their belief had been debunked, they still maintained their belief to some degree.

And rationally so, I say.

Imagine that Baya the Bayesian is doing this experiment. She’s considering how well she’d Perform on a future version of the test—trying to predict the value of a variable, P, that ranges from 1–15 (the number she’d get correct).

She doesn’t know how she’d perform, but she can form an estimate of this quantity, E[P]: a weighted average of the various values P might take, with weighs determined by how likely she thinks they are to obtain.

Suppose she starts out with no idea—she’s equally confident that she’d get any score between 1–15:

Then her estimate for P is just the straight average of these numbers:

E[P]  =  1/15*1 + 1/15*2 + … + 1/15*15  =  8

Suppose now she’s given positive feedback (“You got 14 of 15 correct!”)—label this information “+F”.  Given this feedback, she can form a new estimate, E[P|+F] of how well she’d do on a future test—and since her feedback was positive, that new estimate should be higher than her original estimate of 8.

​Perhaps, given the feedback, she’s now relatively confident she’d get a good score on the test—her probabilities look like this:

Then her estimate for her performance is roughly 13:

​E[P | +F]  ≈  0.25 *15 + 0.2*14 + 0.15*13 + …    =  13

Now in the final stage, she’s told that the positive feedback she received was bunk.   So the total information she received is “positive feedback (+F), but told that it was bunk”. What should her final estimate of her future performance, E[P | +F, told bunk], be?

Well, how much does she believe the experimenters?  After all, they’ve already admitted they’ve lied to her at least once; so how likely does the fact that they told her the feedback was bunk make it that the feedback in fact was bunk?  Something less than 100%, presumably.

Now, what she was told about whether the feedback was bunk is relevant to her estimate of her future performance only because it is evidence about whether the feedback was actually bunk. Thus information about whether the feedback was bunk should “screen off” what the experimenters told her.  (If a hyper-reliable source told her “The feedback was legit—they were lying to you when they told you it was bunk”—she should go back to estimating that her performance would be quite good, since the legitimate feedback was positive.)

Upshot: so long as Baya doesn’t completely trust the psychologist’s claim that the information was bunk—as she shouldn’t, since they’re in the business of deception—then she should still give some credence to the possibility that the feedback wasn’t bunk.

Crucial point: if she’s unsure whether the feedback she received was bunk, her final estimate for her future performance should be an average of her two previous estimates.

​Conditional on the feedback actually being bunk, her estimate should revert to it’s initial value of 8:

E[P | +F, told bunk, is bunk] = 8.

Meanwhile, conditional on the feedback actually being not bunk, her estimate should jump up to it’s previous high value of 13:

E[P | +F, told bunk, not bunk] = 13.

When she doesn’t know whether the feedback is bunk or not, she should think to herself, “Maybe it is bunk, in which case I’d expect to get only a 8 on a future test; but maybe it’s not bunk, in which case I’d expect to get a 13 on a future test.”  Thus (by what’s known as the law of total expectation) her overall estimate should be an average of these two possibilities, with weights determined by how likely she thinks they are to obtain.

So, for example, suppose she thinks it’s 80% likely that the psychologists are telling her the truth when they told her the feedback was bunk.  Then her final estimate should be:

E[P | +F, told bunk]  =  0.8*8 + 0.2*13  =  9.

This value is greater, of course, than her original estimate of 8. (The reasoning generalizes; see the Appendix below.)

By exactly parallel reasoning, if Baya were instead given negative feedback (–F) at the beginning, she would instead end up with an estimate slightly lower than her initial value of 8; say, E[P | –F, told bunk]  =  7.

As a result, there should be a difference between the two conditions.  Even after being told the feedback was bunk, both groups should wonder whether it in fact was.  Because of these doubts, those who received positive feedback should adjust their estimates slightly up; those who received negative feedback should adjust them slightly down. Thus, even if our subjects are rational Bayesians, there should be a difference between the groups who received positive vs. negative feedback, even after being told it was bunk:

E[P | +F , told bunk]    >   E[P]   >   E[P | –F, told bunk].

The belief perseverance effect should be expected of rational people.

A prediction of this story is that insofar as the subjects are inclined to trust the experimenters, their estimates of their future performance should be far less affected by the feedback after being told it was bunk than before so.  (The more they trust them, the more weight their initial estimate plays in the final average.)

This is exactly what the studies find. For subjects who are never told that the feedback was bunk, those who received negative feedback estimated their future performance to be 4.46, while those who received positive feedback estimated it to be 13.18.

In contrast, for subjects who were told the feedback was bunk, those who received negative feedback estimated their future performance at 7.96, while those who received positive feedback estimated it at 9.33.

There is a statistically significant difference between these two latter values—that’s the belief perseverance effect—but it is much smaller than the initial divergence.  As the rational story predicts.

Another prediction of this rational story is that insofar as psychologists can get subjects to fully believe that the feedback really was bunk, the belief perseverance effect should disappear.

Again, this is what they find.  Some subjects where given a much more substantial debriefing—explaining that the task itself is not a legitimate measure of anything (no one can reliably identify the real letters).  Such subjects exhibited no belief perseverance at all.

Upshot: in the lab, the belief perseverance effect could well be fully rational.

Okay. But what about Uncle Ron?

Well, obviously real-world cases like this are much more complicated. But it does share some structure with the above lab example.

Ron originally had some some (perhaps quite low) degree of belief that something fishy was going on with Biden. He then saw a video which boosted that level of confidence. Finally, he was then told that the video was bunk.

So long as he doesn’t complete trust the source that debunks the video, it makes sense for him to remain slightly more suspicious than he was originally.  How suspicious, of course, depends on how much he ought to believe that the video really was bunk.

But even if he trusts the debunker quite a bit, a small bump in suspicion will remain. And if he sees a lot of bits of evidence like this, then even if he’s pretty confident that each one is bunk, his suspicions might reasonably start to accumulate.   

If that's right, the fall into conspiracy theories is an epistemic form of death-by-a-thousand-cuts. The tragedy is that rationality may not guard against it.


What next?
If you​ liked this post, consider sharing it on social media or signing up for the newsletter.
For more on belief perseverance, check out the recent controversy over the "backfire effect"—the result that sometimes people double-down on their beliefs in the face of corrections. See e.g. Nyhan and Reifler 2010 for the initial effect, and then Wood and Porter 2019 for a replication attempt that argues that the effect is very rare.
For more discussion, see the reddit thread on this post.

Appendix

Consider the Baya case. Why does the reasoning go through generally?

P is a random variable—a function from possibilities to numbers—that measures how well she would do on a test if she were to retake it.  Given her (probabilistic) credence function C, her estimate for P is

By parallel reasoning, if she gets negative feedback she should end up with a lower estimate than her initial one:

The most important factor that drove Becca and me apart, politically, is that we went our separate ways, socially. I went to a liberal university in a city; she went to a conservative college in the country. I made mostly-liberal friends and listened to mostly-liberal professors; she made mostly-conservative friends and listened to mostly-conservative ones.

​As we did so, our opinions underwent a large-scale version of the group polarization effect: the tendency for group discussions amongst like-minded individuals to lead to opinions that are more homogenous and more extreme in the same direction as their initial inclination.

The predominant force that drives the group polarization effect is simple: discussion amongst like-minded individuals involves sharing like-minded arguments. (For more on the effect of social influence, see this post.)

Spending time with liberals, I was exposed to a predominance of liberal arguments; as a result, I become more liberal.  Vice versa for Becca.

Stated that way, this effect can seem commonsensical: of course it’s reasonable for people in such groups to polarize. For example: I see more arguments in favor of gun control; Becca sees more arguments against them. So there’s nothing puzzling about the fact that, years later, we have wound up with radically different opinions about gun control. Right?

Not so fast.

In 2010, Becca and I each knew where we were heading. I knew that I’d be exposed mainly to arguments in favor of gun control, and that she’d be exposed mainly to arguments against it.  At that point, we both had fairly non-committal views about gun rights—meaning that I didn’t expect the liberal arguments I’d witness to be more probative than the conservative arguments she’d witness.

This gives us the puzzle. At a certain level of generality, I know nothing about about gun rights now that I didn’t in 2010.  Back then, I knew I’d see arguments in favor, but I was not yet persuaded. Now in 2020 I have seen those arguments, and I am persuaded.

Why the difference? How could receiving the arguments in favor of gun control have a (predictably) different effect on my opinion than predicting that I’d receive such arguments? And given that I could’ve easily gone to a more conservative college and wound up with Becca’s opinions, doesn’t this mean that my current opinion that we need gun control was determined by arbitrary factors. In light of this, how can I maintain my firm belief?

As we’ve seen, ambiguous evidence can in principle offer an answer to these questions. Here I’ll explain how this answer can apply to the group polarization effect.

Begin with a simple question: why do arguments persuade? You might think that’s a silly question—arguments in favor of gun control provide evidence for the value of gun control, and people respond to evidence; so rational people are predictably convinced by arguments. Right?

Wrong. Arguments in favor of gun control don’t necessarily provide evidence for gun control—it depends on how good the argument is!

When someone presents you with an argument for gun control, the total evidence you get is more than just the facts the argument presents; you also get evidence that the person was trying to convince you, and so that they were appealing to the most convincing facts they could think of.

If the facts were more convincing than you expected—say, “Guns are the second-leading cause of death in children”—then you get evidence favoring gun control. But if the facts were less convincing than you expected—say, “Many people think we should ban assault weapons”—then the fact that this was the argument they came up with actually provides evidence against gun control. (H/t Kevin Zollman on October surprises.)

This is a special case of the fact that, when evidence is unambiguous, there’s no way to investigate an issue that you can expect to make your more confident in your opinion.

Why, then, do arguments for a claim predictably shift opinions about it?

My proposal: by generating asymmetries in ambiguity. They make it so that reasons favoring the claim are less ambiguous—and so easier to recognize—than those telling against it.

Here’s an example—about Trump, not gun control.

This is an argument that we should re-elect Trump. Is this a good argument—that is, does it provide evidence in favor of Trump being re-elected?

​I’m not sure—the evidence is ambiguous. What I am sure of is that if it is a good argument, then it’s so for relatively straightforward reasons: “Trump’s presidency has had and will have good long-term effects.”

If it’s not a good argument, then it’s for relatively subtle reasons: perhaps we should think, “Is that the best they can come up with?”; perhaps we should think that relations with China were already changing before Trump; perhaps we should be unsure whether inflaming the confrontation is a good thing; etc.

Regardless: if it’s a good argument, it’s easier to recognize as such than if it’s a bad argument As a result, we can expect to be, on average, somewhat persuaded by arguments like this.

As before, it’s entirely possible for this to be so and yet for the argument to satisfy the value of evidence—and, therefore, for us to expect that it’ll make us more accurate to listen to the argument rather than ignore it.  At each instant we're given the option to listen to a new argument or not, we should take it if we want to make our beliefs accurate.

And yet, because each argument is predictably (somewhat) persuasive, long periods of exposure to pro (con) arguments can lead to predictable, profound, but rational shifts in opinions.

We can model how this worked with me and Becca.  The blue group (me and my liberal friends) were exposed to arguments favoring gun control—that is, arguments that were less ambiguous when they supported gun control than when they told against it. The red group (Becca and her conservative friends) were exposed to arguments disfavoring gun control—that is, arguments that were more ambiguous when they supported gun control than when they told against it.

Suppose that, as a matter of fact, 50% of the arguments point in each direction. Because of the asymmetries in our ability to recognize which arguments are good and which are bad, the result is polarization:

Notably, although such ambiguity asymmetries are a force for divergence in opinion, that force can be overcome.  In particular, as the proportion of actually-good arguments gets further from 50%, convergence is possible even in the presence of such ambiguity-asymmetries.  Here’s what happens when 80% of the arguments provide evidence for gun control:

Thus polarization due to differing arguments is not inevitable—but the more conflicted and ambiguous the evidence is, the more likely it is.

How plausible is this as a model of the group polarization effect? Obviously there’s much more to say, but there is indeed some evidence that group-polarization effects—and argumentative persuasion in particular—are driven by ambiguous evidence.

First, the group discussions that induce polarization also tend to reduce the variance in people's opinions, and the amount of group shift in opinion is correlated with the initial variance of opinion. 

These effects make sense if ambiguous arguments are driving the process. For: (1) the more ambiguous evidence is, the more variance of opinion we can expect; (2) if what people are doing is coordinating on what to think about various pieces of evidence, we would expect discussion to reduce ambiguity and hence reduce opinion-variance; and (3) the more initial variance there is, the more diverse the initial bodies of evidence were—so the more we should discussion to reveal new arguments, and hence to lead participants to shift their opinions.

Second, participation in the discussion (as opposed to mere observation) heightens the polarization effect, as can merely thinking about an issue—especially if people are trying to think of reasons for or against their position or expecting to have a debate with someone about it.  

If what people are doing in such circumstances is formulating arguments (perhaps through a mechanism of cognitive-search), then these are exactly the effects that this sort of model of asymmetrically-ambiguous arguments would predict—for they are in effect exposing themselves to a series of asymmetrically-ambiguous arguments.

Upshot: it is not irrational to be predictably persuaded by (ambiguous) arguments—and the more people separate into like-minded groups, the more this leads to polarization.


What next?
If you liked this post, consider subscribing to the newsletter or spreading the word.
For the details of the models of asymmetrically-ambiguous arguments and the simulations that used them, check out the technical appendix (§7).
Next Post: we'll return to confirmation bias and see how selective search for new information is rational in the context of ambiguous evidence.

​It’s September 21, 2020. Justice Ruth Bader Ginsburg has just died.  Republicans are moving to fill her seat; Democrats are crying foul.

​Fox News publishes an op-ed by Ted Cruz arguing that the Senate has a duty to fill her seat before the election. The New York Times publishes an op-ed on Republicans’ hypocrisy and Democrats’ options.

Becca and I each read both. I—along with my liberal friends—conclude that Republicans are hypocritically and dangerous violating precedent. Becca—along with her conservative friends—concludes that Republicans are doing what needs to be done, and that Democrats are threatening to violate democratic norms (“court packing??”) in response.

In short: we both see the same evidence, but we react in opposite ways—ways that lead each of us to be confident in our opposing beliefs.  In doing so, we exhibit a well-known form of confirmation bias.

And we are rational to do so: we both are doing what we should expect will make our beliefs most accurate.  Here’s why.

​Confirmation bias is the tendency to gather and interpret evidence in a way that can be expected to favor your prior beliefs (Nickerson 1998; Whittlestone 2017). There are two parts two this tendency.

This means that it makes sense to tend to scrutinize evidence for which you expect to be able to recognize its flaws—i.e., often, that which disconfirms your prior beliefs.

In particular, suppose Becca and I started out each expecting 50% of the pieces of evidence for/against replacing RBG to contain flaws, but I am slightly better at finding flaws in the supporting evidence, and she is slightly better at finding flaws in the detracting evidence.

Suppose then we are presented with a series of random pairs of pieces of evidence—one in favor, one against—and at each stage we decide to scrutinize the one that we expect to make us more accurate.  Since accuracy is correlated with whether we expect to find flaws, this means that I will be slightly more likely to scrutinize the supporting evidence, and she will be slightly more likely to scrutinize the detracting evidence.

As a result, we’ll polarize.  Even if, in fact, exactly 50% of the pieces of evidence tell in each direction, I will come to be confident that fewer than 50% of the pieces of evidence support replacing RGB, and she’ll come to be confident that more than 50% of them do:

Upshot: biased assimilation can be rational. People with opposing beliefs who care only about the truth and are presented with the same evidence can be expected to polarize, since the best way to assess that evidence will often be to apply selective scrutiny to the evidence that disconfirms their beliefs.

​There is some empirical support for this type of explanation (though more is needed). Biased assimilation is clearly driven by the process of selective scrutiny (Lord et al. 1979, Taber and Lodge 2006, Kelly 2008). Biased assimilation is more common when the evidence is ambiguous or hard to interpret (Chaiken and Maheswaran 1994, Petty1998).  And the best known “debasing” technique is to explicitly instruct people to “consider the opposite”, i.e. to do cognitive searches that are expected to disconfirm their prior beliefs (Koriat 1980, Lord et al. 1984).

If my explanation is right, this is, in effect, asking people to not let accuracy guide their choice of cognitive searches—and it therefore is no surprise that people do not do this spontaneously.

In fact, it means that we can prevent people from polarizing only by preventing them from trying to be accurate.


What next?
The argument of this post draws heavily on a fantastic paper by Tom Kelly about belief polarization. It’s definitely worth reading, along with Emily McWilliams’s reply.
Jess Whittlestone has a fantastic blog post summarizing her dissertation on confirmation bias—and how she completely changed her mind about the phenomenon.
For more details, as always, check out the technical appendix (§6).
Next post: Why arguments polarize us.

As we learn to live in the ‘new normal’, amidst the easing and tightening of local and national lockdowns, day-to-day decision-making has become fraught with difficulty. Here are some of the questions I’ve been grappling with lately:

• When would be a good time to visit friends and family members?
• Should I head out to the shop/ pub/ restaurant/ hairdresser this week?
• Would it be more sensible to stay home?

A year ago these questions had easier answers. I only needed to check in on my mood, bank balance, or general state of dishevelment. Now it’s far harder to weigh up the costs and benefits of going out and about. There’s so much more hanging on each decision. Is it worth taking the risk of picking up – or passing on – the virus? Of course, we will all settle on our own ways of balancing these concerns. But, in this post, I’m going to look at how our attitudes to risk depend on framing.

Imagine you are offered the following choice:

Option (B) involves risk – if you choose it, you can’t be sure of the outcome. Hence, this is an example of a ‘risky-choice task’. Have a think about which option you’d prefer (and write it down if you’re worried about hindsight bias).
Now consider the following choice:

Which option would you prefer this time?

Typically, people prefer option (A) in the first task and option (D) in the second.

Why is this puzzling? Well, first notice that options A and B in the first task have the same ‘expected value’ as each other. We can work out the expected value of gaining $\$$10 with a probability of 1 as follows:

• $\$$10 x 1 = $\$$10

Meanwhile, the expected value gaining $\$$20 with a probability of 0.5 and gaining $\$$0 with a probability of 0.5 can be worked out like this:

• ($\$$20 x 0.5 = $\$$10) + ($\$$0 x 0.5 = $\$$0) = $\$$10

So, each option is worth 10 dollars.

Why do people tend to prefer option A then? Presumably, because they prefer to have a sure thing than risk walking away with nothing. In other words, people are risk-averse.

But now let’s look at the second task. Again, options C and D have the same expected value. The expected value of losing $\$$10 with a probability of 1 can be worked out as follows:

• $-\$$10 x 1 = $-\$$10

And this is equal to losing $\$$20 with a probability of 0.5:

• ($-\$$20 x 0.5 = $-\$$10) + ($\$$0 x 0.5 = $\$$0) = $-\$$10

In fact, the only difference between the two choice problems is that the first one deals in positives (gaining 10 dollars) while the second deals in negatives (losing 10 dollars).

But in the second task, people tend to prefer option D. That indicates risk-seeking behaviour: other things equal, they prefer to take the risk of losing 20 dollars over the certainty of losing 10 dollars. So, it looks like people flip from being risk-averse when they are faced with gains (as in the first task), to being risk-seeking when they are faced with losses (as in the second task).

This ‘reflection effect’ was first brought to light by the psychologists Daniel Kahneman and Amos Tversky. They factor it into their ‘prospect theory’ of decision-making under risk, which is designed to model how people really make decisions (not how they should!). You can read their seminal paper on prospect theory here.

The ‘shifty’ nature of our risk attitudes is a fascinating topic in its own right. Why do we prefer sure gains but risky losses? For now, I’m going to put that question to one side because I want to focus on a different one: Can options be made to seem like gains or losses just by framing them in particular ways?

The following scenario is adapted from papers by David Mandel, published in 2001 and 2014 (and it is itself inspired by a classic scenario introduced by Tversky and Kahneman here).

Now take a look at this slightly different version:

In the first task, people tend to prefer the sure option, Plan A. But, in the second task they tend to prefer the risky option, Plan D. Perhaps you did as well.

The pattern is reminiscent of our earlier pair of choice tasks. But there’s an important difference: the gains and losses are only apparent now. Plan A and Plan C are supposed to have exactly the same outcome, namely 200 people being saved and 400 people dying. Plan B and Plan D are also supposed to be equivalent: both involve a one-third probability of everyone being saved (i.e. nobody dying) and a two-thirds probability of nobody being saved (i.e. everyone dying). So, it’s not as though the first version of the task involves actual gains while the second one involves actual losses. Even if it’s true that we’re risk-averse for gains and risk-seeking for losses, that can’t completely explain the responses here. Cases like this are especially puzzling.

Tversky and Kahneman put forward a solution. They point out that in the first version of the choice task, positive language is used: the options are described in terms of the number of people who will ‘be saved’. This, they suggest, makes options A and B sound like gains, even though some people could die.

In contrast, the second version of the choice task uses negative language, talking about the number of people who will ‘die’. This makes options C and D sound like losses, even though some people could be saved.

Once we’ve interpreted the options as gains or losses, prospect theory predicts what we’ll do next. On the one hand, since we tend to be risk-averse when we think we’re facing gains, we’ll choose Plan A in the first version of the task. On the other hand, since we tend to be risk-seeking when we think we’re facing losses, we’ll choose Plan D in the second version. And, as we saw, that’s precisely the pattern the psychologists find.

As an aside, I wonder whether the British and Irish governments had this effect in mind early on in the COVID-19 outbreak. In a tweet from 13th March, Billy Bragg wryly comments on their contrasting communication strategies. Whereas Johnson was using a negative frame, warning that many people would die, Varadkar chose a positive frame, claiming that many could be saved. If Tversky and Kahneman are right, Varadkar’s positive framing could have encouraged a relatively cautious response, which would be consistent with Ireland’s swift lockdown. Meanwhile, Johnson could have been encouraging Brits to take a risker approach (herd immunity…?).

This particular case aside, prospect theory has been extremely influential in academia, industry and popular culture. And ‘risky-choice framing effects’ are commonly thought to involve a double dose of irrationality: first, the superficial differences in language affect how we perceive the options facing us (although see my last blog post for a rationalising explanation of the effects of positive and negative frames); second, perceiving options as gains or losses affects which one we prefer.

But Tversky and Kahneman’s account isn’t universally accepted. And one way of challenging it is by questioning the equivalence of the options in each version of a risky-choice task. So, in the above example, is 200 people being saved really the same as 400 people dying? Not necessarily…

This study investigates a different explanation of risky-choice framing effects. The author, David Mandel, suggests that many people interpret number terms like ‘200’ and ‘400’ as having ‘at least’ meanings. In fact, this possibility is well-recognised by linguists and philosophers of language. For instance, when we are instructed to keep two metres apart, this is clearly a minimum social distancing measure – even better if it’s three or four metres.

So, saying that ‘200 people will be saved’ may be interpreted as meaning that at least 200 people will be saved. That leaves open the (good!) possibility that more than 200 people could be saved under Plan A (i.e. less than 400 would die).

On the flipside, saying that ‘400 people will die’ may be interpreted as meaning that at least 400 people will die. That leaves open the (bad!) possibility that more than 400 could die (i.e. less than 200 would be saved). So, strictly speaking, Plan A is a better prospect than Plan C. Perhaps that could be enough to explain why people prefer Plan A in the first version of the problem and Plan D in the second version. Then we needn’t conclude that they have inconsistent attitudes to risk.

Mandel conducted a series of experiments. In summary:

• When he added ‘exactly’ before the number expression (as in ‘exactly 200 people will be saved’, and ‘exactly 400 people will die’), there was no statistically significant framing effect. In other words, people no longer reliably switched between the sure option and the risky option across the two versions of the problem.
• When he added ‘at least’ before the number expression (as in ‘at least 200 people will be saved’, and ‘at least 400 people will die’), there was a particularly large framing effect. So, by making the ‘at least’ explicit, the Plan A/ Plan D response pattern was exacerbated.
• Finally, when he asked people how they interpreted the (unmodified) number terms ‘200’ and ‘400’, only the people who spontaneously adopted ‘at least’ readings were subject to framing effects.

These are really striking results. It looks like risky-choice framing effects could be all down to how people interpret number terms. So…case closed? Well, not quite. Two attempts to confirm the results of one of Mandel’s experiments failed to eliminate framing effects (see here and here, with Mandel’s reply to the first of these here). In both cases, risky-choice framing effects arose even when number terms like ‘200’ and ‘400’ were being understood exactly.

​It seems unlikely, then, that risky-choice framing effects are entirely explained by ‘at least’ interpretations of number terms (and this is a point which Mandel himself is careful to make – in separate work, like this paper written with Michael Tombu, he has developed another proposal which could explain the remainder of the effect). Still, I think these ‘lower-bounded’ interpretations of number terms are an important contributing factor (and I argue in this draft paper that Mandel’s critics should think so too). That’s worth noting because it could allow us to rationalise at least some of people’s risky-choice behaviour.

How does this relate to communications and decision-making during the global pandemic? One useful takeaway is that numbers can be understood in different ways – as ‘exact’ or ‘at least’ (and sometimes ‘at most’, as when locked down Brits were allowed to take ‘one’ outdoor excursion a day). So, when we hear politicians and scientists predicting COVID-19 case numbers or death tolls, it’s worth reflecting on whether these are these are supposed to be point estimates, bare minimums, or upper limits.

In practice, the high degree of uncertainty in the current climate often makes it hard to put numbers on outcomes at all. And that may make the ‘lower-bounding hypothesis’ described above less relevant to our ordinary day-to-day risk judgements. (However, government decision-makers are far more likely to be presented with quantified outcomes and probabilities, so the findings may be more applicable at that level). Some important questions we are left with, then, are:

• What other factors contribute to the risky-choice framing effects we are subject to on a day-to-day basis?
• Are Kahneman and Tversky right to focus on our perception of options as gains or losses?
• And is it rational to be sensitive to such factors?

To sum up, I think the jury is still out on whether risky-choice framing effects can be entirely rationalised. Still, we shouldn’t be too quick to conclude that they can’t. And, in the meantime, we can try to notice when framing is affecting us – perhaps even using that knowledge to our advantage. Recently, I’ve found it particularly useful to reframe my choices, to see how that affects my attitudes and preferences. For instance, it may be true that staying home may offer predictable monotony, compared with the chance of greater enjoyment, but it also offers safety over the risk of infection. Perhaps trying out both perspectives can help challenge our own default ways of thinking – and our understanding of each other’s.

​
What next?
If you’d like to find out more about the different theories of framing effects, this survey chapter by Karl Halvor Teigen is a good place to start.
For more on the philosophical interpretation of the lower-bounding hypothesis, check out Sarah's forthcoming paper on the topic.

So far, I’ve laid the foundations for a story of rational polarization. I’ve argued that we have reason to explain polarization through rational mechanisms; showed that ambiguous evidence is necessary to do so; and described an experiment illustrating this possibility.

Today, I’ll conclude the core theoretical argument. I'll give an ambiguous-evidence model of our experiment that both (1) explains the predictable polarization it induces, and (2) shows that such polarization can in principle be profound (both sides end up disagreeing massively) and persistent (neither side is changes their opinion when they discover that they disagree).

With this final piece of the theory in place, we’ll be able to apply it to the empirical mechanisms that drive polarization, and see how the polarizing effects of persuasion, confirmation bias, motivated reasoning, and so on, can all be rationalized by ambiguous evidence.

Recall our polarization experiment. 

I flipped a coin, and then showed you a word-completion task: a series of letters and blanks that may or may not be completable by an English word.  For example, FO_E_T is completable (hint: where are there lots of trees?); but _AL_W is not (alas, Bernhard Salow is not yet sufficiently famous).
​
One group—the Headsers—saw a completable string if the coin landed heads; the other group—the Tailsers—saw a completable string if the coin landed tails. As they did this for more and more tasks, their average confidence that the coins landed heads diverged more and more:

Question: what drives this polarization—and why think that it is rational?

Our word-completion task provides an instance of what is sometimes called a “good-case/bad-case asymmetry” in epistemology (Williamson 2000, Lasonen-Aarnio 2015, Salow 2018). The asymmetry is that you get better (less ambiguous) evidence in the “good” case than in the “bad” case—and, therefore, it’s easier to recognize that you’re in the good case (when you are) than to recognize that you’re in the bad case (when you are).

In our experiment, the “good case” is when the letter-string is completable; the “bad case” is when it’s not. The crucial fact is that it’s easier to recognize that a string is completable than to recognize that it’s not.  It’s possible to get unambiguous evidence that the letter-string is completable (all you have to do is find a word). But it’s impossible to get unambiguous evidence that it’s not completable.

In particular, what should you think when you don’t find a word?  This is some evidence that the string is not completable—but how much? After all, you can’t rule out the possibility that you should find a word, or that you should at least have an inkling that there’s one. More generally, you should be unsure what to make of this evidence: if there is a word, you have more evidence that there is; if there’s not, you have less; but you can’t be sure of which possibility you’re in.

There are a variety of models we can give of your evidence to capture this idea, all of which satisfy the value of evidence and yet lead to predictable polarization (see the Technical Appendix (§5.1) for some variations).

Here's a simple one:

Either there is a word, or there’s not; and either you find one, or you don’t—but you can’t find a word that’s not there, so there are only 3 types of possibilities (the circles in the diagram).

The numbers inside the circles indicate how confident you should be, beforehand, that you’ll end up in those possibilities: you should be ½ confident that there’ll be no word (and you won’t find one), since that’s determined by a coin flip; and if there is a word, there’s some chance (say, ½) you’ll find one, meaning you should be ½*½ = ¼ confident you’ll end up in each of the Word-and-Find (top right) and Word-and-Don’t-Find (bottom right) possibilities.

Meanwhile, the labeled arrows from possibilities represent how confident you should be after you see the task, if in fact you’re in that possibility. 

If there's a word and you find one, you should be sure of that—hence the arrow labeled “1” pointing from the top-right possibility to itself. If there’s no word and you don’t find one, you should be somewhat confident of that (say ⅔ probability), but you should leave open that there’s a word that you didn’t find (say, ⅓ probability).  But if there is a word and you don’t find one, you should be more confident than that—after all, since there is a word, you’ve received more evidence that there is, even if that evidence is hard to recognize. Any higher number will do, but in this model you should be ⅔ confident there’s a word if there is one and you don't find one.

If you don’t find a word, your evidence is ambiguous because you should be unsure how confident you should be that there’s a word—maybe you should be ⅓ confident; maybe instead you should be ⅔ confident. (In a realistic model there would be many more possibilities, but this simple one illustrates the structural point.)

There are two important facts about this model: (1) it is predictable polarizing, and yet (2) it satisfies the value of evidence.

Why is the evidence predictably polarizing? You start out ½ confident there’ll be a word. But you prior estimate for how confident you should end up is higher than ½. After all, there’s a ½ chance your confidence should go up—perhaps way up, perhaps only somewhat up. Meanwhile, there’s a ½ chance it should go down—but not very far down.  Thus, on average, you expect seeing word-completion tasks to provide evidence that there’s a word.

(Precisely: your prior expectation of the posterior rational confidence is ½*⅓ + ¼*⅔ + ¼*1 = 7/12, which is greater than ½.)

Notice that if you had unambiguous evidence—so that the rational confidence was the same at all possibilities wherein you don’t find a word—this model would not be predictably polarizing. (Then your prior expectation would be ¾*⅓ + ¼*1 = ½.)

So what drives the predicable polarization is the ambiguity—in particular, the fact that when you don’t find a word, you should be more confident the string is completable if there is a word than if there’s not.

This, incidentally, is empirically confirmed: in my experiment, amongst tasks in which people didn’t find a word (had confidence <100%), the average confidence when there was no word was 44.6%, while the average confidence when there was a word was 52.3%—a statistically significant difference. (Stats: t(309) = 2.77, one-sided p = 0.003, and d=0.32.)

Why is the evidence valuable, despite being polarizing? Note that the rational posterior degrees of confidence are uniformly more accurate than the prior rational confidence: no matter what possibility is actual, you become uniformly more confident of truths and less confident of falsehoods. 

This can be seen by noting that, in each possibility, the probabilities always become more centered on the actual possibility. For example, suppose there’s no word. Then initially you should be ½ confident of this, and afterwards you should be ⅔ confident of it. Conversely, suppose there is a word. Then initially you should be ½ confident of this, but afterwards you should be either ⅔ confident of it (bottom right), or certain of it (top right). And so on.

Because of this, the model satisfies the value of evidence: no matter what decision about the word-completion task you might face, you should prefer to get the evidence before making your decision. (Proof in the Technical Appendix, §5.1.)

How, in principle, could this type of evidence lead to profound and persistent polarization?

First note what happens when we divide people into Headsers and Tailsers: we give them symmetric, mirror-image types of evidence. Headsers see completable strings when the coin lands heads; Tailsers see them when it lands tails. Thus Headsers tend to get less ambiguous evidence when the coin lands heads, while Tailsers tend to get less ambiguous evidence when the coin lands tails:

As a result, Headsers should become (on average) more confident in heads, while Tailsers should become (on average) more confident of tails.

​(Precisely: although both start out ½ confident of heads, on average Headsers should be 1/6 more confident of heads than Tailsers should be.)

Now consider: what happens if we present each group with a large number of independent word-completion tasks.  (For simplicity, imagine they all know that they’re 50% likely to find a word if there is one, so they don’t learn anything new about their abilities as they proceed.)

Each time they’re presented with a word-completion task, they face a question: “On this toss, will I find a word, and will the coin land heads or tails?”  Since the coin tosses are each fair and independent, the answer to all of these questions are independent: knowing the answers to some of them has no bearing on the others. Moreover, we’ve seen that with respect to each one of these questions, the evidence is valuable.

In fact, more is true.  Let $Q$ be the question "How will each of the coins land?" By iterating this process in the right way, we can make it such that at each stage $i$ of the process, you should expect that the evidence you'll receive about coin $i+1$ is valuable with respect to $Q$.  (This, mind you, is the most subtle philosophical and technical step—see the Technical Appendix, §5.2, for more discussion.)

Thus at each time, if what you care about is getting to the truth about how any of the coins landed, you should gather the evidence.
​
Suppose Headsers and Tailser both do this.  Then it will predictably lead to profound and persistent disagreement.

Why?  By the weak law of large numbers, everyone can predict with confidence that Headsers should wind up very confident that around 7/12 (≈58%) of the coins landed heads, while Tailsers should wind up very confident that around 5/12 (≈42%) of the coins did.

Now consider the claim:

Everyone can predict, at the outset, that Headsers will become very confident (in fact, with enough tosses, arbitrarily confident) that Mostly-Heads is true, and Tailsers will become very confident it’s false.

​​Thus we have profound polarization.

Moreover, even after undergoing this polarization, Headsers will still be very confident that Tailsers will be very confident that Mostly-Heads is false; meanwhile, Tailsers will be very confident that Headsers will be very confident that Mostly-Heads is true. As a result, neither group will be surprised—and thus neither group will be moved—when they discover their disagreement.

Thus we have persistent polarization.

In short: the ambiguity-asymmetries induced by the sort of evidence presented in word-completion tasks can be used to lead rational people to be predictably, profoundly, and persistently polarized. (See the Technical Appendix, §5.2, for the formal argument.)

This completes the theoretical argument of this series: the type of polarization we see in politics—polarization that is predictable, profound, and persistent--could be rational.

The rest of the series will make the case that it is rational. In particular, I’ll argue that this ambiguity-asymmetry mechanism plausibly helps explain the empirical mechanisms that drive polarization: persuasion, confirmation bias, motivated reasoning, etc.

It’s not hard to see, in outline, how the story will go.

For “heads” and “tails” substitute bits of evidence for and against a politically contentious claim—say, that racism is systemic. Recall how Becca and I went our separate ways in 2010—I, to a liberal university; she, to a conservative college.

I, in effect, became a Headser: I was exposed to information in a way that made it easier to recognize evidence in favor of systemic racism. She, in effect, became a Tailser: she was exposed to information in a way that made it easier to recognize evidence against systemic racism.

If that were what happened, then both of us could've predicted that we would end up profoundly polarized—as we did. And neither of us should be moved now when we come back and discover our massive disagreements—as we’re not.

And yet: although we each should think that the other is wrong, we should not think that they are less rational, or smart, or balanced than we ourselves are.

That is the schematic story of how our polarized politics could have resulted from rational causes.

In the remainder of this series, I’ll argue that it has.


What next?
If you liked this post, consider signing up for the newsletter, following me on Twitter, or spreading the word.
For the formal details underlying the argument, see the Technical Appendix (§5).
Next post: How confirmation bias results from rationally avoiding ambiguity.

(​The Technical Appendix contains all formal statements and proofs.)

In particular, consider me back in 2010, thinking about the political attitudes I’d have in 2020. Unambiguous Bayesianism implies that no matter what evidence I might get—no matter that I was going to a liberal university, for instance—I shouldn’t have expected it to be rational for me to become any more liberal than I was then.

Moreover, Fact 1 also implies that if me and Becca shared opinions in 2010, then we couldn't have expected rational forces to lead me to become more liberal than her. 

Why is Fact 1 true—and what does it mean?

Why it’s true: Return to the simple lottery case. Suppose you are only allowed to ask questions which you know I’ll give a clear answer to.  You’re currently 30% confident that you won. Is there anything you can ask me that you expect will make your more confident of this?  No.

You could ask me, “Did I win?”—but although there’s a 30% chance I’ll say ‘Yes’, and your confidence will jump to 100%, there’s a 70% chance I’ll say ‘No’ and it’ll drop to 0%.  Notice that (0.3)(1) + (0.7)(0) = 30%.

You could instead ask me something that’s more likely to give you confirming evidence, such as “Did Beth or I win?”  In that case it’s 50% likely that I’ll say ‘Yes’—but if I do your confidence will only jump to 60% (since there’ll still be a 40% chance that Beth won); and if I say ‘No’, your confidence will drop to 0%. And again, (0.5)(0.6) + (0.5)(0) = 30%.

This is no coincidence. Fact 1 implies that if you can only ask questions with unambiguous answers, there’s no question you can ask that you can expect to make you more confident that you won. And recall: unambiguous Bayesianism assimilates every scenario to one like this.

What it means: Fact 1 implies that if unambiguous Bayesianism is the right theory of epistemic rationality, then the polarization we observe in politics must be irrational.  

After all, a core feature of this polarization is that it is possible to see it coming. When my friend Becca and I went our separate ways in 2010, I expected that her opinions would get more conservative, and mine would get more liberal. Unambiguous Bayesianism implies, therefore, that I must chalk such predictably polarization up to irrationality.

But, as I’ve argued, there’s strong reason to think I can’t chalk it up to irrationality—for if I’m to hold onto my political beliefs now, I can’t think they were formed irrationally.

This—now stated more precisely—is the puzzle of predictable polarization with which I began this series.

This can be given a precise answer in terms of the value of evidence.

What distinguishes rational from irrational transitions in belief? It’s rational to update your beliefs about the lottery by asking me a question about who won.  It’s irrational to update your beliefs hypnotizing yourself to believe you won. Why the difference?

Answer: asking me a question is valuable, in the sense that you can expect it to make your beliefs more accurate, and therefore to improve the quality of your decisions. Conversely, hypnotizing yourself is not valuable in this sense: if right now you’re 30% confident you won, you don’t expect that hypnotizing yourself to become 100% confident will make your opinions more accurate—rather, it’ll just make you certain of something that’s likely false!

This idea can be made formally precise using tools from decision theory. Say that a transition in beliefs is valuable if, no matter what decision you face, you prefer to make the transition before making your decision, rather than simply making your decision now.  (See the Technical Appendix for details.)

To illustrate, focus on our simple lottery case. Suppose you’re offered the following:

Fact 4 is the foundation for the theory of rational polarization that I’m putting forward. It provides a theoretical “possibility proof”, to complement our empirical one: it shows that, when evidence is ambiguous, you can be rational expect it to lead you to the truth, despite expecting it to polarize you.  This is how we are going to solve the puzzle of predictable polarization.

In particular, it turns out that we can string together a series of independent questions and pieces of (ambiguous) evidence with the following features:

• Relative to each question, the evidence you’ll receive is valuable and yet (slightly) predictably polarizing;
• Yet relative to the collection of questions as a whole, the evidence is predictably and profoundly polarizing.

This, I’ll argue, is how predictable, persistent, and profound polarization can be rational—how, back in 2010, Becca and I could predict that we'd come to disagree radically without predicting that either of us would be systematically irrational.  

What next?
The formal details and proofs of Facts 1 – 4 can be found in the Technical Appendix.
​If you liked this post, consider signing up for the newsletter, following me on Twitter, or spreading the word.
Next Post: an argument that the predictable polarization observed in our word-completion experiment was rational, and an explanation of how predictable, profound, persistent polarization can arise rationally from ambiguous evidence.
​​​

Polarization has always been with us. A set of basic psychological and sociological mechanisms explain why human societies have always been characterized by local conformity and global diversity: there tends to be agreement within small social circles, but disagreement between them. As a result, when people go off on different life trajectories, it’s always been normal for their attitudes to drift apart.

What’s changed is that a series of factors have come together to align these various mechanisms—and kick them into overdrive. As a result, now when people like me and Becca go off on different life trajectories, their opinions diverge in predictable and consistent directions, and do so faster and farther than before.

In sum: normally, the polarization process is a random walk; recently, it’s been transformed into a feedback loop.

Of course, there are many moving parts in a full story of modern polarization.  My goal is not to challenge the standard empirical accounts of it; instead, it’s to challenge their normative interpretations—to argue that our polarized politics arises from reasonable people who care about the truth but face ambiguous evidence. Given that, it’ll suffice to have a simple empirical story on the table; subtle or contested details will be reserved for the Technical Appendix.

As a representative example, focus on me and Becca. There are three questions we want to answer: In what sense have we “polarized”? What mechanisms lead to such polarization? And what has changed, making our current polarization different from past polarizations?


In what sense have we “polarized”?
There are three distinct “polarizations” that Becca and I—and the United States in general—have gone through in recent decades.

The first is ideological sorting: my views have become more consistently liberal and aligned with those of the Democratic Party; hers have become more consistently conservative and aligned with those of the Republican Party.

For example: in 2010 I was pro-choice and Becca was pro-life, but we were both un-opinionated about gun rights. (In high school, I had the experience—which at the time I thought was cool—of firing a friend’s AK-47 at a shooting range. Non-US readers: yes, you can buy those legally.) Yet in the decade since, my views have become more consistently left-leaning—I am now both pro-choice and anti-gun. Meanwhile, Becca’s views have become more consistently conservative—she’s now both pro-life and pro-gun.

The second is affective polarization: our views of the opposing party have become increasingly negative.

For example: in 2010, most of my friends were conservative, and as a result I had quite a bit of respect for the Republican Party.  Today, I have to wrack my brains to think of people I know who might’ve voted for Trump; and—I must admit—I’ve come to dislike the Republicans more, and understand where they’re coming from less.  A similar story, no doubt, governs Becca’s opinions toward Democrats.

The third is attitude polarization: our disagreements over political questions have become much sharper.

For example: if in 2010, you’d asked us both whether it’d be good for the country for a Republican to be elected president in 2020, Becca would’ve been mildly inclined to agree and I’d have been mildly inclined to disagree.  What if you ask us today? Then we’ll both admit that this election feels like a matter of life or death for our country.  Becca thinks, “If Biden wins, the police will be abolished and we’ll be cast into socialism!”  I think, “If Trump wins, the norms that uphold our democracy will be under direct assault!” 

(If your skeptical that American's political opinions have undergone attitude polarization--as some are—see the technical appendix.)

Upshot: Becca and I “polarized” in the senses that (1) our attitudes became more consistently opposed, (2) our feelings toward the other side became more negative, and (3) our disagreements became increasingly sharp.

This process is familiar. The United States has become increasingly ideologically sorted by politics—for example, the proportion of people with consistently liberal or  consistently conservative positions more than doubled between 1994 and 2014. Negative feelings toward the other side have skyrocketed—for example, those with a “very unfavorable” opinion of the opposing party nearly tripled between 1994 and 2016. And disagreements on concrete political questions has become increasingly sharp––for example, the Republican/Democrat presidential approval ratings averaged 75% / 34% in the time of Nixon, 81% / 23% in the time of (W.) Bush, and currently sit at 92% / 4% in the time of Trump.

So Americans have always been polarized, but that polarization has kicked into overdrive in recent decades.  Given that, we need to know two things: Why, in general, do societies polarize? And what has changed to make this more severe in recent decades?


Why do societies polarize?
Psychologists and sociologists have long known of a set of mechanisms that drive people to have stronger (and often more conflicting) attitudes over time—especially when they are put in different social and informational environments.

First mechanism: most obviously, and most simply, people are persuaded by arguments. That means that if two people enter different environments in which they’ll tend to encounter arguments for different positions, their opinions will predictable diverge.

For example, since I was headed to a liberal university, I could expect to hear arguments in favor of progressive taxation and the existence of oppressive sexism; and since Becca was headed to a conservative college, she could expect to hear arguments in favor of the value of capitalism and the importance of being a good woman.  As a result of these opposing arguments, it’s in some sense obvious that we could predict that we’d come to disagree. (But that apparent obviousness is also a bit misleading—as we’ll see, the rationality of this predictable disagreement hinges on the subtleties of ambiguous evidence.)

Second mechanism: when groups of like-minded individuals share and discuss their opinions, they tend to become both more homogenous and more extreme in those opinions.  This is known as the group polarization effect, and is one of the most robust findings in social psychology.

For example, when I started taking about gun rights in groups of mostly-liberal university students (the majority of whom had never held a gun), we all grew more confident that interpretations of the second amendment have been bastardized, leading to the rise in mass shootings.  Meanwhile, when Becca did the same in groups of mostly-conservative friends (many of whom owned guns), she grew more confident that the right to bear arms was central to American identity, and that since we’ve always had guns, the rise in mass shootings has other causes.

These first two mechanisms—persuasion and group polarization—are what gets polarization started: simply by putting me and Becca in different (liberal vs. conservative) social environments, they started to pull our opinions apart.

Now jump to Spring 2011, when our opinions had already been pulled slightly further apart by our new environments. Once this happened, a new set of mechanisms kicked in: our opposing beliefs started to ratchet up on their own.

Much of this phenomenon goes under the rather disunited label of ``confirmation bias”: people’s tendency to gather and interpret evidence in a way that confirms their prior or favored beliefs. This tendency can be helpfully divided into three different mechanisms.

Third mechanism: selective exposure—when given a choice, people tend to prefer to see new information that they expect to confirm their prior beliefs, rather than information they expect to disconfirm them.

For example, in Spring 2011 I began to consistently check the New York Times and the Washington Post to get my news. Becca, meanwhile, became more consistent in checking Fox News and the National Review.

Fourth mechanism: biased assimilation of evidence—when confronted with conflicting or messy evidence, people tend to interpret it in a way that favors their prior beliefs.

For example, in 2012 when Republicans stonewalled Obama’s proposals to use government funds to boost the economy, I took this to show that Republicans tend to put party over country—but Becca took it to show that Democrats tend to resort to government over-reach.

Final mechanism: motivated reasoning, a.k.a. identity-protective cognition—people tend to gather and make use of evidence in a way that confirms the things that they want to believe, especially when the belief is tied to their identity.

For example, when Tara Reade accused Biden of sexual assault, I must admit that I was inclined to be a bit skeptical—to spend a bit longer looking at articles that questioned her credibility than at those that supported it. Yet when Blaise Ford made an accusation against Brett Kavanaugh, I made no such skeptical effort.  Becca, no doubt, reacted in exactly the opposite way.

These mechanisms should sound familiar—we all know that people tend to be persuaded by arguments, that their motivations tend to affect the way they reason, and so on.

This is as it should be. Polarization is a familiar fact of life—it’s always been with us. Thus any adequate explanations of it will be built upon other familiar facts of life; an explanation built solely upon new or surprising findings would be missing the bigger picture.

Nevertheless, there is something new in our polarized politics—and there is something surprising lying behind these familiar mechanisms.

What’s new is that these mechanisms have become collectively aligned and individually kicked into overdrive.

And what’s surprising is that all of them—including howlers like biased assimilation and motivated reasoning—are to be expected from rational people who care about the truth but face systematically ambiguous evidence.

Explaining this latter point will take weeks; today, let’s close with the former.


What has changed?
What has led to recent the increases in American polarization? The story, in outline, is that a variety of societal changes have led to increased social and informational sorting.

Both the southern realignment and the civil rights movement started the process of making Democrats the party of consistent progressives and the Republicans the party of consistent conservatives. In turn, this increase in ideological consistency may have combined with the fading influence of religion to make political party the new key to many people’s identity.

Meanwhile, an increasing urban-rural divide has made it so that the political views of one’s (future) friends has become more predictable than ever. Combined with a precipitous fall in civic engagement, this has led to a decrease in cross-party social pollination and fewer friendships across party lines.

At the same time, an increasingly fragmented and political media landscape, along with the rise of web personalization has allowed people greater freedom in choosing their sources of information and opinion.

In short, decades ago, our social circles were ideologically diverse, our news sources constantly confronted us with differing opinions, and our political beliefs were not terribly consistent or central to our identities. Today, all that has changed. As a result, the effects of persuasion, group polarization, selective exposure, biased assimilation, and motivated reasoning all point in increasingly the same direction––pulling Democrats to the left, and Republicans to the right.

In my case: the more time I spent talking with Democrats, the more persuaded I was of their arguments and the more opinionated we all became. The more informational choices I had, the more inclined I was to listen to and trust liberal sources. The more confident I became, the more inclined I was to interpret ambiguous stories about Republican ideas and politicians in negative ways. The more I came to identify with being a Democrat, the more motivated I was to see new evidence as confirming my Democratic beliefs.  And the stronger each of these processes became, the more it reinforced the others.

Similarly for Becca––and for Americans everywhere.

Our question: what does this story—of old polarizing mechanisms, recently kicked into overdrive—mean for the rationality of politics?

It means that we if understand the old mechanisms, we’ll understand both why we’ve always been polarized, and the route through which polarization has increased. And it means that if we come to see these old mechanisms as rational, then we’ll be able to see our polarized politics—including our political opponents—as the result of individuals doing the best they can with the information they have.

That’s where we’re heading. We’re going to take an in-depth look at the mechanisms of persuasion, group-polarization, selective exposure, biased assimilation, and motivated reasoning. We’ll see how each of these are driven by rational attention to ambiguous evidence—and that as societal changes have made our political evidence systematically more ambiguous, they have all been kicked into overdrive.

But before we can dive into those details, we need to establish some background. What do I mean when I say that these mechanisms are to be expected from “rational people who care about the truth” and yet face “ambiguous” evidence?  And how could it be that rational people—like me and Becca—can be predictably, persistently, and profoundly polarized?



What next?
If you have comments or suggestions about this empirical story, please send them!
​Next post: a philosophical crash course on what it means to be “rational”, what exactly “ambiguous evidence” is, and how it is needed to explain predictable polarization.

In my hand I hold a fair coin. I’m going to toss it twice (…done).  From those two tosses, I picked one at random; call it the Random Toss. How confident are you that the Random Toss landed heads?  50-50, no doubt––it’s a fair coin, after all.

​But I’m going to polarize you on this question. What I’ll do is split you into two groups—the Headsers and the Tailsers—and give those groups different evidence. What’s interesting about this evidence is that we all can predict that  it’ll lead Headsers to (on average) end up more than 50% confident that the Random Toss landed heads, while Tailsers will end up (on average) less than 50% confident. That is: everyone (yourselves included) can predict that you’ll polarize.

The trick? I’m going to use ambiguous evidence.

First, to divide you. If you were born on an even day of the month, you’re a Headser; if you were born on an odd day, you’re a Tailser.  Welcome to your team.

You’re going to get different evidence about how the coin-tosses landed. That evidence will come in the form of word-completion tasks. In such a task, you’re shown a string of letters and some blanks, and asked whether there’s an English word that completes the string. For instance, you might see a string like this:

P_A_ET

And the answer is: yes, there is a word that completes that string. (Hint: what is Venus?) Or you might see a string like this:

CO_R_D

And the answer is: no, there is no word that completes that string.

That’s the type of evidence you’ll get. You’ll be given two different word-completion tasks—one for each toss of the coin. However, Headsers and Tailser will be given different tasks. Which one they’ll see will depend on how the coin landed.

The rule:

• For each toss of the coin, Headsers will see a completable string (like ‘P_A_ET’) if the coin landed heads; they’ll see an uncompletable string (like ‘CO_R_D’) if it landed tails.
• Conversely: for each toss, Tailsers will see a completable string if the coin landed tails; they’ll see an uncompletable string if it landed heads.

Here’s your job. Click on the appropriate link below to view a widget which will display the tasks for your group. You’ll view the first world-completion task, and then enter how confident you are (between 0–100%) that the string was completable. Enter “100” for “definitely completable”, “50” for “I have no idea”, “0” for “definitely not completable”, and so on.

If you’re a Headser, the number you enter is your confidence that the coin landed heads on the first toss. If you’re a Tailser, it’s your confidence that the coin landed tails—so the widget will subtract it from 100 to yield your confidence that the coin landed heads. (If you’re 60% confident that the coin landed tails, that means you’re 100–60 = 40% confident that it landed heads.)

You’ll do this whole procedure twice—once for each toss of the coin. Then the widget will tell you what your average confidence in heads was, across the two tosses. This is how confident you should be that the Random Toss landed heads, given your confidence in each individual toss. And this average is the number that will polarize across the two groups.

Enough set up; time to do the tasks.

• If you’re a Headser, click here. 
• If you’re a Tailser, click here.

Welcome back.  You have now, I predict, been polarized.

This is a statistical process, so your individual experience may differ. But my guess is that if you’re a Headser, your average confidence in heads was greater than 50%; and if you’re  Tailser, your average confidence in heads was less than 50%.

I’ve run this study. Participants were divided into Headsers and Tailsers. They each saw four word-completion tasks. Here’s how the two groups’ average confidence in heads (i.e. their confidence that the Random Toss landed heads) evolved as they saw more tasks:

Both groups started out 50% confident on average, but the more tasks they saw, the more this diverged. By the end, the average Headser was 58% confident that the Random Toss landed heads, while the average Tailser was 36% confident of it.

(That difference is statistically significant; the 95%-confidence interval for the mean difference between the groups’ final average confidence in heads is [16.02, 26.82]; the Cohen’s d effect size is 1.58—usually 0.8 is considered “large”. For a full statistical report, including comparison to a control group with unambiguous evidence, see the Technical Appendix.)

Upshot: the more word-completion tasks Headsers and Tailsers see, the more they polarize.
​
The crucial question: Why?

Getting a complete answer to this—and to why such polarization should be considered rational—will take us a couple more weeks.  But the basic idea is simple enough.

​A word-completion task presents you with evidence that is asymmetrically ambiguous. It’s easier to know what to think if there is a completion than if there’s no completion. If there is a completion, all you have to do is find one, and you know what to think.  But if there’s no completion, then you can’t find one; but nor can you be certain there is none—for you can’t rule out the possibility that there’s one you’ve missed.

Staring at `_E_RT’, you may be struck by a moment of epiphany—`HEART!’—and thereby get unambiguous evidence that the string is completable.

But staring at `ST_ _RE’, no such epiphany is forthcoming; the best you’ll get is a sense that it’s probably not completable, since you haven’t yet found one.  Nevertheless, you should remain unsure whether you’ve made a mistake: “Maybe it does have a completion and I should know it; maybe in a second, I’ll think to myself, `It is completable—duh!’”  This self-doubt is the sign of ambiguous evidence, and it prevents you from being too confident that it’s not completable.

The result? When you’re presented with a string that’s completable, you often get strong, unambiguous evidence that it’s completable; when you’re presented with a string that’s not completable, you can only get weak, ambiguous evidence that it’s not. Thus when the string is completable, you should often be quite confident that it is; when it’s not, you should never be very confident that it’s not.

I polarized you by exploiting this asymmetry. Headsers saw completable strings when the coin landed heads; Tailsers saw them when it landed tails. That means that Headsers were good at recognizing heads-cases and bad at recognizing tails-cases, while Tailsers were good at recognizing tails-cases and bad at recognizing heads-cases.

As a result, if you ask Headsers, they’ll say, “It’s landed heads a lot!”; and if you ask Tailsers, they’ll say, “It’s landed tails a lot!”. They polarize.

Here it’s worth emphasizing a subtle point but important point—one that we’ll return to. The ambiguous/unambiguous-evidence distinction is not the weak/strong-evidence distinction. Ambiguous evidence is evidence that you should be unsure how to react to; unambiguous evidence is evidence that you should be sure how to react to.

Ambiguous evidence is necessarily weak, but unambiguous evidence can be weak too. Example: if I tell you I’m about to toss a coin that’s 60% biased towards heads, that is weak but unambiguous evidence--weak because you shouldn’t be very confident it’ll land heads, but unambiguous because you know exactly how confident to be (namely, 60%).

The claim is that it is asymmetries in ambiguity—not asymmetries in strength—which drive polarization. We can test this by comparing our ambiguous-evidence Headsers and Tailsers to a control group that received evidence that was sometimes strong and sometimes weak, but always (relatively) unambiguous. (It came in the form of draws from an urn; see the Technical Appendix for details.)

Here is the evolution of the Headsers and Tailsers who got unambiguous evidence:

As you can see, there is some divergence (most liked a “response bias” because of the phrasing of the questions), but significantly less divergence than in our ambiguous-evidence case. (Again, see the technical appendix for a statistical comparison.)

Upshot: ambiguous evidence can be used to drive polarization. 

That concludes my possibility proof. The rest of this project will try to figure out what it means.  To do that, we need to look further into both the theoretical foundations and the real-world applications.

To preview the foundations: there is a clear sense which both Headsers and Tailsers are succeeding at getting to the truth of the matter—for each coin flip, they each tend to get more accurate about how it landed. The trouble is that this accuracy is asymmetric, and as a result they end up with very different overall pictures of the outcomes of the series of coin flips.

To preview the applications: these ambiguity-asymmetries can be exploited. Fox News can spin its coverage so that information that favors Trump is unambiguous, while that which disfavors him is ambiguous. MSNBC can do the opposite.  So when we divide into those-who-watch-Fox and those-who-watch-MSNBC, we are, in effect, dividing ourselves into Headsers and Tailsers.  As a result, although we are getting locally more informed whenever we tune into these programs, our global pictures of Trump are getting pulled further and further apart.

In fact, the same sort of ambiguity-asymmetry plays out in many different settings—helping explain why group discussions push people to extremes, why individuals favor news sources that agree with their views, and why partisans interpret shared information in radically different ways.

That’s where we’re headed. To get there, we need to get a few more basics on the table. Empirically: what mechanisms have led to the recent rise in polarization? And theoretically: what would it mean for this polarization—in our Headser/Tailser game, and in real life—to be “rational”?

We’ll tackle those two questions in the coming weeks. Then we’ll put all the pieces together, and examine the role that ambiguity-asymmetries in evidence play in the mechanisms that drive polarization.


What next?
If you liked this post, consider signing up for the newsletter, following me on Twitter, or spreading the word.
For a full statistical analysis of the experiment, see the technical appendix.  No doubt those with more experimental expertise can find ways it could be improved––suggestions most welcome!
Next post: I’ll sketch a story of the empirical mechanisms that drive polarization; with that on the table, we'll move on to evaluating them normatively.


PS. Thanks to Branden Fitelson and especially Joshua Knobe for much help with the experimental design and analysis.  

[9/4/21 update: if you'd like to see the rigorous version of this whole blog series, check out the paper on "Rational Polarization" I just posted.]
​

​​​A Standard Story
I haven’t seen Becca in a decade. I don’t know what she thinks about Trump, or Medicare for All, or defunding the police.

But I can guess.

​​Becca and I grew up in a small Midwestern town. Cows, cornfields, and college football. Both of us were moderate in our politics; she a touch more conservative than I—but it hardly mattered, and we hardly noticed.

After graduation, we went our separate ways. I, to a liberal university in a Midwestern city, and then to graduate school on the East Coast. She, to a conservative community college, and then to settle down in rural Missouri.

I––of course––became increasingly liberal. I came to believe that gender roles are oppressive, that racism is systemic, and that our national myths let the powerful paper over the past.

And Becca?

You and I can both guess how her story differs. She’s probably more concerned by shifting gender norms than by the long roots of sexism; more worried by rioters in Portland than by police shootings in Ferguson; and more convinced of America’s greatness than of its deep flaws.

​In short: we started with similar opinions, set out on different life trajectories, and, 10 years down the line, we deeply disagree.

So far, so familiar. The story of me and Becca is one tiny piece of the modern American story: one of pervasive—and increasing—political polarization.

It’s often noted that this polarization is profound: partisans now disagree so much that they often struggle to understand each other.

It’s often noted that this polarization is persistent: when partisans sit down to talk about their now-opposed beliefs, they rarely rethink or revise them.

But what’s rarely emphasized is that this polarization is predictable: people setting out on different life trajectories can see all this coming. When Becca and I said goodbye in the summer of 2010, we both suspected that we wouldn’t be coming back. That when we met again, our disagreements would be larger. That we’d understand each other less, trust each other less, like each other less.

And we were right.  That’s why I haven’t seen her in a decade.

Told this way, the story of polarization raises questions that are both political and personal. What should I now think of Becca—and of myself? How should I reconcile the strength of my current beliefs with the fact that they were utterly predictable? And what should I reach for to explain how I came to disagree so profoundly with my old friends?

The standard story: irrationality.

The story says, in short, that politics makes us stupid. That despite our best intentions, we glom onto the beliefs of our peers, interpret information in biased ways, defend our beliefs as if they were cherished possessions, and thus wind up wildly overconfident. You’ve probably heard the buzzwords: “confirmation bias”, “the group-polarization effect”, “motivated reasoning”, “the overconfidence effect”, and so on.

This irrationalist picture of human nature has quite the pedigree—it has won  Nobel Prizes, started academic subfields, and embedded itself firmly in the popular imagination.

When combined with a new wave of research on the informational traps of the modern internet, the standard story offers a simple explanation for why political polarization has exploded: our biases have led us to mis-use our new informational choices. Again, you’ve probably heard the buzzwords: “echo chambers”, “filter bubbles”, “fake news”, “the Daily Me”, and so on.

The result?  A bunch of pig-headed people who increasingly think that they are right and balanced, while the other side is wrong and biased.

It’s a striking story. But it doesn’t work.

It says that polarization is predictable because irrationality is predictable: that Becca and I knew that, due to our biases, I’d get enthralled by liberal professors and she’d get taken in by conservative preachers.

But that’s wrong. When I looked ahead in 2010, I didn’t see systematic biases leading to the changes in my opinions. And looking back today, I don’t see them now.

If I did see them, then I’d give up those opinions. For no one thinks to themselves, “Gender roles are oppressive, racism is systemic, and national myths are lies—but the reason I believe all that is that I interpreted evidence in a biased and irrational way.” More generally: it’s incoherent to believe that your own beliefs are irrational. Therefore, so long as we hold onto our political beliefs, we can’t think that they were formed in a systematically irrational way.

So I don’t see systematic irrationality in my past. Nor do I suspect it in Becca’s. She was just as sharp and critically-minded as I was; if conservative preachers changed her mind, it was not for a lack of rationality.

It turns out that tellers of the irrationalist tale must agree. For despite the many controversies surrounding political (ir)rationality, one piece of common ground is that both sides are equally susceptible to the factors that lead to polarization. As far as the psychological evidence is concerned, the “other side” is no less rational than you––so if you don’t blame your beliefs on irrationality (as you can’t), then you shouldn’t blame theirs on it either.

In short: given that we can’t believe that our own beliefs are irrational, the irrationalist explanation of polarization falls apart.

Suppose you find this argument convincing. Even so, you may find yourself puzzled. After all: what could explain our profound, persistent, and predictable polarization, if not for irrationality? As we’ll see, there’s a genuine philosophical puzzle here. And when we can’t see our way to the solution, it’s very natural to fall back on irrationalism.

In particular: since we can’t view our own beliefs as irrational, it’s natural to instead blame polarization on the other side’s irrationality: “I can’t understand how rational people could see Trump so differently. But I’m not irrational—so the irrational ones must be Becca and her conservative friends, right?”

That thought turns our disagreement into something more. Not only do we think the other side is wrong—we now think they are irrational. Or biased. Or dumb. And that process of demonization—more than anything else—is the sad story of American polarization.

A Reasonable Story     
What if it need not be so? What if we could think the other side is wrong, and not think they are dumb? What if we could tell a story on which diverging life trajectories can lead rational people—ones who care about the truth—to be persistently, profoundly, and predictably polarized?

That’s what I’m going to do. I’m going to show how findings from psychology, political science, and philosophy allow us to see polarization as the result of reasonable people doing the best they can with the information they have. To argue that the fault lies not in ourselves, but in the systems we inhabit.  And to paint a picture on which our polarized politics consists largely of individually rational actors, collectively acting out a tragedy.

Here is the key. When evidence is ambiguous––when it is hard to know how to interpret it—it can lead rational people to predictably polarize.

This is a theorem in standard (i.e. Bayesian) models of rational belief. It makes concrete and confirmed empirical predictions. And it offers a unified explanation of our buzzwords: confirmation bias, the group-polarization effect, motivated reasoning, and the overconfidence effect are all to be expected from rational people who care about the truth but face systematically ambiguous evidence.

More than that: this story explains why polarization has exploded in recent decades. Changes in our social and informational networks have made is so that, with increasing regularity, the evidence we receive in favor of our political beliefs tends to be unambiguous and therefore strong—while that we receive against them tends to be ambiguous and therefore weak. The rise in this systematic asymmetry is what explains the rise in polarization.

In short: the standard story is right about which mechanisms lead people to polarize, but wrong about what this means about people. People polarize because they look at information that confirms their beliefs and talk to people that are like-minded. But they do these things not because they are irrational, biased, or dumb. They do them because it is the best way to navigate the landscape of complex, ambiguous evidence that pervades our politics.

That’s the claim going to defend over the coming weeks.

Here’s how. I’ll start with a possibility proof—a simple demonstration of how ambiguous evidence can lead rational people to predictably polarize. Our goal will then be to figure out what this demonstration tells us about real-world polarization.

To do that, we need to dive into both the empirical and theoretical details. In what sense has the United States become increasingly ”polarized”—and why? What would it mean for this polarization to be “rational”—and how could ambiguous evidence make it so? How does such evidence explain the mechanisms that drive polarization—and what, therefore, might we do about them? I’ll do my best to give answers to each of these questions.

This, obviously, is a big project. That means two things.

First, it means I’m going to present it in two streams. The core will be this blog, which will explain the main empirical and conceptual ideas in an intuitive way. In parallel, I’ll post an expanding technical appendix that develops the details underlying each stage in the argument.

Second, it means that this is a work in progress.  It’ll eventually be a book, but—though I’ve been working on it for years—getting to a finished product will be a long process. This blog is my way of nudging it along.

That means I want your help! The more feedback I get, the better this project will become. I want to know which explanations do (and don’t) makes sense; what strands of argument are (and are not) compelling; and—most of all—what you find of value (or not) in the story I’m going to tell.

So please: send me your questions, reactions, and suggestions.

And together, I hope, we can figure out what happened between me and my old friends—and, maybe, what happened between you and yours.


What next?
If you’re interested in this project, consider signing up for the newsletter, following me on Twitter, or spreading the word.
Next post: an experiment that demonstrates how ambiguous evidence can lead people to polarize.

​
PS. Thanks to Cosmo Grant, Rachel Fraser, and Ginger Schultheis for helpful feedback on previous drafts of this post—and to Liam Kofi Bright, Cailin O'Connor, Kevin Zollman, and especially Agnes Callard for much help and advice getting this project off the ground.

Predictive and decision-making algorithms are playing an increasingly prominent role in our lives. They help determine what ads we see on social media, where police are deployed, who will be given a loan or a job, and whether someone will be released on bail or granted parole. Part of this is due to the recent rise of machine learning. But some algorithms are relatively simple and don’t involve any AI or ‘deep learning.’

As algorithms enter into more and more spheres of our lives, scholars and activists have become increasingly interested in whether they might be biased in problematic ways. The algorithms behind some facial recognition software are less accurate for women and African Americans. Women are less likely than men to be shown an ad relating to high-paying jobs on Google. Google Translate translated neutral non-English pronouns into masculine English pronouns in sentences about stereotypically male professions (e.g., ‘he is a doctor’).

When Alexandria Ocasio-Cortez noted the possibility of algorithms being biased (e.g., in virtue of encoding biases found in their programmers, or the data on which they are trained), Ryan Saavedra, a writer for the conservative Daily Wire, mocked her on Twitter, writing “Socialist Rep. Alexandria Ocasio-Cortez claims that algorithms, which are driven by math, are racist.”

I think AOC was clearly right and Saavedra clearly wrong. It’s true that algorithms do not have inner feelings of prejudice, but that doesn’t mean they cannot be racist or biased in other ways.

​But in any particular case, it’s tricky to determine whether a given algorithm is in fact biased or unfair. This is largely due to the lack of agreed-upon criteria of algorithmic fairness.

Statistical Criteria of Fairness
Let’s focus on algorithms like COMPAS. These algorithms make predictions, not decisions, though of course their predictions might be used to feed into decisions about bail, parole, and the like. The algorithms in question take as input a ‘feature vector’ (a set of known characteristics) and output either a risk score, or a binary prediction, or both. For simplicity, let’s focus on algorithms that output both a real-valued risk score between 0 and 1 and a binary (yes/no) prediction. We can think of the risk score as a probability that the individual will fall into the ‘positive’ class, and the prediction as akin to a binary belief about whether the individual will be positive or negative.

What criteria must a predictive algorithm satisfy in order to qualify as fair and unbiased? Some criteria concern the inner workings of the algorithm. Perhaps a fair algorithm must not use group membership as part of the feature vector upon which its predictions are based. It must be blinded to whether the individual is male or female, black or white, and so on. Perhaps fairness also requires that the algorithm be blinded to any ‘proxies’ for group membership. For instance, we might regard ZIP code as a proxy for race, given that housing in the US is highly segregated. It is a difficult matter to say in general when some feature counts as a proxy in a problematic sense, but the basic idea is clear enough.

Fairness also presumably requires that the algorithm use the same threshold in moving from a risk score to a binary prediction. It would be unfair, for instance, if black people assigned a risk score above 0.7 were predicted to recidivate, while only white people assigned a risk score above 0.8 were predicted to recidivate. These criteria are relatively uncontroversial and relatively easy to satisfy (except for the tricky issue of proxies for group membership). But are there any other criteria that an algorithm must satisfy in order to be fair and unbiased?

This post will be concerned with a class of purported fairness criteria that require that certain relations between predictions and actuality be the same across the relevant groups. I’ll call these ‘statistical criteria of fairness.’ These are the sorts of criteria that are at issue in the debate over COMPAS. They are of interest in part because we can determine whether they are satisfied by some algorithm just by looking at what it predicted and what actually happened. We don’t need to look at the inner workings of the algorithm, which may be proprietary or otherwise opaque. (This opacity is itself a problem, and we should seek as much transparency as possible going forward.)

Here are the main statistical criteria of fairness at issue in the debate over COMPAS. See the Appendix for several more that have been considered and discussed in the literature.

(1) Calibration Within Groups: For each possible risk score, the percentage of individuals assigned that risk score who are actually positive is the same for each relevant group and equal to that risk score.
(2) Equal False Positive Rates: The percentage of actually negative individuals who are falsely predicted to be positive is the same for each relevant group
(3) Equal False Negative Rates: The percentage of actually positive individuals who are falsely predicted to be negative is the same for each relevant group.

It’s pretty easy to see why each seems like an attractive criterion of fairness. If an algorithm violates (1) Calibration Within Groups, then it would seem that a given risk score has different evidentiary value for members of different groups. A given risk score would ‘mean’ different things for different individuals, depending on which group they are members of. If an algorithm violates (2), yielding a higher false positive rate for one group than for another, it’s tempting to conclude that it was being more ‘risky,’ or was jumping to conclusions more quickly, with respect to one group versus another. The same goes if it violates (3), yielding a higher false negative rate for one group than for another. And this seems unfair. It seems to conflict with the idea that individuals should be treated the same by the algorithm, regardless of their group membership.

Impossibility Theorems
It would be nice if some algorithms could satisfy all of these criteria. This wouldn’t mean that the algorithm is in fact fair. Even if each of these statistical criteria is necessary for fairness, they are not jointly sufficient – we saw above that there are additional, non-statistical criteria that must be satisfied as well. But still, it would be a promising start if an algorithm could satisfy all of these statistical criteria.

But it is impossible for an algorithm to satisfy all of these criteria, except in marginal cases. This is the upshot of a series of impossibility theorems. Two such theorems are particularly important. Kleinberg et al. prove that no algorithm can jointly satisfy (1) and close relatives of (2) and (3) unless either (i) base rates (i.e. the percentage of individuals who are in fact positive) are equal across the relevant groups, or (ii) prediction is perfect (i.e. the algorithm assigns risk score 1 to all positive individuals and 0 to all negative individuals).  Chouldechova proves that no algorithm can jointly satisfy (2) and (3) and a close relative of (1), again unless base rates are equal or prediction is perfect.

I won’t go through the proofs of these impossibility theorems, but they’re not terribly technical. And here’s a great explanation of the theorems and their importance.

​What should we make of these theorems? Pessimistically, we might conclude that fairness dilemmas are all but inevitable; outside of marginal cases, we cannot help but be unfair or biased in some respect.

​More optimistically, we might conclude that some of our criteria are not in fact necessary conditions on algorithmic fairness, and that we need to take a second look and sort the genuine fairness conditions from the specious ones. This is the tack I will take.

People, Coins, and Rooms
How could we go about determining which (if any) of the above statistical criteria are genuinely necessary for fairness? One methodology would be to go one-by-one, looking at the motivations behind each criterion, and seeing if those motivations stand up to scrutiny.

Another methodology would be to find a perfect, 100% fair algorithm and see if that algorithm can violate any of those criteria. If it can, then this means that the criterion isn’t necessary for fairness. (If it can’t, this doesn’t mean that the criterion is necessary for fairness; perhaps some other 100% fair algorithm can violate it.) But this methodology may seem unpromising. It would be hard, if not impossible, to find any predictive algorithms that everyone agrees is perfectly, 100% fair.

At least, that is the case if we consider algorithms that predict important, messy things like recidivism and professional success. But we can do better by considering coin flips; this will enable us to make use of this second methodology.

Here is the setup: There are a bunch of people and a bunch of coins of varying biases (i.e. varying chances of landing heads). The people are randomly assigned to one of two rooms, A and B. And the coins are randomly assigned to people. So each person has one coin and is in one of our rooms. Our aim is to predict, for each person, whether her coin will land heads or tails. That is, we are trying to predict, for each person, whether they are a heads person or a tails person. Luckily, each coin comes helpfully labelled with its bias.

Here is a perfectly, 100% fair algorithm: For each person, take their coin and read its labelled bias. If the coin reads ‘x’, assign it a risk score of x. If x>0.5, make the binary prediction that it will land heads. If x<0.5, make the binary prediction that it will land tails. (I assume, for simplicity, that none of the coins are labelled ‘0.5.’)

It should be clear that this algorithm is perfectly, 100% fair. This, I think, is bedrock. It’s certainly an odd setup, and there’s some unfortunate randomness, but there’s no unfairness anywhere in the setup––and in particular not in our algorithm itself.

Let’s see how our criteria shake out with respect to this algorithm. A first thing to note is that criteria (1)-(3) were formulated in terms of what outcomes actually result. But with coin flips, anything can happen. No matter how biased a coin is in favour of heads (short of having heads on both sides), it can still land tails, and vice versa. So it’s actually quite easy for our algorithm to violate all of (1)-(3), given the right assignment of coins to people and people to rooms.

This suggests that we should have formulated our criteria in expectational or probabilistic terms:

(1*) Expectational Calibration Within Groups: For each possible risk score, the expected percentage of individuals assigned that risk score who are actually positive is the same for each relevant group and equal to that risk score.
(2*) Expectational Equal False Positive Rates: The expected percentage of actually negative individuals who are falsely predicted to be positive is the same for each relevant group.
(3*) Expectational Equal False Negative Rates: The expected percentage of actually positive individuals who are falsely predicted to be negative is the same for each relevant group.

(There’s a tricky question about how to understand the probability function relative to which these expectations are determined. I’ll think of it as an evidential probability function which represents what a rational person who knew about the workings of the algorithm would expect.)

We can investigate whether our perfectly, 100% fair algorithm can violate any of these starred criteria by considering a case where coin biases match relative frequencies (i.e. exactly 75% of the coins labelled 0.75 land heads, and so on). If our algorithm violates one of the unstarred criteria in this case, then it also violates the starred version of that criterion.

It turns out that our perfectly, 100% fair algorithm must satisfy (1*), but it can violate (2*) and (3*), given the right assignment of coins to people and people to rooms. Moreover, it can violate them simultaneously. And surprisingly, it can violate them simultaneously even when base rates are equal across the two rooms.

The following case illustrates this: Room A contains 12 people with coins labelled ‘0.75’ and 8 people with coins labelled ‘0.125.’ The former are all assigned risk score 0.75 and predicted to be heads people (positive), and nine of them are in fact heads people. The latter are all assigned risk score 0.125 and predicted to be tails people (negative), and seven of them are in fact tails people. Room B contains 10 people with coins labelled ‘0.6’ and 10 people with coins labelled ‘0.4.’ The former are all assigned risk score 0.6 and predicted to be heads people, and six of them are in fact heads people. The latter are all assigned risk score 0.4 and predicted to be tails people, and six of them are in fact tails people.

Note that base rates are equal across the two rooms: exactly ten out of the twenty people in each room are heads people.

While our algorithm in this case satisfies (1) Calibration Within Groups, and hence also (1*), it violates (2) and (3), and hence also (2*) and (3*). For Room A, the False Positive Rate is 3/10, while for Room B it is 4/10. And for Room A, the False Negative Rate is 1/10, while the False Negative Rate is 4/10. This fair algorithm also violates a host of other statistical criteria of fairness that have been suggested in the literature – see the Appendix for details.

​This means that it is possible for a perfectly, 100% fair algorithm to violate (2*) and (3*) when given the right population as input. This suffices to show that neither is a necessary condition on fairness. It also suffices to show that none of the other criteria considered in the Appendix are necessary for fairness, either. Only (1*) Expectational Calibration Within Groups is left standing as a plausible necessary condition on fairness.

Upshots
I think (1*) Expectational Calibration Within Groups is plausibly necessary for fairness. I also think fairness might require that the ‘inner workings’ of the algorithm be a certain way, for instance that the algorithm be blinded to group membership and that it use the same threshold in going from a risk score to a binary prediction. There may be other necessary conditions as well.

But it is misguided to focus on any of the other statistical criteria of fairness considered here or in the Appendix. Those criteria are tempting due to the relative ease of checking whether they are satisfied. But we have seen that an algorithm’s violating those criteria doesn’t mean that the algorithm is in any way unfair.

Now, even a perfectly fair predictive algorithm might have troubling results when used to make decisions in a certain way. It might have a disparate impact on the relevant groups. But the right way to respond to this disparate impact will often be not to modify the predictive algorithm, but rather to modify the way decisions are made on the basis of its predictions, or to intervene in society in other ways, for instance through reparations, changes in the tax code, and so on.

​Of course, some of these responses might be politically infeasible. This is especially true for some of the policies that might be most effective in redressing racial and other injustices. Reparations would be a case in point. It is difficult to imagine reparations becoming policy, despite Ta Nehisi Coates’ influential recent defense. If we can’t deal with racial (or other) injustices in these other ways, perhaps the best response is to chip away at injustice by modifying what was already a fair predictive algorithm. It’s not the ideal solution, but it might be second best. Still, it is important to be clear that an algorithm’s violating Equal False Positive/Negative Rates (or any of the other statistical criteria considered in the Appendix) neither entails nor constitutes the algorithm’s unfairness.


What next?
For an accessible explanation of the impossibility theorems, see this piece on Phenomenal World by Cosmo Grant​.
For the back-and-forth about COMPAS, see the initial ProPublica report,  Northpointe's response, and Propublica's counter-response.
For a more general discussion of the issues surrounding algorithmic fairness, see this article.

Appendix
In this Appendix, I’ll briefly mention some of the other main statistical criteria of fairness that have been considered in the literature:

(4) Balance for the Positive Class: The average risk score assigned to those individuals who are actually positive is the same for each relevant group.
(5) Balance for the Negative Class: The average risk score assigned to those individuals who are actually negative is the same for each relevant group.
(6) Equal Positive Predictive Value: The percentage of individuals predicted to be positive who are actually positive is the same for each relevant group.
(7) Equal Negative Predictive Value: The percentage of individuals predicted to be negative who are actually negative is the same for each relevant group.
(8) Equal Ratios of False Positive Rate to False Negative Rate: The ratio of the false positive rate to the false negative rate is the same for each relevant group.
(9) Equal Overall Error Rates: The number of false positives and false negatives, divided by the number of individuals, is the same for each relevant group.

The first two – (4) and (5) – can be seen as generalizations of (2) and (3) to the case of continuous risk scores. Indeed, Pleiss et al. refer to the measures involved in (4) and (5) as the ‘generalized false negative rate’ and the ‘generalized false positive rate,’ respectively. And, along with (1), it is these two criteria, rather than (2) and (3), that are the target of the aforementioned impossibility theorem from Kleinberg et al.

The next two – (6) and (7) – can be seen as generalizations of (1) to the case of binary predictions. This is how Chouldechova conceives of them. Just as (1) is motivated by the thought that a given risk score should ‘mean’ the same thing for each group, so (6) and (7) can be motivated by the thought that a given binary prediction should ‘mean’ the same thing for each group. Chouldechova’s impossibility result targets (6) rather than (1), showing that (2), (3), and (6) are not jointly satisfiable unless either base rates are equal or prediction is perfect.

The final two – (8) and (9) – are also intuitive. For (8), it would seem that violating this criterion would mean that the relative importance of avoiding false positives versus avoiding false negatives was evaluated differently for the different groups. Finally, (9) Equal Overall Error Rates embodies the natural thought that fairness requires that the algorithm be equally accurate overall for each of the different groups.

​We can now easily see that our perfectly fair predictive algorithm violates all these criteria as well (and hence also their expectational or probabilistic analogues), given the same assignment of coins to people and people to rooms as above:

The fact that the numbers in the Room A column and the Room B column differ for each row means that our fair predictive algorithm violated all of (4)-(9), in addition to (2) and (3). This suffices to show that none of (4)-(9), nor their expectational/probabilistic analogues, is a necessary condition on fairness. Among all these statistical criteria, only (1*) Expectational Calibration Within Groups is left standing as a plausible necessary condition on fairness.

​But wait! If you answered "Pence", or "Pence or Carlson", did you commit the conjunction fallacy? This is the tendency to say that narrower hypotheses are more likely than broader ones––such as saying that P&Q is more likely than Q—contrary to the laws of probability.  Since every way in which "Pence" or "Pence or Carlson" could be true is also a way in which “Pence, Carlson, or Haley” would be true, the third option is guaranteed to be more likely than each of the first two.

Does this mean answering our question with “Pence” or “Pence or Carlson” was a mistake? 

We don’t think so. We think what you were doing was guessing. Rather than simply ranking answers for probability, you were making a tradeoff between being accurate (saying something probable) and being informative (saying something specific). In light of this tradeoff, it’s perfectly permissible to guess an answer (“Pence”) that’s less probable––but more informative––than an alternative (“Pence, Carlson, or Haley”).

Here we'll argue that this explains––and partially rationalizes––the conjunction fallacy.

1. Good Guesses
We make guesses whenever someone poses a question and we can’t be sure of the answer. “Will it rain tomorrow?”, “I think it will”; “What day will the meeting be?”, “Probably Thursday or Friday”; “Who do you think will win the nomination?”, “I bet Pence will”; and so on.

What sorts of guesses are good guesses? The full paper argues that there are a variety of robust and intricate patterns, drawing on a fascinating paper by Ben Holguín.  Here we’ll just focus on the main patterns in our lead example.

Suppose you have the probability estimates from above (Pence, 44%; Carlson, 39%; Haley, 17%), and we ask you: “Who do you think will win?” As we've seen, a variety of answers seem reasonable:

(1) "Pence"   ✓
(2) "Pence or Carlson"   ✓
(3) "Pence, Carlson, or Haley"   ✓

In contrast, a variety of answers sound bizarre:

(4) "Carlson"   ✘
(5) "Carlson or Haley ( = "Not Pence")   ✘
(6) "Pence or Haley"   ✘

We’ve run examples like this by dozens of people, and the judgments are robust––for instance, in a similar Twitter poll in which "Pence or Haley" was an explicit option, it was the least-common answer (6.7%):

What’s going on?  How do we explain why (1)–(3) are good guesses and (4)–(6) are bad ones?

Our basic idea is a Jamesian thought: making good guesses involve trading off two competing goals.  On the one hand, we want to avoid error––to be accurate. On the other, we want to get at the truth––to be informative.  These two goals compete: the more informative your guess, the less likely it is to be true. A good guess is one that optimizes this tradeoff between accuracy and informativity.

More precisely, we assume that guesses have answer values that vary with their accuracy and informativity. True guesses are better than false ones, and informative true guesses are better than uninformative true ones.  Given that, here’s our proposal:

Guessing as Maximizing: In guessing, we try to select an answer that has as much answer-value as possible––we maximize expected answer-value.

To make this precise, we can clarify the notion of informativity using a standard model of a question. Questions can be thought of as partitioning the space of open possibilities into a set of complete answers. For example: the set of complete answers to “Will it rain tomorrow?” is {it will rain, it won’t rain}; the set of complete answers to “Who will win the nomination?” is {Pence will win, Carlson will win, Haley will win}; and so on.

In response to a question, a guess is informative to the extent that it rules out alternative answers. Thus “Pence” is more informative than “Pence or Carlson”, which in turn is more informative than “Pence, Carlson, or Haley”.

Given this, we can explain why (4)–(6) are bad guesses. Consider “Carlson”. It's exactly as informative as “Pence“––both rule out 2 of the 3 possible complete answers––but it is less probable: “Carlson“ has a 39% chance of being true, while “Pence” has a 44% chance. Thus if you’re trying to maximize expected answer value, you should never guess “Carlson”, since “Pence” is equally informative but more likely to be accurate.

Similarly, consider “Pence or Haley”. What’s odd about this guess is that it “skips” over Carlson. In particular, if we swap “Haley” for "Carlson”, we get a different guess that's equally informative but, again, more probable. (“Pence or Carlson” is 44 + 39 = 83% likely to be true, while “Pence or Haley” is 44 + 17 = 61% likely.)

On the other hand, the Guessing as Maximizing account explains why (1)–(3) can all be good guesses. The basic point: if you really care about being informative, you should choose a maximally specific answer (“Pence”); if you really care about being accurate, you should choose a maximally likely answer (“Pence, Carlson, or Haley”); and intermediate ways of weighting these constraints lead to good guesses at intermediate levels of informativity (“Pence or Carlson”).

(For a formal exposition of all this, see the Appendix or the paper.)

3. The conjunction fallacy
With this account of guessing in hand, let’s apply it to our opening observation: guessing leads to the conjunction fallacy.

Recall: this is the tendency to rate narrower hypotheses (like “P&Q”) as more probable than broader ones (like “Q”). It’s the star of the show in the common argument that people’s reasoning is systematically irrational, lacking an understanding of the basic rules of probability and instead using simple heuristics.

The most famous example is from the original paper by Tversky and Kahneman:

A majority of subjects said “feminist bank teller” (FT) was more probable than “bank teller” (T). Again, this violates the laws of probability: every possibility in which Linda is a feminist bank teller is also one in which she’s a bank teller––but not vice versa.

What’s going on here? Our proposal:

Answer-Value Account: People commit the conjunction fallacy because they rank answers according to their quality as guesses (their expected answer-value), rather than their probability of being true.

​In other words, we think the Linda case is analogous to the following one:

Given the above survey, we can expect that around half of people would choose (a).  (Since 35/(35 + 38.2) = 47.8%.) The crucial point is that in both the Linda and Pence cases, option (a) is less probable but more informative than option (b) with respect to the salient question––e.g. “What are Linda’s social and occupational roles?” or “Who will win the nomination?”

In particular, our model of expected answer value predicts that you should rate  “feminist bank teller” as a better guess than “bank teller” whenever you’re sufficiently confident that Linda is a feminist given that she’s a bank teller––whenever P(F|T) is sufficiently high, where the threshold for “sufficient” is determined by how much you value being informative (see the Appendix).

Why is this conditional probability P(F|T) what matters? Because although the probability of “feminist bank teller” is always less than that of “bank teller”, how much less is determined by this conditional probability, since  P(FT) = P(T)•P(F|T). Thus when P(F|T) is high, switching from "bank teller" to "feminist bank teller" has only a small cost to accuracy––which is easily outweighed by the gain in informativity. Our account therefore makes the following prediction:

Prediction: Rates of ranking the conjunction AB as more probable than the conjunct B will tend to scale with P(A|B).

This prediction is borne out by the data; a clean example comes from Tentori and Crupi (2012). They give a vignette in which they introduce Mark, and then say that he holds X tickets in a 100-ticket raffle––where X varies between 0 and 100 across different experimental conditions.  They then ask subjects which of the following is more likely:

The rates of ranking “scientist and will win the lottery” (WS) as more likely than (or equally likely as)  “scientist” (S) scaled directly with the number of tickets Mark held, i.e. with the probability that Mark wins the lottery given that he’s a scientist, P(W|S) (which equals P(W), since S and W are independent):

​4. So is the conjunction fallacy irrational?
Suppose the answer-value account is right: people commit the conjunction fallacy when they rate answers for their quality as guesses rather than for their probability of being true. What would that imply about the conjunction fallacy, and its role in the debate about human rationality more broadly?

The answer is subtle.

One the one hand, it turns out that questions like “What do you think is (most) likely?” are standard ways of eliciting guesses––which in turn have a very different normative profile than probability judgments. For example, when we asked “Who do you think will win?” in our opening question, answering “Pence” is not irrational––nor would it be so if we tweaked the wording to “What’s most likely?”; “What do you bet will happen?”; etc. These are all ways of eliciting guesses. (Note: our second Twitter poll used "What do you think's likely to happen?")

Since these prompts are standard ways of eliciting answers in conjunction-fallacy experiments, this complicates our assessment of such answers. The pragmatic upshot of the question that’s being asked is not a literal question about probability: people will hear these questions are requests to guess—to trade off probability against informativity––rather than to merely assess probability.  And reasonably so.

On the other hand, using such guesses to guide your statements and actions can lead to mistakes. This is clearest in experiments that elicit the conjunction fallacy while asking people to bet on options. Of course, “What do you bet will happen?” is a natural way of eliciting a guess in conversation (“I bet Pence’ll win”). Nevertheless, if we actually give you money and you choose to let it ride on “Pence or Carlson” rather than “Pence, Carlson, or Haley”, then you’ve made a mistake. Moreover, experiments show that people do have a tendency to bet like this––though the rates of the fallacy diminish somewhat.

In scenarios like this, the conjunction fallacy is clearly a mistake. The crucial question: What does this mistake reveal about human reasoning?

If our account is right, it does not reveal––as is commonly claimed––that human judgment works in a non-probabilistic way.  After all, what’s happening is that people are guessing and then acting based on that guess––and guessing requires an (implicit) assessment of probability. Instead, the conjunction fallacy reveals that people are bad at pulling apart judgments about pure probability from a much more common type of judgment––the quality of a guess.

Why are people bad at this?  Our proposal: because guessing is something we do all the time. Moreover, it’s something that makes sense to do all the time. We can’t have degrees of belief about all possibly-relevant claims––no system could, since general probabilistic inference is intractable. So instead, we construct probability judgments about the small set of claims generated by the question under discussion, use them to formulate a guess, and then reason within that guess.

There’s empirical evidence that people do this. For example: poker players decide what to bet by guessing what hands their opponents might have; doctors decide what tests to perform by guessing their patients' ailments; scientists decide what experiments to run by guessing which hypothesis is true; and so on.

People do this, we think, because probability alone doesn’t get you very far.  The most probable answer is always, “Something will happen, somewhere, sometime.” Such certainly-true answers don’t help guide our actions––instead, we need to trade off such certainty for some amount of informativity.

If this is right, the error revealed by the conjunction fallacy is in some ways like that revealed by the Stroop test. Watch the following video and try––as quickly as possible––to say aloud the color of the text presented (do not read the word):

I very much like Jake’s distinction between merely positing an input-output function to explain (ir)rational behavior, vs. positing a particular causal process underlying it.  It seems absolutely right to me that this is a strategy that has the potential to make progress on debates over (ir)rationality.  And I definitely agree that the evidence he presents raises some challenges for a rational picture of rationalization!

But I want to raise a couple questions to probe how severe those challenges are. As I understand it, there are two main pieces of evidence in favor of the (irrational) dissonance-reduction mechanism: (1) people’s tendency to rationalize is mediated by negative affect (the desire to avoid an unpleasant feeling), and (2) the way they rationalize tends to be closely linked to maintaining stable motivation.

My questions: Should (1) be surprising on a rational retelling of the narrative? And how strong is the evidence for (2)?

To (1): if rationalization were generally a result of rational revisions of beliefs, would it be surprising that this process is mediated through discomfort?  In general, processes that take cognitive effort require some motivational push to get people to do them. Given that, why not think that the negative affect associated with dissonance is the mind’s signal to itself, ``Hey––figure this out!”

I think a similar feeling will be familiar to lots of people—especially those who’ve done much math, coding, or logic. There’s a distinctively exciting––but also agonizing––feeling of being stuck on a proof or problem; a strong discomfort with not knowing how to figure out the answer, which can be quite  motivating.

More generally, I suspect curiosity is often driven by some form of negative affect. For an example, consider this riddle:

Think about it. I’m not going to tell you the answer (yet).  But I’ll tell you that the answer is neat––most people have an “Aha!” moment when they figure out.  So keep thinking it over…

As you do so, reflect on how it feels to think it over and not know the answer.  A bit uncomfortable, maybe even frustrating, isn’t it?  Curiosity––the scientists’ famous, “Hm, that’s funny…”––can be quite the motivator!  I appeal to authority:
​

​​The basics: When people encounter evidence that contradicts one of their strongly held beliefs, they experience an unpleasant feeling of cognitive dissonance, a sort of psychological pain. Cognitive dissonance has not only a negative affect—the unpleasant feeling—but also a motivational force, akin to the motivational force to eat that accompanies hunger. But cognitive dissonance doesn’t motivate us to eat (except when it does––meat-eating is often sustained through dissonance reduction); it motivates us to reconcile the contradiction in a way that palliates the negative feeling of dissonance.

Here’s an example. Suppose a kindly experimenter asks you to twiddle a knob for twenty minutes. After completing this boring task, the experimenter then tells you there’s another participant waiting outside, and asks whether you would please tell them that, contrary to the experience you just had, the task was actually fun. If you’re lucky, she offers you 100 dollars to do this, and if you’re unlucky, she offers you 1 dollar. After you tell the participant that the task was fun, you’re then asked for your true beliefs about how fun the task was.

Here’s the result: if you were paid 100 dollars, you think the task was boring and that you lied when you said it was fun. But if you were paid 1 dollar, you think the task was fun and that you were accurately relaying information to the other participant. (This description is adapted from a classic experiment.) 

Why should getting paid less money make you think the task was more fun? Since the task was really boring, acknowledging that fact means acknowledging that you lied when you said it was fun. If you were paid 100 dollars, it’s easy to admit that you’d tell a harmless lie for that much money. But if you were only paid 1 dollar, it’s harder to explain your own behavior without facing some harsh truths. Is it that you’re petty enough that you’ll lie for 1 dollar? Or suggestible enough that you’ll lie for 1 dollar as long as an authority figure requests it? Or oblivious enough that you were incorrect about your own judgment about the task?

There is an irrationalist explanation of effects like this. When faced with a contradiction between your beliefs about yourself––I’m good, I’m rational, I’m competent—and your knowledge of your own behavior––I just told a stranger that this boring task was fun because a psychologist offered me 1 dollar—you experience cognitive dissonance. (This is the generation of dissonance.) This negative feeling motivates you to push your attitudes around in a way that alleviates psychological discomfort. Often enough, the easiest way to do so is simply to change your belief so that your behavior accords with it. In this case, you change your judgment about the task: the task was really fun after all, so when you told that stranger that the task was fun, you were just honestly reporting your beliefs! No obvious negative implications about your personality follow, so you can rest easy. (This is the reduction of dissonance.)

This explanation might strike you as possible, but thoroughly non-obvious and maybe even overly complex. To be sure, alternative rational reconstructions abound. For instance, Daryl Bem (before his forays into ESP) argued that what’s really going on in experiments like this is that people don’t have direct access to their attitudes and must infer them from their behavior. So, since I told this person the task was fun and 1 dollar is not enough to make me lie, I must have done it because I really believe the task is fun. 

In a forthcoming paper called “Rationalization is Rational”, Fiery Cushman argues that we rationalize in order to work out what it would be most rational for us to believe given our behavior, thereby facilitating a form of learning he calls “representational exchange,” wherein we acquire conscious propositional attitudes that repackage the murkier unconscious sources of our original action in a form we can consciously act on. Even if I didn’t say the task was fun because I actually believed it, I rationally ought to adopt that belief anyway because it makes the most sense out of my previous action and is therefore likely to afford beneficial actions in the future.

Now it looks like we’ve ended up in the same position as ever: we have a pattern of belief change in response to evidence, and we can model it as rational or irrational. Fortunately, however, the irrationalist model we get from dissonance theory affords more concrete claims about the causal processes underlying this belief change. Specifically, according to the version of dissonance theory I articulated above, rationalization is self-serving. What I mean by this is twofold:

1. The immediate goal in rationalizing is to make a bad feeling go away—that is, we rationalize to make ourselves feel better, not to get at the truth or be more rational.
2. The broader purpose of rationalization is to avoid damage to core aspects of our self-image and thereby ward off persistent bad feelings (e.g., depression) and maintain stable motivation.

These two claims are clearly opposed to rationalistic models, since they assert that both the immediate aim and broader (possibly evolutionary) function of rationalization is not to acquire true beliefs, knowledge, or other such rationalistic goals. Instead, we push our beliefs around to avoid bad feelings with the broader purpose of maintaining self-esteem and motivation. 

Claims 1 and 2 also give us some empirical predictions: first, rationalization should be mediated by negative affect, and second, rationalization should be responsive to self-esteem. The evidence supports these claims. 

Evidence for Claim #1: When people are in dissonance experiments like the one mentioned above, they report feeling bad. Even more importantly, they show implicit marks of negative affect, such as measurable changes in electrical activity on the skin that signal stress as well as neural activation in regions linked to negative emotion. If these bad feelings are driving the belief change, then mitigating the feelings or making subjects think the feelings derive from an irrelevant source should mitigate the belief change as well. And that’s just what happens: drinking alcohol after dissonance is induced minimizes rationalization, and giving people a sugar pill and telling them it will make them uncomfortable makes them misattribute the negative affect of dissonance to the pill, and therefore makes them less likely to shift their attitudes.

Evidence for Claim #2: The evidence here is more indirect. But a simple prediction is that lowering people’s self-esteem (e.g., by giving them a bogus personality assessment with harsh results) should minimize dissonance-based rationalization. This prediction turns out to be true. And if, after dissonance is generated, you’re reminded of some of your good qualities (which are strictly irrelevant to the task at hand), rationalization is again minimized.

All this evidence suggests that the way we shift our beliefs in response to evidence of our own failings has an irrational character. We first feel bad in response to the attack on our self-esteem, and we then respond by unconsciously shifting our attitudes around in a way that alleviates the negative feeling and preserves our image of ourselves as good, rational, competent people.

My goal here is not to argue that the rational-analysis perspective has no way of modeling these phenomena (though Eric Mandelbaum makes that argument). Instead of testing the limits of rationalistic modeling, we can look “under the hood” at the actual processes driving belief change. Once we do, we find these effects are driven by negative affect and self-esteem. Irrationalists (of a certain sort) have a readymade explanation: we often update beliefs through rationalization, which is geared toward eliminating negative affect and has the purpose of protecting our image of ourselves and thereby maintaining stable motivation and avoiding depression, anxiety, and other maladaptive states of mind. A Bayesian might be able to rationally reconstruct the belief-updating patterns observed in dissonance experiments. But the challenge for them presented by dissonance-based rationalization is not met merely by developing a rationalistic model. Instead, they need also to provide an equally good explanation for why these changes in belief are (i) predictably motivated by negative affect and (ii) predictably responsive to self-esteem.

Now I want to circle back to the question of the “purpose” of belief updating. Kevin’s comparison to visual perception raises the challenge: why should we see rational degeneracy in reasoning when we (apparently) see rational optimality in vision and elsewhere? I think there is a good answer in the case of dissonance-based rationalization. 

Having persistent negative beliefs about yourself––I’m a bad person, I’m stupid, I’m incompetent—is plainly not conducive to mental health. Negative thoughts about the self, and the depression and lack of motivation they engender, are not adaptive. But it doesn’t take a scientific research program to show that human behavior routinely falls short of our ideals; our species is immoral, irrational, and incompetent as a matter of course. A perfectly optimal Bayesian updater saddled with human flaws is therefore at risk of adopting negative self-appraisals in ways that hamper healthy emotional functioning. Thus the rationalistic model of cognition itself, if true, creates a motivational problem that needs to be solved through non-rationalistic means.

The brand of irrationalism that I favor (owing to Gilbert and Mandelbaum) posits a psychological immune system to address this design flaw. Dissonance forms one mechanism in this immune system, and its purpose is to push our beliefs in a direction that allows us to maintain a positive self-image despite evidence to the contrary, and thereby keep us emotionally stable and functional. To fulfil this function, our minds generate cognitive dissonance when our otherwise rational belief-updating processes deliver conclusions that threaten our sense of self-worth, which motivates us to rearrange our beliefs in ways that defang the threatening evidence.

I doubt that this picture of cognition will settle the dispute between rationalistic and irrationalistic models of belief change. But it does put pressure on purely rationalistic models, and it moves the debate past competing models of input-output functions toward competing theories of the structure and function of causal cognitive processes.

It matters whether it’s true that rationalization has the structure I’ve described, and not just for theoretical purposes. Cognitive dissonance is sensitive to group membership—when our group is criticized, we experience dissonance and feel compelled to rationalize it away. This makes dissonance a plausible mechanism contributing to what Charles Mills calls “white ignorance”, including (e.g.) the persistent ignorance of white Americans of the continuing history of racism in the United States. Dissonance pushes people to preserve tribal allegiance as well as seek self-exonerating narratives when their morality is in question (as seen in the meat-eating literature). When these two factors are combined, rampant rationalization is likely.

Changing these tendencies requires understanding the psychological processes responsible for them. Unfortunately, doing so may force us to give up on a rationalistic picture of human nature.


What next?
If you want to hear more about the (potential) adaptive role of irrationality, check out Lisa Bortolloti's The Epistemic Innocence of Irrational Beliefs or––from a different angle––this paper on the evolution of overconfidence.
If you want to hear more about rationalization, check out Jake's full paper, the recent discussion piece by Fiery Cushman, "Rationalization is Rational", and the replies to it (including Jake's!), and recent articles by Jason D'Cruz and Eric Mandelbaum.

I'll spend most of this post defending this claim for koins, and then talk about how it generalizes to real-life random processes––like coins––at the end.

But first: why care? People don't appeal to the gambler's fallacy to explain polarization or to demonize their political opponents––so if you're here for those topics, this discussion may seem far afield.

But I think it's relevant. The irrationality and pervasiveness of the gambler's fallacy is one of the most widespread pieces of irrationalist folklore. It’s been taken to support a variety of unflattering views of the human mind, including a belief in the "law of small numbers", a tendency to use representativeness as a (poor) substitute for probability, an illusion of control, and even an (unfounded) belief in a just world.  Insofar as a general belief that people are irrational leads us to demonize those who disagree with us––as I think it does—scrutinizing such irrationalist claims is important.

So back to gamblers. What is the gambler's fallacy? Many have suggested to me that it's the tendency to think that a heads is more likely after a string of tails, despite knowing that the tosses are statistically independent. But this can't be right––for no one commits that fallacy. After all, knowing that the tosses are independent is just knowing that a heads is not more (or less) likely after a string of tails; therefore anyone who thinks that a heads is more likely after a string of tails does not know that the tosses are independent.

Here's a more plausible account of the (supposed) fallacy. You commit the gambler's fallacy if, purely on the basis of your knowledge that the koin lands heads 50% of the time, you think it's more likely to land heads after a (long string of) tails. That's what I'll argue is rational.

All you know about koins is that they tend to land heads about half the time. You can infer from this that on average––across all flips––the koin's chance of landing heads on a given toss is around 50%. What are the ways that this could be true?

​One (obvious) possibility is that the chance of heads is always 50%. Call this hypothesis:

Given your knowledge about koins, you should leave open that Steady is true.

Should you be sure it’s true? If so, then the gambler's fallacy would indeed be a fallacy. But you shouldn't be sure of it, for here are two other hypotheses that would also vindicate your evidence that koins tend to land heads around half the time:

​The Switchy hypothesis says that the koin has a tendency to switch how it lands. For example, perhaps after landing heads (tails), it's 40% likely to land heads (tails) on the next toss, and 60% likely to switch to tails (heads).  Similarly, the Sticky hypothesis says the koin has a tendency to stick to how it lands. For example, perhaps after landing heads (tails) it's 60% likely to stick with heads (tails) on the next toss, and 40% likely to land tails (heads).

We can represent hypotheses like Steady, Switchy, and Sticky with what are known as Markov chains: a series of states the koin might be in, along with its chance of transitioning from a given state at one time to other states at the next time.  For instance, our example of a Switchy hypothesis can be represented like this:

This diagram indicates that whenever the koin is in state H (has just landed heads), it's 40% likely to land heads on the next flip and 60% likely to land tails on the next flip. Vice versa for when it's in state T (has just landed tails).  We can similarly represent our Sticky and Steady hypotheses this way:

Given their symmetry, all of these hypotheses will make it so that the koin usually lands heads around half the time. (For aficionados: their stationary distributions are all 50-50.) Since that's all the evidence you have about koins, you should be uncertain which is true.

It follows from this uncertainty that, given your evidence, you should commit the gambler's fallacy: when it has just landed tails you should be more than 50% confident the the next toss will land heads; and vice versa when it has just landed heads.

Why? I'll focus on explaining a simple case; the Appendix below gives a variety of generalizations. 

Let's suppose you can be sure that one of the three particular Sticky/Switchy/Steady hypotheses in Figures 1–3 are true, but you can't be sure which.  Suppose you know that the koin has just landed tails (as it has).  Given this, you should be more than 50% confident that it'll land heads––you should commit the gambler's fallacy!  There are two steps to the reasoning.

First, you know that if Switchy is true, it has a 60% chance to land heads; that if Steady is true, it has a 50% chance to land heads; and that if Sticky is true, it has a 40% chance to land heads.  So if you were very confident in Switchy, you'd be around 60% confident in heads; if you were very confident in Steady, you'd be around 50% confident in heads; and if you were very confident in Sticky, you'd be around 40% confident in heads.  More generally, it follows (from total probability and the Principal Principle) that your confidence in heads should be a weighted average of these three numbers, with weights determined by how confident you should be in each of Switchy, Steady, an Sticky.

That is, where P(q)  represents how confident you should be in q, your confidence that the next flip will be heads given that it has just landed tails should be:

And you should be more confident in Switchy than Sticky––this is step two of the reasoning.
​
Why? Since you start out with no evidence either way, you should initially be equally confident in Switch and Sticky. And although both of these hypotheses fit with the observation that the koin tends to land heads about half the time, the Switchy hypothesis makes it more likely that this is so––and therefore is more confirmed than the Sticky hypothesis when you learn that the koin tends to land heads around half the time.  This is because Switchy makes it less likely that there will be long runs of heads (or tails) than Sticky does, and therefore makes it more likely the overall proportion of heads will stay close to 50%.

We can see this in action by working through a small example by hand, and through bigger examples on a computer.

Small example first.  Suppose all you know about the koin is that I've tossed it twice and it landed heads once. Why does Switchy make this outcome more likely that Sticky?

To land heads on one of two tosses is simply to either land HT or TH, i.e. to land one way initially and then switch. Switchy implies that such a switch is 60% likely, whereas Sticky implies that it is only 40% likely. (Meanwhile, Steady implies that it is 50% likely.)  Therefore Switchy makes the "one head in two tosses" outcome more likely than Sticky does.

It follows, for example, that if you were initially equally confident in each of Switchy, Steady, and Sticky, then after learning that it landed heads once out of two tosses, you should become 40% confident in Switchy, 33% confident in Steady, and 27% confident in Sticky.  Plugging these numbers into our above average shows that you should then be a bit over 51% confident that it'll switch again on the next toss––i.e. should commit the gambler's fallacy.

The reasoning in this small example generalizes.  The closer the koin comes to landing heads 50% of the time, the more ways there are to do this that involve switching between heads and tails many times; meanwhile, the closer the koin comes to landing heads 0% or 100% of the time, the fewer switches there could have been. Switchy makes the former sorts of outcomes more likely; Sticky makes the latter sorts of outcomes more likely. So when you learn that the koin tend to land heads roughly 50% of the time, this is more evidence for Switchy than Sticky––and as a result, you should commit the gambler's fallacy.

So far as I know, there's no tractable formula for determining these likelihoods by hand.  But since the systems are Markovian, we can use "dynamic programming" to recursively calculate the likelihoods on a computer.

For example, if we toss the koin 100 times we can plot how likely each our the three hypotheses would make various proportions of heads:

Note that although all three hypotheses generate bell-shaped curves centered around 50% heads, the Switchy hypothesis generates a tighter bell curve around 50% heads.

This is the crucial point. Take any precise statement of what you know about koins––namely, that they "land heads around half the time". A precise version of that claim will take the form "the koin lands heads between (50 – c)% and (50 + c)% of the time" for some c. (For example, "the koin lands heads between 48% and 52% of the time.") Switchy generates a higher bell curve that Sticky, meaning it makes any such claim more likely than Sticky does––and therefore is more confirmed by what you know than Sticky is.

For example here's how likely each of our three hypotheses would make it that the koin lands heads "roughly 50" times out of 100 tosses, under various sharpenings of the claim:

Since Switchy makes each of these more likely than Sticky, learning that the koin lands heads "roughly 50" of 100 times provides more evidence for the former.

​In particular, if you started out ⅓ confident in each of Switchy, Steady, and Sticky, here's how confident you should be in them after updating on various versions of these "roughly 50" claims, along with the resulting you should have that it'll switch on the next flip:

In each case, since you should be more confident in Switchy than Sticky, you should perform the gambler's fallacy.

That said, as has been helpfully pointed out in the comments, as you observe a long string of tails, this provides evidence for Sticky––so sooner or later, that a long enough streak will make it so that you are no longer more confident in Switchy than Sticky.  How quickly this will happen will depend on exactly what version of "the koin tends to land heads roughly half the time" you know beforehand.  If all you know is "it landed heads on 50 of 100 tosses", a short streak of tails will dislodge your confidence in Switchy, and in fact make it rational to perform the "hot hands" fallacy and expect a tails to be more likely to follow a tails (see the discussion in the Appendix for more on this).

But for some versions of "the koin tends to land heads roughly half the time", your confidence in Switchy will be much more robust.  Here's one version that's not an implausible characterization of what people often know about processes like this.

Suppose what you know about koins is: "on every set of tosses I've seen, it's landed heads around half the time––sometimes very close to 50%, sometimes a bit further. I can't remember the details, but it's always been between 40–60%, usually between 45–55%, and often between 48–52%". If this is what you know, then every one of those sets of tosses provides more evidence for Switchy over Sticky, meaning your confidence in Switchy will be quite robust.

For example, suppose you started out ⅓ in each hypothesis and then learned that in 10 sets of 100 tosses each, each set had between 40–60 heads, 7 of them had between 45–55 heads, and 4 had between 48–52 heads. Then you should become 72% confident it's Switchy, 22% confident it's Steady, and 6% confident it's Sticky (see the first "full calculation" section of the Appendix).  As a result, you can see a string of up to 7 tails in a row (with no Switches), and still be more confident in Switchy than Sticky––and, therefore, still commit the gambler's fallacy.

The Fallacy in Real Life
That's what we should say about the gambler's fallacy with koins: it's rational. What should we say about the gambler's fallacy in real life?

I think we should say the same thing.  Most people don't––shouldn't––be sure of how the outcomes from (most of) the random processes they encounter are generated. Many of these outcomes plausibly are either Switchy or Sticky––for example, whether it rains on a given day, or whether a post on Twitter will get significant uptake, or whether the next card drawn from this deck is a face card.  Many others are at least open to doubt.  

So people––especially those who haven't taken statistics courses––should often leave open that various versions of the Sticky and Switchy hypotheses are true. And since they don't (can't) keep track of the full sequence of outcomes they've seen, what they know about the processes is often much more coarse-grained––e.g. that a given outcome tends to happen around 50% of the time. (See the Appendix for generalizations to other percentages.)

As we've just seen, if that's what they know then they are rational to commit the gambler's fallacy.  Instead of revealing a basic misunderstanding about statistics, such a tendency may reveal a subtly tuned sensitivity to statistical uncertainty.

Of course, this doesn't show that the way real people commit the fallacy is rational: they might commit it for the wrong reasons, or in too extreme a way.  (See Brian Hedden's post on hindsight bias for a discussion for how we might probe those questions––and why it is difficult to do so.)  But the mere fact that people commit the gambler's fallacy does not, on it's own, provide evidence that they are handling uncertainty irrationally––after all, it's exactly what we'd expect if they were being rational.

Objection: What about coins? Obviously coins have no "memory", so when it comes to coins, people should be certain that hypotheses like Switchy and Sticky are false, and instead be certain that Steady is true.

Reply: Should they? Should you?  Real coins are much more surprising than statistics textbooks would lead you to think. For example, despite the ubiquity of the notorious   "coin of unknown bias", it's actually impossible to bias a coin toward one of its sides. Perhaps more surprisingly––and more to the point––it turns out that the way real people tend to flip coins leads them to have around a 51% chance to land the side that was originally facing up. So depending on the procedure you use for flipping your coin repeatedly (do you turn it over, or not, when you go to flip it again?), Steady may actually be false and some version of Switchy or Sticky true!

Given subtleties like that, it's rather implausible to insist that someone who has never taken a statistics course nor studied coins in any detail should be certain that hypotheses like Sticky and Switchy are false about real coins, or other more complex gambling mechanisms.  As we've seen, so long as they shouldn't be certain of that, they should commit the gambler's fallacy.

Conclusion
Given people's limited knowledge about the outcomes the random processes they encounter and the statistical mechanisms that give rise to them, they often should commit the gambler's fallacy. So the mere fact that they exhibit this tendency should not be taken to show that they handle statistical uncertainty in an irrational way––if anything, it's evidence that they're handling it as they should!  At the least, we need more detailed information about the way and degree to which people commit the gambler's fallacy for it to provide evidence of irrationality.


What next?
If you have comments, questions, or criticisms, please comment or email me! As I said, this is work in progress.
If you want to see more details, check out the Appendix below.
For more recent work on the gambler's and "hot hands" fallacy, see this fascinating recent paper.

Appendix
Here I’ll give some generalizations I’ve worked out, and some discussion of the robustness of these results.
​
The Full Calculation
Some people have questioned whether the results holds even in the simple case I focus on in the text, so I figured I'd work through the calculations to show that they do.

Take the versions of the Switchy/Steady/Sticky hypotheses we used above. Suppose you are initially ⅓ confident in each: P(Switchy) = P(Steady) = P(Sticky) = ⅓.

Now suppose you learn that 50 of 100 tosses landed heads ("50H"). The likelihoods of this given each of the hypotheses are those from Figure 4, reproduced here:

It follows from total probability that your confidence should be the following weighted average:

It follows from the Principal Principle that P(H | Ch(H) > 0.5) > 0.5 and that P(H | Ch(H) < 0.5) < 0.5.  In our situation you have no reason to treat these two options asymmetrically, so there should be some constant c such that P(H | Ch(H) > 0.5) = 0.5 + c, while P(H | Ch(H)<0.5) = 0.5 - c.  It follows that:

And again, this value will be greater than 50% iff P(Switchy) > P(Sticky).

The trick with using these definitions is that we now need to be careful about what the plausible versions of the Sticky/Switchy hypotheses amount to. We can no longer simply assume they are the 40%-60% hypotheses (from Figures 1–3) I assumed above, so we can’t straightforwardly calculate the likelihoods of various outcomes given Sticky and Switchy. Nevertheless, the plausible versions of these hypotheses will have the same general shape, although some will be more or less extreme in their divergences from 50% probabilities, some may have longer “memories” so that it takes longer streaks to reach these divergences, and so on. See below for direct handling of some of these issues.

Robustness
Since I have no tractable algebraic expression for the likelihoods generated by various Sticky/Switchy hypotheses—even in the simple cases—there are limits on what I can prove about it.  (Hunch: what matters for the difference in likelihoods between Switchy and Sticky is that the former has a shorter mixing time than the latter; perhaps that can be used in a proof? Any mathematicians out there to help a philosopher out?)

Nevertheless, it’s easy to check that these results are robust. For example, here are the likelihoods for various proportions of heads from the three simple hypotheses at 10, 50, 100, and 500 tosses. Clearly we are (quickly) approaching a limit in the ratios of likelihoods of 50% heads, and the differences are not washing out.

And here are graphs that plot the likelihoods considering Switchy and Sticky hypotheses with different levels of probability of sticking or switching at 20, 50, 100, and 300 tosses (for example, Switchy (0.7) has 70% chance of switching; Sticky (0.4) has 40% chance of switching; etc.):

​Longer “Memories”
The explicit versions of the Sticky/Switchy hypotheses we’ve looked at so far all had “memories” of only size 1—the probabilities of outcomes only depend on how the last toss landed.  But both intuitively and empirically, people are much more likely to commit the gambler’s fallacy (or "hot hands fallacy"—see below) with long streaks of outcomes.  It’s only when tails comes up 4 or 5 or more times in a row that people start to expect a heads.

​This can be modeled easily, simply by multiplying the states in our Markov chain.  Instead of simply H and T, they will now include how long the streak of heads or tails has been, with the probabilities shifting gradually as the streak builds up.  For example, here are diagrams representing 2-memory Switchy and Sticky hypotheses, where the probabilities build to a 60% chance to stick or switch, in both diagram and transition-matrix notation. (For the matrix, row i column j tells you the probability of transitions from state i to state j.) For example, the Switchy hypothesis says that after one heads, it's 55% likely to switch back to tails, and after two or more heads in a row it's 60% likely to switch to tails.

And though I'm not going to try to draw the 10-state diagram, here's the transition matrix for a 5-memory Switchy hypothesis that grows steadily to a 60% switch rate.

As you’d expect, the qualitative results from these hypotheses are the same as before, but (very) slightly dampened.  For example, here are the likelihoods of various outcomes of 100 tosses from our original 1-memory 60% hypotheses (reproduced from Figure 4), vs. the likelihoods of outcomes with the 2-memory, 3-memory, and 5-memory 60% hypotheses:

It’s not until we get "memories" of size 10 or more that we start to see significant dampening of the divergence of likelihoods:

And it's worth noting that the qualitative results will be the same in all these cases, though the degree of gambler's fallacy warranted will decrease as the differences in the likelihoods get smaller.

​It seems, empirically, that the versions of the Sticky and Switchy hypotheses that people take seriously are in the 5- to 10-memory range.  For robustness checks, I'll show the likelihoods at 10, 50, 100, and 500 tosses for various 5-memory hypotheses whose probabilities move at constant increments up to a given extreme; for example "5-memory Switchy (0.7)" is the chain that takes 5 steps to become 70% likely to switch, and "5-memory Sticky (0.3)" is the chain that takes 5 steps to become 30% likely to switch:

Here are the robustness checks for these and other hypotheses at 10, 50, 100, and 500 tosses:

Upshot: the qualitative results will be the same under these various more realistic versions of the hypotheses.

Hot Hands
The “hot hands fallacy” is the tendency to think that an outcome is “streaky” in the sense that if a given outcome happens, it is more likely that it’ll happen again on the next trial.   In that sense, it's the opposite of the gambler's fallacy: where gambler's expect things to switch, hot-handsers expect things to stick. (The issue from basketball; see this recent paper for a fascinating discussion of why there were statistical mistakes in the original papers claiming to show that there is not "hot hand" in basketball.)

We saw above that when P(Switchy) > P(Sticky), the gambler’s fallacy is rational, and you should be more than 50% confident that the koin will switch how it lands between tosses.  By parallel reasoning, whenever P(Switchy) < P(Sticky), it follows that you should be less than 50% confident that the koin will land differently to how it did before—i.e. you should be more than 50% confident that it will land the same way. In other words, whenever P(Switchy) < P(Sticky), you should commit the hot hands fallacy!

Upshot: the only time when you should commit neither the gamblers fallacy nor the hot-hands fallacy is when you should be exactly equally confident in Switchy and Sticky: P(Switchy) = P(Sticky). Since such a perfect balance of evidence will be rare, you should almost always commit one of these “fallacies” (though perhaps to only a very small degree).

In particular, suppose you start out equally confident in each of Switchy and Sticky, and then learn what proportion of times the koin landed heads in some series of tosses.  For example, return to our 100-toss example (Figure 4) with the 40/50/60 hypotheses, and recall the likelihoods:

After learning the proportion of heads out of 100 tosses, should be more confident of Switchy than Sticky iff the blue curve is higher than the green one for the outcome (proportion of heads) you observe, and less confident if vice versa. In fact, there is no outcome of a 100-toss sequence that would make these exactly equal, so given any outcome, you should either commit the gambler’s or the hot-hands fallacy on the next toss.


How can you learn it’s Steady?
You might think we’ve run ourselves into bit of a paradox here.  Note that in all the graphs I’ve shown, the Steady likelihoods almost never come out ahead overall.  In the middle of the graphs, they are dominated by the Switchy likelihoods, and in the edges of the graph, they are dominated by the Sticky hypotheses.  This remains true as we crank of the experiment to arbitrarily many tosses of the koins. 

So… what gives? Does our reasoning show that it’s impossible to learn, by tossing a koin, that it is Steady?  If so, it’s gone wrong somewhere.

But it doesn’t show that. What it shows is that if all you learn is the proportion of heads, you won’t be able to get strong evidence that the koin is Steady.  To get that evidence, you’d really need to look closely at the sequences you observe. There Switchy hypotheses will make streaks very unlikely, while Sticky hypotheses will make repeated flips unlikely, and the Steady hypotheses will strike a balance.  If you looked at the full sequence for long enough, you’d  almost surely (in the technical sense) get to the truth of the matter about whether the koin is Sticky, Steady, or Switchy.

But what we have shown is that without that full data—with only tracking the proportions of heads—people will actually not be able to figure out whether the koin is Sticky, Steady, or Switchy. That is an interesting result. Because real people can’t keep track of full sequences of tosses--at best they can keep track of (rough) proportions.  What our results do show is that given only that information, even perfect Bayesians wouldn’t be able to figure out whether the koin is Steady (or Sticky or Switchy). 

Non-50% versions
Every version we’ve looked at so far is one where the number of heads stays around 50%. This is apt for coin tosses, but not so for other chancy events like basketball shots or drawing a face card.  We’ll need to generalize what plausible Sticky and Shifty hypotheses look like for processes where the average number of heads (or “hits”) differs from 50%. For example, in the NBA—where the hot hands fallacy discussion is at home—shooting percentages are often around 45%. 

The reasoning generalizes, but it gets a bit subtle. In the 50% case, all my examples assumed “symmetry” in the sense that the probability added (or subtracted) to getting a heads when it just landed heads is the same as that subtracted (or added) to getting a heads when it just landed tails. 

This isn’t the right version of symmetry when the Steady hypotheses is no longer 50%. For example, suppose the Steady hypothesis is that no matter how it’s landed, the koin has a 40% chance to land heads on each toss.  Then we expect that in the long run, it’ll land heads in 40% of all tosses.  So here's our Steady hypothesis:

You might think natural Sticky and Switchy hypotheses centered around this would simply add or subtract a fixed amount (say, 0.1) to the probabilities depending on the state, as before:

But that’s wrong.  The “stationary distribution” of these two Markov chains is not 40/60—rather, for the Sticky hypothesis it’s around 38/62 and for the Switchy one it’s around 42/58, meaning that in the long run we would expect them to have these proportions of heads to tails, rather than 40-60.  Accordingly, the likelihood graphs are not properly overlapping:

The proper Sticky/Switchy hypotheses if the overall proportion of heads is 40% are ones whose probabilities move depending on where they are, but in a way that leads them to have the same stationary as the Steady hypothesis. I haven’t yet figured out a general recipe for this (any mathematicians care to help?), but here is an example of 1-memory Sticky and Shifty hypotheses that have the correct stationaries:

And here are the likelihood graphs for 100 tosses of these two hypotheses vs. the 40%-heads Steady hypothesis:

Upshot: the same qualitative lessons hold for processes that don't come up "heads" 50% of the time; if all you know is that "roughly x% of the tosses land heads", you should be more confident in (the right version of) Switchy than Sticky, and so should commit the gambler's fallacy.

​

...Phew! Those are all the generalizations and further notes I have (for now). If you have any thoughts or feedback, please do send them along!  ​Thanks!

Of course, the fact that hindsight bias is compatible with––and sometimes even entailed by––models of ideal rationality doesn’t mean that we humans are rational when we display hindsight bias. We might go too far, or base our hindsight bias on bad reasons. And so I’ll close by considering how we might attempt to test whether, when actual humans display hindsight bias, or other alleged biases for that matter, we are doing so in the rational way I identify or are instead doing so in some other irrational way. 

Hindsight Bias 
Consider a medical case. The doctor had evidence consisting of X-rays and blood samples, and on that basis had to reach a conclusion about how likely it was that the patient had a tumour. Hindsight bias means that you’ll estimate that very evidence, the evidence that was available ex ante, as more strongly supporting the existence of a tumour if you know that the tumour’s existence was later confirmed than if you don’t. 

Or consider the case of an accident. The railroad corporation had evidence concerning the safety of its tracks, including a variety of models and records of track inspections. Hindsight bias means that you’ll estimate that evidence as supporting a higher probability of train derailment if you know that a train in fact derailed than if you don’t. 

It’s tempting to think that the judgments you give when you have the benefit of hindsight are irrational and tend to overestimate the degree to which the ex ante evidence supported the relevant hypothesis – that the patient had a tumour, or that a train would derail. The idea is that it’s somehow unfair to take into account evidence that wasn’t available ex ante (namely, evidence that the hypothesis is in fact true) and have it influence your judgment about the upshot of the ex ante evidence. Hindsight is 20/20, but it’s irrational to thereby think that foresight should have been 20/20 too. 

After all, evidence can be misleading or ambiguous, so why not just think that that’s what happened here? That is, why not think that the ex ante evidence didn’t strongly support the diagnosis of a tumour or the prediction of a train derailment, even though that diagnosis and prediction would have in fact been correct?

​As these cases suggest, hindsight bias is of particular importance in the context of negligence lawsuits. In such cases, factfinders need to determine whether the defendants (the doctor or the railroad corporation) took reasonable steps in light of the evidence they had available. If hindsight bias leads those factfinders to overestimate the degree to which the ex ante evidence suggested a need for action (action which wasn’t taken by the defendants), then it will likewise lead them to tend to judge the defendants negligent when in fact they weren’t. 

Lower-Order Evidence
Above I gave a quick-and-dirty argument for thinking that hindsight bias is irrational. Suppose that, before learning whether H was true, you rated the ex ante evidence as not very strongly supporting H. (In jargon, your expectation of the degree to which that evidence supported H was low.) Then you learn that H is true. Well, evidence can be misleading, and so the fact that H is in fact true still leaves open the possibility that the ex ante evidence didn’t strongly support H. So perhaps you shouldn’t change your view about how strongly the ex ante evidence supported H and should instead just conclude that flukes happen. 

This is a bad argument. We can see that it’s a bad argument by considering an analogous case involving coin flips (epistemologists love coin flips!). Suppose that, before tossing a mystery coin of unknown bias, you think it’s probably biased towards tails, though you also think there’s some chance that it’s biased towards heads. (In jargon, your expectation of the coin’s bias towards heads is below 0.5.) 

Then you toss it and see it land heads. Should you change your expectation of the bias of the coin’s bias towards heads? Reasoning analogous to the above would say ‘no.’ After all, even if the coin is biased towards tails, that still leaves open the possibility that the coin would land heads; so instead of raising your expectation of the coin’s bias towards heads, you should still to your original view and just conclude that this toss was a fluke. 

But that’s clearly wrong! Any Bayesian will tell you that upon seeing the coin land heads, you should increase your expectation of the coin’s bias towards heads. Here’s why. The conditional probability of heads given that the coin is biased towards heads is higher than the unconditional probability of heads (that is, the probability of heads given your antecedent views about the coin): P(H|Heads-Biased)>P(H). And positive probabilistic relevance is symmetric, meaning that from the above inequality it follows that P(Heads-Biased|H)>P(Heads-Biased). Similarly, the conditional probability of heads given that the coin is biased towards tails is lower than the unconditional probability of heads: P(H|Tails-Biased)<P(H). It follows by symmetry that P(Tails-Biased|H)<P(Tails-Biased). So upon learning H (i.e. upon seeing the coin land heads), you should be more confident that the coin is biased towards heads and less confident that it is biased towards tails. 

The analogy is clear: Hypotheses about the bias of the coin are like hypotheses about ex ante evidential support. And seeing how the coin landed is like gaining the benefit of hindsight. Just as seeing the coin land heads should increase your expectation of the coin’s bias towards heads, so learning that some other hypothesis is true should increase your expectation of the degree to which the ex ante evidence supported that hypothesis. 

Now, you might think that the coin flip case is importantly different from the case of hindsight bias. Hypotheses about the bias of the coin are contingent and knowable only a posteriori. But hypotheses about evidential support (or, at least, hypothesis about fundamental evidential support, i.e. about how strongly some body of evidence on its own supports some hypothesis) are necessary and knowable a priori. So while you can’t come to know the bias of a coin just by thinking really hard about the coin, you can come to know the how strongly some body of evidence supports some hypothesis just by thinking really hard about that evidence. Ideally rational agents might be uncertain about the bias of some coin, but they won’t ever be uncertain about how strongly any body of evidence supports any given hypothesis. 

That’s a natural picture, but it’s now widely rejected by epistemologists. Even if the fundamental evidential support facts are necessary, ideally rational agents can and often should be uncertain about them. So when two people examine the same exact evidence and come to different and conflicting views about some hypothesis, neither should be certain about how strongly that evidence supported the hypothesis, even if one of them happened to have initially judged things correctly. And this higher-order uncertainty about the fundamental evidential support facts isn’t inert; instead, it should impact your views about first-order matters. Becoming more confident that your evidence in fact supports H should in general make you more confident in H itself. (Note, however, that it’s a very tricky issue how exactly this should go.) So if you initially judged that the evidence supported low confidence in H, but then you learn that your peer thought it supported high confidence in H, you should yourself increase your confidence that the evidence supported high confidence in H, and you should increase your confidence in H itself on that basis. 

On this picture, the case of the coins is in fact analogous to the case of hindsight bias. It’s a picture on which higher-order evidence and higher-order uncertainty (that is, evidence and uncertainty concerning relations of evidence support) have first order consequences, just as evidence and uncertainty about the bias of a coin has consequences for your views about how it will land. Given that, the symmetry of positive probabilistic relevance means that learning those first-order facts has higher order consequences for your views about evidential support, just as learning how the coin lands has consequences for your views about its bias. 

Higher-order evidence, and how to respond to it, is a hot topic in epistemology these days. I’ve shown that recognising the importance of higher-order evidence, and its obverse, lower-order evidence, allows us to see that so-called hindsight bias is often perfectly rational. And the significance of higher-order evidence may help to cast other alleged biases in a new light, sometimes showing that they too can be perfectly rational. 

Testing for Irrationality
While I claim that hindsight bias can be perfectly rational, I don’t claim that humans are always (or even usually, or often) perfectly rational when they display hindsight bias. Indeed, my own suspicion is that real-world hindsight bias is still often irrational. Humans may go ‘too far’ with their hindsight bias, using hindsight to change their views about the upshot of the ex ante evidence to a greater extent than is warranted by the above analysis. And even when humans display just the right ‘amount’ of hindsight bias, they may do so for bad reasons. Rather than displaying hindsight bias as a rational response to lower-order evidence, they may do so on the basis of evidentially irrelevant motivational factors like the need for closure or the desire to see themselves as experts. 

Much the same goes for other alleged biases that have been defended by philosophers. Tom Kelly, for example, argues that some instances of the sunk cost fallacy may in fact be based on perfectly rational ‘redemptive preferences’ whereby the agent is concerned with the narrative arc of her life, preferring a life filled with projects seen through to fruition rather than a bunch of false starts. But he doesn’t claim that all instances of this sort of behaviour are rational in this way. 

Could we run tests to determine the extent to which humans display hindsight bias or sunk cost reasoning irrationally? I am not sure, but I think it would be very difficult and require rather intricate experimental design, not to mention lots of theoretical work to get the normative models right. 

There are two reasons for my scepticism. First, if we assume that it’s always irrational to display some pattern of behaviour to any extent, then things are easy to test. If we find that people ever display that behaviour, then that’s it – end of story. In the case of hindsight bias, if we assume that your judgments about the upshot of the ex ante evidence shouldn’t differ at all depending on whether you know what wound up happening, then if we detect any such difference, no matter how small (provided it’s still statistically significant), then that suffices to show that you’re irrational. There’s a sharp, empirically detectable distinction between ‘no difference’ and ‘some difference.’ But if we assume that your judgments should differ to some degree depending on whether you know what wound up happening, then we can’t determine whether they differ ‘too much’ unless we know exactly how much they should differ. But I think it might well be impossible to say ‘how much’ hindsight bias you should display in any realistic case. 

Second, we might try to determine whether humans display hindsight bias irrationally by determining whether they do so on the basis of good reasons or bad. Do they display hindsight bias in virtue of responding to higher- and lower-order evidence, or on the basis of a need for closure and self-esteem? We might try just asking them for their reasons, but we should be very sceptical about people’s capacity to introspectively determine their own motivations. Perhaps some more complex experimental designs could help determine their motivations, or at least ensure that any bias they display couldn’t be on the basis of the good reasons that I identify. There may be no in-principle reason this couldn’t be done, but at the same time it’s certainly no easy task. 


What next?
If you want to hear more about the details, check out Brian's paper, "Hindsight Bias is Not a Bias."
If you want to learn more about the empirical work on hindsight bias, see this seminal paper by Baruch Fischoff, as well as this recent overview of the literature.
For more on the importance of normative models psychological research, see this paper by Ulrike Hahn and Adam Harris on motivated reasoning.

Why might we make such an error? Kahneman and others have suggested that positive or negative language (as when we hear that a basketball player ‘made’ or ‘missed’ shots) can colour our judgement. This is thought to happen purely by association, perhaps as an emotional response that bypasses our reasoning capacities. 
​
This interpretation implies that we process information fairly shallowly, just responding to surface features (the particular words used) and not thinking through the underlying facts (what’s actually being said with these words). 

Recently, though, a different view of framing effects has begun to emerge. Several theorists have argued that we are right to respond differently to alternative frames. The advocates of this view aim to provide a rationalising explanation of framing effects. 

The idea I’ll focus on here is that frames ‘leak’ information beyond what is explicitly conveyed – a proposal developed here. The proponents agree that the basketball player made 40% of his shots and missed 60%, regardless of which frame is used. But, it is argued, by choosing to describe the player as having ‘made 40%’, some extra information is implicitly conveyed. 

What is this extra information? 

Well, think about how the frame affects your expectations about typical high school players. Do you expect them to make more or less than 40%? How does that change if I tell you the player ‘only made 40%’? 

According to the reference point hypothesis, describing the player as having ‘made 40%’ conveys that the player made relatively many shots – compared, say, to the average high school player. On the flipside, saying that the player ‘missed 60%’ of his shots conveys the opposite information – that the player missed relatively many shots. 

As you might have spotted, it’s possible to switch these expectations with ‘only’. When we hear that the player ‘only made 40%’ we expect that he should have made more. But when we hear that he ‘only missed 60%’ it sounds like he’s a relatively high scorer.

Now, if this theory is right, it would explain why the player is judged to be better under the ‘40% made’ frame than under the ‘60% missed’ frame. Under the ‘40% made’ frame, he is assumed to be a high performer, making more shots than average. In contrast, under the ‘60% missed’ frame, the player is assumed to be a poor performer, making fewer shots than average. 

Importantly, it turns out that we’re right to make these assumptions. The extra inferences we make are actually tracking how speakers choose their words (as shown by studies reported here, here, here, here, and here). For example, when a basketball player made more shots than average, a speaker is more likely to talk about how many shots he ‘made’ rather than how many he ‘missed’. In other words, the speaker’s choice about whether to use ‘made’ or ‘missed’ is a pretty reliable guide to how well the player performed. And it’s perfectly reasonable for our evaluation of the basketball player to be sensitive to this. On the ‘information leakage’ account, then, the framing effect is perfectly rational. Going back to gift-wrapping analogy, if people tend to wrap better gifts in red paper, then you should pay attention to the colour of the paper!

It’s worth emphasising that, according to the ‘information leakage’ account, we fully process the information about how many shots the basketball player made and missed. It is just that we also make an extra inference about whether that represents relatively good or bad performance. In this way, framing effects are seen as evidence of deeper, not shallower, processing. 

Suppose this is the right analysis. That raises a question: what is the status of the additional information ‘leaked’ by frames. Is it part of the meaning of our words, or something we work out using non-linguistic information?

In this draft paper, I argue that the information should be captured under the broad umbrella of pragmatics, as ‘conversational implicatures’. These are all kinds of extra bits of information, which we glean by ‘reading between the lines’ of what people explicitly say. The concept of an implicature was first introduced in a classic philosophy paper by Grice on ‘Logic and Conversation’ (published in a collection of his work here).

Here’s an example: imagine you ask me whether I read your article. I report to you: ‘I did put the article in front of me and I moved my eyes from left to right along each line’. You’ll probably infer that I didn’t actually read it – perhaps I was bored or distracted. But why do you draw that conclusion? After all, reading (in English) usually involves precisely the process I described. 

The answer is that I should simply have said ‘yes’ if that’s what I meant. By replying in such a convoluted way, I implicitly convey that, although I made some attempt to read the article, I didn’t manage to do so. This information is known as an ‘implicature’ of my utterance.

I suggest that framing works a bit like this. If I tell you that a basketball player ‘missed 60%’ of his shots, you’ll probably infer that he’s a poor performer. Why? Because I could have said he ‘made 40%’ but I chose not to do so. Again, what I chose not to say speaks volumes.

If my analysis is right, it suggests that a rational explanation of framing effects was always available, we just needed to appreciate Grice’s insight that the information we communicate to each other can far exceed what we say explicitly. In particular, a speaker’s choice of frame carries implicit information that justifies our sensitivity to what initially seems to be an entirely inconsequential feature of the utterance.

If the case of the basketball player is anything to go by, we shouldn’t automatically jump to the conclusion that framing effects are irrational. We have good reasons to judge the player differently under each frame. Having said that, the research on framing is complex and wide-ranging (sometimes overwhelmingly so!). Various other studies look at the framing of choices, questions, policies, and goals. Some of these investigate not just how information is phrased, but also the order in which it is presented, whether it is in spoken or written form, and all sorts of other possibilities. Different lines of research explore whether someone’s susceptibility to framing effects depends on their age, gender, cognitive ability, or even their political affiliation. It remains an open question whether all the findings in this vast literature can be given a rational explanation. I don’t know the answer to that. But, for now, I’m keeping the glass half full!


What next?
If you want to read more about rational interpretations of framing effects, check out Sarah's website, this 2013 paper by David Mandel, and stay tuned for future posts.

Reds talk to Reds; Blues talk to Blues. What happens? The two groups pull apart:

That’s the group polarization effect. It’s especially strong when there is substantial variation and uncertainty in people’s initial opinions, as we’ve seen there will be with Covid. Thus small variations in the initial reactions of social circles will ramify, leading to substantial group-level differences.

​When we add to this the well-documented social divides between Democrats and Republicans, it’s no surprise that systematic differences would emerge between the groups. Moreover the direction of those divergences (with Republicans less concerned than Democrats) is no surprise either, given that the news sources frequented by the two parties had differing initial reactions to Covid.

In short: Given partisan social separation and some initial nudges from the news, the complexity of Covid-information combined with the group polarization effect to sweep the parties apart. That is my story for the prime driver of pandemic polarization. 

Suppose we accept it. Then we face an evaluative question: What are we to make of it? Should we think of the members of each party as rational or irrational? And—closely related—when people disagree with us about the seriousness of Covid, should we blame them for doing so?  

I think we should not. 

(To be clear, I’m thinking of ordinary party members—rather than political leaders—since leaders’ reactions are subject to very different norms and forces.)

Where would you place the relative positions of the wine glasses? Equally close, no doubt. (And you’d be correct.) But strictly speaking that is underdetermined by the picture––you’d see the exact same pattern of light if instead of two normal-sized wine glasses equally close, one of them were bigger and further away.  (In more complex scenes, more complex alternatives would do the same trick.)

In other words, one of the core problems your visual system faces is that you cannot deduce with certainty the correct 3D map of your surroundings from the 2D projection of them onto your eyes.  And yet––somehow––you do manage to come to the right conclusions, virtually every waking minute.  

How do you do it?

The Solution: The arguably-most-successful models of how you manage to do it fall within the class of Bayesian “ideal observer” models. These models ask, “What would an ideally rational agent who optimally used incoming information conclude about the visual scene?” These models use literally the same formalism as the standard model of rational belief that is so maligned by psychologists working on the judgment system––more on that in a moment.

​Regardless of the details, one thing that is certain is that your visual system uses incredibly sophisticated computations to solve this problem––ones that easily avoid the mistakes that the most sophisticated computer vision systems still routinely make. (And they still make quite laughable mistakes––here’s how a state-of-the-art computer vision program classified the following picture:

For other funny examples, see this Twitter feed or page 16 of this paper.)

The Narrative: In short, our best science tells us that to solve the ill-posed but all-important problem faced by our visual system, our brains have evolved to implement (approximations of) the rationally optimal response to incoming information. Of course, sometimes these mechanisms lead to systematic errors––as in the famous Muller-Lyer illusion, wherein lines of the same length appear to differ due to the angles attached to them:

The Challenge 
The contrast between these two narratives is striking––all the more so since the basic contours of the problems are so similar.

I think we can use this contrast to buttress an ”evolutionary-optimality argument” in favor of rational explanations. (The gist of such arguments are far from original to me––they often help motivate the “rational analysis” approach to cognitive science that has been gaining momentum in recent decades.)

The argument starts with Quine’s famous quip: “Creatures inveterately wrong in their inductions [i.e. judgments based on limited evidence] have a pathetic but praiseworthy tendency to die before reproducing their kind” (Quine 1970, 13). In other words, we have straightforward evolutionary reasons to think that people must make near-optimal use of their limited evidence in forming their beliefs––for otherwise their ancestors would easily have been outcompeted by others who made better use of the evidence.

Of course, that grand conclusion is both too grand and too quick. There are  several reasons the argument could go wrong––here are three important ones:

1. Perhaps the mistakes people make are not important enough to be corrected by evolutionary pressure.
2. Perhaps it was too difficult for evolution to hit upon reasoning methods that avoided these mistakes.
3. Perhaps the reasoning methods people use were near-optimal in our evolutionary past, but are no longer effective today.

My Claim: Although these are all legitimate concerns, they become less compelling once we see the contrast between the narratives on vision and judgment.

First, since few defenders of irrationalist explanations will say that the errors they’ve identified are uncommon or unimportant, option (1) is a unpromising strategy. After all, much of the interest in this research (and the way it gets its funding!) comes from the fact that the reverse is supposed to be true.  Some representative quotes:

• “What would I eliminate if I had a magic wand? Overconfidence.” (Kahneman 2015)
• “No problem in judgment and decision-making is more prevalent and more potentially catastrophic than overconfidence” (Plous 1993, 213). 
• “If one were to attempt to identify a single problematic aspect of human reasoning that deserves attention above all others, the confirmation bias would have to be among the candidates for consideration... it appears to be sufficiently strong and pervasive that one is led to wonder whether the bias, by itself, might account for a significant fraction of the disputes, altercations, and misunderstandings that occur among individuals, groups, and nations.” (Nickerson 1998, 175)

What about option (2)––the proposal that rational Bayesian solutions to the problems faced by the judgment system were too difficult for evolution to hit upon? The narrative from vision science casts doubt on this: the visual system has hit upon (approximations of) the rational Bayesian solutions, suggesting that such solutions were evolutionarily accessible.

Of course, there is much more to be said about potential asymmetries between the evolutionary pressures on the visual and judgment systems––e.g. in terms of time-scales (how long have these systems been under such pressure?) or domain complexity (are optimal solutions harder to hit for judgment than for vision (Fodor 1983)?––Tyler Brooke-Wilson has some interesting work on this). But that way lies the sequence of resistance to resistance to resistance… which none of us have time for. The simple point that Bayesian solutions were hit upon in one domain suggests—at the least––that they are more evolutionarily accessible than we might have thought.

What about option (3)––the claim that our judgment-forming methods were near-optimal in the evolutionary past, but not today? This may be the best response for the defender of irrationalist narratives, but it has at least two problems.

First, we are not talking about the optimality of specific methods of solving specific problems––everyone agrees that humans are not optimal at performing long division.  What we are talking about is the basic architecture of how people represent and reason with uncertainty. Is it (a) a broadly Bayesian picture, assigning probabilities to outcomes and responding to new evidence by updating those probabilities in a rational way, or (b) a heuristics-driven picture, governed by simple rules-of-thumb that systematically ignore evidence, are easily affected by desires, and so on?  There is a general framework for reasoning rationally under uncertainty––that’s the point of Bayesian epistemology. So the question is why evolution would’ve hit upon this framework in one specific domain (vision), but entirely missed it in another (judgment).

Second, option (3) relies on the claim that today there is a radical break from our evolutionary past not only in the problems we face, but also in the types of reasoning that is helpful for solving those problems. The claim must be that the simple heuristics that lead to rampant mistakes today were serviceable (in fact, near-optimal) in our evolutionary history––that people who used such heuristics would not have been outcompeted by people who reasoned better.

That claim is questionable. The problems faced by a forager looking for food or a hunter tracking prey while avoiding predators are different in substance but not in subtlety from a politician looking for a catchy slogan or a millennial writing an eye-catching tweet while avoiding saying something politically unacceptable. Why would the reasoning mechanisms that lead to systematic, pervasive, important, and silly errors in the latter domains (“form a belief based on a single piece of evidence”,  “ignore counter-evidence”, etc.) be perfectly serviceable––in fact, near-optimal––in the former?

Upshot: the defender of irrationalist narratives is left with a genuine challenge: since evolution has found such ingenious solutions to the problem of seeing, why has it landed on such inept solutions to the problem of believing?

That challenge provides reason to doubt that our belief-forming mechanisms are really so inept after all––to go back to that list of 200-or-so cognitive biases with an open mind. As we’ll see, we can offer rational explanations of many of them.  I think we should take those explanations seriously.


What next?
If you want to see some of these rational alternative explanations, see this post on overconfidence or this post on confirmation bias and polarization.
If you want to learn more about “rational analysis” as an approach to cognitive science, chapters 1 and 6 of John Anderson’s The Adaptive Character of Thought are the classic statements of the approach. For summaries of the latest research, see this 2011 article in Science or this TED talk by MacArthur-“genius”-grant-recipient Josh Tenenbaum.

PS. Thanks to Dejan Draschkow for literature suggestions and Tyler Brooke-Wilson for helpful feedback on an earlier draft of this post.

Polarization standardly refers to some measure of how much people disagree about a given topic—say, whether Trump is a good president. Current polling suggests that around 90% of Republicans approve of Trump, while only around 10% of Democrats do.  We are massively polarized—and that is the problem. Right?

Not obviously. Consider religion. On any metric of polarization, Americans have long been massively polarized on religious questions—even those that crisscross political battle lines. For example, 84% of Christians believe that the Bible is divinely inspired, compared to only 32% of the religiously unaffiliated (who now make up over a quarter of the population). Yet few in the United States think that religious polarization is the political problem of our age. Why? Because most Americans who disagree about (say) the origin of the bible have learned to get along: large majorities maintain friendships across the religious/non-religious divide, trust members of each group equally, and are happy to agree to disagree. In short: although Americans are polarized on religious questions, they do not generally demonize each other as a result.

Contrast politics. Whatever your opinion about whether Trump is a good president, consider your attitude toward the large plurality who disagree with you—the “other side.” Obviously you think they are wrong—but I’m guessing you’re inclined to say more than that. I’m guessing you’re inclined to say that they are irrational. Or biased. Or (let’s be frank) dumb.

You are not alone. A 2016 PEW report found that majorities of both Democrats and Republicans think that people from the opposite party are more “close-minded” than other Americans, and large pluralities think they are more “dishonest”, “immoral”, and “unintelligent.”

​This is new. Between 1994 and 2016 the percentage of Republicans who had a “very unfavorable” attitude toward Democrats rose from 21% to 58%, and the parallel rise for Democrats’ attitudes’ toward Republicans was from 17% to 55%:

So we do have a problem, and polarization is certainly part of it. But there is a case to be made that the crux of the problem is less that we disagree with each other, and more that we despise each other as a result.

In a slogan: The problem isn’t mere polarization—it’s demonization.

If this is right, it’s important.  If mere polarization were the problem, then to address it the opposing sides would have to come to agree—and the prospects for that look dim. But if demonization is a large part of the problem, then to address it we don’t need to agree. Rather, what we need is to recover our political empathy: to be able to look at the other side and think that although they are wrong, they are not irrational. Or biased. Or dumb.

The case of religion shows that it’s possible to do this while still harboring profound disagreements: polarization doesn’t imply demonization. And that raises a question: When it comes to politics, why do we suddenly feel the need to demonize those we disagree with? How have we come to be so profoundly lacking in political empathy?

The Story
Here, I think, is part of the story.

Though “rational animal” used to be our catch-phrase, the late 20th century witnessed a major change in the cultural narrative on human nature.  The potted history: 

Upshot: we are living in an age of rampant irrationalism.

How—according to the story that I’m telling—does this rampant irrationalism feed into political demonization?  Two steps.

Step one is simple: we are now swimming in irrationalist explanations of political disagreement.  It is not hard to see how these go. If people tend to reason their way to their preferred conclusions, to search for evidence that confirms their prior beliefs, to ignore opposing evidence, and so on, then there you have it: irrational political polarization falls out as a corollary of the informational choices granted by the modern information age. (Examples: Heuvelen 2007, Sunstein 2009, 2017; Carmichael 2018; Nguyen 2018; Lazer et al. 2018; Robson 2018;  Pennycook and Rand 2019; Koerth-Baker 2019)

Step two is more subtle. Suppose you read one of these articles claiming that disagreement on political issues is caused by irrational forces. Consider your opinion on some such issue—say, whether Trump is a good president. (Whatever it is, I’m guessing it’s quite strong.)  Let’s imagine that you think he’s a bad president. Having come to believe that the disagreement on this question is due to irrationality, what should you now think—both about Trump, and about the rationality of various attitudes toward him?

One thing you clearly should not think is: “Trump is a bad president, but I’m irrational to believe that.” (That would be what philosophers call an “akratic” belief—a paradigm instance of irrationality.)  

What, then, will you think?  You can do one of two things:

1. ​Acknowledge that you have been irrational, and so give up your belief that Trump is a bad president.
2. Maintain your belief about Trump, and so think that you have not been irrational in forming it.

No one will—and arguably no one should—give up such a strong belief, based on such a wide variety of evidence, on the basis of a psychology op-ed they came across in the newspaper. So option (1) is out—you’ll go with option (2).

Now what will you think? You’ve just been told by an authoritative source that widespread irrationality is the cause of the massive disagreement about whether Trump is a good president.  You’ve concluded that your opinion on the matter wasn’t due to massive irrationality. So whose was?  Why, the other side’s, of course!  You were always puzzled about how they could believe that Trump is a good president, and now you have an explanation: they were the ones who selectively ignored evidence, became overconfident, and all the rest. (Meanwhile, of course, those on the other side  are going through exactly parallel reasoning to conclude that your side is the irrational one!)

The result? When we come to think that irrationality caused polarization, we thereby come to think that it was the other side’s irrationality that did so.  

And once we view them as irrational, other terms of abuse follow. After all, irrationality breeds immorality: choices that inadvertently lead to bad outcomes are unfortunate but not immoral if they were rationally based on all the available evidence—but are blameworthy and immoral if they were irrationally based on a biased view of the evidence.  So once we start thinking people are biased, we’ll also start thinking that they are “immoral”, “close-minded”, and all the rest.

In a slogan: Irrationalism turns polarization into demonization.

That’s the proposal, at least. Obviously it’s only part of the story in the rise of political demonization—and I don’t claim to know how big a role it’s played. (I haven’t found any literature on the connection; if you have, please share it with me!) But given how quickly discussions about political disagreements are linked to irrationalist buzz-words like “confirmation bias”, “motivated reasoning”, and the like, I would be shocked if there were no connection at all.

As one small piece of evidence, consider this. Daniel Kahneman and Amos Tversky are the founders of the modern irrationalist narrative—winning a Nobel Prize for doing so, and becoming rock-star scientists known well outside academia. Question: when did their work—which was largely conducted in the 70s and 80s—achieve this rock-star status?  Again, you can see it for yourself.  Use Google Scholar to graph their citations per year. Next to that, graph the trends in political demonization we’ve already seen.  On each of those graphs, draw a line through 2001. Here’s what you get:

First, the test is difficult: it’s likely that only one or two of your answers are correct.  Second, your confidence in your answers probably does not reflect that degree of difficulty: I’ve given the test to 50 people, and their average confidence in their answers was 64%—yet only 44% of those answers were correct. 

When we look closer, this discrepancy between confidence and accuracy only becomes more striking. Question: of the claims people were 90%-confident in, what proportion were true?  It is natural to think that if they are assessing their evidence properly, the answer will be “90%”. More generally, we might expect rational people to be calibrated: exactly 50% of the claims that they’re 50%-confident in are true; exactly 60% of the claims that they’re 60%-confident in are true, etc.

Are people calibrated on this test?  Not even close.  For instance, of all the claims people were 90% confident in, only 63% were true.  We can represent this and other discrepancies in a graph—called a calibration curve—where the x-axis represents degrees of confidence and the y-axis represents the proportion of the answers at that degree of confidence that were true. For instance, the point (0.9, 0.63) represents the fact that of all the claims that people were 90%-confident in, 63% were true.

​If someone is calibrated, this graph will be a diagonal line. How does that line compare to people’s actual calibration curve on my survey? Take a look:

Yikes. 

​What to make of results like that?  Most psychologists take them to provide evidence that people are systematically overconfident in their opinions (summaries: here, here, and here). After all, if people are correct in their opinions far less often than they expect to be, then it's natural to infer that they are irrationally overconfident in those opinions.  

That inference—from “you are correct less often than you expect” to “you are irrationally overconfident”—forms the basis of the (much) repeated claim that people are systematically overconfident in their opinions. That claim has been a key component in the broader cultural narrative of irrationality, and has been cited as a driver of a raft of societal ills—including bad health habits, business failures, market bubbles and crashes, political polarization, intolerance, and even wars.  As one textbook puts it: “No problem in judgment and decision-making is more prevalent and more potentially catastrophic than overconfidence” (Plous 1993, 213). 

But the inference rests on a mistake.

More precisely, it is a mistake to infer simply from a calibration curve like the one I showed you to the conclusion that subjects were overconfident in their opinions.  Since this inference undergirds the general conclusion about overconfidence, it is not at all clear that we have genuine evidence that people tend to be overconfident.

Now, I think there is something importantly right about the inference (for the full story, see the paper this post is based on).  But here I’m going to focus on what’s wrong with it—on why the studies we’ve been running do not, as they are currently constructed, provide a genuine test of overconfidence.

How Not
What exactly are the results of those studies?

The most widely-cited finding is what’s known as the “hard-easy effect“. We can categorize a test based on its hit rate—the proportion of all subjects’ answers that are true. Say that a (binary-choice) test is hard if it has a hit rate of less than 75%, and easy if it has a hit rate of at least 75%. The hard-easy effect is that on hard tests people’s confidence tends exceed the proportion true (hence their calibration curve is to the right of the diagonal line––see the bottom lines in the below graph), whereas on easy tests the proportion true often exceeds their confidence (hence their calibration curve is to the left of the diagonal line––see the top lines):

The standard interpretation of this effect is that people fail to take difficulty into account, and therefore become overconfident on hard tests and under-confident on easy tests.

Is that a sensible interpretation?  To evaluate it, we need to be clear on what it means to say that you are “overconfident.” It does not simply mean that your average confidence in your opinions exceeds the proportion of those opinions there are true. For example: even though people’s average confidence on my test was 64% and the proportion true was 44%, it's still possible that they were fully rational.

Consider an example. I have a coin that is 60% biased toward heads, and I’m about to it 10 times.  How confident are you, of each toss, that it’ll land heads on that toss? 60%, no doubt.  Now I toss the coin… and (surprisingly) it turns out it landed heads only 3 of 10 times—so only 30% of the opinions that you were 60% confident in were true.  Does that mean you were overconfident?  Of course not—your 60% confidence was perfectly rational given your evidence about the coin; it just so happened that you got unlucky with how the coin landed.

The point? There is a difference between rational degrees of confidence and calibrated degrees of confidence.  In saying that you are “overconfident”, we are presupposing that there is some degree of confidence––R––that it would be Rational you to have, and that your actual opinion is more extreme than R. 

Once we make this conceptual point, it’s clear that there is no necessary connection between the (average) rational confidence on a test and the proportion of claims on the test that are true.  In fact, by manipulating the construction of the test, we can distort that connection as much as we like.

Consider Calvin.  Suppose that, given his evidence (memories, general knowledge, etc.) the rational confidence for him to have in the following claims—i.e. R—is as follows:

1. Rome is bigger than Madrid: 60%.
2. London is bigger than New York City: 50%.
3. The printing press came before Machiavelli’s The Prince: 60%.
4. A football field is longer than a home-run lap: 70%.
5. A bowling alley is longer than a half-court shot: 80%.

Then if we gave him a test with these claims on it, his average rational confidence would be 64% and the proportion of truths would be 3 of 5, or 60%—not a bad calibration score. But now suppose that before we give him the test, we first remove two of the true claims: the second and the last. Then we’ve made a new test for which his average rational confidence is 63%, but the proportion true is only 33%—we’ve (quite easily) made him rationally miscalibrated!

This procedure is fully general. Take any test you like, and suppose that Calvin’s rational degrees of confidence would be calibrated on it. Then as we start to remove true claims from the test, his rational confidence will start to exceed the proportion true—both will tend toward 0%, but the latter will do so much faster.  Conversely, if we start to remove falsehoods from the test, both is confidence and the proportion true will tend toward 100%, but again the latter will do so much faster. Either way, he’ll become substantially miscalibrated.

First upshot: Miscalibration is not good evidence for irrationality, since fully rational people will often be miscalibrated––especially on tests that contain a high (or low) proportion of truths. 

Second upshot: Therefore, we cannot test whether people are overconfident by simply testing whether their calibration curves deviate from the diagonal-line calibrated one, as is standardly done.

How, then, can we test for overconfidence?

How To
To do so, we need to know what to expect a rational person’s calibration curve to look like on a test with a given hit rate.

Here’s a helpful analogy. Suppose we have a variety of coins of different biases: some are 90% likely to land heads on a given toss, others 80%, and so on.  Suppose we’ve tossed these coins lots of times and written down the bias of the coin on one side of a piece of paper, and the way it landed on the other.  Now we collect a sample of these slips of paper, show the bias-indicating side of each slip to our subject Bianca, and have her announce her guess as to whether the coin landed heads or tails along with her confidence in that guess.  After doing this, we plot her calibration curve.

The analogy: the bias of the coin is like the evidence that Calvin has about the questions on his test; the fact that Bianca knows the biases is analogous to Calvin knowing the rational confidence R for him to have in each claim; the ways of collecting the slips of paper correspond to different ways of constructing the test.

Finally, note that it is not realistic to suppose that a person like Calvin is always perfectly rational in setting his degrees of confidence—it is far more plausible to suppose that he tends to be rational, but with some random error. To make Bianca’s case analogous, let’s suppose that the bias of the coin is indicated on the slip of paper by an unmarked slider, like this one which indicates a 60% bias toward heads:

Upshot: as I said, the observed miscalibration on my test is not evidence that my subjects were irrational––in fact, a rational person should be expected to have a very similar calibration curve.

What to make of all this?  I think it means that the current methodology for testing for overconfidence is untenable. This methodology involves assuming that a rational person will be calibrated on the test, and then seeing whether real people deviate from this. As we’ve seen, that is a mistake: even rational people will be systematically miscalibrated on tests that are hard or easy. (Similar morals apply, I think, to other empirical effects––see the full paper.) Thus to make a genuine test of overconfidence we must first predict the rational deviations from calibration on our test, and then compare real people’s performance to that prediction.  

As we’ve seen, doing so has the potential to reverse our interpretation of the data. Perhaps we've been too confident that people tend to be overconfident.


What next?
If you want more of the details, check out the full paper here.
If you’re an academic working on these topics and would like to chat—including to point me to parts of the literature that I may have misunderstood, or simply missed—please reach out to me!
If you want to play with the simulators yourself, here are two I’ve posted online. This subject-matter simulator simulates both fully random questions, as well as those pulled from subject-matters that have random degrees of misleadingness. This scrutinized-question simulator simulates a test where questions are scrutinized for their difficulty before it is decided whether they are included. For more details about these simulations, see §5 of the full paper.

PS. The answers are: Madrid; the printing press; a home-run lap.
PPS. Thanks to Cosmo Grant for feedback on this blog post, and many others for feedback on the full paper.