Image titled “Balance” by Akin Bostanci from Getty Images Signature

It’s easy to fall into the trap of confirmation bias—making decisions based on evidence that aligns with preconceived ideas while ignoring signs that challenge them. But what happens when the evidence suggests something contrary to our expectations?

 In their article, "Philanthropy Needs More Disconfirmation Bias", published in the Stanford Social Innovation Review, Kecia Bertermann and Andy Martin explore how philanthropic organizations often struggle to acknowledge and act on disconfirming evidence—data or findings that challenge the assumptions underlying program designs or investments. They argue that as philanthropy shifts toward broader systemic change, there is a growing need to confront the cognitive biases that can distort decision-making. Unlike other sectors, such as law or medicine, where structures exist to regularly test assumptions, philanthropy often lacks mechanisms to address these biases. The article offers lessons from other sectors and outlines ways philanthropy can build practices, structures, and cultures that embrace disconfirming evidence.

Bertermann and Martin emphasize that disconfirming evidence comes in various forms, from clear contradictions of core assumptions to subtle signals that things are not progressing as expected. They argue that acting on such evidence can help guard against confirmation bias and lead to better-designed programs. The challenge, however, is that philanthropic organizations often face incentives to ignore disconfirming evidence—whether due to risk aversion, over-commitment to a particular strategy, or the desire to maintain relationships. The authors draw on examples from various fields to demonstrate how other sectors have built systems to explicitly confront disconfirming evidence. In scientific research, for instance, peer review and replication studies help ensure that findings are robust and reliable, while in medicine, differential diagnosis forces practitioners to rule out multiple possibilities before settling on a diagnosis.

The article offers several actionable recommendations for philanthropy. First, philanthropic organizations must create learning environments that encourage critical thinking and dialogue about disconfirming evidence. This includes establishing external review panels and creating dedicated 'red teams'—groups specifically tasked with challenging assumptions and testing ideas by critically assessing proposed strategies or interventions to identify potential flaws or overlooked evidence. Second, they must build processes that prioritize disconfirming evidence at both the organizational and portfolio levels. This involves setting clear evaluation criteria that include potential disconfirming factors and ensuring that grantees report unexpected outcomes. Finally, philanthropic organizations must foster a culture that values humility, openness, and a willingness to learn from failure. Recognizing disconfirming evidence should be seen as an opportunity for growth, not a threat.

Below we provide three takeaways from the piece relevant for the DATA4Philanthropy Network:

1. Build Mechanisms to Address Disconfirming Evidence: Just as scientific research uses peer review and replication studies, philanthropy should create formal structures to assess and act on disconfirming evidence. This could include external review panels or red teams that challenge assumptions and offer alternative perspectives.

2. Foster a Culture of Continuous Learning: Philanthropic organizations should cultivate a culture that encourages critical thinking, humility, and openness to new information. Hosting regular 'evidence events'—structured meetings or workshops where teams review data, discuss unexpected findings, and openly share lessons learned—can encourage reflection on disconfirming evidence. Establishing anonymous feedback loops can further help organizations stay open to contradictory evidence by providing a safe space for feedback without fear of reprisal.

3. Prioritize Evidence in Evaluation: When designing grant agreements or evaluation processes, organizations should explicitly include metrics and goalposts that challenge assumptions. This ensures that programs are continuously assessed against the full range of possible outcomes, not just those that align with initial expectations.

The ability to recognize and act on disconfirming evidence is not just a matter of integrity but a pathway to greater impact. By adopting these practices, the philanthropic sector can learn to thrive in uncertainty and ensure that the causes they support are truly benefiting from the resources they provide.

The full article can be found here.

***

What other topics would you like to see on DATA4Philanthropy? Let us know by emailing us at DATA4Philanthropy@thegovlab.org. Or, do you know of any great case studies that should be featured on the platform?

Submit a case study to the DATA4Philanthropy website here.

Stay up-to-date on the latest blog posts by signing up for the DATA4Philanthropy Network here.