How can philanthropies better harness the value of data to strengthen impact and evaluation assessments? 

Too many data analysis efforts start from the data supply, rather than the questions one seeks to answer. At the same time, many organizations across sectors are developing their own questions for data science, leading to fragmentation. There is an opportunity for philanthropy to explore new approaches for formulating and prioritizing questions to advance impact and evaluation assessments. 

Towards this end, on September 9th, 2024, Stefaan G. Verhulst and Hannah Chafetz from The GovLab attended Philanthropy Australia’s Phil Eval Network Share and Learn session to present on the value of questions for data science and the need for a new science of questions for philanthropy – drawing on the insights gathered from the 100 Questions Initiative. 

Stefaan began by explaining the potential of new approaches to questioning for impact and evaluation assessments. He noted that among other benefits, new approaches to questioning could help identify the questions that matter most for impact assessments, develop shared data-actionable questions across the philanthropic sector and minimize fragmentation, gain a social license across the impact evaluation process, and bring diverse voices into impact evaluations – reducing question inequity.

He then summarized the 100 Questions methodology – an initiative developed a few years ago that catalyzed a cultural shift by generating new questions and reimagining the questioning process across 10 key domains from migration to air quality. These steps are outlined below:

1. Topic Mapping: Leveraging The GovLab’s R-Search Methodology, conduct a rapid review of the problem landscape and group similar topics. 

2. Identifying and Engaging with Bilinguals: Using a “modified” delphi method, begin engaging with subject and data experts to solicit and prioritize questions. 

3. Using a Taxonomy of Data-Actionable Questions: Understand the hierarchy of types of questions you are looking for (descriptive, diagnostic, predictive, or prescriptive) and solicit questions from bilinguals.

4. Clustering the Questions that Matter: Organize the data-actionable questions based on a set of criteria (e.g. quality, impact, novelty). 

5. Prioritizing Big, Data-Actionable Questions: Work with bilinguals to reformulate questions for clarity and data actionability and release the top ten questions for public voting.

6. Public Voting: Submit an open call to vote on the questions that should be prioritized. 

Following the presentation, Stefaan, Hannah and the Philanthropy Australia team led a discussion on the value of questions for philanthropy with the broader network, emphasizing how this methodology could be applied in impact and evaluation processes. 

A few key takeaways from the discussion:

• A lot of the data needed to address the societal challenges philanthropies are looking to solve has already been collected, but remains privately held. Unlocking these data sources for public good starts by understanding the purposes in which the data is needed. The 100 Questions methodology can help identify these purposes and add rigor to the question formulation process.

• Each part of the 100 Questions methodology serves a unique purpose. For instance, topic mapping allows funders to open up conversations and focus on orphan issues. The taxonomy of data-actionable questions helps identify the type of questions needed as well as the data sources that can address those questions. Additionally, engaging bilinguals can help build communities around questions which is just as important as building data communities.

• Using a place-based approach is key in contextualizing questions at the local level. However, it is also important to understand how participatory approaches for questioning have been applied in other localities and at the global level to learn from best practices. There is an opportunity to develop a set of priority questions for Australia that can be contextualized for local contexts.

• The DATA4Philanthropy platform offers several examples of how participatory sourcing of questions can be applied in the grant making cycle (see here).

***

What other topics would you like to see on DATA4Philanthropy? Let us know by emailing us at DATA4Philanthropy@thegovlab.org. Or, do you know of any great case studies that should be featured on the platform?

Submit a case study to the DATA4Philanthropy website here.

Stay up-to-date on the latest blog posts by signing up for the DATA4Philanthropy Network here.