Choosing research methods for data-driven storytelling

This blog post does not represent Vision Critical. In fact, I think some of my colleagues are going to argue with me vigorously over this one.

Rigorous data gathering and analysis can get in the way of effective storytelling by non-profits.

That proved to be the most controversial part of my talk today on Telling Stories with Data at the Nonprofit Technology Network’s Leading Change Summit. It’s a gathering of nonprofit leaders who are working together and learning from one another about impact leadership, digital strategy and the future of technology.

In other words, a group of people with the potential to significantly advance the visibility and effectiveness of nonprofits by using data to raise public awareness, build public support and make smarter organizational decisions. And judging from the enthusiastic conversation in the room and on Twitter, it’s a group of people who are eager to do that kind of data-driven work.

Yet many non-profit leaders (not to mention government and business leaders) hang back from embracing the power of data because they worry that their organizations lack the expertise, resources or data access to work effectively with data. Data-driven storytelling is a great place for these organizations to start working more intensively with data because it can offer a tangible payoff (hello, social media shares and inbound traffic!) with a relatively limited investment.

But telling stories while making only a modest investment of time, money and effort is easier if you’re willing to cut some corners on your data gathering and analysis. That’s certainly not the approach I was trained in — and in my fortunate position at a company with software and expertise that makes it possible to do high-quality data storytelling, I don’t have to cut a lot of corners myself.

Not every organization has access to a large community of people providing ongoing feedback, however, nor the in-house expertise to do significant data analysis on an ongoing basis. And unlike academic researchers or customer intelligence teams who are trying to get the most accurate picture of a particular topic or a given group of people, nonprofit storytellers often have an investment in what they want their data to show: they’re looking for data stories that support a particular position or approach.

That’s why I encourage organizations to differentiate between the situations in which they need to meet academic standards of rigour and objectivity, and those in which they just need to tell an interesting or compelling story. As ever, a 2×2 is a helpful way of thinking about these different scenarios, so I’m sharing the one I showed in today’s talk — and this version is a little more detailed to clarify some questions that came up today.

Note that this 2×2 is geared towards determining your quantitative research approach; while there is much to be said for qualitative research, this is for organizations thinking about how to do quantitative, data-driven storytelling. It is intended to help you think about how rigorous you need to be in your research design, and also to clarify whether you’re really open to whatever the data shows you: if you’re only prepared to release certain kinds of results, you need to be prepared to walk away from your storytelling project (or better yet, rethink your strategy) if the data doesn’t show you what you’re hoping to see.

METHODS

While researchers and policy makers might prefer to see data that can rise to the standard of rigorous objectivity, organizations often have an agenda in their data projects, and may cut corners in the process of gathering and analyzing that data. In an ideal world, we’d all have access to fantastic data, powerful software and professional expertise to support our data projects…and we’d be comfortable admitting that sometimes, that data raises serious questions about our organization’s tactics, strategy or even mandate.

But successful nonprofits are smart about allocating resources where they’re needed most — and doing the best data work possible isn’t always the top priority. Sometimes you can get the results you need (like inbound traffic generated by an eye-catching infographic) by sharing data that is useful and interesting, if imperfect. That’s why I think it’s time for organizations to get comfortable doing at least some of their data work in the “relaxed” zone, because that is better than missing out on doing any data storytelling at all.

Similarly, it’s time for organizations to admit that they spend an awful lot of time in the bottom half of this 2×2: the zone in which they’re only going to release data that shows a story aligned with a particular mission or approach. Take the example of grant reports: very few organizations are prepared to submit data that shows a funder that they wasted their money. That doesn’t mean organizations can or should fudge the data; it just means admitting that grantees typically choose the metrics that cast the best light on their accomplishments…but should still aim to collect and analyze those flattering metrics as rigorously as possible.

I look forward to hearing from nonprofits who are venturing into the world of data, and using it to tell stories about their issues, their mission and their work. Aspire to produce the best data stories you can! Just don’t let aspirations to data perfection keep you from using data to advance your work.