“Impact,” of course, is the new “it” word — the gold standard (so we are told) for judging the value of social programs. Just check out the Office of Management and Budget’s webpage: according to the OMB the 2014 budget provides funding to increase “the use of evidence and evaluation to spread innovation and drive better results.” Nonprofits that don’t speak the language of results-oriented programming, or who don’t understand how to use data to represent results, may have a hard time holding on to the government support they currently enjoy. But it isn’t only the government that is asking the impact question. Watchdog groups like Charity Navigator are beginning to do so as well, and others will soon do the same.

Metrics measure something, but not everything. How do you measure the impact of arts classes on kids who live in homeless shelters? Or of sports programs that provide after-school options for young people who otherwise have none? If you can’t produce data that show these experiences lead to very concrete outcomes, does this make the experiences meaningless or imply that they have no result?

The truth is, it’s hard to measure most kinds of social value. Social problems are complex, and a good assessment has to begin with a good interpretation: what is the perceived need that an organization exists to address? What are the conditions within which it works? If I create a nonprofit designed to help young people graduate from high school and go on to college, it makes a difference where on the continuum these kids lie. Do they have parents who finished high school (or not)? Do they speak English as a first language (or not)? Are they attending high-performing schools (or not)? If I don’t look at the conditions, at the quality of the need, at the beginning of the story, then my measurement will be incomplete, misleading, and perhaps even wrong.

The point I have been trying to make in this and previous posts is that the work of evaluation is hard. But there is something about the language of impact assessment – using “evidence” to “drive results” – that belies the reality of this, that leads one to think the truth is entirely measurable, not at all in the province of human persons who are thinking and judging and analyzing. It’s no wonder so many nonprofits are hesitant about the process, feel excluded by it in advance, and worry that they won’t be able to meet its standards.

So we need broader and more complex kinds of measurements. But this is only part of the story. Nonprofits need to do important kinds of new thinking as well.

First of all, they need to articulate, examine, and codify their own beliefs, their ways of seeing the world. For a youth-serving arts program, this means articulating what the organization values and why it thinks these values have social meaning. The Boston Youth Arts Evaluation Project helpfully calls this identifying your “sacred bundle”: what do you think matters in the world? What are you trying to cultivate? What values underlie your work?

If you believe, for example, that creating art is empowering, that it generates in young people a grounded sense of self, a bond with adults, and, in turn, a positive connection with one’s community, then you know what you are trying to measure. Your “results” may have a different profile from those of an organization that seeks to reduce homelessness, or prepare adults for jobs, but what you are doing is still very significant: empowering disconnected youth is serious business. It just has a different way of talking about itself.

In other words, NEED and VALUE have to be part of the equation, both for nonprofits and for agencies and groups evaluating them. If we speak about impact without using these key words, without taking the measure of the world we inhabit, and without making judgments about what we value, our society and culture will be the poorer for it. We cannot allow this to happen.

There is a new kid on the block that makes it possible to check reviews on some 11,000 charities. Charity Checker (sponsored by the Tampa Bay Times and the Center for Investigative Journalism) is designed to streamline the process of evaluation by aggregating reviews from the top reviewing entities into one easy-to-use tool. But what exactly is being aggregated, and what are the reviewers reviewing?

Charity Navigator is perhaps the most well-known of Charity Checker’s sources, and it currently examines the finances of about 7,000 of the over 1.5 million US charities. One of the early groups to focus attention on the ratio between administrative and program expenses, it last year signed, along with Guidestar and the BBB Wise Giving Alliance, a “pledge to end the overhead myth” — in other words, to stop focusing on overhead as the chief charity culprit. This amounted to an admission that reliance on a strict ratio of administrative to program expenses so widely hailed as the sign of a charity’s cost-effectiveness was not the holy grail, and that, indeed, nonprofits needed to invest in themselves if they were going to thrive. Charity Navigator (CN) continues to look at the ratio, but it also focuses more broadly on financial transparency as a key indicator and has begun to look at what it calls “results reporting” — how effective a charity is at reporting, in an evidence-based manner, its outcomes and impact. CN admits that too few organizations are in a position to measure impact in this manner and that the process is a “developmental” one, so “CN 3.0, ” as it is called, is not yet here.

Guidestar, another group that powers Charity Checker, is not actually a watchdog site at all. It is, according to Guidestar itself, a “comprehensive” information source. It awards logos (bronze, silver, gold) to participants for successful completion at different levels, of their profiles on the Guidestar Exchange. To a certain extent one can see why a gold rating, for example, might mean something important: filling out the profile demands that an organization reflect on itself, gather information and write about itself, and commit significant time and energy to the process. In other words, full participation registers meaningful organizational capacity. The information, assuming it is accurately reported, can also give a serious investor a lot to consider. But it is important to remember that Guidestar isn’t actually rating the charities or their programs. It assigns its logos/awards based on a charity’s level of participation, and leaves the analysis to the reader.

Great Nonprofits, the third source of Charity Checker’s information, is the Yelp or Zagat of the charity world: it invites readers (and encourages nonprofits to invite supporters) to rate charities the way you or I might rate a restaurant or doctor or retailer. Perhaps predictably, the reviews of about 12,000 charities vary widely in their usefulness. Some are incredibly thoughtful, others are … not; many reviewers simply don’t have the deep information they need to offer an informed opinion. There is something to be said for hearing what volunteers, staff, donors and clients have to say about nonprofits, but how do these ratings stack up against the serious due diligence that we are always urged to perform before we make a donation?

BBB’s Wise Giving Alliance, which covers approximately 1500 national charities (I was unable to find the exact number on its site) focuses on four key measures: good governance, financial accountability, truthfulness and transparency (“willingness to disclose basic information to the public”). These are all important standards, but I wonder how many donors who see the BBB seal on a charity’s site are aware of the fact that, in its own words, it does “not seek to evaluate the quality and content” of a charity’s performance and effectiveness. The standards are all “best practices” in the field, and the very process of seeking to meet them will, at minimum, educate a charity about how it should govern itself and provide relevant information to the public about its operations. But there is no evaluation here of programmatic quality.

Charity Checker combines into one accessible site the ratings of four well-known organizations, making it easy for busy donors to find everything in one place — though it is important to remember that it is dealing with a relatively small number of the over a million and a half US charities, and it is likely dealing with very few of the community-based nonprofits that operate in your hometown.

In any case, it would be a mistake to think that the review process, because aggregated, is necessarily comprehensive. None of the sites that powers Charity Checker assesses the need a charity exists to meet, the programs it has created to meet those needs, or the effectiveness of the work. None of them claims to do this either, but the whole business of awarding stars and badges and seals, and then of aggregating them, creates the illusion of comprehensiveness for a public eager for hard answers about where to give — and short on time to conduct its own research.

So what would it take to make the evaluation process really valuable, and how might it work in communities around the nation?