I was fortunate to attend the 2014 AFP Fund Raising Day
in New York City. The whole day was inspiring, but I found one session particularly compelling. The title was “Getting to Yes: The Role of Outcomes Assessment and Evaluation in Preparing Winning Proposals.” Its purpose was to help nonprofits better understand how to evaluate program outcomes, and communicate meaningful data to foundations and individual philanthropists. The presenters represented both the nonprofit and foundation sectors:
• Benjamin Kim, Public Health Solutions
• Dr. Ruth Finkelstein, Columbia Aging Center
• Dr. Andrew Grant, Hebrew Union College – Jewish Institute of Religion (moderator)
• Sarah Winters, Alfred P. Sloan Foundation
How the Nonprofit Community Views Outcomes Assessment and Evaluation –
Over the past 30 years funders have become increasingly focused on measurable outcomes. Dr. Grant’s opinion (shared by many nonprofits) is that funders are now “outcome-obsessed.” This emphasis on outcome assessment and evaluation is rooted in the “Venture Philanthropy” movement. Venture Philanthropists, like Venture Capitalists, invest in programs with the hopes of seeing significant returns on their investment. In the case of Venture Philanthropy, the anticipated returns are social benefits. The funders’ logic is understandable: “Without assessment, how do you know if the program was/is successful?” However problems arise when time horizons are mismatched. Businesses evaluate success quarter by quarter and year over year. But for nonprofits, meaningful results may take decades to achieve, and interim reporting doesn’t always reflect the actual success of a program. Specific Concerns about the Outcome Assessment and Evaluation Model Expense – Outside evaluation is expensive and time consuming. It pulls resources away from programming and operations. Increases Proposal Complexity – Proposals generally require that nonprofits include an evaluation design. To achieve this, the nonprofit may need to invest in evaluation design, before a program is even funded. This may mean finding outside evaluation consultants who can write the evaluation section of the grant. Grant Periods Aren’t Long Enough for Meaningful Evaluation – It can take five years to see real, measurable results for a new program. It’s difficult (some would say impossible) to get meaningful, measurable outcomes in a 1-year grant period. Working with these Concerns While the emphasis on Outcome Assessment and Evaluation presents a real challenge, there are ways for nonprofits to work successfully with this model. Keep it Simple – Evaluations do not need to be overly complex. Nonprofits should keep the evaluation process as simple as possible, while meeting funder requirements. Use Existing Tools – Many evaluation tools already exist. It’s not necessary to reinvent the wheel. An online search will turn up a wealth of information. For example, the Kellogg Foundation ( www.wkkf.org ) website has good information on logic models and evaluation methods (see Resources below). There are many Logic Models available, which help map out the progression from Input to Outcome. (Keep in mind that a Logic Model is a living document, as inputs, outputs and outcomes will change over time.)
Include Evaluation Costs in Program Budget – Assume that program evaluation will be 5-10% of direct program costs, and budget accordingly. Take a Creative Approach to Evaluation Design – Evaluation consultants can be expensive, but there are approaches that can lower or defer the costs. Some nonprofits enlist university faculty (or grad students) to design the evaluation.
The Funder’s Perspective
Sarah Winters of Alfred P. Sloan ( www.sloan.org ) acknowledged that her Foundation is very outcome oriented, and offered this perspective: Funder’s View of Evaluation Design
• Grant funding is the R&D money of the nonprofit world
• R&D always assumes there will be many failures along the way
• Success doesn’t mean meeting every goal
• Grantors want to see that nonprofits are taking a thoughtful approach
• Trustees’ Bottom Line: “Did you do what you said you would do?”
• Learning curve: “If you were to do it again, how would you do things differently?”
Advice for Nonprofits Ms. Winters emphasized that funders want nonprofits to be successful and she encouraged organizations to work closely with them after they receive a grant. She offered the following words of wisdom: Indirect Costs – Nonprofits should understand that most foundations don’t fund indirect program costs. Sloan’s guidelines allow for some indirect costs, but recognize that the amount may not cover everything. Formative Evaluations – Formative evaluations are made over the course of the program. It’s usually acceptable to make mid-course corrections. But it’s essential to let funders know about desired changes and ask permission before making them. Summative Evaluation – The summative evaluation is the final report. It should spell out exactly what happened, supported by measurable results. There should not be any surprises for the funder. Proxy Variables – For programs where long-term outcomes can’t be measured, it’s acceptable to use proxy variables, which stand in for the actual variables that you can’t measure.
A Grantee’s Perspective
Dr. Ruth Finkelstein represented one of the Alfred P. Sloan Foundation’s grantees. She offered these highlights from her experience.
• The Columbia Aging Center uses sophisticated evaluation designs, which are then shared with other nonprofits.
• She suggests a “What do I need to know at what standard of convincingness?” approach to evaluation design.
• Reality is infinite. You can’t measure everything
• Set reasonable standards. Evaluation goals need to be meaningful, but not so high they can’t be reached.
• Types of Evaluations:
Feasibility : New, truly innovative projects should begin with a feasibility evaluation, which includes a cost evaluation that provides a baseline for measuring future cost effectiveness. Process – It’s important to state whether or not the organization did what it said it would do. Outcome – This is where measurable results are particularly useful: Who was involved; how were they affected? Impact – This is a long-term evaluation, which can’t be measured within one grant cycle. Dr. Finkelstein calls this “the holy grail.” ROI – Return on investment is assessed based on how much improvement is gained for each dollar spent. Dr. Finkelstein pointed out that once an award is given, the relationship with the funder changes. The Program Officer becomes much more available and the nonprofit should take advantage of this. Find out exactly what kind of assessment and evaluation the funder wants. A grantor may also be willing and able to help the nonprofit build relationships with other foundations. As a Resource Development Consultant, I write grants on behalf of many organizations and have personally witnessed this shift in emphasis to measurable outcome assessment and evaluation. At times I do find it challenging to meet funders requirements and am not always convinced that this is the best approach to philanthropy. As Albert Einstein famously said, “Not everything that can be counted counts, and not everything that counts can be counted.” I valued the balanced views presented during this session and want to share what I learned with others in the grant-seeking community.
Resources W.K. Kellogg Foundation Logic Model Development Guide Alfred P. Sloan Foundation Grant Application Guidelines