The Importance and Challenge of Evaluation and Measurement in the Nonprofit Sector
“Well-run organizations and effective programs are those that can demonstrate the achievement of results. Good management is based on good decision-making. Good decision-making depends on good information. Good information requires good data and careful analysis of the data. These are all critical elements of evaluation.”- GuideStar
Evaluation has become critical to nonprofit performance and can support every aspect of both leading and managing an organization. Data from evaluation can help to establish which strategies are working and which ones need to be adjusted. It can also guide hiring practices and help identify areas of need, so that resources can be allocated properly and effectively. Moreover, evaluation can strengthen fundraising efforts by providing information about measurable outcomes and charting impact.
A nonprofit or community-based organization that includes evaluation plans and data in funder reports demonstrates that it takes its work seriously and is open to continuous learning and ongoing improvement. More and more funders, individual donors and government agencies expect grantees to have an evaluation process in place. However, few directly allocate resources for this work and this is largely due to the high cost associated with traditional evaluation methods.
Rigorous evaluation requires professional researchers with advanced degrees in scientific methods and statistical analysis. In large nonprofit organizations, 10% of the annual budget is routinely dedicated to evaluation activities. Extrapolate this down to a smaller nonprofit or community-based organization with a $1 million dollar budget and it would be expected to spend $100,000 a year, which is simply not realistic.
While small nonprofits and community organizations understand the importance of having an evaluation plan, many simply do not have sufficient resources to support an ongoing evaluation process. Therefore, smaller nonprofits are far less likely to have the organizational tools in place to measure their impact. A 2016 report by the Innovation Network discovered that only 16% of smaller nonprofits conduct evaluations, compared with 69% of larger ones. This discrepancy is particularly troubling because small nonprofits represent the vast majority of those registered as 501(c)(3) organizations in the United States.
This problem raises some critical questions for the nonprofit sector and demands a reappraisal of our approaches to evaluation. The disparity between those that have evaluation resources and those that do not mirrors and exacerbates the inequities already experienced by the communities that smaller nonprofits serve. The inability of these smaller nonprofits to finance evaluation and articulate their impact negatively effects their chances of raising money for their work and limits their ability to pay staff a living wage.
"Evaluations have become critical to nonprofit performance, but they are prohibitively expensive for many organizations. Solving this problem will make the sector more equitable and successful." – Kim Sabo Flores, Stanford Social Innovation Review
To address issues of equity, every nonprofit, regardless of size, must have access to evaluation and be able to use data for ongoing learning and improvement, advocacy for their community needs and communication about promising practices and outcomes.
Being able to measure the quality and impact of nonprofit work should be available to all, not an expensive luxury afforded only to big budget organizations. Fortunately, there are some important developments unfolding which promise greater access to evaluation.
In-House Evaluation Training and Development
To train internal staff in evaluation methods and strategies many nonprofits have engaged in what is known as Evaluation Capacity Building (ECB). Leslie Fierro, assistant clinical professor of evaluation at the Claremont Graduate University, defines evaluation capacity building (ECB) as:
“Implementing one or more interventions to build individual and organizational capacities to engage in and sustain the act of evaluation—including, but not limited to, commissioning, planning, implementing, and using findings from evaluations.”
ECB is a proven strategy for meeting funders' increased demands for evaluation for both accountability and strengtheingn the long-term capacity and sustainability of nonprofit organizations.
ECB is often done through cohort-based workshops and one-to-one coaching and can be a more cost-effective way to meet funder mandates for evaluation and strengthen an organization's ability to measure the outcomes of its activities over time. Building evaluation capacity this way can improve staff attitudes toward evaluation, increase their knowledge about research methods and data, enhance leadership skills and create a culture of learning and improvement for the entire organization. The cost of most ECB is far less than traditional evaluations and the impact is often far greater, displaying positive results at multiple levels of organizational effectiveness.
Although there are distinct benefits to investing in employees and volunteers so that they have the knowledge and confidence to engage in certain types of evaluations, ECB can only go so far. The increasing demand for research-based tools and rigorous scientific analytics asks too much of most nonprofit staff members, even if they have taken every evaluation certificate program available to them! Without an advanced-level statistics qualification or the help of an expensive consultant, most simply cannot develop the evaluation skills required to meet the requirements of funders and donors. So, organizations need to foot the bill for hiring evaluation expertise at some point.
Shared Measurement Platforms
There are now a growing number of shared measurement platforms on the market and several innovative web-based solutions that allow thousands of nonprofits to easily administer statistically valid measures and access real-time reports that include high-quality analyses of outcome growth, program quality and benchmarks. These solutions offer significant benefits, including lower cost, greater efficiency, expert guidance, improved credibility and consistency in reporting, cross-site learning and collaboration and an increased ability to make informed decisions in the long-term.
A 2009 report written by FSG titled “Breakthroughs in Shared Measurement and Social Impact argues: “The level of research that goes into the development of these systems, the timeliness and consistency in data reporting that they permit, and the standardization that they enable across multiple grantees all contribute to a significantly better quality of data about grantee performance.” Further, they state that “these systems offer nonprofit organizations a” degree of credibility that individual organizations’ idiosyncratic evaluation approaches often lack.”
These platforms have the potential to even the playing field for all nonprofits, effectively crowd sourcing measurement and sharing the cost of rigorous scientific methods and analytics across many organizations which really benefits those with limited resources. In many cases, evaluation becomes a fraction of the cost of a traditional evaluation study, with quality on par with randomized controlled studies that can cost hundreds of thousands of dollars.
These platforms can also leverage the larger dataset to understand and provide feedback about what is working, for whom and in which contexts. As more organizations participate, the data sets become richer and more diverse and the platforms smarter and more effective. Complex analytics inform staff what has proven effective in the past, so that they can tailor their work accordingly and readily move from compelling reports to decisive action without having to become data experts. In this way, shared measurement platforms that value ongoing feedback, continuous improvement, and inclusivity can serve as massive joint experiments that enrich the sector’s knowledge and benefit every organization that gets involved.
"Shared measurement systems can help us move beyond the fragmented and disconnected efforts of more than a million nonprofits by creating a new degree of coordination and learning that can magnify the impact of funders and grantees alike.” - Breakthroughs in Shared Measurement and Social Impact, Mark Kramer, Marcie Parkhurst, Lalitha Vaidyanathan, FSG, 2009
Shared Measurement for Youth Serving Nonprofits
Hello Insight is a shared online measurement platform designed specifically for youth-serving agencies that support positive youth development and social and emotional learning (SEL). The platform was developed to address the opportunity gap in the sector and to make rigorous scientific evaluation cost effective and readily available to all. Currently serving over 2,000 organizations nation-wide, the cost of evaluation is significantly lower than traditional evaluation activities, with many members paying less than $2,500 per year.
The platform was designed with young people at its heart, with the aim of giving youth workers access to real-time data to help them truly understand the needs and development of their participants. But more than that, Hello Insight uses AI technology, leveraging the broader data to create practical recommendations that let front-line staff know what has worked for other young people, just like those with whom they are working. How does it do this? It centers youth voice, inviting individuals to asses their experiences in programs. This data is central to developing a large body of knowledge about what works, for whom and in what circumstances. The more people that use Hello Insight the smarter it gets, providing more nuanced recommendations to organizations and to the field.
Spotlight on HI:Sports
One of the tools in the Hello Insight platform, HI: Sports, was created in concert with Laureus Sport for Good USA, Up2Us Sports, Vita Sports Partners, Play Rugby USA, and The NY Community Trust. It was developed and validated with grantee partners who tested the tool and discussed findings through ongoing learning cohorts.
Now more than five years old, the tool has been used by more than 200 sports-based youth development programs across the country and has had adequate time to accumulate a wealth of valuable data. It is now possible to mine this data to articulate the actual impacts of these types of social impact programs. In a recent report –Sport Based Youth Development: Hitting a Home Run in SEL supported by Laureus Sport for Good USA and The All State Foundation, data showed that more young people attending sport-based youth development programs developed social and emotional learning skills than peers in other non-sport programs. This was especially true for young men of color. Further, the study showed what types of approaches worked best for promoting SEL and how these differed by gender. For example, more boys developed SEL when they were able to set and manage their own goal, while girls needed opportunities to make their own decisions and have choices.
These findings have ignited several other studies in the field and have continued so support these types of programs in their fund development efforts, allowing them to effectively articulate their impact to donors, government agencies, board members, families and community stakeholders. When information is gathered from and valued, not only by all stakeholders and organizations, but by entire fields of practice, it can unearth new insights, understandings, and solutions that will produce sustained, informed and equitable social change.
Ready to enhance your evaluation? Contact us today to find out how we can help.