Evaluation Training organized by Nonprofit Prince George’s County

MUPJ at Evaluation Training
January IDEAS & INFO Luncheon:

Dr. Taj Carson and the team from Carson Research Consulting presented a workshop on program evaluation. The workshop met its goals in providing participants with the understanding of what process and outcome measures are.  The workshop further offered best practices for measurement and data collection.   The workshop raised several questions for organizations to re-examine their impact. How does your organization measure impact? ‘You say your program works, but why should I believe you? Are your staff and board members aware of your programs’ effectiveness? Are potential funders able to see the results of your most popular programs?
The workshop further corroborates the importance of Program evaluation, and why “it is essential for nonprofits to gauge social impact, and organizations large and small have the ability to conduct them”.
According to Dr. Taj Carson of Carson Research Consulting (CRC) , organizations, funders, donors use evaluations for an array reasons.

To ensure investments are making the desired impact. 

To determine future investments. 

To demonstrate efficient use of resources.

To engage with grantees about program work and possible improvements.

The event took place on Wednesday, January 11, 2016 From 12 Noon – 2:30 pm At the Vista Gardens Marketplace 10201 Martin Luther King. Jr. Highway  Second Floor Community Room A  Bowie, MD 20720

Why Evaluation? The importance of evaluation is to improve the organization, increase effectiveness of management and administration, objectively assess performance, address problems related to cost, recognize and understand secondary or unanticipated effects, program replication and marketing, to meet accountability requirements.

STRATEGIC THINKING
One of the first presenters of the Workshop Dr. Taj Carson, has done 14 years of Evaluation work with Data of all types. She works participants through the format of the workshop and says she wants to make it interactive. She described the five steps cycle of Program Planning and Evaluation.
She listed the following five steps of Program Planning and Evaluation:
 Step 1. Conduct needs assessment  Step 2. Define goals and objectives  Step 3. Design and implement Research-based activities.  Conduct Evaluation(s)  Step 5. Report and use Evaluation Findings.

She reiterated the importance for organizations to consider evaluation as a key to their success. It is important to therefore conduct Program Evaluation as an essential part of a grant that your  organization has received, or to provide a clear idea to your Board and provide answers about outcomes, or to also let your  staff become clear about the effectiveness of  a particular program.
The facilitator also went on to say that ‘evaluation is critical whether organizations are planning a new project or starting a program from the ground up, re-working an existing project or program to make it more effective, or trying to make sense of findings from previous evaluations, the Evaluation method is critical in helping organizations get off the ground in their current environment and a formal needs assessment.
She mentioned and explained the various types of evaluation methods which are the Process Evaluation, which is used for the following.
 Used to measure the activities of the program, its quality and who it is reaching. (Has the project reached the right group?)  have all the activities being implemented as intended and what happens afterwards if it is not?   All information and materials suitable for the target market? She mentioned that in program planning, and outcome, things sometimes don’t happen the way they are planned or intended like how one expects projects to be. There should be internal conversation with staff
and reassess the program as well with program beneficiaries.
The Outcome Evaluation is concerned with the long term effects of the program. This process addresses the critical question of whether the overall program goals have been achieved. And if the goals were not achieved, what factors have contributed or hindered the desired change. Some dynamics of the program may change due to shift in needs or unexpected changes in the environment. There should be formal needs assessment about the quality of the program and the number of people who are asked about their experiences of program impact and their feedback and evaluation helps.
Defining Goals
The workshop facilitator says every organization should first of all have goals they need to reach and it is important to ‘defining these goals and objectives’. She described goals as “overarching statements of what the program hope to accomplish. Goals may vary depending on the work you do or resources needed. There are small programs with a specific focus might conceivably have one goal, whilst a more complex or broader program may have multiple goals. The facilitator drew the participants attention to a workforce development program’s goals which she says might include:
Examples of goals include:
 Help clients reduce their barriers to employment  Help clients become job ready  Increase client’s economic self-esteem.
 Evaluation Training          organized by Nonprofit Prince George’s County

She says that in writing measurable goals, organizations should also consider the ‘who, what and when’. Who refers to the target audience, ad what will refer to the nature of that needs, and when the timeline to meet these needs. It is important to write reasonable goals and consider the following points.
 Audience – Who? The population/target audience for whom the desired impact/outcome is intended.   Behavior – What? A clear statement of the behavior change/results expected  Condition – when? Under what condition will it happen.
The facilitator asked participants to give examples of objectives eg. Providing access to job training’.
She mentioned the common mistakes that organizations should avoid when writing goals and objectives. These include
 Stating activities as goals  Stating implementation or operational benchmarks as goals  Writing compound goals as objectives.
She admonished organizations that whilst they are in need of the grants, they must not be swayed by what the funders want them to do but to stick to what you are good at so that you can better deliver and reach your goals.
Logic Models
The importance of Logic Model in program evaluation was also discussed. The facilitator expressed that ‘everyone has a theory about the impact that their programs have or will have on the people in the community, but wouldn’t you want to know why your program is expected to have an impact?  That’s where logic models come in. Logic models help lay out the chain of causation that stakeholders often look for. Why will a certain strategy work? How will this new strategy affect the program as a whole? Organizations need to develop these tools in order for the need to properly plan their programs and improve them for grant applications, evaluation, and strategic decision-making.
The illustration above shows the basic Logic model format which includes ‘Your Planned Work’ and ‘Your Intended work’.
Logic models ‘Allow us to show connections among program components’ show how program activities will lead to the accomplishments of objectives, and how accomplishing objectives will lead to the fulfillment of goals.

Participants went through an exercise of the Graphical Model (elevation pitch) which is a great way to share what your program is doing via graphics.
Participants also were asked to develop their ‘list of outcomes which are the results of program operations or activities’. An example of a work place development program may include:  Clients having reduced barriers to success, Clients access and completed job-training programs
Organizations also need to research, evaluate their programs often and analyze data understand the needs of their program and address the challenges they face in the tasks ahead. It is also significant that organizations can share or tell what their impact has been and how to demonstrate their worth to others, by ensuring that they communicate their findings in a proper manner or in way that makes sense to funders, donors or partners.
Data Collection and Analysis
The world of evaluation and data collection evolves as technology improves and needs change and so it is important to look out for new changes.
The second facilitator and presenter was Sarah McCruden who is also from the CRC. She says that  ‘We love data’! but data is expensive and cost money so it is important to save and store data and collect as much data as possible whilst looking at indicators every two years. Eg if an organization has added new funders to their list, that needs to be included in their data. Sarah says that Data is important depending on your organization’s needs. In choosing data, organizations can select and design surveys, implement the most appropriate data collection methods, conduct interviews, or compile existing data within your organization. There is need to collect data through surveys and provide survey results, focus groups, needs assessments.
She mentioned the quantitative and qualitative data analysis, including use of SPSS, Excel, and ATLAS.ti. software packages tools to gather data. Qualitative data she says, ‘involve interviews and it is good to bring in researchers too or consultants who understand the background of those you are collecting data from. Universities can provide graduate students as interns who already have experiences in using data software to help organizations provide quality data and useful data with fewer expenses from the organization. There are also Google services, Google drive, Microsoft tools (Office Suite) and survey monkeys some of which are free and others can be upgraded with fees’.

While we think data is fun on its own, we know that you have goals you need to reach, so you need to be able to access and use it.
Organizations must be able to generate and use evaluation findings, design programs, and use data for outcome-based assessments and to show funders how their support has the desired measuring impact through your work.  Data when used effectively will help organizations access resources and secure the funding they need and to communicate your project or program’s impact to different audiences successfully.
Sarah also mentioned the importance of using Dashboards in Data Collection as part of several data tools.

The use of dashboard at the close of organization’s research and evaluations is important as organizations may have collected a huge volume of data that needs to be accessed. Organizing these data is important and the tools with dashboard  to combine new data with data you already stored can be done with dashboard to straighten tables, graphs and  so that they are interpreted correctly and updated constantly. The use of dashboards helps in the dynamic data collection. She mentioned that easily accessed online, so you can answer questions about your program, school, agency, or other organization quickly and knowledgeably.
Dashboards also allow organizations to use multiple visual elements to explain the benefits of your programs and communicate findings clearly. And they’re easy to update.
Database Management And Reporting
Data management and reporting is therefore key and so designing and building  or managing it needs such software like  ‘Efforts to Outcomes (ETO) and Microsoft Access  which are popular software .

By: Andrew Greene MUPJ 2017