Evaluation guide hands program school

->>>> Click Here to Download <<<<<<<-

Michael Patton, prominent researcher, writer and consultant in evaluation, suggests that the most important type of evaluation to carry out may be this implementation evaluation to verify that your program ended up to be implemented as you originally planned.

Decide how can that information be efficiently and realistically gathered see Selecting Which Methods to Use below. Consider program documentation, observation of program personnel and clients in the program, questionnaires and interviews about clients perceived benefits from the program, case studies of program failures and successes, etc. You may not need all of the above. Analyze and report the findings see Analyzing and Interpreting Information below. The following table provides an overview of the major methods used for collecting data during evaluations.

Also consider Appreciative Inquiry Survey Design. Note that if you plan to include in your evaluation, the focus and reporting on personal information about customers or clients participating in the evaluation, then you should first gain their consent to do so. They should understand what you're doing with them in the evaluation and how any information associated with them will be reported.

You should clearly convey terms of confidentiality regarding access to evaluation results. They should have the right to participate or not. Have participants review and sign an informed consent form. See the sample informed-consent form. The overall goal in selecting evaluation method s is to get the most useful information to key decision makers in the most cost-effective and realistic fashion.

Consider the following questions: 1. What information is needed to make current decisions about a product or program? Of this information, how much can be collected and analyzed in a low-cost and practical manner, e. How accurate will the information be reference the above table for disadvantages of methods? Will the methods get all of the needed information? What additional methods should and could be used if additional information is needed? Will the information appear as credible to decision makers, e.

Will the nature of the audience conform to the methods, e. Who can administer the methods now or is training required? How can the information be analyzed? Note that, ideally, the evaluator uses a combination of methods, for example, a questionnaire to quickly collect a great deal of information from a lot of people, and then interviews to get more in-depth information from certain respondents to the questionnaires.

Perhaps case studies could then be used for more in-depth analysis of unique and notable cases, e. There are four levels of evaluation information that can be gathered from clients, including getting their: 1.

Usually, the farther your evaluation information gets down the list, the more useful is your evaluation. Unfortunately, it is quite difficult to reliably get information about effectiveness. Still, information about learning and skills is quite useful. Analyzing quantitative and qualitative data is often the topic of advanced research and evaluation methods.

There are certain basics which can help to make sense of reams of data. Always start with your evaluation goals: When analyzing data whether from questionnaires, interviews, focus groups, or whatever , always start from review of your evaluation goals, i.

This will help you organize your data and focus your analysis. For example, if you wanted to improve your program by identifying its strengths and weaknesses, you can organize data into program strengths, weaknesses and suggestions to improve the program. If you wanted to fully understand how your program works, you could organize data in the chronological order in which clients go through your program. If you are conducting an outcomes-based evaluation, you can categorize data according to the indicators for each outcome.

Basic analysis of "quantitative" information for information other than commentary, e. Make copies of your data and store the master copy away. Use the copy for making edits, cutting and pasting, etc. Tabulate the information, i. For ratings and rankings, consider computing a mean, or average, for each question.

For example, "For question 1, the average ranking was 2. This is more meaningful than indicating, e. Consider conveying the range of answers, e. Basic analysis of "qualitative" information respondents' verbal answers in interviews, focus groups, or written commentary on questionnaires : 1.

Read through all the data. Organize comments into similar categories, e. Label the categories or themes, e. Attempt to identify patterns, or associations and causal relationships in the themes, e. Keep all commentary for several years after completion in case needed for future reference.

Attempt to put the information in perspective, e. Consider recommendations to help program staff improve the program, conclusions about program operations or meeting goals, etc.

Record conclusions and recommendations in a report document, and associate interpretations to justify your conclusions or recommendations. The level and scope of content depends on to whom the report is intended, e. Be sure employees have a chance to carefully review and discuss the report.

Translate recommendations to action plans, including who is going to do what about the program and by when. Bankers or funders will likely require a report that includes an executive summary this is a summary of conclusions and recommendations, not a listing of what sections of information are in the report -- that's a table of contents ; description of the organization and the program under evaluation; explanation of the evaluation goals, methods, and analysis procedures; listing of conclusions and recommendations; and any relevant attachments, e.

The banker or funder may want the report to be delivered as a presentation, accompanied by an overview of the report.

Or, the banker or funder may want to review the report alone. Be sure to record the evaluation plans and activities in an evaluation plan which can be referenced when a similar program evaluation is needed in the future.

An example of evaluation report contents is included later on below in this document. Ideally, management decides what the evaluation goals should be. Then an evaluation expert helps the organization to determine what the evaluation methods should be, and how the resulting data will be analyzed and reported back to the organization. Most organizations do not have the resources to carry out the ideal evaluation.

If they can afford any outside help at all, it should be for identifying the appropriate evaluation methods and how the data can be collected. The organization might find a less expensive resource to apply the methods, e. If no outside help can be obtained, the organization can still learn a great deal by applying the methods and analyzing results themselves.

However, there is a strong chance that data about the strengths and weaknesses of a program will not be interpreted fairly if the data are analyzed by the people responsible for ensuring the program is a good one. Program managers will be "policing" themselves. This caution is not to fault program managers, but to recognize the strong biases inherent in trying to objectively look at and publicly at least within the organization report about their programs.

Therefore, if at all possible, have someone other than the program managers look at and determine evaluation results. Develop an evaluation plan to ensure your program evaluations are carried out efficiently in the future.

Note that bankers or funders may want or benefit from a copy of this plan. Ensure your evaluation plan is documented so you can regularly and efficiently carry out your evaluation activities. Record enough information in the plan so that someone outside of the organization can understand what you're evaluating and how. Consider the following format for your report: 1.

Table of Contents 3. Executive Summary one-page, concise overview of findings and recommendations 4. Purpose of the Report what type of evaluation s was conducted, what decisions are being aided by the findings of the evaluation, who is making the decision, etc.

Don't balk at evaluation because it seems far too "scientific. There is no "perfect" evaluation design. Don't worry about the plan being perfect. It's far more important to do something, than to wait until every last detail has been tested. Work hard to include some interviews in your evaluation methods. Questionnaires don't capture "the story," and the story is usually the most powerful depiction of the benefits of your services. Johnson suggested that there are long-range, intermediate, immediate, and unplanned-for results that need consideration.

According to Johnson, long-range results focus on how programs affect students after they have left school. Usually long-range results are gathered using follow-up studies. Intermediate results focus on the knowledge and skills all students may gain by graduation from participating in the guidance program. Immediate results are the knowledge and skills students may gain from participating in specific guidance activities. Finally, the possibility of unplanned-for results that may occur as a consequence of guidance activities conducted as a part of the guidance program also need to be taken into account.

Johnson outlined the following procedures to answer this question for immediate results. First the competencies to be mastered need to be identified. Second what results what students should be able to write, what they may be able to talk about, or what they may be able to do are specified. Then who will conduct the evaluation is decided. This is followed by when the evaluation is done. Then criteria are established so that judgments can be made about students' mastery of guidance competencies.

Finally, how all of this is done is specified. Do students develop and use career plans? In making judgments concerning the career plans of students, criteria need to be identified as to what makes good plans. Four criteria are recommended; plans need to be comprehensive, developmental, student-centered and student-directed, and competency based. In order to fully evaluate comprehensive school guidance programs, three forms of evaluation are required.

Second, guidance program personnel need job descriptions derived directly from the program so that evaluation forms can be developed and used for formative and summative personnel evaluation. Third, results evaluation that focuses on the impact of the guidance and counseling activities in the guidance curriculum, individual planning, responsive services, and system support components of a comprehensive guidance program is mandatory.

American School Counselor Association Professional development guidelines for secondary school counselors: a self audit. Evaluating School Guidance Programs. ERIC Digest. For example, here is a program standard: The school district is able to demonstrate that all students are provided the opportunity to gain knowledge, skills, values, and attitudes that lead to a self-sufficient, socially responsible life.

Using the standard listed above, evidence that the standard is in place might include the following: 1. Linking program performance to program budget is the final step in accountability. The early steps in the program evaluation approach such as logic modeling clarify these relationships, making the link between budget and performance easier and more apparent.

While the terms surveillance and evaluation are often used interchangeably, each makes a distinctive contribution to a program, and it is important to clarify their different purposes. Surveillance is the continuous monitoring or routine data collection on various factors e.

Surveillance systems have existing resources and infrastructure. Data gathered by surveillance systems are invaluable for performance measurement and program evaluation, especially of longer term and population-based outcomes. There are limits, however, to how useful surveillance data can be for evaluators.

Also, these surveillance systems may have limited flexibility to add questions for a particular program evaluation. In the best of all worlds, surveillance and evaluation are companion processes that can be conducted simultaneously. Evaluation may supplement surveillance data by providing tailored information to answer specific questions about a program. Data from specific questions for an evaluation are more flexible than surveillance and may allow program areas to be assessed in greater depth.

Evaluators can also use qualitative methods e. Both research and program evaluation make important contributions to the body of knowledge, but fundamental differences in the purpose of research and the purpose of evaluation mean that good program evaluation need not always follow an academic research model.

Research is generally thought of as requiring a controlled environment or control groups. In field settings directed at prevention and control of a public health problem, this is seldom realistic.

Of the ten concepts contrasted in the table, the last three are especially worth noting. Unlike pure academic research models, program evaluation acknowledges and incorporates differences in values and perspectives from the start, may address many questions besides attribution, and tends to produce results for varied audiences.

Program staff may be pushed to do evaluation by external mandates from funders, authorizers, or others, or they may be pulled to do evaluation by an internal need to determine how the program is performing and what can be improved. While push or pull can motivate a program to conduct good evaluations, program evaluation efforts are more likely to be sustained when staff see the results as useful information that can help them do their jobs better.

Data gathered during evaluation enable managers and staff to create the best possible programs, to learn from mistakes, to make modifications as needed, to monitor progress toward program goals, and to judge the success of the program in achieving its short-term, intermediate, and long-term outcomes. Most public health programs aim to change behavior in one or more target groups and to create an environment that reinforces sustained adoption of these changes, with the intention that changes in environments and behaviors will prevent and control diseases and injuries.

Through evaluation, you can track these changes and, with careful evaluation designs, assess the effectiveness and impact of a particular program, intervention, or strategy in producing these changes. The Working Group prepared a set of conclusions and related recommendations to guide policymakers and practitioners. Program evaluation is one of ten essential public health services [8] and a critical organizational practice in public health.

The underlying logic of the Evaluation Framework is that good evaluation does not merely gather accurate evidence and draw valid conclusions, but produces results that are used to make a difference.

You determine the market by focusing evaluations on questions that are most salient, relevant, and important. You ensure the best evaluation focus by understanding where the questions fit into the full landscape of your program description, and especially by ensuring that you have identified and engaged stakeholders who care about these questions and want to take action on the results.

The steps in the CDC Framework are informed by a set of standards for evaluation. The 30 standards cluster into four groups:. Utility: Who needs the evaluation results? Will the evaluation provide relevant information in a timely manner for them? Feasibility: Are the planned evaluation activities realistic given the time, resources, and expertise at hand? Propriety: Does the evaluation protect the rights of individuals and protect the welfare of those involved?

Does it engage those most directly affected by the program and changes in the program, such as participants or the surrounding community? Accuracy: Will the evaluation produce findings that are valid and reliable, given the needs of those who will use the results? Sometimes the standards broaden your exploration of choices.

Often, they help reduce the options at each step to a manageable number. Feasibility How much time and effort can be devoted to stakeholder engagement? Propriety To be ethical, which stakeholders need to be consulted, those served by the program or the community in which it operates? Accuracy How broadly do you need to engage stakeholders to paint an accurate picture of this program?

Similarly, there are unlimited ways to gather credible evidence Step 4. Asking these same kinds of questions as you approach evidence gathering will help identify ones what will be most useful, feasible, proper, and accurate for this evaluation at this time. Thus, the CDC Framework approach supports the fundamental insight that there is no such thing as the right program evaluation.

Rather, over the life of a program, any number of evaluations may be appropriate, depending on the situation. Good evaluation requires a combination of skills that are rarely found in one person.