Using Evaluation - Strategies and Capacity

Making Evaluation Data Actionable

Instructor: Dr. Ann Doucette, TEI Director, Research Professor, Columbian College of Arts and Sciences, George Washington University

Description: Interventions and programs are implemented within complex environments that present challenges for collecting program performance information. A general problem for performance measurement initiatives — and what often causes them to fall short of their intended objectives — is the failure to choose performance measures that are actionable, meaning that they are linked to practices that an organization or agency can actually do something about, and the changes in those practices can be linked directly to improved outcomes and sustained impact.

This class introduces complex adaptive systems (CAS) thinking and addresses the implication of CAS in evaluating the outcomes and impact of interventions and programs. Examples used in this case range from healthcare, education, transportation and safety, developing countries, and research and development environments. The class examines performance measurement strategies that support actionable data. The focus will be on data-based decision making, value-based issues, and practice-based evidence that can assist in moving performance measurement and quality monitoring activities from a process, outcome, and impact evaluation approach to continuous quality improvement. Business models such as Toyota Production System, Six-sigma, Balanced Scorecards, as well as knowledge management and benchmarking strategies will be discussed in terms of how they can inform improvement strategies.

Note: Persons with some experience in program evaluation, and those with interest in a systems perspective will likely derive the most benefit from this course.

Upcoming schedule for this class:
• Chicago, April 2012  [ course listing ]
• Washington, DC, July 2012  [ course listing ]

>> Return to top

Strategy Mapping

Instructor: Dr. John Bryson, McKnight Presidential Professor of Planning and Public Affairs, Hubert H. Humphrey Institute of Public Affairs, University of Minnesota, Minneapolis

Description: The world is often a muddled, complicated, dynamic place in which it seems as if everything connects to everything else – and that is the problem! The connections can be problematic because – while we know things are connected – sometimes we do not know how, or else there are so many connections we cannot comprehend them all. Alternatively, we may not realize how connected things are and our actions lead to unforeseen and unhappy consequences. Either way, we would benefit from an approach to strategizing, problem solving, conflict management, and evaluation that helps us understand just how connected the world is, what the effects of those connections are, and what might be done to change some of the connections and their effects.

Action-oriented strategy mapping (AOSM) is a simple and useful technique for addressing situations where thinking – as an individual or as a group – matters. An action-oriented strategy map is a word-and-arrow diagram in which ideas and actions are causally linked with one another through the use of arrows. The arrows indicate how one idea or action leads to another – and specifically how coherent sets of actions, strategies, and goals can be created, communicated, managed, and implemented. AOSM makes it possible to articulate a large number of ideas and their interconnections in such a way that people can know what to do in an area of concern, how to do it, and why, because the arrows indicate the causes and consequences of an idea or action. AOSM therefore is a technique for linking strategic thinking, acting, and learning; helping make sense of complex problems; communicating to oneself and others what might be done about them; and also managing the inevitable conflicts that arise. The technique is useful for formulating and implementing mission, goals, and strategies and for being clear about how to evaluate strategies. The bottom line is: AOSM is one of the most powerful strategic management tools in existence. AOSM is what to do when thinking matters!

When can mapping help? There are a number of situations that are tailor-made for mapping. Mapping is particularly useful when:

• effective strategies need to be developed
• persuasive arguments are needed
• effective and logical communication is essential
• effective understanding and management of conflict are needed
• it is vital that a situation be understood better as a prelude to any action
• organizational or strategic logic needs to be clarified in order to design useful evaluations

These situations are not meant to be mutually exclusive. Often they will overlap in practice. In addition, mapping is very helpful for creating business models (that link competencies and distinctive competencies to mission, goals, and critical success factors) and Balanced Scorecards (which are more useful for strategy implementation than for strategy formulation). AOSMs are related to logic models, as both are word-and-arrow diagrams, but are more tied to goals, strategies, and actions and are more careful about articulating causal connections.

Objectives:

At the end of the course, participants will:

• Understand the theory of mapping
• Know the difference between action-oriented strategy maps, business model maps, and balanced scorecard maps
• Be able to create action-oriented strategy maps for individuals – that is, either for oneself or by interviewing another person
• Be able to create action-oriented maps for groups
• Be able to create a business model map linking competencies and distinctive competencies to goals and critical success factors
• Know how to design and manage change processes in which mapping is prominent
• Have an action plan for an individual project

Upcoming schedule for this class:
• Chicago, April 2012  [ course listing ]
• Washington, DC, July 2012  [ course listing ]

>> Return to top

Effective Reporting Strategies for Evaluators

Instructor: Dr. Kathryn Newcomer, Professor and Director, The Trachtenberg School of Public Policy and Public Administration, The George Washington University, Washington, DC

Description: The use and usefulness of evaluation work is highly affected by the effectiveness of reporting strategies and tools. Care in crafting both the style and substance of findings and recommendations is critical to ensure that stakeholders pay attention to the message. Skill in presenting sufficient information -- yet not overwhelming the audience -- is essential to raise the likelihood that potential users of the information will be convinced with both the relevance and the validity of the data. This course will provide guidance and practical tips on reporting evaluation findings. Attention will be given to the selection of appropriate reporting strategies/formats for different audiences and to the preparation of: effective executive summaries; clear analytical summaries of quantitative and qualitative data; user-friendly tables and figures; discussion of limitations to measurement validity, generalizability; causal inferences, statistical conclusion validity, and data reliability; and useful recommendations. The text provided as part of course fee is Torres et al., Evaluation Strategies for Communicating and Reporting (2nd Ed., Sage, 2005).

Upcoming schedule for this class:
• Chicago, April 2012  [ course listing ]
• Washington, DC, July 2012  [ course listing ]

>> Return to top

Evaluating the Decision for Contracting, Outsourcing, and Partnering

Instructor: Dr. James Edwin Kee, Professor, Trachtenberg School of Public Policy and Public Administration Department of the George Washington University

Description: Organizations of all types—government, non-profit, and for profit—are faced with a bewildering array of outsourcing decisions, including contracting out to another organization, engaging in a public-private partnership, or providing grants to accomplish a given task. This course will provide an evaluation mechanism that organizations can apply to a variety of outsourcing decisions, whether in their decision-making stage or upon implementation. The course will be useful for decision makers, those involved in the contracting process, and those evaluating contracting decisions. Included are an overview of the legal and political environment, methods for comparing costs and effectiveness, case studies to evaluate, and the use of spreadsheets to aid the analysis process.

>> Return to top

How to Build a Successful Evaluation Consulting Practice

Instructor: Dr. Michael Quinn Patton, Director, Utilization-Focused Evaluation and Independent Evaluation Consultant

Description: This class offers the opportunity for participants to learn from someone who has been a successful evaluation consultant for 30 years. Issues addressed include: What does it take to establish an independent consulting practice? How do you find your consulting niche? How do you attract clients, determine how much to charge, create collaborations, and generate return business? Included will be discussion on such topics as marketing, pricing, bidding on contracts, managing projects, resolving conflicts, professional ethics, and client satisfaction. Participants will be invited to share their own experiences and seek advice on situations they've encountered. Then course is highly interactive and participant-focused.

>> Return to top

Implementation Analysis for Feedback on Program Progress and Results

Instructor: Dr. Arnold Love, Internationally-recognized independent consultant based in Toronto, Canada

Description: Many programs do not achieve intended outcomes because of how they are implemented. Thus, implementation analysis (IA) is very important for policy and funding decisions. IA fills the methodological gap between outcome evaluations that treat a program as a "black box" and process evaluations that present a flood of descriptive data. IA provides essential feedback on the "critical ingredients" of a program, and helps drive change through an understanding of factors affecting implementation and short-term results. Topics include: importance of IA; conceptual and theoretical foundations of IA; how IA drives change and complements other program evaluation approaches; major models of IA and their strengths/weaknesses; how to build an IA framework and select appropriate IA methods; concrete examples of how IA can keep programs on-track, spot problems early, enhance outcomes, and strengthen collaborative ventures; and suggestions for employing IA in your organization. Detailed course materials and in-class exercises are provided.

>> Return to top

Using Program Evaluation in Nonprofit Environments

Instructor: Dr. Kathryn Newcomer, Professor and Director, The Trachtenberg School of Public Policy and Public Administration, The George Washington University, Washington, DC

Description: Funders and oversight boards typically need data on the results obtained by the programs they fund. Within foundations program officers want information about grantees, and about lines of effort they fund to guide planning and future allocation of resources. Executive officers and members of the boards that oversee non-profit service providers also want to know what works and what does not. This class provides background that program officers and overseers need to understand how evaluation can serve their information needs, and how to assess the quality of the evidence they receive.

Drawing upon cases from foundations and nonprofits, the session will help attendees:

• Learn what/who drives program evaluation and performance measurement in public and non-profit service providers;
• Explore uses of evaluation and outcomes assessment in the non-profit sector
• Understand how to frame useful scopes of work (SOWs) and requests for proposals (RFPs) for evaluations and performance measurement systems;
• Identify and apply relevant criteria in choosing contractors and consultants to provide evaluation assistance;
• Discuss challenges to measurement of social service outcomes;
• Understand what questions to ask of internal evaluation staff and outside consultants about the quality of their work; and
• Clarify where to start in using evaluation to improve non-profit social service programs.
  

Upcoming schedule for this class:
• Washington, DC, July 2012  [ course listing ]

>> Return to top

Culture and Evaluation

Instructor: Dr. Leona Ba, international development consultant

Description: This course focuses on examining how culture affects the effectiveness of evaluation. It begins with a definition of culture and a brief overview of major cultural theories and models. Participants will be encouraged to reflect on their own cultural sensitivity, a prerequisite for conducting culturally sensitive evaluations. The course will discuss cultural factors affecting the effectiveness of evaluation at different levels, including the evaluator, the evaluation context and the evaluation process. Participants will explore strategies for applying cultural sensitivity to evaluation practice using examples from first-hand experience and from reviews of various program evaluations. In order to make the most of this one-day course, students will be sent reading materials to review prior to the course.

Prerequisite: Understanding of evaluation and research design

Upcoming schedule for this class:
• Chicago, April 2012  [ course listing ]
• Washington, DC, July 2012  [ course listing ]

>> Return to top