Evaluation Foundation Classes

Basics of Program Evaluation

(Also taught as Foundations of Evaluation: Theory, Method, and Practice)

Instructor: Dr. Arnold Love, internationally-recognized independent consultant based in Toronto, Canada

Description: This course focuses on design of program evaluation for human service programs in such areas as education, mental health, crime, poverty, employment, etc. A step-by-step approach is used starting with formulation of questions the evaluation should address and continuing with overviews of approaches for answering those questions. It is organized around the key concepts in program evaluation design and methods for application of the concepts to actual program situations, e.g., assessing need for a program; assessing program design and theory; assessing program process and implementation; assessing program outcome/impact; and assessing program cost and efficiency. The format will be a mix of presentation, discussion of concepts, and application exercises using case study materials. (Participants in this class will each receive one copy of the new 7th Edition (Sage, 2003) Rossi et al. text, Evaluation: A Systematic Approach.)

Note: This course will be an introduction to evaluation for persons familiar with social science research, but unfamiliar with evaluation, and a refresher course for evaluators who wish to review current practice and concepts.

Upcoming schedule for this class:
• Washington, DC, July 2012  [ course listing ]

>> Return to top

Evaluation Research Methods: A Survey of Quantitative and Qualitative Approaches

Instructor: Dr. David B. Wilson, Professor in the Department of Criminology, Law & Society at George Mason University

Description: This course will introduce a range of basic quantitative and qualitative research methods with emphasis on application of these to the evaluation of social programs. Topics will include observational and qualitative methods, survey and interview (structured and unstructured) techniques, using data from records and archives, experimental and quasi-experimental designs, and data analysis. Within these approaches, attention will be given to quantitative, qualitative, and mixed-method design considerations; and applications to both process and impact evaluation questions will be addressed. This course is for those who want to update their existing knowledge and skills and will serve as an introduction for those new to the topic. Text provided: Creswell, J. Research Design (Sage 2002).

Upcoming schedule for this class:
• Washington, DC, July 2012  [ course listing ]

>> Return to top

Informing Practice Using Evaluation Models and Theories

Instructor: Dr. Melvin M. Mark, Professor of Psychology at the Pennsylvania State University

Description: Evaluators who are not aware of the contemporary and historical aspects of the profession. "are doomed to repeat past mistakes and, equally debilitating, will fail to sustain and build on past successes." Madaus, Scriven and Stufflebeam (1983).

"Evaluation theories are like military strategy and tactics; methods are like military weapons and logistics. The good commander needs to know strategy and tactics to deploy weapons properly or to organize logistics in different situations. The good evaluator needs theories for the same reasons in choosing and deploying methods." Shadish, Cook and Leviton (1991).

These quotes from Madaus et al. (1983) and Shadish et al. (1991) provide the perfect rationale for why the serious evaluator should be concerned with models and theories of evaluation. The primary purpose of this class is to overview major streams of evaluation theories (or models), and to consider their implications for practice. Topics include: (1) why evaluation theories matter, (2) frameworks for classifying different theories, (3) in-depth examination of 4-6 major theories, (4) identification of key issues on which evaluation theories and models differ, (5) benefits and risks of relying heavily on any one theory, and (6) tools and skills that can help you in picking and choosing from different theoretical perspectives in planning an evaluation in a specific context. The overarching theme will be on practice implications, that is, on what difference it would make for practice to follow one theory or some other.

Theories to be discussed will be ones that have had a significant impact on the evaluation field, that offer perspectives with major implications for practice, and that represent important and different streams of theory and practice. Case examples from the past will be used to illustrate key aspects of each theory's approach to practice.

Participants will be asked to use the theories to question their own and others' practices, and to consider what characteristics of evaluations will help increase their potential for use. Each participant will receive Marvin Alkin's text, Evaluation Roots (Sage, 2004) and other materials.

The instructor's assumption will be that most people attending the session have some general familiarity with the work of a few evaluation theorists, but that most will not themselves be scholars of evaluation theory. At the same time, the course should be useful, whatever one's level of familiarity with evaluation theory.

Upcoming schedule for this class:
• Washington, DC, July 2012  [ course listing ]

>> Return to top

Professional Standards and Principles for Evaluation Practice

Instructor: Dr. Michael Morris, Professor of Psychology at the University of New Haven

Description: Professional standards and principles are essential guides for effective and ethical evaluation practice. They provide criteria for designing comprehensive evaluations; for stimulating use of findings; and for teaching clients what evaluation is and what should be expected from it. And, when used in proposal development, they provide evidence of evaluator competence. This course introduces the Joint Committee's Program Evaluation Standards and AEA's Guiding Principles for Evaluators, examines strategies for using them to enhance evaluation quality, and explores issues that can arise in their application (e.g., dealing with conflicting priorities, addressing cultural and subcultural differences). The course also addresses how the Standards/Principles can be employed to prevent, and respond effectively to, ethical problems encountered as an evaluation unfolds. Participants will have an opportunity to apply their learning to case studies involving evaluation planning and ethical challenges. Copies of the Guiding Principles and the Program Evaluation Standards text are provided as a part of the registration fee.

>> Return to top

Working with Evaluation Stakeholders

Instructor: Dr. John Bryson, McKnight Presidential Professor of Planning and Public Affairs, Hubert H. Humphrey Institute of Public Affairs, University of Minnesota, Minneapolis

Description: Working with stakeholders is a fact of life for evaluators. That interaction can be productive and beneficial to evaluation studies that inform decisions and produce positive outcomes for decision makers and program recipients. Or that interaction can be draining and conflictual for both the evaluator and the stakeholders and lead to studies that are misguided, cost too much, take too long, never get used, or never get done at all. So this is an incredibly important topic for evaluators to explore. This course focuses on strategies and techniques to identify those who can and will be most beneficial for the achievement of study goals and how to achieve a productive working relationship with them. Stakeholder characteristics like knowledge of the program, power and ability to influence, willingness to participate, etc., will be analyzed and strategies and techniques are presented to successfully engage stakeholders for effective collaboration. Detailed course materials, case examples, and readings are provided to illuminate course content and extend its long-term usefulness.

Upcoming schedule for this class:
• Chicago, April 2012  [ course listing ]
• Washington, DC, July 2012  [ course listing ]

>> Return to top

Project Management & Oversight for Evaluators

Instructor: Mr. James Bell is the president of James Bell Associates, Inc., a firm that has specialized in national health and human services program evaluation for more than 30 years

Description: Evaluation theory and methods meet the reality of program operations and decision-making when a program evaluation is implemented. Against a backdrop of demanding technical requirements and a dynamic political environment, the goal of evaluation management is to develop, with available resources and time, valid and useful measurement information. Management responsibility rests with a project director and, to a lesser extent, other project staff. Those who oversee evaluations, usually as sponsors' representatives, also have management responsibilities, such as developing specifications for a call for proposals, and selecting and monitoring the evaluation team. Evaluation management is challenging because the choice of methods (quantitative or qualitative) and the prevailing political context varies across projects.

The purpose of this course is to increase the evaluation management knowledge and skills of those who conduct evaluations (as project directors or staff) and others who have oversight responsibility (for programs and studies). Participants will have opportunities to improve their ability to directly manage or oversee each phase of an evaluation. Using a combination of short presentations, seminar discussions, and small group exercises, the course will focus on six areas: (1) selecting the evaluator(s); (2) developing rational work plans; (3) organizing staff for results; (4) making assignments productive; (5) monitoring interim progress; and (6) ensuring product quality and usefulness. Practical suggestions will focus on how common problems can be avoided or resolved. In addition, cross-cutting topics will be addressed such as: how the six areas of evaluation management are interrelated; how project management determines the quality and usefulness of evaluation products; and, why maintaining agreement with sponsors on the evaluation mandate is crucial. Case examples of evaluation management challenges and solutions will be examined and used in class exercises. The case examples will be provided by the instructor as well as drawn from course participants' evaluation experiences.

Upcoming schedule for this class:
• Washington, DC, July 2012  [ course listing ]

>> Return to top

Applied Measurement for Evaluation

Instructor: Dr. Ann Doucette, TEI Director, Research Professor, Columbian College of Arts and Sciences, George Washington University

Description: Successful evaluation depends on our ability to generate evidence attesting to the feasibility, relevance and/or effectiveness of the interventions, services, or products we study. While theory guides our designs and how we organize our work, it is measurement that provides the evidence we use in making judgments about the quality of what we evaluate. Measurement, whether it results from self-report survey, interview/focus groups, observation, document review, or administrative data must be systematic, replicable, interpretable, reliable, and valid. While hard sciences such as physics and engineering have advanced precise and accurate measurement (i.e., weigh, length, mass, volume), the measurement used in evaluation studies is often imprecise and characterized by considerable error. The quality of the inferences made in evaluation studies is directly related to the quality of the measurement on which we base our judgments. Judgments attesting to the ineffective interventions may be flawed - the reflection of measures that are imprecise and not sensitive to the characteristics we chose to evaluate. Evaluation attempts to compensate for imprecise measurement with increasingly sophisticated statistical procedures to manipulate data. The emphasis on statistical analysis all too often obscures the important characteristics of the measures we choose. This class content will cover these topics:

• Assessing measurement precision: examining the precision of measures in relationship to the degree of accuracy that is needed for what is being evaluated. Issues to be addressed include: measurement/item bias, the sensitivity of measures in terms of developmental and cultural issues, scientific soundness (reliability, validity, error, etc.), and the ability of the measure to detect change over time.
• Quantification: Measurement is essentially assigning numbers to what is observed (direct and inferential). Decisions about how we quantify observations and the implications these decisions have for using the data resulting from the measures, as well as for the objectivity and certainty we bring to the judgment made in our evaluations will be examined. This section of the course will focus on the quality of response options, coding categories - Do response options/coding categories segment the respondent sample in meaningful and useful ways?
• Issues and Considerations - using existing measures versus developing your own measures: What to look for and how to assess whether existing measures are suitable for your evaluation project will be examined. Issues associated with the development and use of new measures will be addressed in terms of how to establish sound psychometric properties, and what cautionary statements should accompanying interpretation and evaluation findings using these new measures.
• Criteria for choosing measures: assessing the adequacy of measures in terms of the characteristics of measurement - choosing measures that fit your evaluation theory and evaluation focus (exploration, needs assessment, level of implementation, process, impact and outcome). Measurement feasibility, practicability and relevance will be examined. Various measurement techniques will be examined in terms of precision and adequacy, as well as the implications of using screening, broad-range, and peaked tests.
• Error - influences on measurement precision: The characteristics of various measurement techniques, assessment conditions (setting, respondent interest, etc.), and evaluator characteristics will be addressed.

Participants will be provided with a copy of the text: Measurement Theory in Action (Case Studies and Exercises) by Shulz, K.S. and D.J. Whitney (Sage, 2004).

Upcoming schedule for this class:
• Chicago, April 2012  [ course listing ]
• Washington, DC, July 2012  [ course listing ]

>> Return to top