Instructor: Dr. Ann Doucette
Description: Successful evaluation depends on our ability to generate evidence attesting to the feasibility, relevance and/or effectiveness of the interventions, services, or products we study. While theory guides our designs and how we organize our work, it is measurement that provides the evidence we use in making judgments about the quality of what we evaluate. Measurement, whether it results from self-report survey, interview/focus groups, observation, document review, or administrative data must be systematic, replicable, interpretable, reliable, and valid. While hard sciences such as physics and engineering have advanced precise and accurate measurement (i.e., weigh, length, mass, volume), the measurement used in evaluation studies is often imprecise and characterized by considerable error. The quality of the inferences made in evaluation studies is directly related to the quality of the measurement on which we base our judgments. Judgments attesting to the ineffective interventions may be flawed - the reflection of measures that are imprecise and not sensitive to the characteristics we chose to evaluate. Evaluation attempts to compensate for imprecise measurement with increasingly sophisticated statistical procedures to manipulate data. The emphasis on statistical analysis all too often obscures the important characteristics of the measures we choose. This class content will cover these topics:
-
Assessing
measurement precision: examining the precision of measures in relationship
to the degree of accuracy that is needed for what is being evaluated.
Issues to be addressed include: measurement/item bias, the sensitivity
of measures in terms of developmental and cultural issues, scientific
soundness (reliability, validity, error, etc.), and the ability of the
measure to detect change over time.
-
Quantification: Measurement
is essentially assigning numbers to what is observed (direct and inferential).
Decisions about how we quantify observations and the implications these
decisions have for using the data resulting from the measures, as well
as for the objectivity and certainty we bring to the judgment made in
our evaluations will be examined. This section of the course will focus
on the quality of response options, coding categories - Do response
options/coding categories segment the respondent sample in meaningful
and useful ways?
-
Issues and Considerations - using
existing measures versus developing your own measures: What to look
for and how to assess whether existing measures are suitable for your
evaluation project will be examined. Issues associated with the development
and use of new measures will be addressed in terms of how to establish
sound psychometric properties, and what cautionary statements should
accompanying interpretation and evaluation findings using these new
measures.
-
Criteria
for choosing measures: assessing the adequacy of measures in terms
of the characteristics of measurement - choosing measures that fit your
evaluation theory and evaluation focus (exploration, needs assessment,
level of implementation, process, impact and outcome). Measurement feasibility,
practicability and relevance will be examined. Various measurement techniques
will be examined in terms of precision and adequacy, as well as the
implications of using screening, broad-range, and peaked tests.
- Error - influences on measurement
precision: The characteristics of various measurement techniques,
assessment conditions (setting, respondent interest, etc.), and evaluator
characteristics will be addressed.
Participants will be provided with a copy of the text: Measurement Theory in Action (Case Studies and Exercises) by Shulz, K.S. and D.J. Whitney (Sage, 2004).
Certificates:
CEP IB.b or
CAEP IIB.b; and
CQEM III.d
|
