Linking Evaluation Questions |
Instructor: Dr. Melvin M. Mark Description: Statistics are a mainstay in the toolkit of program and policy evaluators. Human memory being what it is, however, even evaluators with reasonable statistical training, over the years, often forget the basics. And the basics aren't always enough. If evaluators are going to make sensible use of consultants, communicate effectively with funders, and understand others' evaluation reports, then they often need at least a conceptual understanding of relatively complex, recently developed statistical techniques. The purposes of this course are: to link common evaluation questions with appropriate statistical procedures; to offer a strong conceptual grounding in several important statistical procedures; and to describe how to interpret the results from the statistics in ways that are principled and will be persuasive to intended audiences. The general format for the class will be to start with the evaluation question(s) and then discuss the choice and interpretation of the most-suited statistical test(s). Emphasis will be on creating a basic understanding of what statistical procedures do, of when to use them, and why, and then on how to learn more from the data. Little attention is given to equations or computer programs, with the emphasis instead being on conceptual understanding and practical choices.* Within a framework of common evaluation questions, statistical procedures and principled data inquiry will be explored.(A) More fundamental topics to be covered include (1) basic data quality checks and basic exploratory data analysis procedures, (2) basic descriptive statistics, (3) the core functions of inferential statistics (why we use them), (4) common inferential statistics, including t-tests, the correlation coefficient, and chi square, and (5) the fundamentals of regression analysis. (B) For certain types of evaluation questions, more complex statistical techniques need to be considered. More complex techniques to be discussed (again, at a conceptual level) include (1) structural equation modeling, (2) multi-level modeling, and (3) cluster analysis and other classification techniques. (C) Examples of methods for learning from data, i.e., for "snooping" with validity, for making new discoveries principled, and for more persuasive reporting of findings will include (1) planned and unplanned tests of moderation, (2) graphical methods for unequal treatment effects, (3) use of previously-discussed techniques such as clustering, (4) identifying and describing converging patterns of evidence, and (5) iterating between findings and explanations. Each participant will receive a set of readings and current support materials. Prerequisites: Familiarity with basic statistics. *This course is not the equivalent of nor a substitute for Dr. Poister's Applied Statistics for Evaluators course (offered at previous Institute programs). Similarly, the course is not the equivalent of nor a substitute for Dr. Mark's previously taught course on Understanding Data that was offered at several previous Institute programs. This course looks at a longer chain (from evaluation question to choice of statistical procedure to interpretation), and will spend less time on the interpretation side than was possible when an entire two days of time was devoted to the topic of interpretation. Certificates: CEP, IB.e; or CAEP, IIB.e; and CQEM, III.g. |