Designing, Managing, and Analyzing Multi-Site Evaluations

Instructor: Dr. Debra J. Rog

Description: Guidance on how to carry out multi-site evaluations is scarce and what is available tends to focus on quantitative data collection and analysis and usually treats diverse sites in a uniform manner. This course will present instruction on designing, managing, and analyzing multi-site studies and focus on the differences that are required due to the specifics of the situation, e.g., central evaluator control vs. interactive collaboration; driven by research vs. program interests; planned and prospective vs. retrospective; varied vs. standardized sites; exploratory vs. confirmatory purpose; and data that are exclusively quantitative vs. qualitative vs. mixture. Topics include stakeholder involvement, collaborative design, maintaining integrity/quality in data, monitoring and technical assistance, data submission, communication and group process, cross-site synthesis and analysis, and cross-site reporting and dissemination. Practical strategies learned through first-hand experience as well as from review of other studies will be shared. Teaching will include large- and small-group discussions and students will work together on several problems. Detailed course materials are provided. Text provided: Herrell, J.M. & R.B. Straw, Conducting Multiple Site Evaluations in Real-World Settings, New Directions in Evaluation #94 (Jossey-Bass, 2002).

Prerequisite: Understanding of evaluation and research design.

Instructor: Dr. Debra J. Rog directs the Washington office of the Vanderbilt University Center for Mental Health Policy, Institute for Public Policy Studies. She has 20+ years of experience in program evaluation/applied research and has directed numerous multi-site evaluations and research projects, most for vulnerable populations. Current projects include multi-site/multi-component studies of supported housing for persons with serious mental illnesses, mental health and substance abuse interventions for homeless families, housing services for persons with HIV/AIDS, and community partnerships and collaborations focused on violence preventions. She has numerous publications on evaluation methodology, housing, homelessness, poverty, mental health, and program and policy development, and is co-editor of the Applied Social Research Methods Series and the recently published Handbook of Applied Social Research Methods (1997, Sage). She has a recent article on the methodological lessons learned from cross-site collaboration in the multi-site evaluation of supported housing and has another multi-site effort in press, "Sustaining Collaboratives: A Cross-Site Analysis of the National Funding Collaborative on Violence Prevention." She served on AEA's Board of Directors; and the Advisory Committee of Women's Services for the U.S. Substance Abuse and Mental Health Services Administration.

Certificates: CEP IC.g or CAEP IIC.g

CEU: 1.4