Rapid Learning: Methods for Testing and Evaluating Change in Social Service Programs

Meeting Topic

Background and Context. There is increasing desire for data-driven approaches to examine program implementation, impact, and improvement efforts faster – even in real-time – to provide feedback and input for program modification. Numerous approaches exist to help support such rapid learning (i.e. Rapid Cycle Evaluation and Continuous Quality Improvement, including specific strategies like Plan Do Study Act cycles). All use data, some employ methods to determine causality, and some incorporate advanced statistical methods to make predictions. Unfortunately, this broad array of methods—along with overlapping and sometimes unclear terminology—may make it difficult to determine which methods are most appropriate given the specific challenges facing a program.

OPRE’s 2018 research methods meeting seeks to increase the field’s understanding of how rapid learning methods can be helpful, and to share information about methods for rapid learning so that researchers and practitioners can use the most appropriate methods to evaluate programs. We focus on rapid learning methods designed to quickly test program improvements and evaluate program implementation or impact. These approaches often use program data to monitor and measure improvements, and are typically embedded in a cycle of learning. Selecting the most appropriate methodology depends on numerous factors including: the research questions being asked; what data are available to address the questions; who is asking the questions and conducting the research; how the results will be used; the required level of confidence in analysis results; the level of disruption to the program; whether the program being evaluated is new or established; potential costs of implementing the method; and the potential decision cycle.

Meeting Topics and Goals. The 2018 research methods meeting will provide attendees with a deeper understanding of rapid learning methods and their underlying methodologies. Speakers will describe key questions that can be addressed by rapid learning methods, along with relevant considerations for selecting the most appropriate method for different contexts. They will share their expertise in understanding and supporting rapid learning efforts, and provide examples of successful implementation among Federally-funded programs as part of a broader conversation about supporting program improvement and decision-making. The meeting may include presentations and discussions on the following questions:

  • What are rapid learning methods? What specific methodological approaches and frameworks exist for collecting, analyzing, and responding to data in a compressed timeframe?
  • What factors or considerations affect the selection of the right method (e.g., stakeholders involved; appropriate level of confidence in the results; availability and frequency of data; maturity of the program being studied)? How can researchers and program staff apply the highest quality, most reliable methods possible while balancing other concerns including budget, speed, and relevance?
  • What data are typically available to inform rapid learning? What challenges do researchers and program staff face in accessing appropriate data? What challenges are there around identifying appropriate outcome measures that are both meaningful and sensitive to change?
  • What can social services learn about rapid learning methods from fields that have a longer history of using the methods, such as health care and business?
  • How can Federal staff and researchers support local programs in building capacity to collect, analyze, and use data to inform practice? What does collaboration among multiple stakeholders look like in the context of rapid learning? What role do management information systems and performance measures with multiple functions (e.g., program management and continuous learning) play in this process?
  • How can rapid learning methods be evaluated? How can they be integrated into larger impact and/or implementation evaluations?

The goals of the meeting are to:

  1. Identify and describe high-quality rapid learning methods, discuss how they can complement traditional research and analysis methods, and describe the considerations for selecting a particular methodological approach.
  2. Provide illustrative examples of rapid learning methods to help attendees explore how these methods can be designed and implemented in high-quality, effective ways.
  3. Encourage thinking on ways Federal offices can facilitate effective use of high-quality rapid learning methods.

Agenda and Presentations

Day 1: Thursday, October 25

Greetings

9:00 – 9:30 a.m.

Naomi Goldstein (Deputy Assistant Secretary for Planning, Research, and Evaluation)

Emily Schmitt (Deputy Director for the Office of Planning, Research, and Evaluation)

Rapid Learning: Methods for Testing and Evaluating Change in Social Programs

9:30 – 10:30 a.m.

Rapid Learning: Methods for Testing and Evaluating Change in Social Programs

Scott Cody (Insight Policy Research)

MaryCatherine Arbour (Harvard University)

Supporting the Implementation of Rapid Learning Methods: Perspectives from the Field

3:15 – 5:00 p.m.

Panelists:

Jodi Sandfort (University of Minnesota)

Tyson Barker (University of Oregon)

Robert Goerge (Chapin Hall)

Bi Vuong (Harvard University)

Moderator:

Virginia Knox (MDRC)

Informal Gathering

Evening

All are welcome to continue the discussion at the 21st Amendment Bar & Grill (inside hotel). This is an informal, no-host gathering; drinks and food are available for purchase.

Day 2: Friday, October 26

Using Rapid Learning Methods to Answer Policy Relevant Questions

9:00 – 10:45 a.m.

Panelists:

Kinsey Dinan (NYC Department of Social Services)

Nick Hart (Bipartisan Policy Center)

Jennifer Lloyd (The Centers for Medicare & Medicaid Services)

Moderator:

Erica Zielewski (Office of Management & Budget)

Creating a Lasting Culture of Rapid Learning

11:00 a.m. – 12:00 p.m.

Developing, Maintaining, and Spreading a Culture of Rapid Cycle Learning Within Home Visiting: Lessons Learned

Mary Mackrain (Education Development Center)

Julia Heany (Michigan Public Health Institute)

Meeting Products

Coming Soon