Meeting Topic

Background and Context. There is increasing desire for data-driven approaches to examine program implementation, impact, and improvement efforts faster – even in real-time – to provide feedback and input for program modification. Numerous approaches exist to help support such rapid learning (i.e. Rapid Cycle Evaluation and Continuous Quality Improvement, including specific strategies like Plan Do Study Act cycles). All use data, some employ methods to determine causality, and some incorporate advanced statistical methods to make predictions. Unfortunately, this broad array of methods—along with overlapping and sometimes unclear terminology—may make it difficult to determine which methods are most appropriate given the specific challenges facing a program.

OPRE’s 2018 research methods meeting sought to increase the field’s understanding of how rapid learning methods can be helpful, and to share information about methods for rapid learning so that researchers and practitioners can use the most appropriate methods to evaluate programs. We focus on rapid learning methods designed to quickly test program improvements and evaluate program implementation or impact. These approaches often use program data to monitor and measure improvements, and are typically embedded in a cycle of learning. Selecting the most appropriate methodology depends on numerous factors including: the research questions being asked; what data are available to address the questions; who is asking the questions and conducting the research; how the results will be used; the required level of confidence in analysis results; the level of disruption to the program; whether the program being evaluated is new or established; potential costs of implementing the method; and the potential decision cycle.

Meeting Topics and Goals. The 2018 research methods meeting provided attendees with a deeper understanding of rapid learning methods and their underlying methodologies. Speakers described key questions that can be addressed by rapid learning methods, along with relevant considerations for selecting the most appropriate method for different contexts. They shared their expertise in understanding and supporting rapid learning efforts, and provided examples of successful implementation among Federally-funded programs as part of a broader conversation about supporting program improvement and decision-making. The meeting included presentations and discussions on the following questions:

• What are rapid learning methods? What specific methodological approaches and frameworks exist for collecting, analyzing, and responding to data in a compressed timeframe?
• What factors or considerations affect the selection of the right method (e.g., stakeholders involved; appropriate level of confidence in the results; availability and frequency of data; maturity of the program being studied)? How can researchers and program staff apply the highest quality, most reliable methods possible while balancing other concerns including budget, speed, and relevance?
• What data are typically available to inform rapid learning? What challenges do researchers and program staff face in accessing appropriate data? What challenges are there around identifying appropriate outcome measures that are both meaningful and sensitive to change?
• What can social services learn about rapid learning methods from fields that have a longer history of using the methods, such as health care and business?
• How can Federal staff and researchers support local programs in building capacity to collect, analyze, and use data to inform practice? What does collaboration among multiple stakeholders look like in the context of rapid learning? What role do management information systems and performance measures with multiple functions (e.g., program management and continuous learning) play in this process?
• How can rapid learning methods be evaluated? How can they be integrated into larger impact and/or implementation evaluations?

The goals of the meeting were to:

1. Identify and describe high-quality rapid learning methods, discuss how they can complement traditional research and analysis methods, and describe the considerations for selecting a particular methodological approach.
2. Provide illustrative examples of rapid learning methods to help attendees explore how these methods can be designed and implemented in high-quality, effective ways.
3. Encourage thinking on ways Federal offices can facilitate effective use of high-quality rapid learning methods.