Meeting Topic

Background and context

The Office of Planning, Research, and Evaluation (OPRE) is tasked with using rigorous research to evaluate Administration for Children and Families’ programs and build evidence to inform policy and practice. To fully describe how, when, and why a program works, OPRE periodically employs mixed methods designs, which integrate qualitative and quantitative approaches and data, drawing on the strengths of each to ask and answer research questions.  

When implemented well, a mixed methods approach to program evaluation offers several significant advantages. Integrating qualitative and quantitative methods can answer research questions that one approach alone cannot. For example, analysis of survey or aggregated administrative data may reveal a trend in program participation; combining these data with interviews with program participants can illuminate motivations for seeking program services and experiences receiving them.^[1] The context revealed through incorporating qualitative methods can help explain trends in the quantitative findings. Qualitative findings can also reveal options to modify the program design, delivery, outreach, or other important factors which may not be discovered through quantitative methods alone. Mixed methods can also add value beyond providing context by increasing the validity of findings—for example, by supplementing survey data with interviews or focus groups to expand the reach to other study groups or enhancing available quantitative data. Qualitative research can illustrate different perspectives, creating space to learn more about program participant motivations and experiences: this factor is critical for promoting equity in research. Some qualitative methods, such as Photovoice and arts-based approaches, broaden the ways in which people can express themselves and promote a participatory approach. Importantly, using qualitative approaches independently or mixing methods can provide opportunities for wider involvement from communities that are the focus of research.  

While more systematic integration of mixed methods approaches into OPRE’s work holds promise, it is critical that standards of rigor and reproducibility apply to each phase of the research process if the research will inform policy and program decision making.^[2] Researchers and evaluators should understand how to develop a mixed methods study design that is appropriate for the type of questions that decision-makers need answered. Similar to quantitative data, qualitative data analysis requires rigor. It is also important for study teams to understand how to effectively interpret findings from mixed methods research. In addition, qualitative research may present an opportunity to disseminate findings in a different way than quantitative findings, for example, with images and corresponding descriptions used from a Photovoice data collection. Researchers should be transparent about how all of the components of a study, including qualitative methods, are conceptualized and executed to make it easier for other teams to replicate the study with the same data.  

OPRE’s 2022 methods meeting convened researchers, evaluators, Federal staff, community members, and others working in different human services areas. This meeting broadly focused on how mixed method research approaches, along with qualitative methods within those approaches, can be applied to social policy and program evaluation. 

Meeting topics and goals

The meeting had the following goals: 

• Describe what mixed methods research and mixed methods evaluations are. Explore how researchers can integrate qualitative and quantitative methods to increase the rigor of results. 
• Define qualitative methods and understand how to write qualitative research questions, compared with the types of questions best answered by quantitative methods and mixed methods research designs. 
• Expose attendees to examples of innovative mixed methods and qualitative research questions and designs. 
• Help attendees consider how to integrate mixed methods and qualitative methods into their work from the start of a project. 

• Provide project officers and contractors with an understanding of how mixed methods designs, including qualitative approaches, fit into project life cycles. 
• Explore examples of high-quality, mixed methods approaches that have been applied in the context of federally-sponsored research and evaluation, particularly in the field of social policy. 

The meeting included presentations and discussions on the following questions: 

• What role do mixed and qualitative methods play in policymaking? What role could these methods play? 
• How can researchers and evaluators ensure rigor, credibility, and other considerations for high-quality mixed and qualitative methods in their work? 

• How can mixed and qualitative methods promote equity in research and evaluation by incorporating participant voices from a diversity of groups? 
• What are the potential benefits of using mixed and qualitative methods in research in various human services settings? What are the challenges to conducting this type of research? 
• What study designs are appropriate for the research questions that decision-makers need answered? What are some examples of mixed methods approaches that have proven successful? 
• How can researchers ensure that appropriate human subjects protections are applied with the expanded use of qualitative methods? 
• How does the terminology used in qualitative approaches differ from that of quantitative approaches? How can these terminology differences be reconciled? 

• What are examples of mixed methods approaches that researchers and evaluators have successfully applied in the context of human services evaluation (i.e., how can researchers use qualitative data to inform the appropriateness or applicability of a quantitative measure for a given population)? 
• How can we integrate qualitative and quantitative data in the context of evidence reviews? 
• How can we effectively and accurately interpret and communicate the results of qualitative and mixed methods research? 
• What are considerations when implementing mixed methods designs in federally-sponsored research? 

Meeting attendees and logistics

The meeting convened Federal staff and researchers interested in exploring mixed methods and qualitative approaches to research and evaluation. It was held virtually October 19 and 20, 2022. More than 500 participants and 20 speakers attended. Participants and speakers included representatives from the Federal Government, community members who have participated in research, research firms, and academia. 

Footnotes

1. Halpern-Meekin, S., Hoiting, J. (2022, February 18). Abdul Latif Jameel Poverty Action Lab. Strengthening randomized evaluations with qualitative research: Baby’s First Years household measurement. https://www.povertyactionlab.org/blog/2-18-22/strengthening-randomized-evaluations-qualitative-research-babys-first-years-household

2. Brannen, J., & Moss, G. (2012). Critical issues in designing mixed methods policy research. American Behavioral Scientist, 56(6), 789–801.