Meeting Topic

Background and context

The Office of Planning, Research, and Evaluation (OPRE) routinely administers surveys to evaluate Administration for Children and Families’ programs and better understand the communities in which programs operate. Surveys enable OPRE to collect and summarize perspectives from large numbers of program staff, partners, and participants with the goal of improving the implementation and delivery of human services. However, declining survey participation rates over the past decade,^[1] exacerbated by the COVID-19 pandemic, threaten to undermine the validity of survey data to inform important program improvements. When participants do not respond to a survey request, uncertainty about the accuracy or bias of estimates increases, leading to the possibility that survey findings may not adequately represent the needs, perspectives, or experiences of the population of focus. Such nonresponse bias can occur if the attitudes, characteristics, and experiences of respondents are systematically different from those of nonrespondents.

Of particular concern is that marginalized populations or communities with specific communication considerations (e.g., communities where English is not the primary language or Internet access is limited) may be less likely to participate in surveys than other groups. For instance, in 2020, researchers found that households with high incomes were more likely to respond to the U.S. Census Bureau’s Current Population Survey Annual Social and Economic Supplement than those with lower incomes, upwardly skewing reported income data.^[2] When survey respondents do not represent the full population of focus, data collection is inequitable and the data can be biased, limiting the survey’s utility for informing equitable programmatic and policy decisions. Therefore, it is important for OPRE to understand how to minimize unit missingness to ensure evaluations are rigorous and resulting data are high-quality and representative.

Researchers can, however, act at each step of the study design process to avoid, mitigate, and address the impact of nonresponse. Practices to improve outreach and recruitment of high priority populations—such as engaging community members to build trust and rapport, addressing privacy considerations, and using a range of methods and tools to reach participants—can significantly reduce nonresponse. Innovative questionnaire design and administration techniques, combined with usability and cognitive testing to design more respondent-centered and accessible surveys, can also encourage response.

Data from other sources can also be used to supplement or fill gaps left by missing survey data. For example, researchers can use observational methods to collect data, such as monitoring behaviors, observing interpersonal interactions, collecting passive data using sensors and geolocation, and collecting social media and digital trace data. Researchers may also link administrative data from private, federal, state, and program sources into their studies to augment missing data. Finally, many analytic techniques are available to measure and mitigate the impact of missing data on survey estimates, assess nonresponse bias, and adjust estimates for nonresponse. However, each of these strategies is associated with specific underlying assumptions, trade-offs, and limitations that researchers should carefully consider.

The 2023 OPRE Methods Meeting will convene researchers, evaluators, federal staff, and others working in human services research to address nonresponse in survey data. Experts in the field will discuss the reasons behind declining survey response rates and the potential for increased nonresponse bias before discussing the implications for human services research, programmatic decisions, and policy decision-making. Meeting presenters will also address research design strategies to reduce nonresponse and mitigate the impact of missing data on resulting estimates. They will also explore the use of other data sources to supplement and minimize the risk of missing data on surveys.

Meeting Topics and Goals

The meeting has the following goals:

• Define unit missing data and discuss potential implications for human services research and evaluation.
• Introduce strategies attendees can use to reduce survey nonresponse and mitigate the impact of missing data in their work.
• Identify data collection methods and non-survey data sources that can be used to supplement or replace survey data.
• Describe projects that successfully implemented strategies to mitigate survey nonresponse.

The meeting will include presentations and discussions on the following questions:

• Why is nonresponse an important issue in survey research?
• How can researchers identify patterns of missing data, including which populations have disproportionate nonresponse rates?
• How do missing data affect the quality of survey estimates and what implications does this have for policymaking and programmatic decisions?
• How can study and questionnaire designs reduce survey nonresponse?
• What data analysis techniques and strategies can researchers use to address missing data in their projects?
• What strategies can researchers use to improve outreach to populations that experience barriers to completing surveys.

Footnotes

[1] Office of Survey Methods Research. (2022). Household and establishment survey response rates. U.S. Bureau of Labor Statistics. https://www.bls.gov/osmr/response-rates/home.htm

[2] Rothbaum, J., & Bee, A. (2021). Coronavirus infects surveys, too: Survey nonresponse bias and the Coronavirus pandemic. https://www.census.gov/content/dam/Census/library/working-papers/2020/demo/sehsd-wp2020-10.pdf

Attendee Logistics

Join the discussion by tagging @OPRE_ACF and #OPREmethods on Twitter! 

You can also sign-up for the OPRE Newsletter, like OPRE on Facebook, and follow OPRE on Instagram.

Virtual Meeting Access for Attendees

The virtual meeting will take place on Wednesday, October 18 and Thursday, October 19, 2023 from 1:00 p.m. to 5:00 p.m. EDT each day.  Please see below for details about accessing and participating in the event. If you have additional questions, please contact OPREmeeting@insightpolicyresearch.com.

How will I access the OPRE methods meeting event?

The livestream site for the Methods Meeting is https://opre23.performedia.com/registration/.  Only people who have registered for the meeting can enter the site. If you have registered, click the Login button, which takes you to the Login page, then enter the email address you registered with and the passcode that was sent to you in the event reminder emails from opremeeting@insightpolicyresearch.com. You may need to check your ‘Other’ rather than ‘Focused’ tab of Outlook, or your junk box to find the emails.

If you have issues logging in to the livestream site, please contact OPREmeeting@insightpolicyresearch.com for assistance.

Are there any other technology requirements to view the event?

To ensure you can see the event, you should have an up-to-date version of your preferred internet browser (preferably Chrome).

Additionally, for the best experience, you should disconnect from VPN before the meeting and ensure you have a strong internet connection. You can go to speedtest.net to check your connection (10-12 Mbps download for HD streaming and 6 Mbps for SD streaming is recommended).

Will I be expected to turn on my video camera?

No, meeting participants will not be visible or audible and will not need to use their webcams. Instead, the presentations will be livestreamed on the meeting website. You will be able to interact with the speakers and other attendees by typing your questions, upvoting other comments and questions, and participating in polls. We also encourage you to engage via social media using the information at the top of this page!