Meeting Summary

Panel 1 Summary

The discussion session focused on methods for standardizing the language used when reporting effect sizes and the information that is presented in journals and reports. To assist in the reporting of effect sizes, one panelist suggested collaborating with the publishing editors from professional associations to establish guidelines rather than working with individual journal editors. This suggestion was based on the observation that there are disciplinary differences in the reporting of effect sizes and in providing the context for the interpretation. When calculating effect sizes, common errors may result in multilevel models when researchers use statistical software packages without understanding the underlying formulas or functions—particularly if the variance component used is unknown or not clear. The panelists emphasized that effect sizes are not necessary when reporting natural units, though they may be useful for making comparisons across different measures. Also, percentage change may be an appropriate reporting mechanism when the measure has a natural zero point, such as dollars earned, but is of limited use in psychological studies (e.g., cognitive outcomes, affective measures). In conclusion, the suggested guidelines for reporting results included clearly defining the target population, providing a description of the comparability of measures to assess variability for contextual interpretation, and preparing tables containing standard deviations and regression coefficients.

Panel 2 Summary

The implications of measurement error differ depending on whether one is using natural or standardized units. One panelist noted that psychological outcomes are typically measured in arbitrary units, whereas economic indicators are typically measured in natural units. Measurement error for an outcome measured in natural units will affect standard errors and significance tests, but will not bias coefficient estimates. In contrast, measurement error for outcomes measured in standardized units is more problematic and introduces bias in coefficient estimates. In addition to reporting the measurement procedures, it is important to identify the intervention study design (e.g., random assignment versus pre-post) since these factors can influence the magnitude of the effect size expected. Standardized tests provide useful benchmarks from which to contextualize outcomes. How the contextualization markers are used depends on the goals of the comparison. When calculating effect sizes, researchers are comparing an impact to a standard deviation, but there are many standard deviations from which to select depending upon the research question. Some examples of reference points from which to select include: criterion reference points; social, normative, and moral standards; outcomes; and cost. Though usually not presented in detail, cost of an intervention is an important factor, though not the only factor, when interpreting the magnitude of the effect size. In terms of policy implications, intervention benefits for one population should not suggest that the intervention does not have potential negative effects for another population. Analyses must probe to find potential subgroup differences in effects.

Panel 3 Summary

Panelists indicated that intervention effects can be translated into financial benefits, but emphasized that this kind of translation introduces an additional form of uncertainty. In addition to the statistical uncertainty reflected in confidence intervals, projecting the long-term benefits of an intervention’s impact requires numerous assumptions. Panelists recommended examining the results of varying key assumptions. A theoretical framework can justify multiple outcomes, however, complications may arise when outcomes are all from the same family. When using a developmental trajectory approach for calculating benefits, it was noted that early outcomes may be difficult to quantify from a financial benefits perspective. Finally, when reporting effect size results, authors should take into account that stakeholders, such as policymakers and the public, have varying levels of knowledge on understanding and interpreting research results.