Ignoring Non-English-Language Studies May Bias Ecological Meta-Analyses

Konno, K., Akasaka, M., Koshida, C., Katayama, N., Osada, N., Spake, R. and Amano, T. (2020) Ecology and Evolution.

By reanalysing existing meta-analyses including both English- and Japanese-language studies, we show that effect sizes differ significantly between English- and Japanese-language studies, causing considerable changes in overall mean effect sizes and even their direction when Japanese-language studies are excluded (see figure below). The differences in effect sizes are likely attributable to systematic differences in reported statistical results (language bias in statistical results – see diagram below) as well as study characteristics, particularly taxa and ecosystems (language bias in study characteristics), between English- and Japanese-language studies. This finding has a broad, yet simple implication: future meta-analyses—particularly those conducted at global extents or in regions where English is not widely spoken—should actively search for relevant non-English-language studies, and if appropriate, include them.

View full publication, here.

Differences in mean effect sizes between English- (blue) and Japanese-language studies (red) included in existing ecological meta-analyses (left: Koshida & Katayama (2018) Conservation Biology, right: Osada et al. (2013) Japanese Journal of Ecology).

The fate of ignoring studies published in relevant language(s). Studies providing certain information may be more likely to be published in non-English languages (language bias in study characteristics) because, for example, those studies are often not of great interest from an international perspective. After the analysis, statistically significant or positive results may be more likely to be published in higher-impact, English-language journals (language bias in statistical results).