Errors and secret data in the Italian research assessment exercise. A comment to a reply

Authors

DOI:

https://doi.org/10.13130/2282-5398/8872

Keywords:

iInformed peer review, research assessment, bibliometric evaluation, Italian VQR, peer review, open data, undisclosed data, sound science, replicability crisis, reproducibility crisis

Abstract

Italy adopted a performance-based system for funding universities that is centered on the results of a national research assessment exercise, realized by a governmental agency (ANVUR). ANVUR evaluated papers by using “a dual system of evaluation”, that is by informed peer review or by bibliometrics. In view of validating that system, ANVUR performed an experiment for estimating the agreement between informed review and bibliometrics. Ancaiani et al. (2015) presents the main results of the experiment. Alberto Baccini and De Nicolao (2017) documented in a letter, among other critical issues, that the statistical analysis was not realized on a random sample of articles. A reply to the letter has been published by Research Evaluation (Benedetto et al. 2017). This note highlights that in the reply there are (1) errors in data, (2) problems with “representativeness” of the sample, (3) unverifiable claims about weights used for calculating kappas, (4) undisclosed averaging procedures; (5) a statement about “same protocol in all areas” contradicted by official reports. Last but not least: the data used by the authors continue to be undisclosed. A general warning concludes: many recently published papers use data originating from Italian research assessment exercise. These data are not accessible to the scientific community and consequently these papers are not reproducible. They can be hardly considered as containing sound evidence at least until authors or ANVUR disclose the data necessary for replication.

References

Abramo, Giovanni, Cicero, Tindaro, and D'Angelo, Ciriaco Andrea (2013), 'National peer-review research assessment exercises for the hard sciences can be a complete waste of money: The Italian case', Scientometrics, 95 (1), 311-24. DOI: 10.1007/s11192-012-0875-6

Abramo, Giovanni and D'Angelo, Ciriaco Andrea (2015), 'The VQR, Italy's second national research assessment: Methodological failures and ranking distortions', Journal of the Association for Information Science and Technology, 1-13. DOI: 10.1002/asi.23323

Abramo, Givanni and D’Angelo, Ciriaco Andrea (2017), 'On tit for tat: Franceschini and Maisano versus ANVUR regarding the Italian research assessment exercise VQR 2011–2014', Journal of Informetrics, 11 (3), 783-87. DOI: 10.1016/j.joi.2017.06.003

Ancaiani, Alessio, et al. (2015), 'Evaluating scientific research in Italy: The 2004–10 research

evaluation exercise', Research Evaluation, 24 (3), 242-55. DOI: 10.1093/reseval/rvv008

ANVUR (2013), 'Rapporto finale. Valutazione della qualità della ricerca 2004-2010 (VQR 2004-2010)', (Roma: http://www.anvur.org/rapporto/).

Baccini, Aaberto (2016), 'Napoléon et l’évaluation bibliométrique de la recherche. Considérations sur la réforme de l’université et sur l’action de l’agence national d’évaluation en Italie.', Canadian Journal of Information and Library Science-Revue Canadienne des Sciences de l'Information et de Bibliotheconomie, 40 (1), 37-57. DOI: 10.1353/ils.2016.0003

Baccini, Alberto and De Nicolao, Giuseppe (2016a), 'Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise', Scientometrics, 108 (3), 1651-71. DOI: 10.1007/s11192-016-1929-y

--- (2016b), 'Reply to the comment of Bertocchi et al', Scientometrics, 108 (3), 1675-84. DOI: 10.1007/s11192-016-2055-6

--- (2017), 'A letter on Ancaiani et al. ‘Evaluating scientific research in Italy: the 2004-10 research evaluation exercise’', Research Evaluation. DOI: 10.1093/reseval/rvx013

Benedetto, Sergio, et al. (2017), 'Reply to the letter on Ancaiani et al. ‘Evaluating Scientific research in Italy: The 2004–10 research evaluation exercise’', Research Evaluation. DOI: 10.1093/reseval/rvx017

Bertocchi, Graziella, et al. (2015), 'Bibliometric evaluation vs. informed peer review: Evidence from Italy', Research Policy, 44 (2), 451-66. DOI: 10.1016/j.respol.2014.08.004

Bonaccorsi, Andrea, et al. (2017), 'Do social sciences and humanities behave like life and hard sciences?', Scientometrics, 112 (1), 607-53. DOI: 10.1007/s11192-017-2384-0

Jappelli, Tullio, Nappi, Carmela Anna, and Torrini, Roberto (2017), 'Gender effects in research evaluation', Research Policy, 46 (5), 911-24. DOI: 10.1016/j.respol.2017.03.002

Downloads

Published

2017-07-22

Issue

Section

Discussion notes