Errors and secret data in the Italian research assessment exercise. A comment to a reply
Keywords:iInformed peer review, research assessment, bibliometric evaluation, Italian VQR, peer review, open data, undisclosed data, sound science, replicability crisis, reproducibility crisis
AbstractItaly adopted a performance-based system for funding universities that is centered on the results of a national research assessment exercise, realized by a governmental agency (ANVUR). ANVUR evaluated papers by using “a dual system of evaluation”, that is by informed peer review or by bibliometrics. In view of validating that system, ANVUR performed an experiment for estimating the agreement between informed review and bibliometrics. Ancaiani et al. (2015) presents the main results of the experiment. Alberto Baccini and De Nicolao (2017) documented in a letter, among other critical issues, that the statistical analysis was not realized on a random sample of articles. A reply to the letter has been published by Research Evaluation (Benedetto et al. 2017). This note highlights that in the reply there are (1) errors in data, (2) problems with “representativeness” of the sample, (3) unverifiable claims about weights used for calculating kappas, (4) undisclosed averaging procedures; (5) a statement about “same protocol in all areas” contradicted by official reports. Last but not least: the data used by the authors continue to be undisclosed. A general warning concludes: many recently published papers use data originating from Italian research assessment exercise. These data are not accessible to the scientific community and consequently these papers are not reproducible. They can be hardly considered as containing sound evidence at least until authors or ANVUR disclose the data necessary for replication.
Abramo, Giovanni, Cicero, Tindaro, and D'Angelo, Ciriaco Andrea (2013), 'National peer-review research assessment exercises for the hard sciences can be a complete waste of money: The Italian case', Scientometrics, 95 (1), 311-24. DOI: 10.1007/s11192-012-0875-6
Abramo, Giovanni and D'Angelo, Ciriaco Andrea (2015), 'The VQR, Italy's second national research assessment: Methodological failures and ranking distortions', Journal of the Association for Information Science and Technology, 1-13. DOI: 10.1002/asi.23323
Abramo, Givanni and D’Angelo, Ciriaco Andrea (2017), 'On tit for tat: Franceschini and Maisano versus ANVUR regarding the Italian research assessment exercise VQR 2011–2014', Journal of Informetrics, 11 (3), 783-87. DOI: 10.1016/j.joi.2017.06.003
Ancaiani, Alessio, et al. (2015), 'Evaluating scientific research in Italy: The 2004–10 research
evaluation exercise', Research Evaluation, 24 (3), 242-55. DOI: 10.1093/reseval/rvv008
ANVUR (2013), 'Rapporto finale. Valutazione della qualità della ricerca 2004-2010 (VQR 2004-2010)', (Roma: http://www.anvur.org/rapporto/).
Baccini, Aaberto (2016), 'Napoléon et l’évaluation bibliométrique de la recherche. Considérations sur la réforme de l’université et sur l’action de l’agence national d’évaluation en Italie.', Canadian Journal of Information and Library Science-Revue Canadienne des Sciences de l'Information et de Bibliotheconomie, 40 (1), 37-57. DOI: 10.1353/ils.2016.0003
Baccini, Alberto and De Nicolao, Giuseppe (2016a), 'Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise', Scientometrics, 108 (3), 1651-71. DOI: 10.1007/s11192-016-1929-y
--- (2016b), 'Reply to the comment of Bertocchi et al', Scientometrics, 108 (3), 1675-84. DOI: 10.1007/s11192-016-2055-6
--- (2017), 'A letter on Ancaiani et al. ‘Evaluating scientific research in Italy: the 2004-10 research evaluation exercise’', Research Evaluation. DOI: 10.1093/reseval/rvx013
Benedetto, Sergio, et al. (2017), 'Reply to the letter on Ancaiani et al. ‘Evaluating Scientific research in Italy: The 2004–10 research evaluation exercise’', Research Evaluation. DOI: 10.1093/reseval/rvx017
Bertocchi, Graziella, et al. (2015), 'Bibliometric evaluation vs. informed peer review: Evidence from Italy', Research Policy, 44 (2), 451-66. DOI: 10.1016/j.respol.2014.08.004
Bonaccorsi, Andrea, et al. (2017), 'Do social sciences and humanities behave like life and hard sciences?', Scientometrics, 112 (1), 607-53. DOI: 10.1007/s11192-017-2384-0
Jappelli, Tullio, Nappi, Carmela Anna, and Torrini, Roberto (2017), 'Gender effects in research evaluation', Research Policy, 46 (5), 911-24. DOI: 10.1016/j.respol.2017.03.002
LicenseAuthors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).