Blue skies, impacts, and peer review


  • James Britt Holbrook University of North Texas
  • Steven Hrotic University of Vermont



peer review, impact, expertise, policy, autonomy, evaluation


This paper describes the results of a survey regarding the incorporation of societal impacts considerations into the peer review of grant proposals submitted to public science funding bodies. The survey investigated perceptions regarding the use of scientific peers to judge not only the intrinsic scientific value of proposed research, but also its instrumental value to society. Members of the scientific community have expressed – some more stridently than others – resistance to the use of such societal impact considerations. We sought to understand why. Results of the survey suggest that such resistance may be due to a lack of desire rather than a lack of confidence where judging impacts is concerned. In other words, it may be less that scientists feel unable to judge broader societal impacts and more that they are unwilling to do so.

Author Biography

James Britt Holbrook, University of North Texas

Assistant Director, Center for the Study of Interdisciplinarity

Research Assistant Professor, Department of Philosophy and Religion Studies


APS News (2007) ‘NSF‘s “broader impacts” criterion gets mixed reviews’, American Physical Society News 16. Available at, last accessed 8 June 2011.

Bhattacharya, Ananyo (2012) ‘Science Funding: Duel to the Death’, Nature 488. Available at, last accessed 19 March 2013.

Boden, Margaret, Ash, Eric, Edge, David, Reece, Charles, Skehel, John and Williams, Peter (1991) ‘Peer review: A report to the advisory board for the research councils from the working group on peer review’, Medical Research Council of England. Available at, last accessed 8 June 2011.

Bornmann, Lutz, Mutz, Ruediger and Hans-Deiter, Daniel (2007) ‘Gender differences in grant peer review: A meta-analysis’, Journal of Informetrics, 1: 226-238.

Bornmann, Lutz (2012) ‘Measuring the societal impact of research’, EMBO Reports, 13(8): 673-676.

Bornmann, Lutz (2013) ‘What is societal impact of research and how can it be assessed? A literature survey’. Journal of the American Society of Information Science and Technology, 64(2), 217-233.

Bozeman, Barry and Boardman, Craig (2009) ‘Broad impacts and narrow perspectives: Passing the buck on science and social impacts’, Social Epistemology, 23: 183-198.

Burggren, Warren W. (2009) ‘Implementation of the National Science Foundation’s “broader impacts”: Efficiency considerations and alternative approaches’, Social Epistemology, 23: 221-237.

Bush, Vannevar (1945) ‘Science: The Endless Frontier’, US Government Printing Office, Washington, DC. Available at, last accessed 8 June 2011.

CAPR Midterm Workshop Report. Available at, last accessed 8 June 2011.

Chubin, Daryl E. and Hackett, Edward J. (1990) Peerless Science: Peer Review and U.S. Science Policy. Albany: State University of New York Press.

Cole, Jonathan R. and Cole, Stephen (1981) Peer Review in the National Science Foundation: Phase Two of a Study. Washington, DC: National Academy Press.

Cole, Stephen, Rubin, Leonard and Cole, Jonathan R. (1978) Peer Review in the National Science Foundation: Phase One of a Study. Washington, DC: National Academy Press.

Edgerton, David (2009) ‘The “Haldane Principle” and other invented traditions in science policy’, History and Policy. Available at, last accessed 8 June 2011.

Eigenbrode, Sanford D., O‘Rourke, Michael, Wulfhorst, J. D., Althoff, David M., Goldberg, Caren S., Merrill, Kaylani, Morse, Wayde, Nielsen-Pincus, Max, Stephens, Jennifer, Winowiecki, Leigh and Bosque-Pérez, Nilsa A. (2007) ‘Employing philosophical dialogue in collaborative science’, BioScience, 57: 55-64.

Frodeman, Robert and Holbrook, J. Britt (2007) ‘Science‘s social effects’, Issues in Science and Technology, 23: 28-30.

Frodeman, Robert and Holbrook, J. Britt (2011) ‘NSF’s struggle to articulate relevance’, Science, vol. 333, July 8, 2011: 157-58.

Frodeman, Robert (2010) ‘Experiments in field philosophy’, New York Times, 23 November. Available at, last accessed 8 June 2011.

Frodeman, Robert and Briggle, Adam (2012) ‘The dedisciplining of peer review’, Minerva. DOI 10.1007/s11024-012-9192-9198.

Frodeman, Robert, Briggle, Adam, and Holbrook, J. Britt (2012) ‘Philosophy in the Age of Neoliberalism’, Social Epistemology: A Journal of Knowledge, Culture, and Policy. 26 (2-4). DOI: 10.1080/02691728.2012.722701.

Funtowicz, Silvio O. and Ravetz, Jerome R. (1993) ‘Science for the post-normal age’, Futures, 25: 739-755.

Gorman, Michael E. (Ed.) (2010) Trading Zones and Interactional Expertise: Creating New Kinds of Collaboration. Boston: Massachusetts Institute of Technology Press.

Hackett, Edward J. (1997) ‘Peer review in science and science policy’. In: Frankel, M.S. and Cave, J. (eds.) Evaluating Science and Scientists: An East-West Dialogue on Research Evaluation in Post-Communist Europe, pp. 51-60. Budapest: Central European University Press.

Healy, Stephen (1999) ‘Extended peer communities and the ascendance of post-normal politics’, Futures, 31: 655-669.

Holbrook, J. Britt (2005) ‘Assessing the science – society relation: the case of the U.S. National Science Foundation‘s second merit review criterion’, Technology in Society, 27: 437-451.

Holbrook, J. Britt (2010a) ‘Peer review’. In: Frodeman, R., Klein, J.T. and Mitcham, C. (eds.) The Oxford Handbook of Interdisciplinarity, pp. 321-332. Oxford: Oxford University Press.

Holbrook, J. Britt (2010b) ‘The use of societal impacts considerations in grant proposal peer review: A comparison of five models’, Technology and Innovation – Proceedings of the National Academy of Inventors, 12: 213-224.

Holbrook, J. Britt and Frodeman, Robert (2011) ‘Peer review and the ex ante assessment of societal impacts’, Research Evaluation, 20(3): 239–246. DOI: 10.3152/095820211X12941371876788.

Holbrook, J. Britt (2012a) ‘What is interdisciplinary communication? Reflections on the very idea of disciplinary integration’, Synthese. DOI: 10.1007/s11229-012-0179-7.

Holbrook, J. Britt (2012b) ‘Re-assessing the science - society relation: The case of the US National Science Foundation's broader impacts merit review criterion (1997 - 2011)’, in Peer Review, Research Integrity, and the Governance of Science – Practice, Theory, and Current Discussions. Robert Frodeman, J. Britt Holbrook, Carl Mitcham, and Hong Xiaonan. Beijing: People’s Publishing House: 328 – 62. [Denton, Texas]. UNT Digital Library. Accessed November 1, 2012.

Holbrook, J. Britt and Robert Frodeman (2012) ‘Resistance to impact criteria can lead to a tightening of the accountability noose’, London School of Economics Impact of Social Sciences Blog. March 15, 2012.

Von Humboldt, Wilhelm (1970) ‘On the spirit and the organizational framework of intellectual institutions in Berlin’, Minerva, 8: 242-250.

Johnson ,Valen E. (2008) ‘Statistical analysis of the National Institutes of Health peer review system’, Proceedings of the National Academy of Sciences of the United States of America, 105: 11076-11080.

Kamenetzky, Julia R (2013) ‘Opportunities for impact: Statistical Analysis of the National Science Foundation’s broader impacts criterion’ Science and Public Policy, 40 (1): 72-84. DOI: 10.1093/scipol/scs059.

Kitcher, Philip (2001) Science, Truth, and Democracy. New York: Oxford University Press.

Lamont, Michèle and Mallard, Grégoire (2005) ‘Peer review in international perspectives: US, UK and France. Report commissioned by the Social Sciences and Humanities Research Council of Canada.’ Available at, last accessed 8 June 2011.

Lamont, Michèle (2009) How Professors Think: Inside the Curious World of Academic Judgment. Cambridge, MA: Harvard University Press.

Landry, Réjean, Amara, Nabil and Lamari, Moktar (2001) ‘Climbing the ladder of research utilization: Evidence from social science research’, Science Communication, 22: 396–422.

Langfeldt, Liv (2001) ‘The decision-making constraints and processes of grant peer review, and their effects on the review outcome’, Social Studies of Science, 31: 820–841.

Langfeldt, Liv (2004) ‘Expert panels evaluating research: decision making and sources of bias’, Research Evaluation, 13: 51–62.

Laudel, Grit (2006) ‘Conclave in the Tower of Babel: how peers review interdisciplinary research proposals’, Research Evaluation, 15: 57-68.

Lightbourne, Jim (2010) ‘NSF merit review criteria and process’. Available at, last accessed 8 June 2011.

Lok, Corie (2010) ‘Science funding: Science for the masses’, Nature 465, 416-418. Available at, last accessed 8 June 2011.

Mallard, Grégoire, Lamont, Michèle and Guetzkow (2009) ‘Fairness as appropriateness: Negotiating epistemological differences in peer review’, Science, Technology and Human Values, 34(5), 573-606.

Marsh, Herbert W., Jayasinghe, Upali W. and Bond, Nigel W. (2008) ‘Improving the peer-review process for grant applications: Reliability, validity, bias, and generalizability’, American Psychologist, 63: 160-68.

Nightingale, Paul and Scott, Alister (2007) ‘Peer review and the relevance gap: ten suggestions for policymakers’, Science and Public Policy, 24: 543-53.

Nowotny, Helga, Scott, Peter and Gibbons, Michael (2001) Re-thinking Science: Knowledge and the Public in an Age of Uncertainty. Cambridge, MA: Polity Press.

O’Rourke, Michael and Crowley, Stephen J. (2012) ‘Philosophical intervention and cross-disciplinary science: the story of the Toolbox Project’ Synthese. DOI: 10.1007/s11229-012-0175-y.

Peters, Douglas P. and Ceci, Stephen J. (1982) ‘Peer-review practices of psychological journals: The fate of published articles, submitted again’, Behavioral and Brain Sciences, 5(2): 187-255.

Polanyi, Michael (1962) ‘The Republic of Science: Its political and economic theory’, Minerva, 1: 54-73.

RCUK (2010) ‘RCUK expectations for societal and economic impact’. Available at, last accessed 8 June 2011.

Rip, Ari (2000) ‘Higher forms of nonsense’, European Review, 8: 467-86.

Roberts, Melanie (2009) ‘Realizing societal benefit from academic research: Analysis of the National Science Foundation‘s broader impacts criterion’, Social Epistemology, 23: 199-219.

Sarewitz Daniel (1996) Frontiers of Illusion: Science, Technology, and the Politics of Progress. Philadelphia: Temple University Press.

Scott, Alister (2007) ‘Peer review and the relevance of science’, Futures, 39: 827-45.

Stokes, Donald E. (1997) Pasteur's Quadrant: Basic Science and Technological Innovation. Washington, D.C.: Brookings Institution Press.

Viner, Neil, Powell, Philip and Green, Rod (2004) ‘Institutionalized biases in the award of research grants: A preliminary analysis revisiting the principle of accumulative advantage’, Research Policy, 33: 443-54.

Wennerås, Christine and Wold, Agnes (1997) ‘Nepotism and sexism in peer review’, Nature, 387: 341-43.