Navigating from the heights of technical rigour to the swampy reality: Lessons from New Zealand in population health outcomes monitoring

Abstract

Background: The objective of this paper is to describe and discuss two documents produced by the New
Zealand Ministry of Health concerning the monitoring of outcomes of public health programs.
The New Zealand Government is increasingly expecting planners and managers of publicly funded services to
shift their focus from the delivery of ‘outputs’ towards achievement of ‘outcomes’. Intervention logic models
and outcomes monitoring are promoted by central government agencies as suitable management methods
for implementing the change. [1-3]
Methods: To help managers design and implement comprehensive, effective and measurable population
health programmes the Ministry of Health recently published two guidance documents. The first document
provided guidance about how to plan programmes using a generic logic model approach.[4] The second set
out in detail a process on how to monitor population health programmes.[5] The intent of the documents was
to help managers navigate between the heights of technical rigour and the swamps of reality in the delivery
of population health programmes.[6]
Results: A number of issues and implications for how population health programmes are planned monitored
and performance assessed have been identified by the guidance documents. Issues include the problem of
small numbers, understanding the difference between outcomes monitoring and traditional forms of
evaluation, and outcomes monitoring being seen as a tool for punitive performance management rather than
‘continuous programme improvement’. Implications include more time spent on the design of programmes.
Planners will need to focus upon better sequencing of activities, setting more specific and time limited goals,
and to be more informed about how to use research to inform the selection of interventions.
Conclusions: The guidance documents promoted by the Ministry of Health have provided useful advice about
how to develop and use logic models and outcomes monitoring in the planning of population health
programmes. Practitioners particularly welcomed the inclusion of a glossary that included explanatory
comments and examples. Two key issues have been identified with the application of outcomes monitoring
that need to be managed. The first is the issue of statistically small numbers associated with relatively rare
health events. The second is to promote outcomes monitoring as a tool for continuous programme
improvement, rather than as a potentially punitive.