1. Evaluation of Investments in Strengthening Management and Governance

1.1 Introduction

Te Puni Kōkiri has for several years invested extensively in building the capacity and capability of whānau, hapū, iwi, Māori communities and Māori organisations. Through programmes such as Tāhua Kaihoatu (Māori Provider Development), the Capacity Building Programme, and, more recently, the Māori Potential Fund, investments have enabled Māori organisations to develop sound governance practices and implement robust management processes and systems.

This report provides an outcome evaluation of Te Puni Kōkiri’s Strengthening Management and Governance (SMG) programme.

1.2 The Strengthening Management and Governance Programme

The Strengthening Management and Governance programme aims to develop strong and effective institutional governance and management capacity in established Māori organisations and businesses contracted to deliver government services.¹ The programme was developed by Te Puni Kōkiri at a time when public and political scrutiny was heavily focused on Māori organisations delivering, or failing to deliver, government-funded health, welfare, education and justice programmes. Te Puni Kōkiri regards the SMG programme as a means of increasing the capacity of established Māori organisations by assisting them to:

• establish high value, sustainable relationships with government funders; and
• avert the risk of high profile failures that in the past have highlighted weak governance leading to poor accountability and decision making.²

The SMG programme aims to achieve four key high level outcomes:

• Outcome 1: Māori organisations involved in the delivery of government service contracts achieve high quality outcomes and are accountable to stakeholders.
• Outcome 2: Government funding is targeted appropriately.
• Outcome 3: To contribute to the growth of strong Māori organisations able to meet organisations’ goals and take advantage of future opportunities.
• Outcome 4: To increase stakeholder confidence in the accountability of Māori organisations.

To achieve these outcomes, the SMG programme assists Māori organisations by providing an independent organisational analysis and, where required, assists them with tailored interventions to enhance or improve their performance.

Phase 1: Organisational analysis

The key component of the SMG programme is the provision of an independent report to Māori organisations on their operating performance in key governance and management areas. Te Puni Kōkiri contracts external consultants (assessors) to undertake independent assessments of organisations. Six assessor organisations have been engaged as preferred providers to the SMG programme.

Te Puni Kōkiri selects and assigns assessors to work with organisations based on a preliminary (informal) match of the assessors’ skills and the requirements of organisations.

The assessments are based on a standardised, yet flexible, framework that assessors follow to identify strengths and specific gaps in the organisations’ governance and management practices. From this analysis, the assessors recommend areas for improvement and practical solutions. The assessments are voluntary – organisations decide if they wish to participate – and full assessment reports are confidential to the organisation. Te Puni Kōkiri receives only a summary copy of the key findings due to the confidential and commercially sensitive nature of the report contents.

Phase 2: Tailored remedial activities

The second component of the SMG programme is the provision of remedial support to organisations that completed the initial assessment process. The purpose of the remedial support is to address the areas for improvement identified in the assessors’ reports. This phase is achieved through limited funding support from Te Puni Kōkiri or the facilitation and brokering of assistance from other government funders. In this phase, organisations are able to choose who undertakes the remedial work. The majority of organisations choose to have the remedial work but some choose not to. Some advised that they will undertake remedial work as their timeframes and resources allow.

Programme criteria

Although the programme is voluntary in nature, Māori organisations are required to meet certain criteria to be eligible for the SMG programme. Organisations are required to be in either the ‘development’ or ‘established’ stage of the business lifecycle; have significant responsibilities for service delivery and/or asset management; and have a minimum annual income of $300,000 in government service contracts.

When the programme was first introduced, the qualifying threshold was an income of $800,000 per annum. The lowering of the threshold has allowed more organisations to enter the programme. Proactive targeting by Te Puni Kōkiri of organisations with development needs (such as Māori radio stations) has enabled a wider catchment of Māori organisations to participate in the programme.

Investment quantum

Since the inception of the SMG programme, Te Puni Kōkiri has invested in excess of $4 million in the programme.

On average, each assessment costs approximately $20,000 (including disbursements of up to a maximum of $5,000).

Programme milestones

The table below provides an overview of the key project milestones for the SMG programme.

Table 1: Chronological milestones for the SMG programme

Table 1: Chronological milestones for the SMG programme

Milestones	Implementation
SMG programme established	Mid-2003
Pilot phase – assessment of 26 organisations completed	January-May 2004
Trial review – internal paper completed	July 2004
Independent evaluation of the programme (process evaluation) completed	June 2005
Over 110 organisational assessments completed	December 2007
Outcomes evaluation commenced	March 2008

Overview of findings of SMG evaluation report June 2005

The June 2005 evaluation report³ found that the SMG programme was widely supported by organisations and assessors. The report identified critical factors leading to the success of the SMG programme. These included: the programme’s non-compulsory nature; the quality of the assessors and their reports, and the respect assessors showed towards the tikanga, kawa and kaupapa of organisations. The report also noted the willingness of Te Puni Kōkiri to adapt the programme where necessary.

The report noted that it was too early to make any definitive statements about the achievement of the programme outcomes and that these outcomes would only be visible after a longer period of time. However, some outcomes were evident and these included:

• improving management and governance in Māori organisations – the SMG programme helped organisations to work out priorities for action and provided strategies and resources to address the issues; and
• improving confidence in the business capacity of Māori organisations – organisations reported increased confidence within themselves and with a range of stakeholders including other government agencies.

The report also identified areas for improvement. Some of these included:

• leveraging the policy and operational feedback loop within Te Puni Kōkiri – an opportunity for Te Puni Kōkiri to maximise the lessons from the SMG programme;
• developing the facilitation and brokerage role of Te Puni Kōkiri – more preparatory work at a national and regional level was still required to move Te Puni Kōkiri from solely a funding role to a mixture of funding and brokering;
• reconsidering the role of assessors in the remediation work with organisations; and
• ensuring the follow-up remedial process was timely.

Overall, the evaluation found that the assessment phase was popular, the assessor reports were valid and accurate, and the remediation phase needed to be improved.

 

¹ CAB Min (03) 30/6 refers.
² Paper to Cabinet Social Development Committee, 24 July 2003, p1.
³ Te Puni Kōkiri (2005).

1.3 Evaluation Objectives

The growth and popularity of Te Puni Kōkiri investments in the SMG programme provide an opportunity to learn about the investments and the impact they are having on Māori organisations.

The objectives of this evaluation are to:

• understand what is good governance and management practice for Māori organisations;
• understand the extent to which investments in governance and management have been effective, and the range of outcomes achieved; and
• guide future investments in governance and management initiatives and inform future programme design.

1.4 The Evaluation Approach

A mixed method approach, using both quantitative and qualitative techniques, was utilised for this evaluation. This included an online survey of all participants in the SMG programme to gain baseline data on the overall effectiveness of the programme; key informant interviews with a sample of Māori organisations, assessors and Te Puni Kōkiri staff to gain more ‘rich’ information on achievement of outcomes and specific examples; and a review of all relevant documentation and assessors’ summary reports.

Evaluators used the judgement of key informants to assess the level of contribution of the SMG programme to the achievement of outcomes or to a change in how organisations operated. This approach is considered subjective because key informants may be likely to attribute any change that takes place to the SMG programme rather than to other factors. However, it provides some evidence of the change.

A more detailed discussion of the methodology is provided in Appendix one.

1.4.1 Selection of sample

The evaluation focused on all Māori organisations that participated in the SMG programme from 2003/04 through to the 2007/08 period.

Online survey

All 110 organisations that had participated (as at March 2008) in the SMG programme were selected as part of the population cohort for the online survey. An email list was generated from the Te Puni Kōkiri administrative database and validated by programme and regional staff. As a result of the validation process, the survey frame was reduced to a population of 89 organisations.

Key informant interviews

From the 110 organisations identified, a random list of 20 organisations was selected. A purposeful sample of six organisations was then drawn based on the:

• investment reach – a mix of investments at a regional and national level;
• assignment of the assessors – a mix of assessors assigned to respective organisations;
• investment status – organisations that participated at different stages of the programme (i.e. in different financial years); and
• nature of involvement in the SMG programme – assessment solely, or a combination of both assessment and remedial work.

The six organisations were drawn from five Te Puni Kōkiri regions: Te Taitokerau, Tāmaki Makaurau, Te Tairāwhiti, Te Whanganui-a-Tara and Te Waipounamu.

Three of the six assessor organisations were randomly selected to be part of the key informant interview process. All assessors interviewed had conducted over 10 organisational assessments and/or undertaken remedial work through the SMG programme.

1.4.2 Evaluation questions

Te Puni Kōkiri developed a draft outcomes model (see Figure 1 below)⁴ to inform the overall evaluation design, test the assumptions around the outcomes and, ultimately, inform Te Puni Kōkiri’s investment strategy. The model describes the key outcomes required to achieve healthy, vibrant and excellent Māori organisations. For this to be achieved, a series of low level outcomes needs to be present. These low level outcomes (such as appropriate financial management, relevant governance structure and strong management) provide the focus of the evaluation, and some attempt was made to assess (to the extent possible) whether the higher order outcomes, such as strong Māori organisations and high performing Māori organisations, were achieved. (Refer to the third tier of boxes in Figure 1).

Figure 1: Draft outcomes model – Healthy, vibrant and excellent Māori organisations

All evaluation questions were derived from the draft outcomes model and the programme outcomes for the SMG programme. The key evaluation questions were:

• Did the SMG programme promote improvements in governance and management practices/ outcomes for the participating organisations, such as improved systems, infrastructure, communications and strategy?
• What impact, if any, did the SMG programme have on the performance of Māori organisations?
• What areas of the SMG programme could be enhanced?

1.4.3 Information gathering

Document analysis, an online survey and key informant interviews were undertaken to inform the objectives of the evaluation.

Document analysis

SMG programme summary reports, briefing materials, SmartFund⁵ and the 2005 process evaluation were analysed to understand the SMG programme and to inform the evaluation design. These documents were also used to develop the detailed question guides for the key informant interviews and the online survey.

Online survey

The online survey was initially run for two weeks, from 3 to 17 March 2008, with a 46% response rate. The evaluation team extended the survey for another three weeks (until 7 April 2008), resulting in a small increase in the response rate.

Overall, a 51% response rate was achieved, which is above the average response rate⁶ for online surveys.

Key informant interviews

The evaluation team conducted semi-structured interviews with a range of stakeholders including organisation managers, directors, operational staff, board members and trustees, as well as assessors and Te Puni Kōkiri regional directors and kaiwhakarite. Interviews were held either individually or in group settings (two or more participants) within the organisations.

The interviews provided a ‘rich’ source of information on stakeholders’ perceptions and experiences of the SMG programme.

1.4.4 Analysis

The qualitative information was analysed using a thematic approach noting the ‘significant’ findings from the interviews. The quantitative information was analysed using a descriptive statistical tool and the findings integrated with the key themes identified through the qualitative findings.

Information collected is presented as aggregated results and quotes are attributed anonymously to maintain the confidentiality of participants. However, where quotes are attributed to assessors or Te Puni Kōkiri staff, this is noted in brackets.

 

⁴ The draft outcomes model has not been validated by Te Puni Kōkiri.
⁵ Te Puni Kōkiri administrative contracts management database.
⁶ The literature noted that the average response rate for online surveys is between 30% and 40%. See Ritter & Sue (2007).

1.5 Evaluation Team

The evaluation was conducted by three members of the evaluation team from Risk and Assurance, Te Puni Kōkiri.

1.6 Ethical Considerations

The evaluation was conducted in accordance with Te Puni Kōkiri’s Evaluation for Māori⁷ guidelines and the draft Social Policy Evaluation and Research (SPEaR) guidelines for Evaluation with Māori.⁸ Adherence to the principles of respect, integrity, responsiveness, competency and reciprocity was observed throughout the evaluation design, fieldwork and reporting.

The evaluation was conducted in a way that ensured all participants were comfortable in the evaluation process and participants were able to openly express their views in confidence. All participants were briefed on their rights prior to the interview process and were able to withdraw or decline to answer questions at any stage during the interviews.

 

⁷ Te Puni Kōkiri (1999).
⁸ www.spear.govt.nz.

1.7 Limitations of the Evaluation

The following limitations apply to this evaluation:

• The evaluation is not intended to assess whether investments in the SMG programme provided value for money (cost benefit analysis) for government.
• The scope of the evaluation did not extend to questioning other government agencies on their investment practices and whether Māori service providers they invested in produced high quality outcomes. Therefore the evaluation was unable to address two of the four high level outcomes for the SMG programme, namely whether:
  • Māori organisations involved in the delivery of government service contracts achieved high quality outcomes and were accountable to stakeholders (Outcome 1); and
  • government funding was targeted appropriately (Outcome 2).
  
  Assessment of the achievement of these outcomes is more appropriately addressed through the standard contract monitoring and evaluation activities of those agencies that are
  contracting for services.
• The achievement of Outcomes 3 and 4 is addressed in section 2.3 of this report: Outcomes for Te Puni Kōkiri. These outcomes are as follows:
  • To contribute to the growth of strong Māori organisations able to meet organisations’ goals and take advantage of future opportunities (Outcome 3); and
  • -To increase stakeholder confidence in the accountability of Māori organisations (Outcome 4).

The evaluation noted that in the absence of information from other government agencies, the extent of achievement of these two outcomes is based primarily on assessments made by the evaluators. On that basis, some caution should be taken when interpreting the discussion.

• Back to Executive Summary
• Next: 2. What Has the SMG Programme Achieved?