4. Insights

Throughout this evaluation, several themes arose that did not fit neatly into the six performance measures. These themes all demonstrate the way in which wider social and public sector context impacted the delivery and reception of the Inquiry. They are shared here, contextualised within the Commission’s impact measures.

Beyond performance measures, the Commission must consider three types of impacts that its work will have over the longer term. These impact measures address influence and change that reach further than the immediate time period following an inquiry. These impact measures provide a helpful way to articulate the broader themes that arose during this evaluation, both as an illustration of how context affects any piece of public sector policy or research. Broadly, these impact measures capture what might be called system evolution, which will ideally occur as a result of the research and recommendations that Commission inquiries present to the public.

The Commission must consider the following impact indicators:²⁶

• Policies and behaviours change as a result of the Inquiry work;
• Discussion and debate is generated on the Inquiry's findings and recommendations;
• Levels of engagement and response lift the standard of quality analysis and advice.

Expectations for action were high for this Inquiry. It may be that a report of this nature, dealing directly with discomfiting evidence around the inequity and disadvantage that some New Zealanders face, creates more of an onus for action than previous inquiries. Respondents named this throughout the evaluation process.

This report is trying to tackle the holy grail, the big issue of inequity around the world. One nation state tries to crack it. I think we need small agile dialogue. A social policy report is a very expensive doorstop … do you think some well thought out thing is actually going to deliver? No. Focus Group Participant

If this is about changing things, a lot of change happens through social movements. Documents are important milestones to articulate, but what’s really important is networks and deep dialogue. There’s a dominant set of constructs in a report, but the real value is the ongoing dialogue and intentional networks trying to make sense of it. This is a really big opportunity for the Productivity Commission - what are the dialogues and networks for ongoing conversation? The advantage of NZ is our small degree of separation, so use it. Focus Group Participant

This call for action relates directly to the impact measure of ‘policies and behaviours change as a result of the inquiry work’, which also relates to the Commission’s function to ‘promote public understanding of productivity-related matters’. The Commission is expected to influence the conversation towards changing policies, but the Commission does not possess a mandate for ensuring action, nor is it resourced for facilitating and convening dialogues and networks for ongoing conversation. It is not a policy agency and it does not have policy levers. One explanation for the frequent ‘calls for action’ heard during this evaluation could be the unfamiliarity of new (for the Commission) stakeholders regarding the boundaries of the Commission’s role. However, expectations of action came from public sector and academic respondents as well, who are presumably more familiar with the role and purpose of the Commission. This could indicate a general frustration with a perceived slow pace of change, particularly in a topic area like persistent disadvantage, where people are suffering. It may also be that the discussion of accountability mechanisms outlined in the Inquiry itself, were at play in some public sector respondents’ comments around the need for action, particularly in cases where people responded that action only happens ‘above our level’.

Furthermore, there is no requirement for the Government of the day to respond directly to Commission inquiries. Although the Government does often issue a formal response to an inquiry report, the onus on Government could be considered further, particularly in regards to any departure from or inaction on Commission recommendations.²⁷

Despite a call to action, with respondents expressing a wish for something to happen with the inquiry, the Commission does not currently have levers to do this, beyond influencing discussion and understanding.

Some respondents understood the current mandate and bemoaned it as ineffective:

There’s no formal commitment from Government to respond to recommendations in any way, so that diminishes confidence in the process - what prospect is there for an impact? Focus Group Participant

We need to think about how reports can be more enduring. The answer lies in, reports shouldn’t read like they’ve been commissioned by the Government - they should have durability to live on. Focus Group Participant

Other respondents did not understand where the Commission’s mandate ended with the delivery of the Inquiry:

Curious about how process works from here. How do they go about engaging with incoming Government, is there opportunity to do some collective work? … Would be good for them to let us know what the next steps are and how we could support. Focus Group Participant

There was an initial discussion / reaction when released, but it seems there’s no clear plan going forward. Focus Group participant

Others simply wanted to see some change and were already working towards this:

We’re still in direct connection with the Productivity Commission and presenting at leadership forum on this soon. Still trying to find opportunities to keep the ideas moving, while report is a bit on hold. Protecting the things that are working is so important - we can use this as motivation to keep going. They’re in this job trying to transform the system, at a regional level, we’re trying to work to drive change, and that’s a motivation for them. Focus Group Participant

No matter which way the issue is cut, it was clear that respondents wished for more clarity around next steps, with a strong preference to see policy change and action as a more immediate result from the Inquiry. This report notes, that even if the Commission had possessed the resources to foster further public debate, pre-election period guidelines, including advice from Te Kawa Mataaho, informed a decision not to.

In response to these calls for action, the Commission could act within its current mandate to place greater emphasis and proportionality of its resources onto the education, promotion of understanding, and influence of its findings in the public arena. Within its current purpose, functions and impact measures, the Commission already has a mandate to influence and educate. The Commission could rebalance its work programme to put greater weight and resources on educating and influencing, relative to its role researching and analysing. Note this would likely mean a significant review of the Commission’s existing work programme and the recommendation below does not suggest it would be a simple exercise.

Finding 10: Although it is clearly stated in the governing legislation that the Commission does not have a policy design nor implementation role, a significant number of stakeholders called for action from the Commission. This likely arose from the significance of the topic, which created a strong desire to see immediate reduction in persistent disadvantage, as well as the nature of some stakeholders from the inquiry, who may be less familiar with the Commission’s mandate.

Another consideration could be seeking a review of its mandate, either broadly, or more narrowly concerning Government response. This could expose the tension inherent in being an entity with a great amount of knowledge of complex social issues (following an inquiry) but without the policy levers or mechanisms to act on this knowledge, or the resourcing to convene ongoing discussion and debate. As discussed in section 1.3, the Treasury commissioned advice in 2020 around the role of productivity institutions, which appears to have been quite narrow. Any mandate review could also present an opportunity to consider more broadly the role and purpose of the Commission, including whether the Government of the day should be required to respond to future Commission inquiries, and the implications of any such change. For example, requiring a direct response from Government could require the Commission to grow more responsive, policy-like functions whilst in an environment where Government is not required to respond, or not required to address all inquiry recommendations, then engagement and partnership incentives for the Commission may lean more towards non-government actors, particularly when an inquiry topic is valued highly by them.

It may also be worth considering the recent trend of establishing response units to consider recommendations made by other inquiries. For example, the Ministry of Justice is currently establishing a new response unit to work through findings and recommendations from Waitangi Tribunal inquiries.²⁸ Establishing such a body or specifying a policy unit that should receive and respond to a future inquiry's findings could be one way to mitigate actual and/or perceived risk of no pathway for action. Any consideration of this approach should include consideration of the wider system-cost of such tools, and the degree to which they may inadvertently create more costs (resources) than they confer benefits (increased policy change).

Finding 11: The expectation of action also materialised in frustration with what some stakeholders saw as weak levers on Government to respond to the recommendations of the Inquiry. Considering that the mandate of the Commission has not been thoroughly considered in recent times, some form of review may be valuable in the near future.

Another theme that emerged during the evaluation was differentiation in the methodology that people use and find valid. This relates directly to the impact measures of ‘generating discussion and debate on inquiry findings and recommendations’ and to ‘levels of engagement and response that lift the standard in quality analysis and advice’.

As evidenced throughout this report, some sectors and types of agencies embraced this Inquiry more than others. Beyond specific views around the focus and scope of the Inquiry, there may be a broader social experience at play, where competing frames of validity are talking past each other. This is most starkly demonstrated within the ‘right focus’ and ‘clear message delivery’ sections of this report, where some people disengaged with this Inquiry because for them it did not use sufficient frames of relevance for their work. However, other people engaged heavily with the Inquiry because they saw direct relevance for their sector and work, for many of them in a way that they had not seen previously (when they may have been the ones disengaging from an earlier report or Commission inquiry).

It may be that whānau-centred, place-based initiatives and more traditional, economic spheres of work and policy analysis do not speak a common language. Furthermore, these differing groups likely hold different values, although confirming that was beyond the scope of this evaluation. The Final Report named this phenomenon in a Chapter 4 discussion on broadening values.

These different responses to the Final Report points to these differing frames of validity, and highlights the importance of pluralist approaches to research, analysis and policy-making.

Considering this, the nature of this Inquiry means it operated in this nexus ‘talking past each other’ phenomenon. This may point to a current culture of separate ‘bubbles’, perhaps amplified by social media norms, reducing the onus on people to engage constructively with frames or findings they do not agree with.²⁹

My worry is, there’s some really good stuff in there, but [people] being frustrated with parts of it has made us dismiss the whole thing. So how do we give voice to the parts that are good in it, valid and useful? It’s up to us, we can make change, let’s not dismiss the whole lot. I feel a responsibility to take the good bits to my organisation and work to try and push it through the system. I have been doing this. Focus Group Participant

Another theme that arose was around the timing of the Final Report, particularly the rue some respondents felt about the delay of the Quantitative Report and that being a missed opportunity. This report has already outlined how those circumstances were beyond the Commission’s control. However, the broader theme of limited attention spans, potentially due to information overload, is worth mentioning. It relates most closely to the impact measure of ‘discussion and debate is generated on the inquiry's findings and recommendations’.

Some respondents expected to be given all the information in a single package. When that did not happen, they quickly disengaged from the conversation. This occurrence is well beyond the Commission’s control or mandate. However, Recommendation 6 of this report identifies an approach to mitigating future disconnect between quantitative findings with the rest of an inquiry.

Another timing considering was the way the Commission leveraged the proximity of the Inquiry launch to other relevant public conversations. Specifically, the decision to cross-reference common themes with the Future for Local Government Review launch of its Final Report, may have increased the level of public discussion for the Commission.³⁰ This may be something the Commission considers even more in future inquiries, especially considering the feedback from this evaluation on the importance of hitting a ‘window of attention’ for busy people with lots of information to sift through.

Beyond generating a source of evaluation data, focus groups were valuable for facilitating reflections between stakeholders. Both Focus Groups for this review stimulated connections and/or reconnections between parts of the system. Participants left the Focus Groups with actions to connect across their agencies. Other participants shared that they found the process of sharing their experience and reflections of the Inquiry ‘helpful’ and ‘cathartic’ as a way to process the complexity of the topic.