Search the site by keyword

How effective are ALMPs?

30 October 2019
Jon contract

The success of an Active Labour Market Policy (ALMP) will depend a lot on the type of programme and who it is targeted at, as well as on the effectiveness of delivery. Before we rush into spending more on ALMPs because we spend less than other countries we should be clear what we are trying to achieve with ALMPs and for whom.

I’ve been having a look at the Ministry of Social Development’s approach to ALMPs – MSD are an important funder of ALMPs in New Zealand. MSD’s ALMPs are narrowly targeted. They are almost exclusively for people on a benefit or for people who have recently left the benefit system. MSD publishes an annual report evaluating the effectiveness of its employment services.

MSD recently evaluated the effectiveness of services that were delivered in the 2016/17 year. Some programmes could not be evaluated because it was too soon to test the programme’s effectiveness, or it was not technically feasible to do so. MSD evaluated each intervention based on the impact on participant outcomes across five domains:

  • Increasing the time participants spend in employment
  • Increasing participant’s overall income
  • Reducing the time participants are in corrections services
  • Improving participant’s education qualifications
  • Increase the time participants are independent of welfare assistance.

Each intervention received an overall rating. For instance, an “effective” rating means that the intervention has a positive impact on more than one domain (and no negative impact on any other domain), while a “promising” rating means that trends across domains suggest the intervention will have a significantly positive impact in the medium to long-term. This method does not consider the size of the impacts, just whether they are statistically significant. 

How effective are MSD’s ALMPs?

Based on the rating system described above, just under 60% of evaluated interventions for 2016/17 were rated either effective or promising. A quarter of interventions were rated as making no difference while one intervention (the service for at-risk youth) was rated as having a negative impact.

Ratings for evaluated MSD employment interventions, 2016/17

Ratings for evaluated MSD employment interventionsSource: My analysis of data from De Boer and Ku (2019)

It’s great that MSD is systematically measuring the effectiveness of its employment assistance programmes, on a range of criteria, for current recipients.

People in different circumstances have very different characteristics and needs

The overarching aim of ALMPs is to support people into work in a timely way. But different types of interventions aim to help different types of people and will therefore have different objectives.

For example, a policy that helps workers who have lost a job due to technology-driven change might be focused on connecting them with a new job that is a good match for their skillset. Whereas a policy targeted at people who do not have the core skills for work might be more focused on developing those skills and getting them into some form of sustainable work.

Because of MSD’s narrow targeting, a lot of its programmes are for people who have relatively weak attachments to the labour market. If the government were to expand eligibility for ALMPs to a wider group of people, it’s important to recognise that those people might have very different needs and characteristics.

It would also be worth taking a step back and thinking about the needs and characteristics of people who might lose their jobs from, for example, technological change, and what types of support would be as valuable or more valuable than expanding current ALMPs and employment services.

Image: from a video about ALMPs for youth in Kosovo


Have your say

Please tick the box to prove you're a human and help us stop spam.

Comments

  • Gravatar for Tony Burton

    Tony Burton 7/11/2019, 5:21 am (4 years ago)

    Hi Tim
    It's great to see the Prod Comm looking at this colossal area of expenditure, and the attention you are bringing to Marc's long term (and mostly thankless) efforts to have meaningful evidence in assessments of NZ welfare interventions. A couple of comments:
    - The total expenditure is $516m of which only $206m is included in the graphs above (page 1&2 of the MSD report). There should be another bar here to make clear that we have no assessment of most spend. That remainder is primarily childcare assistance, which is very badly targeted if it is meant to be an ALMP, so the suspicion has to be that spend is, overall, less effective than the graphic suggests.
    - The MSD analysis notes that it does not take account of non-participant effects (page 8). Basically this is where the MSD intervention means someone not on MSD's books loses out. From memory, work by Mare in 2006 had these effects at close to 90% for wage subsidies. ie in some domains the effectiveness may be 10% of those reported.
    - An important point raised in the MSD report is the interaction between interventions. Most obviously that effective case management would mean choosing appropriate interventions. More broadly, individual interventions are part of a suite of actions. This works both ways. Some intervention may seem effective because of selection biases, while others may only be effective if considered as part of a suite. Surely it's the overall impact of the employment service that matters?
    - The role, and failings, of case management are not well represented. This is no small cost since this is a service industry where staff costs are the major cost. Case management is mentioned in the MSD report, but is treated with kid gloves for institutional reasons. There are lots of ways this would have a big impact on the assessment, including:
    a) Case managers choosing people for interventions when they know they will find jobs anyway (Marc's methodology does try to account for this but there are plenty of "unobservables" that are observable to case managers but not researchers);
    b) Case managers finding ways to pre-select "difficult" clients to streams that mean the client gets no interventions;
    c) Case managers being defined as providing a payment service or an ALMP depending on whether or not the case manager finds jobs for their clients;
    d) Cost overheads associated with case management not being included.

    Good luck

    Tony

    Please tick the box to prove you're a human and help us stop spam.
    • Gravatar for Tim Maddock

      Tim Maddock 8/11/2019, 1:55 pm (4 years ago)

      Thanks for your comment Tony - some really helpful thoughts. We calculated MSD's ALMP spending at about $400m when we included wider case management spending and excluded Childcare subsidies (since they aren't classified as targeted ALMPs). Your point about the indirect non-participant effects is a really good one, one we haven't forgotten about (though could have been worth a mention in this post) but is a useful reminder. And it is certainly seems difficult to get a handle on the efficacy of spending on case management services, and the extent that selection bias might be distorting evaluation results...

RSS feed for comments on this page | RSS feed for all comments