Even before the events of 22 February 2011, Finance Minister Bill English was signalling that it would be a tough budget this year.  The Minister of Finance must always choose his or her words precisely and on 4 April Mr English said “the Government [has] to carefully consider its spending priorities and set a credible path back to budget surplus. Only then can we begin repaying our debt and building a buffer against the next economic shock.”  In other words, the 2011 Budget will be as far away from an election-year lolly scramble as it fiscally possible.

While this position is both understandable and reasonable, the problem with being in cost-cutting mode is that it often precludes Governments from taking bold or visionary steps to drive change.  Bold steps don’t necessarily have to cost more.  We should not confuse being fiscally careful with being overly cautious with regard to policy.

These days, a number of Budget announcements are “foreshadowed” or simply announced by Ministers well before Budget Day proper.  The Budget went to print on 7 April and, before these pre-announcements begin, Independent Tertiary Institutions (ITI) decided to carry on our nascent annual tradition of making some predictions about what we expect to see in the Budget, and some policies we would love to see but don’t expect will happen this year.  These will generally be limited to issues which relate to the Private Training Establishment (PTE) sector.

Looking back at our predictions for Budget 2010, we got a credible 6 out of ten correct (though one was not announced in the Budget but happened later), 1 partially correct and 3 wrong (including one where we actually were pleased to be wrong).  So, that hit ratio means we are not quite Paul the Octopus but we are usually pretty close to the mark.

So, here is what we expect to see:

• A modest but welcome increase in Government SAC funding for the PTE sector of between $3m and $8m – probably $5m – to cover inflation and maybe a little more.
• Expansion of the Youth Guarantee scheme utilising the five vocational pathways announced by the Minister recently.  Given the limited policy work done so far, there may only be some indicative or contingency funding attached – possibly around $10m for the first year.
• Confirmation that PTEs will be able to offer level 1 and level 2 courses to ensure they can fully participate in the expanded Youth Guarantee scheme.  This has been well-signalled and will be well received.
• Some level of restriction on the ability of older students to access allowances and/or loans.  This could be one of the more controversial aspects of the tertiary education Budget.  PTEs have a slightly higher level of older students than TEIs.
• Additional support for providers affected by the Christchurch earthquake.  Over 120 PTEs were impacted with more than 20 still shut entirely.
• A small funding boost for the new Education NZ Crown Entity – possibly $3m-$5m.  This would be a combination of reassigned funding from efficiency savings created by the merger and some additional student levy money.
• A fix for the GST issue with travel payments for a number of trainees.  Currently, providers are being short-changed and the Government has said the problem will be addressed.

What we would like to see but don’t expect to:

• A workable solution to the 103% over-delivery policy.   This policy disproportionately affects PTEs because of the historical nature of their EFTS caps.  There is a review of the policy currently underway but progress has been slow over the last three years.
• Greater flexibility to allow Government funding to follow performance between tertiary sub-sectors.  Currently, each has a discrete silo of money.  We think the Government is moving this way however.
• A performance funding system which actually rewards the best providers.  Under the currently proposed scheme, the top providers will only receive the original funding promised to them.  There is no real incentive to excel.  This ‘carrot’ might be added after the scheme has operated for a year or two.

I guess we will only find out for certain on 19 May 2011.

ITI is a recognised peak body representing a diverse range of high-quality private providers around New Zealand.

• Best Policy The release of performance data on tertiary education organisations wins hands down. We can all find reasons why the data isn’t perfect but it makes it pretty hard to hide a decision to run a lot of rubbish courses and it shifts funding towards results, rather than enrolments. It will change the tertiary landscape.
• Worst Policy Interest-free loans continues to be a major drain on taxpayers and diverts tertiary education investment unnecessarily. If we even adjusted loans for inflation we would save hundreds of millions of dollars and it would have no or minimal effect on access. Luckily for other policy-makers and lobbyists, their policy is unlikely to displace free loans as worst policy of the year for the foreseeable future
• Best News Story Well, it had to be the husband asking the MOE to do something about his wife sleeping with a Brazilian homestay student. Blogging gold that one was.
• Best Regulatory Response I’ll give a joint award to NZQA and Immigration NZ for finally pulling finger and doing something about those bottom-feeding providers, mainly in Auckland, that exploit international students and/or NZ by poor education, visa fraud and cheating. It took them long enough, but they have been firm on a few places and it is great to see. There is more to do in 2011.
• Best TEO initiative The work on commercialisation that both the Metro ITPs and the universities did was great, both in websites and other work. There is much more to go in terms of actions and results, but they did something very positive and it deserves to be recognised. Many TEOs did other good things, but these were substantive efforts to address a largely unserved need.
• Best Use of Social Media I can’t go past Jason Nockels’ work up at UCOL. He’s built up a following of 3,700 fans on Facebook and he and the team are still working on new ideas.
• Ministerial Assessment Well, I liked Steven Joyce’s work this year and since I wrote about his appointment in my second post of the year, I’ve covered all of his time in the role. He hasn’t presented an awe-inspiring vision for the sector but he’s pushed through some important issues , (like performance reporting and performance funding), appointed a strong chair to TEC, encouraged NZQA in its review of quals (Anne Tolley started that), approved sensible TEI mergers and generally done a very effective job. He gets 8/10.
• Opposition Assessment Grant Robertson picked up Labour’s tertiary education role in June and he’s been reasonably effective over that time. He stepped away from Maryan Street’s focus on ACE and has spent more time on issues like caps and student support changes (these became bigger after the May budget too). He hasn’t landed any blows on Joyce and I think that’s partly because it is a big hurdle to take over a new portfolio, but also because it is hard to get traction essentially pushing for more spending when the deficit is so high. He gets 5/10. He’ll improve next year as he’s very capable, but tertiary education won’t be an election issue unless National puts loans on the table, so you may hear more of him in his other (or new) portfolios. (In terms of the Greens, I’d give them 2-3/10 for being there, but they made no impact.)
• CEs in the Media The straight talking to the universities and the TEU about why graduate outcomes matter by Unitec’s Rick Ede was great. On the other hand it would have been good to hear more from Massey’s Steve Maharey on campus/course closures and the cleaning staff dispute.
• Worst Last Minute Document Drop This one goes to NZQA, which released its detailed guidelines for L1-6 qualification approval late on 21 December, two days after their last newsletter of the year! I’m sure you’ll all read that on the beach…

By the way, we didn’t trawl through every blog post, shortlist finalists and invite in neutral judges to help in the decision. Marina and I spent a few minutes chatting at the end of a cafe meeting (yup, that’s what consultants do all day), so feel free to add your own take on the year below!

This post will probably be the last until the 10th of January, unless something really exciting comes up. If it’s just exciting, I’ll pop something on Facebook or Twitter – sign up on the site, top right. And if your media team just put out a puff piece to get some easy coverage in the news dead period, I’ll get back to reading my book.

Qualification completions are not as clear cut as course completions, which I covered earlier today. That is because many students do enrol in a qualification but only intend to complete one or a few courses within that qual. Of course, TEC funds and desires quals, so the stat is important. Many providers also purposely enrol people in courses, knowing they won’t do the whole qual, so that’s a choice that will depress stats.

As with course completions, universities are at the top of this table, filling the first 5 places, as well as 8 and 9. Massey University, however, is down at 17th place, probably reflecting the lower completion rates on extramural qualifications (although I suspect that doesn’t explain the whole difference, as Massey has a lot of face to face students).

The 10 TEIs below 50% deserve some extra scrutiny, but I’m willing to give those ten a little more time to contextualise their situation (and take some time to think about it), as the issues are more complex. The Open Polytechnic has very, very low completion rates, but they’re at the sharp end of the course vs qual enrolment issue. Raukawa and Waiariki are next from the bottom of the list, but that’s probably related to them being on the bottom of the course completion table.

44 PTEs had qualification completion rates below 50%. On the other hand, 19 had 100% and another 20 had 90-99% qualification completion rates.

The universities fill the top 7 places in the table, with Massey in 10th (probably due to extramural courses). That’s unsurprising, given they have some entry limitations on courses and are dealing with a group that is probably the best academically prepared. Even so, well done to them. I don’t think we should worry too much about institutions above 70%, or even the differences in rankings between them. They’ll probably run some courses with very low completion rates that will need attention, but their average performance is OK. I’ll deal with issues after the table!

The four TEIs with course completions below 60% owe the public an explanation, and especially Waiariki Institute of Technology at 50% and Te Wananga O Raukawa at 42%. It is simply unacceptable to enrol people that have a lower than 50% chance of completing your course. It is a waste of students’ time and the government’s money. To achieve those poor levels of completions, some or all of entry standards, teaching, assessment or pastoral care are stuffed. You might wonder why I have included UCOL and Tairawhiti as well – the reason is that that average will probably hide a large number of sub-50% completion rate courses. Just as I was about to publish this post, I received a media release from UCOL with some marketing bumph but also some thoughtful comments on their rankings – good on them. I’ll link to it when it goes online)

By the way 16/199 PTEs had course completions of under 50% , and another 12 had 50-59%. They range from Practical Education Institute at 49% down to Directions Appearance Technology Training at 0% (I suspect they’re no longer funded). They also need to explain that performance.

TEC’s CE, Roy Sharp, said that “Today’s release of tertiary education performance information will allow the general public to understand more about how their local tertiary providers are performing”. The indicators being released by TEC are (full TEC technical report here):

• successful course completion (all EFTS completed a course/all EFTS enrolled in a course, but only covering courses expected to end in the relevant year);
• completion of qualifications (all qual completions multiplied by EFTS value/total EFTS enrolled in courses);
• student progression to higher level study (number of students enrolled in a higher qual within 12 months following qual completion/number of students completing a qual in year n); and
• students retained in study (the proportion of students that re-enrol in year n+1 or complete a qual in year n or n+1/students enrolled in year n).

Roy went on to say “For the first time we are releasing information that can be used to make year on year comparisons of tertiary providers’ performance…Providers have put in a lot of effort to prepare their information and to make it accurate. I would like to thank them for their work in this process.” He also pointed students and parents to tertiary provider websites and Career Services for more info about tertiary education.

Of the four indicators released today, course completions and qualification completions are the most useful, as they provide an indicator of students’ progression in their goals. The other two indicators are TEC creations that are unlikely to be very relevant to the general public, largely because they are only meaningful at a course or qualification level – if a qual is designed to get you a job, then a low progression to further education will not matter to students.

The data only relates to domestic students funded at providers through the Student Achievement Component. ITO data will be released at a later date.

Every provider has a snazzy PDF provided for them, listing the aggregate results for their subsector (eg ITPs, PTEs, universities) on p.1, the provider stats on p.2 and the subsector rankings on p.3. Below I’ve provided the main subsector stats and a further post will have TEI stats.

If you look at the course and qual completions above, universities seems to be highest, with PTEs in second – what that means will wait for another post. A Courses v Qualifications chart by David Choat, my colleague at ED, shows that there is an apparent relationship between the two, as you would expect.

The detailed info is being published at an organisation, rather than a course or programme, level. This will make it largely useless for most students seeking info on public tertiary education institutions, as the aggregate figures will not provide any useful pointer to a specific course or programme. The figures will be a useful guide for students looking at  PTEs and OTEPs, which usually have a much tighter subject focus and fewer programmes. Of course, this release is largely intended to focus on organisational performance and is a preparation for more detailed data releases in future years (that’s not TEC’s official line, but my prediction). Releasing the data will expose the differences between organisations and drive some policy and practice changes.

• There is an enormous amount of data available on tertiary education inputs (eg enrolments), but very little on outputs (course or qual completions) or outcomes (did people get a job, make a living, enrol for another qual, etc). TEC’s data release will help a lot with filling in the outputs area, but graduate outcome data will be the next step.
• The data that will be released will have various errors, stemming from two main issues.
  1. The first is that tertiary education organisations (TEOs) have had no good incentive to tidy up their data until now and it will take time to fix things. That is not a reason to delay publication any further as TEOs will continue to delay until data is published – it’s a natural reaction. Exactly the same arguments about poor quality data were raised in 2003 when then-Minister Steve Maharey proposed releasing similar data.
  2. The second is that TEC has produced data using techniques (and sometimes raw data) that TEOs do not understand or agree with. The TEC has, in response, delayed publication several times to fix things, which is good. I suspect, however, that TEC has made things too complicated and will continue to face reasonable criticism over their techniques. TEC will need to improve things before funding is based on their data, but I’m not convinced that the data should be withheld.
• Data on whether students stay in the course (retention) are important because they give us a sense of whether the course material and teaching is engaging and whether students were appropriately assessed before enrolment. If students are bored or overwhelmed they are likely to leave early.
• Data on whether students succeed in the course (course completion) are important. Most people should pass a well-targeted course and low pass rates will usually point to lax entry standards or poor teaching (or an earthquake in some rare cases).
• Data on whether students gain the qualification (qualification completion) are important because the government’s policy is to encourage qualification completion, as that is associated with higher incomes. TEOs will argue that many students enrol just to get a course – that’s true, but the government has been clear for a long time that it does not want to fund or encourage that. If TEOs have been ignoring that position for many years, then it is hard to complain when the government reinforces it through funding shifts. (NB I am supportive of course enrolments being funded outside of wider qualifications, as a segment of overall funding.)
• Data on whether students enrol in a further qualification (progression) are useful, but are less clear-cut than the other measures. The utility of further quals will vary by industry/subject and I suspect this measure will be the least used of the four to be published.
• Every TEO will come up different reasons for why their enrolments are affected by student factors, but they should mostly be disregarded unless they are accompanied with a serious research report. The fact is that every TEO will have a good number of students shifting, getting pregnant, losing a job, going to jail, getting bored or playing X-Box. At an individual course level those factors will skew results, but at an organisational level, the results will be a fair reflection of the enrolment, delivery and student support policies and practices of an organisation.
• There is a reasonable argument that enrolling some population groups may depress completion rates, given those groups’ worse prior academic achievement. But that is a difficult issue, as some TEOs do quite well with groups that otherwise have poor performance – one good thing about the release of stats is that we will  get a clearer picture of who does well and who doesn’t with different groups.
• Publishing performance data may lead people to game the system and artificially increase pass rates, but I think the more likely response that professionals in the tertiary education sector will take is solid work to help improve students’ performance. I expect that the attention to detail will far outweigh the gaming, and that gaming will often (but not always) be picked up at the margins.

Overall, I’m looking forward to this data helping students to choose their courses and redirecting providers’ efforts. Students who enrol and do not stay in the course, do not pass their courses or do not get a qual are a silent group in education. They usually do not get surveyed on their experience (that is left to the graduates) but they are saddled with debt, foregone earnings and quite probably discouraged from further education. I support publication of performance stats for the impact it may have on those students.

However, a great deal of data has been available for years on the Ministry of Education’s Which Course Where site. I was speaking at a Marketing Education Services conference  on the impact of regulation on marketing this week, so I downloaded a few figures on course completion rates and put them up on the screen for discussion (I used a selection of providers with sites in the Wellington region – there are a lot more PTEs in the area). I asked five volunteers to respond to a ficititious media enquiry about the results. It was an interesting exercise, but I thought I’d share the figures with blog readers as well.

Before commenters decry my release of such data, I want to emphasise that this info has been on an MOE website as “a student information resource” for years. The MOE chose to release this data for students, but has, unfortunately never publicised it – accordingly not many people go to it. Alexa is a global Web ranking service and the MOE’s main site has an Alexa ranking of 255,000, while the WCW site has an Alexa ranking of 8,342,000.

The top of this particular list is the Institute of Professional Legal Studies. They have a one course programme that is only available to law degree graduates seeking admission as a lawyer. I would expect a high pass rate from such a highly qualified and motivated student group. The next two on the list are the dance and drama schools, that also have highly motivated student bodies, and very tough entry standards. After that, things get a bit more complex. Victoria University has the same rate as Sir George Seymour College, a L3-4 tourism provider (aren’t L3-4 courses supposed to have bad completion rates?). The bottom rankings surprised me – I expected the Open Poly to have low completion rates, because extramural courses usually do (but should we still fund them at the same rate as other courses? The Minister is asking that question). Te Wananga o Raukawa’s rate did surprise me (they also offer extramural study BTW) and their rates seem so low as to invite intervention, and to make one wonder why nothing has been done to date.

The completion rates above are only one part of the picture, but they provide important information. Off the top of my head, I’d expect most rates to be in the 70-90% range – above that and you might want to question whether standards are tough enough and below that you might want to question entry standards and teaching quality. Of course, those rates are just a rule of thumb that I’d use for asking questions, rather than shifting funding. But if the course completion rate is below 50%, then I believe that you have a clear case of a waste of government money and students’ money and time. I’m looking forward to the release of TEC’s more comprehensive data in the next couple of weeks, because it will provide much more depth and allow open discussion of providers’ performance and TEC’s investment decisions.

But anyway, what do you think? What would you like to see (or not) in performance data?

(Brief) Technical Notes

• The figures above are from a range of providers with a Wellington region presence, as defined by the MOE. Many PTEs have not been listed, mainly because I wanted a list that would fit on a PowerPoint slide. You can access the full search results here.
• Course completion measures whether students have passed a course (there can be extra complexities to this, but I couldn’t find the MOE’s methodology on the site). A programme is made up of courses, and completion of a programme results in a qualification.
• The figures in the table are for the provider as a whole and sourced from Which Course Where this week.
• All figures are an average for the 2007-09 period, other than WelTec, for which only 2008-09 figures were available.