MR JOYCE … Currently we’re writing off about 45c in the dollar for every student loan because of the interest-free policy. That’s down from about 48c. We want to get it down to 43c-odd in this Budget and then I’d love to see it down below 40c. Two ways you can do that: one is get a much better repayment rate from international borrowers; second way is actually looking at some of the high-cost borrowers in New Zealand, and there are some. I’ll give you one example which might be dear to your heart. We are currently loaning $30 million a year to people training to be pilots. $30 million a year. Most of them don’t go on to get jobs as pilots, either in the commercial or general aviation sphere. The write-off for pilot training in terms of write-off of student loans is in excess of 60%. So we’re looking pretty closely at whether we’re doing that right.

PAUL What, are you saying possibly no more student loans for pilot training?

MR JOYCE Well, not necessarily no more student loans for pilot training, but we could be looking at things like saying, for example, “We won’t fund your PPL or your Private Pilot’s Licence,” or, “We won’t fund your solo-instructor hours.” Those sort of things are options.

PAUL What about older people – people over 55 getting student loans for going to…?

MR JOYCE That is another one where there’s a big write-off. I mean, once you get past 55 – if you take out a student loan at 55, 56 – then there’s every likelihood that about 70% of that will be written off, because you’re at the other end of your working career, so we are looking at some things around that. I think we always want to be able to provide some access, so for perhaps borrowing for fees, but borrowing for living costs and borrowing for the compulsory course costs is not necessarily where we want to be.

What I love about this is that Joyce is laying out a very clear rationale for why he is proposing to cut the loan access. He may be right or wrong, but by giving out some facts he has enabled other to engage with him. Unfortunately the Greens and Labour have mainly rolled out the assertion that all over-55s are career changers anxious to improve themselves, but the fact that they’re not paying back their loans rather undercuts that. Labour and the Greens will need to lift their game and PROVE that is the case in order to undercut Joyce’s proposal. It is unlikely they will, as Joyce is probably completely right on his (admittedly limited) choice of measurement, but he has given them the ability to engage.

The aviation industry, is another area where a clear outlier is being targeted. The average annual loan is $50,000 ($30m for 600 EFTS) and repayment rates aren’t good. The aviation industry will have been in Joyce’s hair this week about it, but they’re well aware of the issues. If you want to see the detail about pilot training poor completions and other issues, read this report. Aviation hasn’t been delivering results for the investment in it and it’s good to see hard questions being asked. (On the other hand, it’s good to see Air NZ streamlining their training pathways and there are ways to make things better.)

So, this week’s thumbs-up goes to policy based on data as it allows us to have a serious discussion about where best to put our resources. And, in this case, whether we should shift resources from a current, to a better, use, just as Steven Joyce has been doing this year and last year for student loans.

Thumbs-Up celebrates good ideas or outcomes in the tertiary education sector each week.

Based on the data released so far, I have major reservations over the validity of the qualification completion EPI.  Like most others, I strongly support the release of the EPIs, and believe that there has been plenty of time for providers to get their data right.  I also believe that the other three EPIs are robust, and that they are difficult to manipulate.  I don’t think that either is as true for the qualification completion measure.

My concerns are highlighted in the comparisons between qualification completion and retention data among ITPs.  I know that these are based on different views: one is EFTS based the other person based: one is mainly historical the other forward looking.  However, I do not see how qualification completion can consistently exceed retention.  Surely if a student is retained the best that can happen is that they finally complete.  Yet for 14 of the 20 ITPs the qualification completion is greater.  Some of the differences are quite large: in one instance qualification completion is almost double retention.

Variations in timing, changes in cohort sizes, changes in the mix of full and part time students, and normal variability of results will lead to fluctuations in results at qualification, band, and provider level, but it is hard to see how these could give an average qualification completion across the whole ITP sector that is higher than the average sector retention.  Interestingly, other sectors have qualification completions lower that retention, as do ITPs for higher level qualifications.

I believe that the most likely explanation lies in the award of multiple qualifications from one programme.  Two examples of this are a local and national award that cover similar content, and the stair-casing of 1 EFTS certificate, 2 EFTS diploma and 3 EFTS degree all awarded within one three year, 3 EFTS programme.  I have absolutely no problem with either of these approaches, as long as each award is based on a thorough needs analysis and meets a clearly defined graduate profile.  What does concern me is that the decisions around the use of these award structures will be influenced in future by their impact on the qualification completion EPI as much as the value and integrity of the awards.

In their notes on the EPIs the Tertiary Education Commission acknowledge that “embedded qualifications” will affect the qualification completion EPI, and that this “may need to be taken into account in comparing qualification completion rates across providers.” This seems to significantly reduce the value of this measure.

Finally it is interesting to note that the qualification completion EPI does not take into account completion at another provider, even if this follows a planned articulation pathway.  If significant emphasis is to be placed on this EPI it may, ironically, make such pathways out of their institution less attractive for providers.

In conclusion, I understand that completion of qualifications is an important objective.  The current measure is elegant in its simplicity and in the way it correctly treats part time study and acknowledges that some students shift direction during their study.  However, I do not think it is a robust indicator of what it aims to measure, and I believe that it is open to manipulation by providers.  Perhaps a cohort based, person based approach is the only way to get a meaningful and reliable measurement of qualification completion.

Richard wrote a letter to the Minister for Tertiary Education on 9 August and, since he hasn’t had a reply, he wanted to share it with ED Blog readers. You can read the five-page letter but I think the key messages are around the difficulties in generating meaningful data on performance. I think some of the issues raised by Richard have been addressed (he cites some early release of draft data), but the issue that most interested me was the low correlation between PTE course and qualification completion, which you would expect to be reasonably high (if you complete your courses, you should get a  qual, especially in PTEs, which have been put under more pressure than TEIs over qual completion). Richard had that correlation at 0.24 on some incomplete data in August and reckons it was about 0.49 for last week’s release. I looked at the TEI correlation and it was .80, so there’s a big difference there that deserves some further exploration.

I should emphasise that Richard is not against the release of performance data, as shown by the quote below, but he does want to be much more certain about the quality of the data being released. I think robust discussion will help us get there.

“meaningful, accurate and fair indicators will:

1. Provide students with guidance when selecting a programme at a TEO.
2. Ensure that tax payers’ money is spent wisely buying the best results.
3. Allow TEOs to see how they are doing and, if necessary, improved their performance.

I believe the performance indicators are a good thing, and should be published: providing that they are meaningful, accurate and fair.”

More immediately, Richard has also identified in the list below what he says are some errors in the EPI release last week. I checked the first one, on which Richard is right, and I’ve made comments in italics on that and a few others. The errors and discrepancies are unfortunate but it’s great that people are checking on this data so that it can be improved.

1. NZ College of Early Childhood Education appears twice in list of Students Retained in Study – once with 0% and once with 70% He’s right.
2. Some organisations appear in list of Students Retained in Study but not Qualification or Course Completion – e.g. Beauty Therapy College. This organisation is no longer registered with NZQA. New Zealand Institute of Education also appears only in list of Students Retained in Study but is still registered with NZQA. Curiously “New Zealand Institute of Education 2007” has values in all three lists. I couldn’t find Beauty Therapy College on the list of 2009 funded PTEs from TEC.
3. Directions Appearance Technology Trng appears in all three lists but has 0% for Qualification Completion and Course Completion. This organisation is no longer registered with NZQA and, as far as I know, ceased training in 2008. My info from TEC on funded PTEs in 2009 does not include Directions Appearance Technology Trng
4. There are three organisations which may be the same:
   • New Zealand Career College Limited has values for all three indicators
   • New Zealand Career College 2008 Ltd has only a Qualification Completion entry.
   • New Zealand Career College (2010) Ltd has Qualification and Course Completion but not Students Retained in Study. This is probably an error but if the company name has changed, there really should be a note to reduce confusion.
5. It seems odd that some organisations have 0% Qualification Completions but Course Completions over 90% – e.g. Occupational Safety Management (NZ) Ltd at 95% and Tipu Ora Charitable Trust at 98%. That does seem a bit odd.
6. Equally it seems odd that Manaakitanga Aotearoa Charitable Trust could have 100% Course Completion but 98% Qualification Completion. I can see how this could happen as the qual could take longer than a year on average, students could enter late in the year or the entering population could be larger than the leaving population.
7. The correlation between Qualification and Course Completion is only 0.49. In practice it is probably lower than this because TEC has arbitrarily truncated values at 100% and we know that quite a number of providers had values in excess of 100%.  As noted above, this is an issue, especially since the TEIs have a .80 correlation (ITPs were 0.66, wananga o.87 and universities 0.90).

Overall, none of this discourages me too much. TEC knew that it would have work to do to tidy up this data and they’re obviously missed a few things, but they seem to have been pretty dedicated to fixing problems as they arise and I’m sure they’ll address the specific errors raised by Richard (I’ve flicked the list to them separately). It would be great if others that have concerns about the TEC’s data also fronted up with the same sort of detail as Richard has done. I’m sure some at the TEC think he’s a bit difficult, but he puts up a clear argument and data. Some others put up unclear assertions with no data and that doesn’t move the debate along at all.

1. Educational Performance Information
   • TEC released education performance information yesterday – there was also a TEC media release and one from the Minister. I liked the Minister’s ending statement: The Minister reaffirmed that, over time, the government wants to see publication of performance indicators and performance-linked funding to include employment outcomes for students. ”Ultimately the vast majority of students are studying to improve their employment prospects.  It’s important that we develop timely ways of showing them which pathways through study are best at achieving that.”
   • Sector groups have responded to the release. The TEU takes the view that any low performance is simply a matter of the students targeted and delivery style and the government should support providers to improve. Universities New Zealand noted the high level of uni performance. NZUSA noted data gaps and seems to want more, and more coherent info. NZ ITP suggests that the data will help improve performance.
   • Of course, providers have also responded to the release. MIT is the best because it is “the only institute ranked in the top five across all indicators of success” (they have done well, but their measure might be a bit creative!). Whitireia Community Poly brings together their recent EER report and the stats to argue that their teaching is great.  WelTec said they did well and that lower course completions were a 2009 aberration. The Open Polytechnic has defended its extramural delivery mode in the face of some poor qualification completion stats – at the link they also have links to more info. They state that many students just enrol to do one or a few courses. Massey made a similar case about its stats. Massey extramural students felt that the numbers mislead and confuse (no, they expose the lower results of extramural study – what is done about is is a different matter). UCOL is committed to improvement after some poor completion rates and puts out a lot of detail. Unitec welcomed the results and also provided extra detail.
   • The ODT has good coverage of TEIs in the surrounding area while the Timaru Daily Herald covers Aoraki.
   • The Greens’ Gareth Hughes thinks we should spend more money. Grant Robertson made what I think is the most constructive media release I’ve yet seen from him. He says that some polys and PTEs have some explaining to do, while more detailed info would be useful – he also thinks we should spend more money. 
2. Quake Update
   • The University of Canterbury has pushed off the student start date to 20/9, with staff back on 13/9 – their FAQ is great and includes no central library access for the rest of the semester due to the collapse of shelving.
   • CPIT closed until further notice after yesterday’s big aftershock but is now planning to open to all on 13/9.
   • SIT will be back on 13/9.
   • Avonmore staff are back today.
   • The beautiful 7-storey high rise brick building Manchester Courts, which is likely to be demolished, is the home of an ITO’s Chch office – the Retail Institute. Careerforce ITO got back to work on Monday, while I hear BCITO staff have been offered time off to help older people secure their homes.
   • UC Accommodation Student Village has made a great offer, putting up 50 free places for Civil Defence approved quake victims until 20/9, plus 50 places for UC approved quake victims, plus study space for students. Good work Darel!
   • Canty Uni’s Mark Quigley writes about aftershocks.
   • Otago Uni had a lecture on the earthquake and check out the picture here of a row of trees that has ben shifted out of alignment.
   • The Reverse Undie 500 will happen,with the OUSA van bringing up water and food – pity the water came back on today!
   • Some students have helped in the cleanup.
3. Nice Canty News A UC associate professor, Chris Cree Brown, has won 2010 SOUNZ Contemporary Award, the most prestigious annual award for ‘classical’ composition in New Zealand.
4. Fee Increases Massey’s fees rises are not welcomed by students, while NZUSA has blamed the government for fee increases.
5. Uni Rankings The QS World Rankings have been released, and the University of Auckland is the only one in the top 100 and it dropped 7 spots (they put out a release). The ODT covered Otago Uni’s ten places drop.
6. Software Split Waiariki is going for a Tribal student management system, moving on from Artena. Tribal is a UK company that runs the ITP benchmarking software.
7. Bits and Pieces Bay of Plenty Poly music students are doing a showcase. A Waiariki BFA student is exhibiting in Melbourne soon and may also do so in the US (PS she helped design the tino rangatiratanga flag). Here’s another story on the Dyson Award finalists. Otago Uni researchers want top replicate some Aussie research on the effetciveness of public health measures. NMIT is a finalist in the Tourism Industry Awards. An AUT academic comments on Samoan IP rights. Grant Robertson used World Literacy Day to challenge government funding policy. SIT CE Penny Simmonds has been slightly dragged into the mayoral campaign in Invercargill.
8. Aquaculture Local dignitaries visit Bay of Plenty Poly’s new aquaculture centre, building on their long-term involvement in marine studies. NMIT yesterday announced that they had launched NZ’s first aquaculture diploma - I don’t care who is first but both ITPs seem to be responding well to the coming aquaculture boom.

Qualification completions are not as clear cut as course completions, which I covered earlier today. That is because many students do enrol in a qualification but only intend to complete one or a few courses within that qual. Of course, TEC funds and desires quals, so the stat is important. Many providers also purposely enrol people in courses, knowing they won’t do the whole qual, so that’s a choice that will depress stats.

As with course completions, universities are at the top of this table, filling the first 5 places, as well as 8 and 9. Massey University, however, is down at 17th place, probably reflecting the lower completion rates on extramural qualifications (although I suspect that doesn’t explain the whole difference, as Massey has a lot of face to face students).

The 10 TEIs below 50% deserve some extra scrutiny, but I’m willing to give those ten a little more time to contextualise their situation (and take some time to think about it), as the issues are more complex. The Open Polytechnic has very, very low completion rates, but they’re at the sharp end of the course vs qual enrolment issue. Raukawa and Waiariki are next from the bottom of the list, but that’s probably related to them being on the bottom of the course completion table.

44 PTEs had qualification completion rates below 50%. On the other hand, 19 had 100% and another 20 had 90-99% qualification completion rates.

The universities fill the top 7 places in the table, with Massey in 10th (probably due to extramural courses). That’s unsurprising, given they have some entry limitations on courses and are dealing with a group that is probably the best academically prepared. Even so, well done to them. I don’t think we should worry too much about institutions above 70%, or even the differences in rankings between them. They’ll probably run some courses with very low completion rates that will need attention, but their average performance is OK. I’ll deal with issues after the table!

The four TEIs with course completions below 60% owe the public an explanation, and especially Waiariki Institute of Technology at 50% and Te Wananga O Raukawa at 42%. It is simply unacceptable to enrol people that have a lower than 50% chance of completing your course. It is a waste of students’ time and the government’s money. To achieve those poor levels of completions, some or all of entry standards, teaching, assessment or pastoral care are stuffed. You might wonder why I have included UCOL and Tairawhiti as well – the reason is that that average will probably hide a large number of sub-50% completion rate courses. Just as I was about to publish this post, I received a media release from UCOL with some marketing bumph but also some thoughtful comments on their rankings – good on them. I’ll link to it when it goes online)

By the way 16/199 PTEs had course completions of under 50% , and another 12 had 50-59%. They range from Practical Education Institute at 49% down to Directions Appearance Technology Training at 0% (I suspect they’re no longer funded). They also need to explain that performance.

TEC’s CE, Roy Sharp, said that “Today’s release of tertiary education performance information will allow the general public to understand more about how their local tertiary providers are performing”. The indicators being released by TEC are (full TEC technical report here):

• successful course completion (all EFTS completed a course/all EFTS enrolled in a course, but only covering courses expected to end in the relevant year);
• completion of qualifications (all qual completions multiplied by EFTS value/total EFTS enrolled in courses);
• student progression to higher level study (number of students enrolled in a higher qual within 12 months following qual completion/number of students completing a qual in year n); and
• students retained in study (the proportion of students that re-enrol in year n+1 or complete a qual in year n or n+1/students enrolled in year n).

Roy went on to say “For the first time we are releasing information that can be used to make year on year comparisons of tertiary providers’ performance…Providers have put in a lot of effort to prepare their information and to make it accurate. I would like to thank them for their work in this process.” He also pointed students and parents to tertiary provider websites and Career Services for more info about tertiary education.

Of the four indicators released today, course completions and qualification completions are the most useful, as they provide an indicator of students’ progression in their goals. The other two indicators are TEC creations that are unlikely to be very relevant to the general public, largely because they are only meaningful at a course or qualification level – if a qual is designed to get you a job, then a low progression to further education will not matter to students.

The data only relates to domestic students funded at providers through the Student Achievement Component. ITO data will be released at a later date.

Every provider has a snazzy PDF provided for them, listing the aggregate results for their subsector (eg ITPs, PTEs, universities) on p.1, the provider stats on p.2 and the subsector rankings on p.3. Below I’ve provided the main subsector stats and a further post will have TEI stats.

If you look at the course and qual completions above, universities seems to be highest, with PTEs in second – what that means will wait for another post. A Courses v Qualifications chart by David Choat, my colleague at ED, shows that there is an apparent relationship between the two, as you would expect.

The detailed info is being published at an organisation, rather than a course or programme, level. This will make it largely useless for most students seeking info on public tertiary education institutions, as the aggregate figures will not provide any useful pointer to a specific course or programme. The figures will be a useful guide for students looking at  PTEs and OTEPs, which usually have a much tighter subject focus and fewer programmes. Of course, this release is largely intended to focus on organisational performance and is a preparation for more detailed data releases in future years (that’s not TEC’s official line, but my prediction). Releasing the data will expose the differences between organisations and drive some policy and practice changes.

• There is an enormous amount of data available on tertiary education inputs (eg enrolments), but very little on outputs (course or qual completions) or outcomes (did people get a job, make a living, enrol for another qual, etc). TEC’s data release will help a lot with filling in the outputs area, but graduate outcome data will be the next step.
• The data that will be released will have various errors, stemming from two main issues.
  1. The first is that tertiary education organisations (TEOs) have had no good incentive to tidy up their data until now and it will take time to fix things. That is not a reason to delay publication any further as TEOs will continue to delay until data is published – it’s a natural reaction. Exactly the same arguments about poor quality data were raised in 2003 when then-Minister Steve Maharey proposed releasing similar data.
  2. The second is that TEC has produced data using techniques (and sometimes raw data) that TEOs do not understand or agree with. The TEC has, in response, delayed publication several times to fix things, which is good. I suspect, however, that TEC has made things too complicated and will continue to face reasonable criticism over their techniques. TEC will need to improve things before funding is based on their data, but I’m not convinced that the data should be withheld.
• Data on whether students stay in the course (retention) are important because they give us a sense of whether the course material and teaching is engaging and whether students were appropriately assessed before enrolment. If students are bored or overwhelmed they are likely to leave early.
• Data on whether students succeed in the course (course completion) are important. Most people should pass a well-targeted course and low pass rates will usually point to lax entry standards or poor teaching (or an earthquake in some rare cases).
• Data on whether students gain the qualification (qualification completion) are important because the government’s policy is to encourage qualification completion, as that is associated with higher incomes. TEOs will argue that many students enrol just to get a course – that’s true, but the government has been clear for a long time that it does not want to fund or encourage that. If TEOs have been ignoring that position for many years, then it is hard to complain when the government reinforces it through funding shifts. (NB I am supportive of course enrolments being funded outside of wider qualifications, as a segment of overall funding.)
• Data on whether students enrol in a further qualification (progression) are useful, but are less clear-cut than the other measures. The utility of further quals will vary by industry/subject and I suspect this measure will be the least used of the four to be published.
• Every TEO will come up different reasons for why their enrolments are affected by student factors, but they should mostly be disregarded unless they are accompanied with a serious research report. The fact is that every TEO will have a good number of students shifting, getting pregnant, losing a job, going to jail, getting bored or playing X-Box. At an individual course level those factors will skew results, but at an organisational level, the results will be a fair reflection of the enrolment, delivery and student support policies and practices of an organisation.
• There is a reasonable argument that enrolling some population groups may depress completion rates, given those groups’ worse prior academic achievement. But that is a difficult issue, as some TEOs do quite well with groups that otherwise have poor performance – one good thing about the release of stats is that we will  get a clearer picture of who does well and who doesn’t with different groups.
• Publishing performance data may lead people to game the system and artificially increase pass rates, but I think the more likely response that professionals in the tertiary education sector will take is solid work to help improve students’ performance. I expect that the attention to detail will far outweigh the gaming, and that gaming will often (but not always) be picked up at the margins.

Overall, I’m looking forward to this data helping students to choose their courses and redirecting providers’ efforts. Students who enrol and do not stay in the course, do not pass their courses or do not get a qual are a silent group in education. They usually do not get surveyed on their experience (that is left to the graduates) but they are saddled with debt, foregone earnings and quite probably discouraged from further education. I support publication of performance stats for the impact it may have on those students.