This guest post is by Richard Hamilton-Williams, a partner in Meta-Office, which provides the Take 2 student management system. Richard has been involved in education-related information systems since 1992.

Remember about this time last year when TEC trotted out a set of performance indicator information which nobody could understand, which ignored a Statistics New Zealand standard and which contained figures affected by a “processing error” that had to be corrected?

Well its that time of year once more and TEC is doing it again. After abandoning the performance measures used last year a new set of Educational Performance Indicator values (EPI) are to be published by TEC, and TEOs (tertiary education organisations) are being given a chance to comment on the accuracy of figures that have been compiled before they are made public.

Word is that there are some pretty unhappy TEOs out there. They have reviewed the figures and found them wanting. The common theme is that the figures do not accurately report the performance of the TEOs and that TEC’s intention to publish them is unreasonable. And when you look at what has happened it is easy to understand why TEC has managed to upset so many people.

1. The new indicator measures were not published until the end of March and should therefore not be used retrospectively for 2009. In a rational world the earliest they could be deployed would be for the 2011 year.
2. The new measures are in any case poorly thought out and do not reliably or consistently measure what they claim to measure.
3. Although the new measures use the same titles as the old measures they are not the same. It is therefore not valid to compare a TEO’s 2009 figures with those published in earlier years.
4. The new measures are specified in a manner which means that some TEOs will have an apparently poorer performance than their peers simply because of factors such as the length and start dates of their programmes. It is therefore not valid to compare one TEO’s figures with those of another.
5. The data used by TEC to compile their figures is drawn from the SDR (Single Data Return). The SDR does not provide the full set of data necessary to support the specified indicators, in particular qualification enrolment records. Therefore TEC manipulates the SDR data to create pseudo enrolment records. This does not work consistently.
6. There is some evidence that TEC has again experienced a “processing error” – as they did last year – and that the error means that in some cases perfectly valid records are not being counted. This sometimes results in the figures they have presented being better than they should be and sometimes worse; in short it is a lottery.

It is my view that should TEC persist in its plans to publish figures notwithstanding the problems detailed above TEOs would have a good case to contact the Auditor General who “provides Parliament with independent assurance that public sector organisations are operating, and accounting for their performance, in keeping with Parliament’s intentions.”

On a more positive note, and in order to keep the show on the road, maybe TEC should dig into the archives, dust off the old “Statement Of Service Performance (SSP) – for Student Achievement Component-funded qualifications” template last used for the 2008 year, and send that out for use for the 2009 year.