Quality assurance of a programme of workplace-based assessment
In viewing the quality assurance of a programme of workplace-based assessment it is helpful to review the utility equation described earlier. Workplace-based assessment has strengths in the areas of validity (by virtue of its authenticity), educational impact and acceptability (because it reconnects teaching and learning), and feasibility (through local assessment).
There are, however, problems with demonstrating its reliability using traditional psychometric approaches. As Southgate et al. (2001) point out, ‘establishing the reliability of assessments of performance in the workplace is difficult because they rely on expert judgements of unstandardized material’.
In workplace assessment like any other form of assessment there are several potential threats to reliability:
- inter-observer variation (the tendency for one observer to mark consistently higher or lower than another)
- intra-observer variation (the variation in an observer’s performance for no apparent reason - the ‘good/bad day’ phenomenon)
- case specificity (the variation in the candidate’s performance from one challenge to another, even when they seem to test the same attribute) (Crossley et al., 2002).
Therefore it might be helpful to reframe the term reliability with the phrase maximising ‘consistency and comparability’. Baker et al. (1991) have proposed a framework that can do this by incorporating the following five elements within the assessment design.
- Specification of standards, criteria, scoring guides.
- Calibration of assessors and moderators.
- Moderation of results, particularly those on the borderline.
- Training of assessors, with retraining where necessary.
- Verification and audit through quality assurance measures and the collection of reliability data.
It is clear, therefore, that the implementation of workplace-based assessment will require a complementary training programme, arrangements for calibration, a procedure for the moderation of results and a raft of quality control and reliability checks.
The more that teachers can be engaged in assessment, in selecting methodologies, generating standards, discussing criteria, etc., the more they will be empowered in the educational process.
- What do you think are the training requirements for trainers and supervisors in relation to the above tools?