Principles of assessment
The traditional educational paradigm below indicates the importance of integrating teaching, learning and assessment, where assessment should figure as an integral part of educational planning. However, it is all too easy to view assessment as a ‘bolt-on’ component at the end of an educational programme.
The utility, or usefulness, of an assessment has been defined as a product of its reliability, validity, cost-effectiveness, acceptability and educational impact (van der Vleuten, 1996). In later years the term ‘feasibility’ has been added. Utility can be expressed as an index:
|Utility = educational impact x reliablity x validity x cost effectiveness x acceptability|
Utility can be applied to an entire assessment system or to an individual assessment method or component of the system. The concept is important in that no single element of utility should be regarded as a panacea. Instead, assessment design should pay attention to all the elements within the utility equation, although it is recognised that there may be a ‘trade off’ between the elements.
Thus, traditional approaches to maximise the reliability or reproducibility of assessments can have a negative educational impact on the learner by reducing the opportunity for meaningful developmental feedback. Whereas workplace-based assessments can offer high educational impact but might not be performed as reliably as other medical assessments, e.g. MCQs.
From methods to programmes
Traditional approaches to developing medical assessments have been founded on the notion that domains of medical competence (e.g. problem solving, communication skills) are stable and generic. As a result, it is possible to design tests that assess these domains separately and reliably. This has led to the ‘one trait, one instrument’ approach to assessment (Schuwirth and Van der Vleuten, 2004).
However, there is a growing realisation that medical competence is specific to particular clinical situations or clinical contexts, and not generic. This is called case specificity and poses particular problems in assessing doctors. To overcome this, the most important thing is to sample widely across the content of the curriculum.
In recognition of the complexity of assessing professional competence there is a growing realisation that assessment should be considered as a programme of activity requiring the assimilation of quantitative and qualitative information from different sources. Assessing doctors in their actual working environment therefore offers the opportunity to gather sufficient information to build up a ‘rich picture’ of them.
This also has a powerful impact on work-based learning through the provision of feedback and encouragement of reflection, and takes us right back to the educational paradigm.
- Try applying the utility index to an assessment method with which you are familiar. What are the assessment’s strengths and weaknesses?