Some methods for gathering evidence
A brief overview of some of the main methods for gathering evidence currently used in the United Kingdom is shown below.
Observations of patient encounters
Mini-CEX (the clinical encounter)
It has been traditional for clinical skills to be assessed by the ‘long case presentation’. The problem of case specificity using this technique, limiting the potential to sample widely across different clinical cases and contexts, has given rise to a version of the ‘short case examination’, popularly known as the mini-CEX (Norcini et al., 1995). The technique has been developed to assess the clinical skills that trainees most often use in real clinical encounters. It is based on assessment of multiple encounters within a hospital setting observed by an educational supervisor or other clinician.
Direct observation of skills
This category is again an assessment of real-life activities where the focus of the assessment is the skill with which the activity was performed, e.g. direct observation of technical skills (DOPS), teaching skills and presentation skills. ‘The consistent feature is that one or more assessors, who are trained in the assessment of that skill, make a judgment about a real life performance’ (PMETB, 2007).
Discussion of clinical cases
Case-based discussion (CbD)
The technical development for the use of CbD in UK training assessment systems is based on the use of case-based discussion in the General Medical Council (Southgate et al., 2001), which itself is derived from chart-stimulated recall oral assessments used extensively in the US and Canada.
CbD is one of the evidence gathering tools used in the framework for workplace-based assessment in the UK Foundation programme and is also being used in specialty training programmes, e.g. the nMRCGP licensing assessment of the Royal College of General Practitioners.
Norcini (2003) classifies methods of work-based assessment in two dimensions.
- The basis for making a judgement:
o outcomes of care, while being the most desirable, are limited by problems of attribution (to the individual), complexity, case mix and numbers
o process of care is more directly in the control of the doctor but does not necessarily guarantee the best patient outcome
o volume of activity information is premised on the basis that the more of a given activity that a doctor performs, the better the likely quality of care in relation to that activity. This method of judgement is more likely to be applicable to surgeons who are engaged in numbers of specific technical procedures.
- The actual methods of collecting data:
o external audit of medical records
o use of administrative databases
o log diaries
o direct observation.
Lockyer (2003) describes the principles involved in multi-source feedback (MSF), which is seen as a practical approach to assessing doctors in the workplace. The goal is to view a person’s work from a variety of perspectives. In medical settings, physician colleagues (peers), co-workers and patients can be asked to complete surveys about the doctor.
The person being assessed receives feedback with their own aggregate ratings, along with ratings for others being assessed at the same time. The opportunity for comparing self-assessment data with those provided by raters is clear. Early evaluation suggests that 6–8 raters per cycle of MSF are required to make the tools sufficiently reliable.
MSF tools can be further classified into:
- peer–rating tools, e.g. the TAB (Team Assessment of Behaviour) which is used in foundation training
- patient satisfaction questionnaires (PSQs). The following list indicates some of the tools being used within the UK at present (Chisolm and Askham, 2006):
o Consultation and Relational Empathy (CARE) questionnaire
o SHEFFPAT (Sheffield patient assessment tool)
o General Practice Assessment Questionnaire (GPAQ)
o Consultation Satisfaction Questionnaire (CSQ)
o Doctors Interpersonal Skills Questionnaire (DISQ).
- Think of a professional competency area from your own specialty, or the Foundation curriculum. What do you think would constitute ‘sufficient evidence’?