Unseen assessment feedback

Matthew Weait, Open University

I begin with a very brief description of the Open University (OU) law programme and the way in which our courses are assessed. There are currently four 60 point courses, which cover the seven foundation subjects. These are all assessed by means of coursework and unseen summative examination, worth 50% each. The examination is in two parts, each of which relates to one of the foundations. To pass the course students must pass both the coursework component and both parts of the exam.

In common with other OU courses, the coursework is assessed by means of tutor marked assessments, or TMAs, and are marked according to explicit criteria known both to the student and the tutor. Tutors are trained to comment constructively in both general and specific terms on the student's work, and each student receives detailed feedback on the six or sevenpieces of work they are required to submit during the course. The marking and comments of tutors during their probationary period is subject to continuous monitoring, and tutors are encouraged to reflect critically on their approach.

The reasons for introducing personalised unseen examination feedback

Until last year, comments to students on their unseen examination performance was limited, as it is in many other institutions, to general feedback about the performance of the cohort as a whole in respect of each of the questions. While useful in some respects, this is obviously less than ideal, since it does not help the individual candidate, either in knowing why they received the mark they did, or in identifying particular areas of weakness which they could focus on in order to improve their performance in subsequent examinations.

In order to rectify this we decided to pilot a means, used elsewhere in the university, by which candidates would be given information about their individual performance in each question attempted.

The feedback mechanism

Essentially the mechanism we used was this:

• each of the scriptmarkers for the examination in criminal and constitutional law was provided with a computer readable form
• on this form are listed a set of criteria against which each question attempted on the paper is evaluated and a range of possible options from excellent to missing/absent to use in respect of each of those criteria
• markers were given an explanation of what was meant by each of the criteria, and what was meant by excellent, very good etc
• they were then asked to fill in each candidate's form as they marked the question the completed form was sent back to the OU along with the scripts, and a letter generated for each candidate which explained the criteria and the gradings

In this way each candidate was provided, in addition to their overall pass Grade (1,2,3,4 etc), with detailed information about the quality and attributes of their answers.

Justification for feedback

The basic justification for feedback is, I think, self-evident but needs emphasising. Put simply, it enhances the learning experience for our students.

We tend, I think, to see summative unseen examinations as a means primarily of establishing what our students have learned and determining their current status. We provide them with a mark, or with a class of achievement, which - depending on how clear we are about what that mark or class means - with some idea of where they stand in relation to their peers, or whether they have improved on their overall performance since the last time they were tested. However, that mark or class, despite being necessary and important, can also have an unduly negative impact, and/or give the wrong message, especially to students in the mid- to lower range of exam attainment:

1. It goes without saying that marks and classes are necessarily reductive they can neither capture nor reflect, except in the most general terms, the range of skills and attributes demonstrated in a student's exam performance. We reduce the essentially qualitative to the quantitative, ennumerate the literate. In this process of quantification much is lost.
2. Marks and classes analogise the performance of unlike candidates. The person who gets marks of 75, 65, 45 and 35 on a paper gets an average of 55, despite producing nothing of 2:2 quality, and so generic feedback on what a 2:2 performance means will be of little assistance to him or her. Furthermore, that person will (in the ranking) be 'equivalent' to the person who achieved 57, 56, 54 and 53, despite being utterly different from him or her in any meaningful sense.
3. The person who achieves 55 and is told nothing about why may become disenchanted and characterise him or herself as a 2:2 type student, despite the fact that there may be many aspects of the performance worth praising.
4. More generally, marking without feedback can create the impression that examination is about control and about judgement about us telling them what they're worth. This is to some extent inevitable in any process of testing or evaluation, but if our concern is to create independent and autonomous learners, able to reflect critically on their own work and take control of it themselves, then we are surely under an obligation to provide them with the information which will enable them to do so

Recasting, or re-imagining the unseen summative exam as part of an on-going learning process, as formative and developmental, can enable students to identify what aspects of their approach to learning and of communicating that learning need to be improved.

Other functions

Those then are some of the reasons why students can benefit from feedback, but we can benefit from it too. Put simply, if we want to understand, as we no doubt do, how and why our students perform as they do in the examinations we set, structured feedback can be very helpful.

We, just as students, tend to see exams as the end of the road in course terms. We mark the scripts, write relatively cursory comments on them, bundle them up and forget about them. But just as they ought properly be seen as part of the learning process for students, so they can be part of the teaching and learning process for us.

These are just a few ways in which I think they can help:

1. Using the type of feedback form we have developed, it will be possible to see more clearly what the strengths and weaknesses of a cohort of students taking an exam are, whether this be generally, in relation to different types of question, or in relation to the different subject areas covered by the questions. It will be possible, for example, to see whether there was a general problem with the use of primary materials, doctrinal analysis, or grammar; or whether candidates were better at applying the relevant rules and principles in problem-type or essay-type questions; or whether knowledge and understanding was more apparent in answers to questions about offences against the person than in questions about property offences. This knowledge, which we tend to gain by impression, will now be more firmly grounded in fact and will enable us to make appropriate changes to the materials and teaching strategy.
2. The feedback information will give us a far clearer idea of what is valued or prioritised by markers. It will be possible to see, for example, whether people whose knowledge and understanding was excellent but whose application of that knowledge was merely good or adequate received better marks, on average, than those whose attainment in respect of those criteria was the converse. This will help us reflect critically on the marking process, and on the (often implicit and unarticulated) values that inform it.
3. The feedback information will enable us to see what deficiencies are most common in those candidates receiving marks in the 2:2 range and below. We will then be in a position to focus our teaching and exam preparation on developing these skills and attributes.

All this will of course take work, but the advantage of a computerised system is that the information we want can be generated relatively easily and will be far more reliable than anything we have managed to achieve so far.

Disadvantages, problems and issues

I now want briefly to mention some of the disadvantages, problems and issues which feedback of this kind raises, and which I think could usefully be discussed.

As far as disadvantages go, these are both of a theoretical/pedagogical and practical kind. Pedagogically, some have argued that feedback can create a kind of dependency which inhibits, rather than promotes, independent learning. The student who relies on feedback from others is less likely, according to this line of reasoning, to develop the self-criticism which is the mark of an independent learner.

Although I can see how this can happen, I don't think the argument carries much weight, especially if students are also encouraged and trained to reflect on their own performance. And I don't see how this can be done unless we give them something to reflect on.

Practically, feedback takes time. We are all under enormous pressure as it is, and this is yet one more thing. However, I think that by using pro-forma, and/or computer readable forms, and by filling them in at the same time as marking, the amount of extra work is more than compensated for by the advantages it brings both us and the students.

As for the problems and issues, I would raise the following they are ones I am still grappling with.

1. There is the question of formulating the feedback criteria. We decided that the Quality Assurance Agency's law benchmark statement provided as good a basis as any for this, and have adapted these for the pilot. They cover a good range of skills and attributes, and are ones which we are building explicitly into our courses. It is important, I think, that there is congruence between the way in which we teach, the way in which we want our students to learn and the way that we assess.
2. A related question is how general or specific the criteria should be. It seems to me that they need to be sufficiently explicit and focused to enable targeting of particular areas of weakness or difficulty in a subject-specific context, but sufficiently general to be of use to students who may be going on to study courses other than in law during the course of their degree (this is very common at the OU, but may be less of an issue in other institutions).
3. There is the question of how the feedback criteria relate to the marking scheme. This is a really difficult one. At the moment, the feedback we have provided is supplemental and diagnostic. The examination which the students took and for which feedback was provided was graded according to a more general set of criteria of the kind normally used in determining whether a paper merits a particular grade. This was unavoidable, because the decision to provide feedback was taken too late in the year to change the marking scheme for the exam. However, this year, when we will be providing feedback for all our courses, the plan is to correlate the criteria for determining marks with those used in the feedback. They will also be used for the purpose of determining coursework marks, in order to ensure consistency within the course and to ensure that students get used to them before they take the examination. That said, there is still the issue of whether particular marks should be available in respect of particular criteria (10% for doctrinal analysis etc). I am open-minded about this, but can see real problems. How, for example, should the criteria be weighted? This problem comes into even sharper relief when considering whether different criteria should be used in courses at different levels. My gut feeling is that they should be different; but determining how is complex.
4. A final problem I want to raise before finishing is more fundamental. By providing feedback to the students in this way, we have effectively created a dialogue where we have had the last word. Ideally students should be given the opportunity to reflect on, and respond to, the feedback; but I have no idea how this could be achieved. Unless we are prepared to give the exam papers back to the candidates, along with the feedback, and respond to any subsequent concerns or questions they may have then the feedback may simply end up being perceived as another form of judgment, with negative consequences.