The use of a research trail as an assessment: a practical case study

Hazel Dawe, London Metropolitan University

The reasons for introducing a research trail

Assessment for the Law of Contract 2 module was 100% by exam with one seen question. The fact that we handed out a seen question for the students to research was deemed to be an assessment of their research skills. The theory was that this would mean that students would research the seen question, and when marking the seen question we would be assessing their research skills. However, when marking scripts it became obvious that whilst the majority of students were doing the research, a substantial minority were neglecting to do any research at all for the seen question, to the point where some of them did not even obtain a copy of the question before the exam. The implication was that we were neither teaching them research skills nor giving them any real practice in research skills.

The course specifications included the following:

Benchmarking skills specifically taught and assessed:

• sources and research

Expected learning outcomes:

On successfully completing this unit the student will be able to:

• read, understand and use primary sources
• undertake guided legal research

It was however not clear that we were, in fact, teaching and assessing sources and research appropriately. The form of assessment we were using did not seem to provide sufficient encouragement to students to acquire the skills we were claiming to be assessing. “The main purpose of assessment is to discover if students have achieved the learning outcomes of the course studied” (Bone, 1999) We were discovering that a substantial minority of our students were not achieving some of the learning outcomes specified for the course.

Research skills were taught to some extent in tutorials, as students were shown how to solve problem questions, but this was non-specific and incidental. There was also no express mechanism for motivating students to do the research: “assessment is often the major driving force which gets students down to serious studying” (Race, 1995). It therefore became obvious that, if we wanted students to take research skills seriously and learn them, we would have to tailor an assessment to assess research skills specifically. “Students can escape bad teaching, they can’t escape bad assessment” (Boud, 1994).

The merger of the London Guildhall University with the University of North London to become London Metropolitan meant that the degree programme was rewritten and now had two full contract modules instead of the previous one and a half. This created the opportunity to introduce a research trail on the seen question as part of the assessment for the new Contract 2 module. More time was available for skills teaching, as some topics were moved to the new Contract 1 module slimming down the curriculum for Contract 2.

Although there is abundant literature on how to do research, it proved impossible to find any literature on how to write a research trail, despite extensive research on the subject. We were therefore obliged to formulate our own guidelines as to how to write a research trail. This paper examines how experience led to changes in teaching methods and in the instruction to students, as well as how we formulated our marking guidelines. Comments are invited as to how to improve what is being presented here.

Research exercises in tutorials: the pilot project

The year before the new assessment was introduced research exercises were piloted in the old Contract 2 module during tutorials, using the problem-based learning (PBL) method. This was a two stage process, with preparation for the research exercise discussed in one tutorial then the result of the students’ efforts presented in the following tutorial. The students were encouraged in the first tutorial to think about how they would approach this particular research task. They were asked how they intended to do the research and what sources they intended to use.

Although the choice of research methodology was voluntary, tutors attempted to ensure that some students in each tutorial group would use electronic sources and some would use hard copy. Depending on the size of the group, a spread of different electronic and hard copy sources was also used. For example, of those students using hard copy an attempt would be made to ensure that some used law reports, some used text books and some used journals. Of those students using electronic sources, where possible a spread of different databases was explored. In the subsequent research exercise the students were encouraged to swap and use a type of resource they had not used on the previous occasion. In the second week of the research exercise the students discussed not only the answer but also the relative merits of the various sources and how useful had they found the different sources. In consequence, they were led towards an evaluative discussion of research methods.

Originally three research exercises were planned - each one taking two weeks, with the first one starting in week three of the eleven week course to allow tutorial groups to settle. However, once the students had received the seen question in week eight of the course it seemed pointless to ask them to do another research exercise when they were already busy with the actual assessed research trail. Therefore after the first pilot only two research exercises were scheduled each semester.

As this was not a formal assessment student participation was voluntary. Students therefore tended to use their favourite method of research first and then their least favourite. Most students participated with some enthusiasm. Participation was excellent on the first exercise, but non-participation increased with later exercises. This was probably because the research exercises were not assessed and were perceived to be over and above normal tutorial work.

Experience of the research exercises with the assessed research trail

The first research exercises were based on the students finding a particular case. They were given some basic facts and some legal issues covered in the case. The two exercises were graded according to their difficulty, the first being easier than the second. In the first exercise the students were merely asked to report verbally in the tutorial on how they had found the case. This was in order to introduce them gradually to a novel form of academic activity. In the second exercise they were asked to write a written report to increase the degree of formality and to give them initial experience of writing a mini research trail before the actual assessment.

The research exercises done in the tutorials were such that students were given a brief outline of the facts and/ or legal issues of a particular case and then asked to find that case. This differed in nature from the actual assessment, where they were asked to write a research trail on how they had researched the answer for a contract law problem question. Unfortunately, this led to considerable confusion, with many students trying to find 'the case' instead of researching the answer to the problem question. One student managed to answer the question in the exam by only referring to one case, which he had assumed was 'the case'. This occurred despite the fact that students had queried whether this was the object of the exercise. It had also been announced in three successive lectures that the research trail was not about finding 'the case' but about researching the answer to the problem question. Written instructions were also handed out with the seen question as follows:

You must write a research trail on this question and hand it in by Friday 6th May.
Only report on your research; do NOT write an answer. You should answer the question in the exam.

New research trails have now been written which are similar in nature to the assessment task (see research exercise A).

Another problem was that students were unfamiliar with the term 'research trail' and therefore did not understand what was being asked of them and how they were supposed to tackle this. Instructions had been given, but the students were still uncertain. Guidelines which incorporated a mini example of how to write a research report based on researching one of the tutorial research exercises were written, but explanations to students who enquired at the end of lectures about the research trail still seemed to leave them in some confusion. The illuminating moment came when the lecturer explained that what was wanted was a report on how the research had been done. The one small word 'report' seemed to alleviate the confusion. The students appeared to be satisfied and feel they had a better understanding of what was being asked of them.

The assessment has therefore been renamed a research report rather than a research trail, and the instructions have been expanded to include style tips based on some of the most common student errors and are now incorporated into the handbook.

Law of Contract 2 has a very low pass rate, generally the lowest of any law module in the first year. However, the students’ marks for their research trail were almost uniformly better than their exam results. Anomalous results with very high marks for the research trail and extremely low marks for the exam have been detected in at least two cases. On analysis one student appears to have answered the exam using none of the material mentioned in the research trail. In the second case the student appears to have a problem with exam nerves, and has been recommended to seek special arrangements so she can take the exam in a less intimidating environment. Contract 2 has student numbers of over 200, and the exam hall experience can be very intimidating.

Marking guidelines

Again, we could find no template for marking guidelines for the research trail and were forced to invent our own guidelines. This was done on the basis of marking the first formative written research trails from students on the course. This was necessarily limited both by the nature of the exercise we were marking and by the small number of students who participated in a voluntary non-assessed exercise, giving us a small sample of student work.

However, the disparate nature of the research exercises did not seem to make much difference to the ability to extrapolate guidelines from the students’ work. Later experience showed that the basic principles for both the tutorial research exercises and the assessed research trail still seemed to be the same, with the main difficulty being in not having any work in the 2.1 range. This meant that although we could draft guidelines for 1st, 2.2, 3rd and Fail it was very difficult to draft guidelines for 2.1s. Our marking guidelines were refined and extended after the first experience of marking the actual assessed research trail - feedback and suggestions for improvements would be appreciated.

Research trails: marking criteria

First

• Explains in detail what the students has done and how but also why they did it and evaluates why some methods of research are better than others.
• Demonstrates past experience of research and a grasp of research methods. Uses the full range of sources. Cites cases and statutes correctly and gives book and journal references in full.
• Clear explanations.
• Well structured. First explains how the student found the material and then what was found.

Upper second

• Explains how research was done in some detail. Evaluates research methods. Grasp of research methods is good. Wide range of sources used. Statutes and cases may be correctly cited. References to books and journals are correct and reasonably clear but probably not full.
• Explanations reasonably clear – covers what student did, how and sometimes why as well.
• Clear structure.

Lower second

• Gives explanation of how the research was done but does not evaluate research methods. Grasp of research methods is not very good. Does not use the full range of sources. May not be familiar with electronic databases. Cases and statutes are probably not cited using the correct notation.
• References to books and journal articles are not made in full.
• Structure may be poor or unclear. May include elements of an “answer” rather than a proper research report.

Third

• Does not explain fully how the research was carried out or why. No evaluation of research methods. Poor grasp of research methods. Approach to research may be incoherent. Uses very limited sources. Probably is not familiar with electronic databases.
• May be too much of an “answer” to the question instead of a research report.
• Poorly structured. Citations and references are incomplete or incorrectly stated.

Fail

• No evidence of a grasp of research methods May have approached the research badly and used inappropriate resources. No evaluation of different research methods. Might say what the student has done but not why and perhaps also not how. Citations and references are done incorrectly. Very limited use of sources. Probably unfamiliar with electronic databases.
• Probably too much of an 'answer' to the question rather than a research report.
• Poorly structured.

In the guidelines we concentrated on how the students should report their research. Agreement was reached that we would not mark the students on the accuracy of their research. It is interesting to note that two markers who had not heard the agreement before they started marking gave average marks considerably below the other markers - 12% and 16% below the averages of the other markers.

As the marking outline shows we used criteria related to:

• How well did the students explain how they had done the research?
• Did they demonstrate a grasp of research methods?
• Had they used a wide range of sources?
• Had they explained why they used particular sources and why they did not use others?
• Had they evaluated the various sources?
• How well did they structure their report?

Obviously these are all indicators rather than absolutes. For example, if a student had not used the full range of sources but had explained clearly and reasonably why not, this was often taken into account, especially if the explanation showed clear evidence of past experience of using the rejected sources. Students were penalised quite heavily if what they wrote was an 'answer' rather than a research report. They had clear instructions at the top of the handout with the seen question that they were to write a research report, not to write an answer.

Despite these instructions a considerable minority of students did write an answer rather than a research report. Most of these failed the coursework. The most recent set of instructions now includes a reference to the section in the module handbook on 'how to write a research trail', with page numbers:

You must write a research trail on this question and hand it in by Friday 16th December
Only report on your research; follow the instructions on pps. 24-25 of the module handbook; do NOT write an answer. You should answer the question in the exam.

As this was the very first time we had set such an assessment we had no previous experience to draw on as an indicator of how many words students were likely to need to write a research trail. We therefore gave an indicative word limit of 1,000 words. Experience showed that most of the best work was in the 1,200-1,500 word range. We have therefore now given a fixed word limit of 1,500 words. Students who had used less than 1,000 words did not generally give good answers.

Conclusion

Our original assessment of one seen question as part of the exam was obviously not fulfilling the function of assessing students on the benchmarking skill of 'sources and research'. We hope that the redesign of the assessment to include a research trail on the seen question has now remedied the problem, and that students are now not only being assessed on research skills but that we are now teaching research skills better and encouraging our students to become more autonomous.

The best answers came from students who followed the instructions handed out to them on how to structure their research trails. The instructions to students and the marking guidelines are very much our own creation. We are happy to share them with others and would be grateful to receive any constructive feedback.

References

• Boud D (1994) Keynote speech at Staff and Educational Development Association conference (unpublished)
• Bone A (1999) Ensuring successful assesssment Coventry: National Centre for Legal Education
• Race P (1995) 'The art of assessing' The New Academic 5:1

---

Biography of Hazel Dawe

Hazel is a Senior Lecturer at the newly merged London Metropolitan University, where she teaches contract and European law. She has also taught at the University of Kent. Her interest in the variety of teaching and assessment methods and their effects on students was stimulated when she passed the PGCTLHE at one of London Metropolitan's predecessor institutions, London Guildhall University.