I had a chance to have a look at the workings of
ReView
while I was at UNSW. It is an impressive feedback tool with some affordances
that aren’t yet available in competing tools but also some limitations
which may be deal breakers for most institutions.
Pros
1 Mapping
ReView is really well suited to the Australian Tertiary Education
sector which has made significant investments in the identification of
Graduate Attributes (GAs). Institutions are now being (or will soon be)
required to map learning outcomes and student achievement against those
GAs. ReView’s key design feature is its ability to do this as a natural
part of the marking process. Academics identify assessment criteria and
link them directly to GAs so when they come to generate the feedback,
these automatically map onto them. This constitutes a huge time saving
in what could otherwise have become a very onerous process. In the HE
sector in the UK we are not yet required to do this – but I don’t think
it is far off. Designing this kind of mapping in where we can to
pre-empt such a requirement and finding the right tools to help us do it
may be worth considering.
2 Transparency
As tutors mark using ReView, they generate a grade based on student
achievement against defined Assessment Criteria. This is, I believe, a
great way of improving the transparency of marking for students (making
it clearer to them how their final grade was arrived at). While I
acknowledge the critical scholarship on the use of scored or calculated
rubrics and assessment criteria in this way (particularly Royce Sadler’s
work) I feel that the benefits it affords students outweigh the
potential or actual drawbacks in terms of integrity. The tutors using
this tool determine student attainment using ‘sliders’ which I do take
issue with (below) but which work in much the same way as the rubric
calculator in Grademark.
3 Analytics
Spitting out the back of ReView is a really interesting ‘dashboard’
which shows rich and valuable data on the student achievement harvested
from the marking. I didn’t get to see this in action because (as is
always the case with these tools) unless there is live student activity
in it, it’s difficult to ‘mock up’ demos of things like this. But I saw
enough to get the gist of it and it looks more advanced than simply the
raw data which is generated by Grademark.
4 Mobility
This tool is designed to work on mobile devices – particularly
tablets and especially iPads. This makes them portable. It means that
this is a tool which is very well suited to the marking of studio-based
work (such as design, textiles, fine art) and it’s also fantastic for
marking hand written exams because it doesn’t need assessment to be
submitted to it in order for it to be marked and returned.
5 Self-evaluation
Unlike Grademark, this tool includes a student self-evaluation tool.
Students can indicate what they think their work deserves. To achieve
the same thing in Grademark requires a workaround and lots of data
entry. The student self-evaluation is clear to the tutor as they mark
and this may influence their judgement in unhelpful ways. I feel that if
there is going to be a student self-evaluation function, it must be
‘blind’ to the tutors as they mark.
Cons
1 Integration
Currently ReView is not well integrated or integrable in that it is a
stand alone tool (it’s not yet a building block for any of the major
VLEs) and in that it is only a feedback tool not a marking tool. In
other words – students can’t submit their work to it and tutors can’t
comment directly on their work with it. The danger of this is that it
will generate false economy. So even if it saves tutors time in the
marking of student work, it may cost them or their institutions more
time in terms of mark entry, handling submissions, returning student
work etc. Tutors may find themselves moving between two or even three
different systems to received, read, annotate, plagiarism check, return
and enter the marks for a piece of student work. Additionally, the
transparency it achieves through the rubric ‘sliders’ may be
counteracted by the lack of clarity as to precisely where the strengths
and problems in the work are located if they can’t be marked on the work
itself. For instance – a comment saying that some sentences are poorly
constructed is useless to students unless they are clear which ones are
poor and which ones aren’t. The integration within VLEs will no doubt
come with time, but it’s looking unlikely that the marking tool is going
to emerge. As such, while it does some lovely analytics, its not the
‘granular’ level that GradeMark achieves.
2 Clarity
I have concerns about the ‘sliders’ themselves. If we are using
rubrics and assessment criteria to improve transparency, we need to take
great care not to then obfuscate what we are doing. The ‘sliders’ allow
tutors to decide whether a piece of work is in the high or low range
within a classification (i.e. that against a single criteria it can be a
‘high 2.1' or a ‘low 2.1'). This to me is one step forward and two
steps back. When you have five or more criteria (averaging 20% or less
per criteria) the difference between one classification and another is
going to be 2% or less of the total. To be making judgements
within
that classification (within 2%) is marking to a level of accuracy
which is simply not reliable or helpful to the students. It is for this
reason that I think the ‘radio button’ approach of the Grademark rubric
calculator is more transparent. In other words, it doesn’t leave them
wondering what makes their achievement a ‘high’ rather than ‘low’ 2.1
(for instance) against a particular criteria. It does allow tutors to
tweak a final grade away from a borderline (eg 69%) but my hunch is that
when rubric calculators are used, students are less inclined to
complain about a mark that ‘comes out in the wash’ to that number than
one which is arrived at holistically by the tutor. As a result – I don’t
think we should shy away from awarding borderline marks if that’s what
the rubric (which has been clearly communicated to the student before
hand) calculates. Anything else is duplicitous.
3 Cost
This tool looks like it’s going to be quite expensive in comparison
to its competitors. Given that this is likely to be a tool that would
need to be used in conjunction with other marking and submission tools and that we can
probably achieve many of the affordances it offers with some workarounds
within Grademark, it’s going to prove a hard sell to many cash-strapped
institutions at the moment.
Final Evaluation
This looks like a fine tool that will almost certainly be the right
tool for many marking jobs. I think it will be especially attractive to
colleagues marking physical objects (like artworks, models etc) and
performances (music, drama, presentations etc). I suspect many will find
it useful for marking exams, especially if feedback is required on
them. It won’t replace marking tools like Grademark and it may be hard
to justify the investment if we can find workarounds which achieve
similar things within the tools for which we already hold sight
licenses.
Response column: This is a new feature in the assignment inbox which
shows if and when students have collected their feedback (image right). Of
course it can’t tell us if they’ve engaged with their feedback,
understood it and/or addressed it, but at least we now know if it’s been
collected. I’ve been amazed at how quickly students have accessed their
mark and feedback. For instance, a bunch of assignments were released
at 1pm today and by 1.05pm, eight students had collected their feedback.
Given I’d only sent the email alerting them to the release time having
been moved forward 15 minutes earlier, I was pretty amazed by this! I
checked an hour or so later and the number was up to 27 students:
roughly a third of the total.
Rubric viewer: In
the move to Turnitin 2, one of the things that a lot of people missed
was the ability to see the rubric clearly. The new version of the rubric
required you to mouse over each segment in order to see the words. A
recent adjustment means that with the click of a single button (the
four-arrow button in the image to the left) the whole rubric can be
opened out into a full view and can be moved to a second screen. I tend
to use two monitors (I dock my laptop to a screen whenever I can) so
I’ve now got into the habit of opening the rubric straight away and
moving it onto my second screen. This means I can adjust it as I go
rather than having to remember everything at the end of the paper. It
also means that you can put a lot more detail into the rubric segments
and still see it all clearly which is a vast improvement on the old view
of the rubric.
