Reviewing ReView

I had a chance to have a look at the workings of ReView while I was at UNSW. It is an impressive feedback tool with some affordances that aren’t yet available in competing tools but also some limitations which may be deal breakers for most institutions.
Pros
1 Mapping
ReView is really well suited to the Australian Tertiary Education sector which has made significant investments in the identification of Graduate Attributes (GAs). Institutions are now being (or will soon be) required to map learning outcomes and student achievement against those GAs. ReView’s key design feature is its ability to do this as a natural part of the marking process. Academics identify assessment criteria and link them directly to GAs so when they come to generate the feedback, these automatically map onto them. This constitutes a huge time saving in what could otherwise have become a very onerous process. In the HE sector in the UK we are not yet required to do this – but I don’t think it is far off. Designing this kind of mapping in where we can to pre-empt such a requirement and finding the right tools to help us do it may be worth considering.
2 Transparency
As tutors mark using ReView, they generate a grade based on student achievement against defined Assessment Criteria. This is, I believe, a great way of improving the transparency of marking for students (making it clearer to them how their final grade was arrived at). While I acknowledge the critical scholarship on the use of scored or calculated rubrics and assessment criteria in this way (particularly Royce Sadler’s work) I feel that the benefits it affords students outweigh the potential or actual drawbacks in terms of integrity. The tutors using this tool determine student attainment using ‘sliders’ which I do take issue with (below) but which work in much the same way as the rubric calculator in Grademark.
3 Analytics
Spitting out the back of ReView is a really interesting ‘dashboard’ which shows rich and valuable data on the student achievement harvested from the marking. I didn’t get to see this in action because (as is always the case with these tools) unless there is live student activity in it, it’s difficult to ‘mock up’ demos of things like this. But I saw enough to get the gist of it and it looks more advanced than simply the raw data which is generated by Grademark.
4 Mobility
This tool is designed to work on mobile devices – particularly tablets and especially iPads. This makes them portable. It means that this is a tool which is very well suited to the marking of studio-based work (such as design, textiles, fine art) and it’s also fantastic for marking hand written exams because it doesn’t need assessment to be submitted to it in order for it to be marked and returned.
5 Self-evaluation
Unlike Grademark, this tool includes a student self-evaluation tool. Students can indicate what they think their work deserves. To achieve the same thing in Grademark requires a workaround and lots of data entry. The student self-evaluation is clear to the tutor as they mark and this may influence their judgement in unhelpful ways. I feel that if there is going to be a student self-evaluation function, it must be ‘blind’ to the tutors as they mark.
Cons
1 Integration
Currently ReView is not well integrated or integrable in that it is a stand alone tool (it’s not yet a building block for any of the major VLEs) and in that it is only a feedback tool not a marking tool. In other words – students can’t submit their work to it and tutors can’t comment directly on their work with it. The danger of this is that it will generate false economy. So even if it saves tutors time in the marking of student work, it may cost them or their institutions more time in terms of mark entry, handling  submissions, returning student work etc. Tutors may find themselves moving between two or even three different systems to received, read, annotate, plagiarism check, return and enter the marks for a piece of student  work. Additionally, the transparency it achieves through the rubric ‘sliders’ may be counteracted by the lack of clarity as to precisely where the strengths and problems in the work are located if they can’t be marked on the work itself. For instance – a comment saying that some sentences are poorly constructed is useless to students unless they are clear which ones are poor and which ones aren’t. The integration within VLEs will no doubt come with time, but it’s looking unlikely that the marking tool is going to emerge. As such, while it does some lovely analytics, its not the ‘granular’ level that GradeMark achieves.
2 Clarity
I have concerns about the ‘sliders’ themselves. If we are using rubrics and assessment criteria to improve transparency, we need to take great care not to then obfuscate what we are doing. The ‘sliders’ allow tutors to decide whether a piece of work is in the high or low range within a classification (i.e. that against a single criteria it can be a ‘high 2.1' or a ‘low 2.1'). This to me is one step forward and two steps back. When you have five or more criteria (averaging  20% or less per criteria) the difference between one classification and another is going to be  2% or less of the total. To be making judgements within that classification (within 2%)  is marking to a level of accuracy which is simply not reliable or helpful to the students. It is for this reason that I think the ‘radio button’ approach of the Grademark rubric calculator is more transparent. In other words, it doesn’t leave them wondering what makes their achievement a ‘high’ rather than ‘low’ 2.1 (for instance) against a particular criteria. It does allow tutors to tweak a final grade away from a borderline (eg 69%) but my hunch is that when rubric calculators are used, students are less inclined to complain about a mark that ‘comes out in the wash’ to that number than one which is arrived at holistically by the tutor. As a result – I don’t think we should shy away from awarding borderline marks if that’s what the rubric (which has been clearly communicated to the student before hand) calculates. Anything else is duplicitous.
3 Cost
This tool looks like it’s going to be quite expensive in comparison to its competitors. Given that this is likely to be a tool that would need to be used in conjunction with other marking and submission tools and that we can probably achieve many of the affordances it offers with some workarounds within Grademark, it’s going to prove a hard sell to many cash-strapped institutions at the moment.
Final Evaluation
This looks like a fine tool that will almost certainly be the right tool for many marking jobs. I think it will be especially attractive to colleagues marking physical objects (like artworks, models etc) and performances (music, drama, presentations etc). I suspect many will find it useful for marking exams, especially if feedback is required on them. It won’t replace marking tools like Grademark and it may be hard to justify the investment if we can find workarounds which achieve similar things within the tools for which we already hold sight licenses.

grademark 1 from PlagiarismAdvice on Vimeo.
This is a screencast I've produced on my experiences of using Grademark focussing on the benefits it has brought to me, my students and my institution. 

Grademark Shortcuts

I’m in the middle of ploughing through around one hundred 2500 word essays using Grademark with a scored rubric to mark them. Along the way I’ve been searching out shortcuts to make things even quicker and I’ve found quite a few so I thought I’d share a few of them here.

1 Highlighting matched text in Grademark

This is the biggest and most important improvement in the new version of Turnitin and in itself is saving me around 3-4 minutes per essay. When you go into Grademark you can now select a button which highlights matched text in pale pink. As you read and mark the essay you can instantly dismiss passages of matched text which are properly referenced as you go. Given the vast majority of student essays contain no plagiarism at all, this means you only have to go back into the originality report for the essays which look dodgy.



In this image I’ve selected a section of a student's essay which includes the essay question which, of course, always comes up as matched and which I can, consequently, ignore each time. Unfortunately, when you open an essay in document viewer to mark it, it doesn’t default to this setting, so you have to get into the habit of turning it on with each essay you open. To do this, simply click in the darker grey button to the left of the Originality button on the top left hand corner of the screen.




2 – Quickmark Search Function

This has been a real breakthrough for me. The Quickmarks (QM) are great little timesavers in their own right, allowing me to basically ‘stamp’ aspects of the essay with rich comments for common problems. But sometimes it takes longer to find the right one than it would have taken to type it out in full. There is now a search function at the top of the Quickmark window which allows you to find the QM you’re looking for quickly and easily.

It searches both the name and the content of QMs, so searching for 'para' will bring up all the QMs with 'paragraph' in them. This has made my use of QMs feel faster and more fluid and I’m not using the pull down menus for QMs sets at all.

3 – Keystrokes for Saving Comments

When you add a QM to the essay it automatically saves it, but when you add a comment bubble you need to save it yourself. You can do this by using the mouse to click on the blue ‘save’ button but a quicker and more convenient way is to use the Tab key to select the ‘save’ button and then hitting return. By the time I’ve moved my hand back to the mouse, the Comment is saved and I’m ready to keep reading. If you hit the Tab key twice it selects the Cancel button and hitting it three times selects the ‘More Options’ link. This works also in the ‘General Comments’ window.

4 – Adding comments to Highlighted Passages

There are two ways you can connect a bubble comment to a specific section of text. You can highlight the text and then click on the ‘comment’ button at the top of the Quickmark window. Alternatively, you can click directly on the highlighted passage and it will open a Comment bubble for you. I find this second way quicker and more convenient than clicking the ‘comment’ button and appears in much the same way in the print-out version.


Find Out More


If you want to find out more about Grademark or Turnitin, there will be a free event prior to the JISC conference in Liverpool this year. To sign up, go to:


http://www.plagiarismadvice.org/component/content/article/38-frontpage/252-maximising-turnitin-grademark-peermark-and-new-features-

Grademark and eMarking

Glamorgan demo on Prezi

Tweckling, iconoclasm and lecturing as a normative discourse: reflections on two ALT-C keynotes

When I discovered what the topic of Donald Clark’s ALT-C keynote was I was pretty interested. ‘Don’t Lecture Me’ augured well, I thought, because challenging the orthodoxy of lecturing is a matter dear to my heart. But I honestly thought that this was going to be a classic example of preaching to the proverbial choir. Surely, I thought, settling into their seats in the auditorium were people, like me, who had read Bligh’s What’s the Use of Lectures? and Gibb’s ‘Twenty Terrible Reasons For Lecturing’ and were all going to enjoy the ride Clark was about to take us on. I couldn’t have been more wrong.

Before long, the twitter back channel started to fill with defensive and at times aggressive reactions to the points he was making. And these weren’t particularly radical points. These were points that have been made countless times before by countless other scholars in countless other settings over countless years. And I don’t say this out of disrespect for Clark’s scholarship. Clearly, it all still needs to be said. Again!

There were tweets attacking Clark for using a lecture to denounce lecturing, attacking him for critiquing the lecturers rather than the lecture, picking on physicists and his swearing and there were many more being just plain rude. I won’t counter these here as he’s done a perfectly good job of doing that himself in his blog and, as an old friend of mine would say, ‘he’s big enough to look after himself’.

What really struck me was that even here, at a teaching and learning technology conference, how many apparently well-informed, innovative thinkers about teaching and learning practice were defending the lecture as a pedagogic technique. Astonishingly, when I sent a tweet about this shock I was feeling I received two @ replies that, you guessed it, defended the lecture.

In his keynote Clark discussed lecturing as operating as a default. I would challenge him on this and argue that it is probably better understood as a normative discourse. A default is something to which things automatically reset or return. While I think this is true – I think there is more going on than this. I think lecturing operates, within HE, as a normative discourse. A normative discourse is a way of thinking and talking about something whereby it is understood to be true that something is normal. Heterosexuality is usually understood to operate as a normative discourse for instance – and this assumption is often referred to as heteronormativity. Similarly – in HE it is widely considered to be true that lecturing is normal. This is evident in our job titles, our institutional architecture, our workload models, our quality assurance strategies, our timetabling software and countless other systems and principles that define and demarcate our working lives.

One of the things about normative discourses is that they are really hard to talk against or to critique. Because something is seen as normal, saying anything which is critical of it can be taken personally by those who subscribe to this ‘truth’. That was what happened at the ALT-C keynote. While Clark was clearly critiquing the lecture as a pedagogical strategy, many people in the room who were engaging with the twitter back channel (and therefore probably many others who weren’t) clearly took his critique personally – as being an attack on them as lecturers and as people who had been lectured themselves as undergraduates.

Speaking out against the lecture is hard. Bligh, for instance, spends the first chapter of his book arguing why lecturing is not a great pedagogical strategy and then spends the rest of the book suggesting ways we could do it better. Gibbs’ ‘Twenty Terrible Reasons For Lecturing’ is written explicitly in response to the defences lecturers routinely offer of the lecture (and by extension of themselves). I hear these defences all too frequently myself when I muster the courage to challenge the normative discourse and Gibbs’ article comes in really handy at these moments.

In effect, speaking out against lecturing is hard because it amounts to a kind of iconclasm – literally icon smashing. As Clark’s subsequent blog post made clear, he is big enough to look after himself. I still can’t help thinking he must have felt at least a little bruised by the experience. Which leaves me wondering about those without his confidence and self-employed status. And this is precisely how normative discourses maintain and protect themselves – it becomes too risky and too scary to speak out against them so people choose not to and the ‘truth’ of the normalcy gets to sail on calmly.

What really drilled the point home to me were the tweets which saw the second keynote delivered by Dr Sugata Mitra as incontrovertible evidence that the lecture isn’t dead. While I agree with them all that it was a fantastic example of a brilliant, inspiring, spell-binding lecture, so was Clark’s. (A key point that Gibbs makes very effectively is that inspiring lectures aren’t in and of themselves evidence of the effectiveness of lecturing as a pedagogical strategy). What these tweeters failed to see, however, was that Mitra, like Clark was arguing persuasively and passionately about the ineffectiveness of teacher-centred learning strategies – at the core of which is lecturing. At one point he said something to the effect that students learn better when teachers aren’t even in the room! In essence, Mitra’s critique of lecturing was as powerful and passionate (albeit with fewer swear words) thank Clark’s but he escaped without the at times brutal tweckling simply because he wasn’t being so directly and overtly iconoclastic.

To view to the keynotes at the ALT-C YouTube channel, go here:

Donald Clark's 'Don't Lecture Me'

Sugata Mitra's 'The Hole in the Wall: Self Organising Systems'

To view them with the twitter backchannel as subtitles, (with thanks to Martin Hawksey for pointing these out) go here:

Donald Clark's 'Don't Lecture Me'

Sugata Mitra's 'The Hole in the Wall: Self-Organising Systems'

Turnitin2 - some first impressions.

As I’ve made clear in a previous blog post, I’m a fan of Turnitin and particularly the marking tool: GradeMark. After some shifting of dates, the long awaited upgrade, Turnitin 2, was launched on September 4^th. There’s never a good time for upgrades, but this one has come very close to the start of the academic year in Higher Education in the UK but at a time when not many people are doing any marking. Given I had three MA dissertations to mark, I thought it would be an ideal opportunity to give it a proper test run. So here are my first impressions of Turnitin2 – focusing particularly on the new version of GradeMark. I tried it in three browsers on two platforms:

• Firefox 3.6.9 on Mac OSX
• Safari 5.0.1 on Mac OSX
• Internet Explorer 8 on PC

First impressions

· The assignment inbox looks a lot cleaner and less cluttered: losing the column of red apples is a significant breakthrough.

· Grademark opens with a ‘getting started’ screen which offers some key tips for first-time users plus a link to a helpful 4-minute walkthrough which offers enough information to get started although it has the odd annoying typo in it.

· The quickmark comments have moved from the left hand side of the screen to the right which, as a right handed person, feels more intuitive to me, but cannot be moved so may make left handers grumpy.

· The use of screen ‘real estate’ is much better with a trimmed down header taking up much less room than previously and other useful stuff (such as the word count) tucked away in an information menu at the bottom left hand corner.

· The option to view originality within GradeMark is a real breakthrough. This alone, I estimate, will save me heaps of time as I can see quite easily where unoriginal text falls with respect to citations. This will save me having to go into each essay twice. It's done in a way which is unobtrustive so that it doesn’t get in the way of my reading and marking, but in any case is easy to turn on and off.

New things:

· There is a new ‘comments list’ view which lists comments made on the paper in a column on the right hand side of the screen. This should make it easier to navigate through the essay comment by comment – although the ‘show on paper’ link works haphazardly in all the browsers and platforms I tried.

· The comments look cleaner and less obtrusive as they sit in the paper as the icons are in blue and sit transparently behind the text. It’s still possible to highlight text and attach comments to it.

· It’s now considerably easier to add comments directly into my set of QuickMarks than previously. This will make a huge difference to academic staff coming to use it for the first time who often find that the process of setting up personal QuickMarks makes eMarking feel more rather than less time consuming. The time efficiencies don’t really kick in until your personal QuickMarks are built so making this process as quick and easy as possible is a big breakthrough.

Things I’ve not explored in depth yet:

· I’ve not yet used the rubrics, but I’ve heard on the grapevine that this has improved considerably. I’ll blog separately on that when I’ve had a chance to check it out.

· Also – I’ve not even started looking at PeerMark. Even though I’ve used it a few times now (in the old version) I was never confident I fully understood how it worked. I hope that the new version is more obvious and user-friendly. Again – I’ll blog about it when I get a change to explore it.

· On the Originality Report side of things, I’ve not done a huge amount with it yet, but my first impressions are that it now feels like the three different ways of viewing the reports have merged into one which most people should find easier to manage. It’s certainly a cleaner interface and it’s much easier to drill down into the matches than it was previously.

Things that have disappeared:

· The inbox still defaults to a hierarchy of originality reports – with the highest percentage at the top – but it’s no longer as clear that it can be ordered by author, title and date columns as there is nothing to distinguish the titles of these columns from those which can’t be ordered (grademark, file and paper ID).

· It’s now not possible to type words directly onto the paper (in the margins for instance). I didn’t use this much so won’t miss it a great deal, but I did find it handy to type the correct spelling on the top of misspelled words.

· It’s no longer possible to change the icons of comments (from the speech balloon to, for instance, a stop sign, a question mark or a tick). I used these quite a lot and it was good to be able to put ticks through an essay.

Having said all this – my overall impression on this first use of Grademark has been frustrating and disappointing. To put it simply – so much of it simply doesn’t work and it’s very, very buggy. There are some things which aren’t working on any browser:

· It’s not yet possible to print or download from Document Viewer (the new version). To do this you still have to go back to the previous version. The icon to do this is greyed out but recently an error message has been added which advises users to go to the old version.

· There is a tantalizing ‘time spent’ area on the information menu which, if it worked, could provide really useful information about how much time this tool is saving us – but it simply shows 00:00:00 and it’s not clear if I’m required to turn this feature on somewhere.

But there are things that aren’t working on some browsers.

· The ‘show on paper’ link in the Comments List, as I said, works haphazardly in some browsers or not at all in others. It’s potentially very useful – but only if it works.

· It’s not possible to highlight text to add a comment to it in Safari and difficult to highlight text in other browsers.

In Firefox in Mac, I repeatedly got the error message illustrated above when adding comments which required me to shut the essay down, open it up again and retype my comment. This happened around 15-20 times while marking a 10,000 word dissertation.

The overwhelming impression is that it’s very buggy and unstable which leads me to think that the launch has, as many suspected, been rushed and, as an end user, I’m doing beta testing for them. I’m now quite hesitant about running training sessions for staff on Turnitin2 until I have a clearer sense of if and when these bugs will be fixed.

Beyond plagiarism checking: exploiting the power of Turnitin

I've been spending a lot of time recently talking about Turnitin and it's struck me how so many people think of it as only a plagiarism checking tool. In the first place I find this ironic because it doesn't actually check plagiarism - academics do. Turnitin merely provides another tool, through its originality matching, that academics can use to identify instances of plagiarism. But secondly, the reason I find it striking is because the plagiarism, or originality, checking is such a minor part of my use of it these days.

There's still not a great deal of awareness that Turnitin has two other great tools - a marking tool called Grademark and a peer marking tool called Peermark. I've found Grademark particularly beneficial in that it's made my marking:

1 Quicker: I type faster than I write and the capacity it gives me to automate common comments means that I'm able to offer students much more detailed comments in considerably less time.
2 Better: Because I'm spending less time writing common comments (which are often on the 'surface features' of writing: misplaced apostrophes and the like) I find that the comments I do write for each student are more likely to be on the content (what they're writing about) than on the structure (how they're writing it). This is also more rewarding and feels less laborious making marking feel much less of a chore. I'm also able to offer students better feedforward by pasting in links through to useful online resources that they can use to develop their skills and understanding.
3 Easier: I'm not having to lug heavy piles of essays to and from work, I can erase mistakes I've made in my comments and I find it much easier to type than write with a pen on paper.
4 Retrievable: My dog can't eat their homework! And while I've never actually lost any student work, I have come awfully, awfully close (and spent a lot of time hunting). I also like having a fully annotated copy of students' previous work available for when they come to see me in a tutorial or consultation. It also makes simultaneous marking and moderation possible, and it's a whole heap easier to send work to the external examiner and to archive it at the end of the academic year.

I've found that students also like their work being marked and returned this way. Key reasons they cite are:
1 Privacy: they hate the scrum at the end of class when essays are returned and everyone's comparing their mark - especially if they've not done so well. Being able to log on in the privacy of their own home and take in their mark and concentrate on reading through their feedback without the pressure of friends and student colleagues around them is important to them.
2 Accessibility: Quite simply, they don't have to decipher my handwriting. They also like the transparency of the rubric scorecard - but more on that below.
3 Convenience: they like being able to submit their work online and get it back online. After all, they do everything else online these days, as do we (can you imagine not being able to submit a paper to a journal electronically these days?)

I've also come to appreciate the analytical potential that Grademark offers. I find the rubric scorecard to be a particularly powerful tool to use for criteria-based marking. I design the rubric and publish it to students at the start of term and then use it to arrive at their mark using the scorecard that's built into Grademark. For me it is vital in achieving the transparency in marking to which I believe students are entitled. In other words, the rubric allows me to make it clear to students exactly how I've arrived at their mark. It also identifies where they should concentrate their efforts to improve.

Grademark allows me to export a report for each assessment task that I can then use for diagnostic purposes. This chart is derived from data exported out of Grademark and shows a particular student cohort's achievement against six criteria. It's clear from this in which criteria most students are struggling (the green 2.2 'bulge' on the third criteria from the top for instance). This allows me to target skills development work with that cohort to the area of most need as well as undertaking dedicated work with the previous year group to develop their skills in that particular area. Evidence like this can be compared across and between cohorts and offers a potentially powerful evaluative tool.

I know that many academics don't like the idea of marking online because of the difficulty of reading from a screen. I used to feel that way, but am so accustomed to reading from a screen these days that I actually prefer it. I have no difficulty reading email after all. I think there is further potential that I'm yet to uncover and eagerly anticipate the arrival of Turnitin2 this September which promises a friendlier user interface and more seamless integration between the three main Turnitin tools.