Jennifer Lape, OTD Occupational Therapy


Project Overview

As part of year 1 of my Technology Fellowship (2015-2016), I wanted to focus on enhancing feedback to online doctoral students on their capstone projects, and improving the peer review process already in place within the occupational therapy doctorate (OTD) capstone courses.  As a result, I explored the use of Turnitin’s GradeMark and PeerMark in detail, and piloted use of these tools in several courses.  As part of this process, I also undertook the task of revising the analytic rubrics for each of the 6 capstone chapters.


Planning Process

In planning this project, I had to consider both the course learning objectives as well as my personal goals for the project.  In the OTD program, students take a series of evidence-based practice courses designed to guide them through the development, implementation, and evaluation of their doctoral capstone projects.  This process includes the writing of 6 capstone chapters with peer review integrated throughout the courses.  Goals of peer review include helping the students to increase the quality of their work and to emulate the peer review process inherent in pursing publication, since this is also an objective of the courses/program.

Previously, the peer review process involved instructor pairing of peers, exchange of papers among peers, and general provision of feedback to each other using the assignment rubric as a guide.  In the past, both instructor feedback and feedback from peers was delivered via the Track Changes feature in Microsoft Word.  This process entailed downloading the student’s file, pasting in the rubric, saving to your computer, adding comments, completing the rubric, resaving, and then uploading the feedback file to Moodle.  This process is cumbersome and time consuming, so my personal goal was to streamline the process and be able to provide each student with richer feedback in a timely manner.

Goals for the project included:

  1. Improve the quality of feedback/grading provided on student assignments, to increase quality of student work and student satisfaction and decrease instructor time commitment. (Technology used to augment, modify)
  2. Improve peer review process to improve quality of student writing/publication. (Technology used to modify)

Implementation

The first step in the project was to redesign the analytic rubrics for the capstone courses.  The prior rubrics were analytic in the sense that they listed the assignment criteria with each criteria having 4 possible scores, including outstanding, meets criteria, approaching criteria, and below expectations.  Since these courses are taught by several full time faculty, as well as adjunct faculty, it became apparent that the scoring needed to be more objective.  A variety of resources on Bloom’s taxonomy and rubrics were consulted in development of these rubrics.  Weighting was also used for assignment criteria to emphasize categories according to course objectives.

redesigned rubric

Click to see redesigned rubric.

Next, to improve the quality of feedback that students both give and receive in the peer review process, structured peer review questions were developed for each capstone chapter via modification of questions within the PeerMark library.  For example, these are the peer review questions for the chapter 1:

Scan this paper for errors in formatting of in-text citations, direct quotes, and the reference list. Give several examples of these errors, if they exist.
Question type: Free Response
Minimum answer length: 5
Does the writer use sufficient evidence/references to support the existence of and the need to address the identified problem? If yes, explain your rationale for this answer. If no, explain where support is lacking and how this section of the paper could be strengthened.
Question type: Free Response
Minimum answer length: 100
How effective was the writer’s use of language related to readability and clarity of the subject matter? Very effective would be similar to the language used in professional journals.
Question type: Scale
Highest: very effective, Lowest: very ineffective
Does the writer give a clear and concise description of the setting (omitting all extraneous details and leaving no unanswered questions)? Please provide the rationale for your answer as well as suggestions to improve this section if necessary.
Question type: Free Response
Minimum answer length: 100
Does the writer acknowledge all applicable supports and barriers in the setting? Provide suggestions of additional supports and barriers to be considered if applicable.
Question type: Free Response
Minimum answer length: 1

Next, the revised rubrics and peer review questions had to be entered into Turnitin within Moodle, and I had to test/pilot these features to be sure that I understood the functionality and settings available.  An additional benefit of using Turnitin, is the availability of the originality report, since these capstone assignments involve increased use of external resources, quoting, and citations.

I also had to consider that this would likely be NEW technology for most of the students, so tutorials on how to navigate the technology would be necessary.  As a result, 4 videos demonstrating how to upload a paper to Turnitin, how to retrieve instructor feedback, how to complete a peer review, and how to access peer review comments were created by Instructional Technology and posted within the courses.


Assessment

I assessed the project both formally, through a survey created within SurveyMonkey, and informally via dialogue with students during synchronous classes, an onsite visit, and phone conversations.  Some info about the project was also gleaned from Chatham course evaluations as several students commented on this process in those evaluations.  These formative assessment methods revealed the following:

  • Some students struggled with navigation of the technology, but not all students took advantage of the how-to videos posted within the course. An extra synchronous online class was held to answer students’ questions specifically about Turnitin & PeerMark.
  • 56% of students who responded to the survey said they preferred feedback via Turnitin (as opposed to the Track Changes files within Microsoft Word) or liked both methods equally.
  • Features that students liked best about Turnitin: the originality reports, audio feedback from the instructor, ease of use and retrieval of feedback, variety of options to mark papers with ease.
  • Students struggled with the use of PeerMark to complete the peer review process. Issues included: difficulty with technology, not viewing how-to videos, mismatched pairs for review resulting in some students getting multiple reviews of their papers and other students getting none.
  • Despite these glitches, the average of all student responses to the question “How valuable do you feel the peer review process is to the capstone process on a scale of 1 to 10? (1=not valuable at all; 10 = extremely valuable) was 7.5.
  • 88% of students reported utilizing outside sources to verify information when completing their reviews of peers’ papers and reviewing others work helped them to better understand course content and strengthen their own work.
  • As an instructor, I also felt the comments students made on their peer reviews were more appropriately directed toward the content and of higher quality than previous.

Reflections and Next Steps

I consider the use of Turnitin’s GradeMark a success.  Students had little issue with submission and retrieval of feedback via this system and I found it easier to give detailed feedback.  I particularly valued the ability to record an audio comment with each assignment and to save custom QuickMarks for use in future papers.

The use of PeerMark for the peer review was definitely a challenge on many levels.  Going through the process helped me to hone the questions that students answered about their peers’ papers, and to realize that the students do understand the purpose and value of the activity.  As a result of the issues encountered with this process, I’ve moved the peer review process to an online forum within Moodle, but continue to have students answer the more detailed questions.  I’d consider piloting the use of PeerMark again in another course, but would likely opt to hold a live synchronous class to review the process, in addition to posting how-to videos in the course.

My goals for year 2 include:

  1. Trying to use Turnitin on the iPad for grading on the go!
  2. Exploring a reference manager, such as Mendeley or Zotero.
  3. Exploring software for qualitative data analysis that could be accessed remotely for online students.

Katie Cruger, Ph.D. Communications

Using Turnitin’s “Grademark” features to Increase Efficiency and Efficacy of Written Comments

Dr. Katie Cruger
Assistant Professor, Communication
Director, Professional Writing Program

Overview

While I used this first year as technology fellow to tweak my courses (online, on ground, and hybrid) in many different ways, I’ve chosen to use this space to share my success incorporating Turnitin for efficient and effective paperless grading at the undergraduate level.

I wanted to achieve the following outcomes:

  • Decrease the amount of time I spent downloading papers, making comments, uploading feedback to moodle, etc., which was much greater than hard copy grading practices
  • Increase usefulness of comments to students
  • Increase the likelihood that students would actually read/incorporate electronic feedback in future work or revisions
image 1

Image 1: The grademark screen in Turnitin.

Implementation

Beginning in Summer of 2013, once Moodle 2.0 was available, I used Turnitin Assignments for all student submissions of written work (both group assignments and individual papers and proposals). I utilized the following key features of Turnitin:

1)    Originality Report: This is the part of Turnitin that faculty members are most familiar with, and we often use it as a way of catching/proving plagiarism. However, in lower level courses, this is also a great tool for reinforcing information literacy. It lets students see, in a relatively low-stakes environment, when they are relying too heavily on any particular source, when they haven’t done enough to move from a quotation to a paraphrase, etc.

2)    Quickmarks: This feature allows an instructor to drag and drop a standardized comment anywhere in the document. You can edit or make additions to any of these comments, as well as create your own quickmarks for feedback you provide often to students. For sentence level errors and suggestions, it both decreases the time spent giving comments and increases the amount of information students receive about a punctuation rule or the difference between there/they’re/their.

Image 2: A customized quickmark with additional comments for the student.

Image 2: A customized quickmark with additional comments for the student.

Image 3: The quickmark as it appears in the paper. Students hover over the mark or click to read full comments.

Image 3: The quickmark as it appears in the paper. Students hover over the mark or click to read full comments.

3)    Grademark General Comments: Turnitin allows instructors to leave overall feedback on the paper either in text format or as a voice comment.

Image 4: General Comment view, including text feedback, voice comment, numerical grade, and percentage similarity with other sources.

Image 4: General Comment view, including text feedback, voice comment, numerical grade, and percentage similarity with other sources.

Assessment

When I compared the amount of time I spent grading a Turnitin Assignment (vs. a regular moodle assignments the previous semester, or using iAnnotate and a stylus to mark PDF copies of papers two semesters ago) I spent less than a third of the time (7.5 minutes vs. 25 minutes) per 5-page paper.

Furthermore, I was much happier with the quality and quantity of feedback I was able to provide students. Quickmark comments are thorough, and offer students enough information (in clear text they can read easily) that they are able to incorporate changes in future writing. Their customizability allowed me to tailor a few comments to the style of writing we were working on for a particular assignment without recreating the feedback each time. Voice comments (which limit me to 3 minutes) allow me to quickly talk through the good and the bad of student work and convey enthusiasm and emotion through my voice. This felt particularly important during online courses, where I didn’t have the same face-to-face connection to students.

However, Turnitin does require a little bit more effort from students to retrieve my comments than some other forms of electronic grading. They must first log into moodle, then click on the actual assignment, at which point they are directed to their paper on Turnitin’s server. The process takes about 1 minute, but we know this might be too much effort for some. Once there, students need to toggle between different views in order to see general comments, specific Quickmarks, and the originality report. Although we think of our students as technologically savvy, this has not been my experience when it comes to instructional technologies. Anecdotally, I know a few students struggled to gain access to my comments and asked for help. I can only assume that others struggled and chose not to pursue the matter further. However, I was able to address some of these challenges in productive ways.

Overcoming Challenges

1)    Student User Error: Many students said they “couldn’t see” my comments in Turnitin. This was usually because of one of two problems: either they hadn’t actually clicked on the link and entered turnitin (meaning that they expected all the info to be present in moodle) or they were using a browser that was not compatible. Both were relatively easy fixes. I provided students with instructions for how to use the Grademark feature in Turnitin (see “Resources” below) and stated explicitly in assignment descriptions and in the syllabus that Firefox is the optimal browser for use with moodle and Turnitin. In the future, I may devote classtime to showing students a sample and getting them familiar with the platform.

2)    Instructor User Error: Moodle isn’t always the most user-friendly interface. One quirk with the Turnitin Assignment plugin is that, unprompted, it creates randomized deadlines for all assignments. These parameters are not something that an instructor can amend or disable while they are creating a new assignment. Instead, they must go in and amend the assignment after the fact, so there are two steps to this process. Until I recognized this, students were erroneously notified that their submissions were late, which created some panic.

Next Steps

I will continue to use Turnitin for all my grading in all courses in the future; it’s the best option for my needs and the needs of my students. Additionally, I’d like to incorporate the Peermark feature in my writing-intensive courses, where students must often review one another’s work. Currently, I use Moodle forums or hard copies of papers for these exercises.

I’m also continuing my work exploring VoiceThread to increase student engagement and interaction around course readings in hybrid and online courses.

Resources

Student training and GradeMark overview for students