Project Overview
As part of year 1 of my Technology Fellowship (2015-2016), I wanted to focus on enhancing feedback to online doctoral students on their capstone projects, and improving the peer review process already in place within the occupational therapy doctorate (OTD) capstone courses. As a result, I explored the use of Turnitin’s GradeMark and PeerMark in detail, and piloted use of these tools in several courses. As part of this process, I also undertook the task of revising the analytic rubrics for each of the 6 capstone chapters.
Planning Process
In planning this project, I had to consider both the course learning objectives as well as my personal goals for the project. In the OTD program, students take a series of evidence-based practice courses designed to guide them through the development, implementation, and evaluation of their doctoral capstone projects. This process includes the writing of 6 capstone chapters with peer review integrated throughout the courses. Goals of peer review include helping the students to increase the quality of their work and to emulate the peer review process inherent in pursing publication, since this is also an objective of the courses/program.
Previously, the peer review process involved instructor pairing of peers, exchange of papers among peers, and general provision of feedback to each other using the assignment rubric as a guide. In the past, both instructor feedback and feedback from peers was delivered via the Track Changes feature in Microsoft Word. This process entailed downloading the student’s file, pasting in the rubric, saving to your computer, adding comments, completing the rubric, resaving, and then uploading the feedback file to Moodle. This process is cumbersome and time consuming, so my personal goal was to streamline the process and be able to provide each student with richer feedback in a timely manner.
Goals for the project included:
- Improve the quality of feedback/grading provided on student assignments, to increase quality of student work and student satisfaction and decrease instructor time commitment. (Technology used to augment, modify)
- Improve peer review process to improve quality of student writing/publication. (Technology used to modify)
Implementation
The first step in the project was to redesign the analytic rubrics for the capstone courses. The prior rubrics were analytic in the sense that they listed the assignment criteria with each criteria having 4 possible scores, including outstanding, meets criteria, approaching criteria, and below expectations. Since these courses are taught by several full time faculty, as well as adjunct faculty, it became apparent that the scoring needed to be more objective. A variety of resources on Bloom’s taxonomy and rubrics were consulted in development of these rubrics. Weighting was also used for assignment criteria to emphasize categories according to course objectives.
Click to see redesigned rubric.
Next, to improve the quality of feedback that students both give and receive in the peer review process, structured peer review questions were developed for each capstone chapter via modification of questions within the PeerMark library. For example, these are the peer review questions for the chapter 1:
| Scan this paper for errors in formatting of in-text citations, direct quotes, and the reference list. Give several examples of these errors, if they exist. Question type: Free Response Minimum answer length: 5 |
|
| Does the writer use sufficient evidence/references to support the existence of and the need to address the identified problem? If yes, explain your rationale for this answer. If no, explain where support is lacking and how this section of the paper could be strengthened. Question type: Free Response Minimum answer length: 100 |
|
| How effective was the writer’s use of language related to readability and clarity of the subject matter? Very effective would be similar to the language used in professional journals. Question type: Scale Highest: very effective, Lowest: very ineffective |
|
| Does the writer give a clear and concise description of the setting (omitting all extraneous details and leaving no unanswered questions)? Please provide the rationale for your answer as well as suggestions to improve this section if necessary. Question type: Free Response Minimum answer length: 100 |
|
| Does the writer acknowledge all applicable supports and barriers in the setting? Provide suggestions of additional supports and barriers to be considered if applicable. Question type: Free Response Minimum answer length: 1 |
Next, the revised rubrics and peer review questions had to be entered into Turnitin within Moodle, and I had to test/pilot these features to be sure that I understood the functionality and settings available. An additional benefit of using Turnitin, is the availability of the originality report, since these capstone assignments involve increased use of external resources, quoting, and citations.
I also had to consider that this would likely be NEW technology for most of the students, so tutorials on how to navigate the technology would be necessary. As a result, 4 videos demonstrating how to upload a paper to Turnitin, how to retrieve instructor feedback, how to complete a peer review, and how to access peer review comments were created by Instructional Technology and posted within the courses.
Assessment
I assessed the project both formally, through a survey created within SurveyMonkey, and informally via dialogue with students during synchronous classes, an onsite visit, and phone conversations. Some info about the project was also gleaned from Chatham course evaluations as several students commented on this process in those evaluations. These formative assessment methods revealed the following:
- Some students struggled with navigation of the technology, but not all students took advantage of the how-to videos posted within the course. An extra synchronous online class was held to answer students’ questions specifically about Turnitin & PeerMark.
- 56% of students who responded to the survey said they preferred feedback via Turnitin (as opposed to the Track Changes files within Microsoft Word) or liked both methods equally.
- Features that students liked best about Turnitin: the originality reports, audio feedback from the instructor, ease of use and retrieval of feedback, variety of options to mark papers with ease.
- Students struggled with the use of PeerMark to complete the peer review process. Issues included: difficulty with technology, not viewing how-to videos, mismatched pairs for review resulting in some students getting multiple reviews of their papers and other students getting none.
- Despite these glitches, the average of all student responses to the question “How valuable do you feel the peer review process is to the capstone process on a scale of 1 to 10? (1=not valuable at all; 10 = extremely valuable) was 7.5.
- 88% of students reported utilizing outside sources to verify information when completing their reviews of peers’ papers and reviewing others work helped them to better understand course content and strengthen their own work.
- As an instructor, I also felt the comments students made on their peer reviews were more appropriately directed toward the content and of higher quality than previous.
Reflections and Next Steps
I consider the use of Turnitin’s GradeMark a success. Students had little issue with submission and retrieval of feedback via this system and I found it easier to give detailed feedback. I particularly valued the ability to record an audio comment with each assignment and to save custom QuickMarks for use in future papers.
The use of PeerMark for the peer review was definitely a challenge on many levels. Going through the process helped me to hone the questions that students answered about their peers’ papers, and to realize that the students do understand the purpose and value of the activity. As a result of the issues encountered with this process, I’ve moved the peer review process to an online forum within Moodle, but continue to have students answer the more detailed questions. I’d consider piloting the use of PeerMark again in another course, but would likely opt to hold a live synchronous class to review the process, in addition to posting how-to videos in the course.
My goals for year 2 include:
- Trying to use Turnitin on the iPad for grading on the go!
- Exploring a reference manager, such as Mendeley or Zotero.
- Exploring software for qualitative data analysis that could be accessed remotely for online students.
