Manually Scored Exams
Overview
You can set an Exam to be a Manually Scored exam in the settings. The Registrar needs assign a person an 'Instructor' role, and then that Instructor needs to be assigned to the Module which contains the Manually Scored Exam.
When the Training Developer is editing the module and adding questions to that exam, they will have an option to add a 'Written Answer' type question. It is only a question, with no possible answers to be entered. You can also add any of the other types of questions to this same exam.
When Student submits exam for scoring, they are told that it is 'Waiting For Scoring' by the Instructor, and the Instructor is sent an email notifying they have an exam to score. Instructor logs in and sees a list of Student exams waiting for scoring. The Instructor opens the exam. The Multiple Choice and Hotspot questions will already be scored. The Manually scored questions will need to scored as correct or incorrect by the instructor.
There is also an optional feature which enables the Training Developer to set up key words and/or phrases to watch for in the written answer of a question, and mark the question as correct when a specified minimum of these are found in the answer. Even when this is automatically scored, the Instructor still needs to open the exam and verify that the answers are correct.
When the instructor is finished scoring the exam, the Student is notified by email and can then go back into the Module and either re-take exam (if failed) or continue in the module (if passed) and/or submit the Module.
Set As A Manually Scored Exam
In Training Developer, inside a Module in Edit mode, on an Exam page, click the Exam Settings icon and then turn ON the option for 'Manually Scored Exam'. Setting an exam to Manually scored exam, allows a Training Developer to add the Written Response (‘Written Answer’) type questions to an exam. The exam will permanently remain as a Manually Scored exam. Only an Instructor can mark (set the score) for a 'Manually Score' exam.
Setting the Instructor
The Registrar sets the Instructor(s) for the module in the 'Module Settings'. Click 'Content' tab on left, then 'Modules'. Find the Module that contains the Manually Scored exam and click on the Settings icon for that module. A list of users in your system who have been given an Instructor role appears. Select the Instructor(s) who you want to assign to this module (these people are the 'Subject Matter Experts' for this module.
In this case, Smith Thomson is the Author and Ethan Morgan is designated as “Instructor”. This allows Ethan to score the exam(s) for this specific module.
The multiple choice and hotspot image questions are marked automatically by the website engine, but the “Written Answer” questions require input from the designated “Instructor(s)”.
Scoring the Exam
When a student completes an exam containing “Written Answer” questions, the Instructor(s) receives an email stating that “Student has completed an exam that needs marking”.
After switching or sign in to Instructor role, the homepage will show that a student is awaiting an exam marking.
Click on that and you’ll see who/what exam needs marking. Click the blue radio button to enter the exam.
Once the exam is open, if the exam contains Multiple Choice or Hotspot image questions, the instructor will see that those are already marked. The green highlighting shows which is the correct answer, but the little grey button and the red [Incorrect] shows the answer that the student selected. No action is required on these pre-marked questions.
Besides the Written response questions (top right) there will be a “Check mark” and an “X” that you will use to ASSIGN a Pass or Fail mark to that question.
If you feel they successfully answered the question, select the “Check mark”.
If you feel that they did NOT successfully answered the question, select or leave the “X” selected.
Notes/Suggestions
If you feel the Student did not answer successfully, you can provide guidance by adding hints/suggestions/SOP references at the bottom.
Assigning Scores
The website engine will “suggest” a mark in the top right based on the correct multiple choice, hotspot image, and long answer questions that you have marked (either with a “check mark” or “X”). In this case 74%.
You select the check box for “Set score Manually” and enter either the “suggested” mark or whatever you feel is correct as the instructor.
If you enter a mark less than 100% and select “Assign Mark” a window will pop up, warning you that the student will FAIL. This is fine.
Select CONFIRM and the candidate will receive an email telling them that the exam has been marked, and they have failed.
The student then logs in, go to the exam and at the top see the “NOTE from Instructor” in which you provided hits/suggestions/SOP references etc. They select RETAKE EXAM and carry on until they pass!
Automatic Scoring Feature (optional)
You can specify certain key words or key phrases for the system to watch for in a Written Answer and then score it as correct or incorrect based on finding a threshold number of these, which you set. Even though the system shows a score for these questions, the Instructor still must view and confirm the exam since there is opportunity for questions to be incorrectly scored by the system.
When you Add a 'Written Answer' question to an exam, below the question is a toggle for "Enable Automatic Marking'. Turn this ON, and then a 'Minimum Target Phrases' menu appears. Select the number of Phrases that are required in the written answer to determine it as correct, then click the Add button to add phrases. You can also select which of those phrases MUST be in the written answer.
Example:
Question: "What should a driver do when approaching an un-controlled intersection?"
Minimum Target Phrases: 3
Phrases To Watch For: "slow, brake, yield, watch for pedestrians, cars on right, vehicles on right, stop for traffic, stop if car, stop if vehicle". Of these, the following had "Required" selected: slow, yield, stop.
Incorrect Answer: "Check if intersection is clear and yield if necessary" ("yield" being a detected phrase. Only one phrase was detected, minimum 3, so answer is incorrect.
Correct Answer: " Slow down and check for pedestrians and if there are cars on right side, yield to them or stop if necessary." (4 phrases were found so system AND all three of the 'required' phrases were found, so system scored this as Correct).
NOTE: As you can see, this is only an AID to scoring written answer questions, and the instructor is still responsible for verifying the answer and, if necessary, overriding the system's score for answer.
IMPORTANT NOTE: The student will need to log back in to "Submit Module" after the instructor is done marking it. Otherwise, the student will notice the module progress has 99% complete on their student desk. Here's what the "Submit Module" will look like on the students module page.
