Configuring Items for Manual Scoring in TAO Portal

By default, a test-taker receives one point per completely correct interaction. However, there are instances where a test author may need to manually configure individual items for alternative scoring methods (i.e manual scoring to be completed by a scorer, or awarding partial credit).

In TAO Portal, scorers will be able to view which assessments need scoring from a session’s monitoring page. As soon as a test taker has submitted a test and the automatic scoring of any relevant items has been carried out, the Score button for an individual test taker will trigger the process. For more information on manual scoring, see documentation on TAO Grader.

To ensure that test-takers’ responses are processed correctly, the relevant test items need to be configured for manual scoring when the item is authored. Follow the steps below to do this.

1. 1. Label the item as one which requires manual scoring.
      Click on Outcome Declarations in the Response Properties panel to the right of the item. The outcome(s) for your item will appear, as in the image below. The outcomes are defined by the user. By default, SCORE and MAXSCORE are automatically added, as they are part of the QTI standard.Clicking on the edit icon to the right of an outcome will open the values associated with it. In the example below, the first custom outcome (created by the user) listed for this item of the test (i.e. the first aspect of the response which needs to be scored) is WORDCHOICE. The first value is Interpretation – the name of the outcome. This is the name which will be displayed to the person carrying out the manual scoring. In the example below, this is Word Choice.

      

      In the  drop-down menu under External Scored, select Human.
      This value indicates the type of scoring which is needed for this particular outcome (in this case, scoring by a human).
      

   2. Add a description of the scoring criteria.

      The manual scorer needs to know what scoring system to use for the item (or outcome) in question. This is shown in the form of a rubric which contains the scoring criteria you wish to be employed, and is accessed via a link to a hosted PDF file.

      In the box labeled Long Interpretation, add a link to the hosted PDF file that contains the scoring rubric. This will be accessed later by the scorer.

   3. Define the score range.

      Enter the range (e.g. 1-4) in the box labeled Value.

      It is possible to define different ranges, as long as the following rules are respected: both the minimum and maximum should be defined, using only integers. The maximum should not be negative and should be greater than the minimum.

      Note:In order for an item’s responses to be flagged for manual scoring, there must be a valid score range under Value and human must be selected in the externalScored box.

      Note:If an item has more than one outcome (i.e. if different aspects of a response are awarded separate scores, meaning that multiple scores are needed for one item), the scoring method must be configured for each one. This is described as ‘multi-trait scoring’ on the manual scoring platform. In the example on the image above, two other aspects of this test item are visible: Understanding and Mechanics.

   4. Define the subject category of your item.

      The subject category of an item needs to be defined. This is to ensure that it goes to an appropriate scorer in cases where scorers are assigned on the basis of the subject category of the items.

      This is done in the properties of the test section in question (it can also be defined for individual items – i.e. each item can be classified differently, under item properties). See the image below for an example.
      Click on the Properties icon for the area of the test you wish to define (the cogs at the top, or the cogs next to the test-part, section, or item). In the properties panel on the right, click on Categories. Enter the subject category you need for the current test, section, or item. Each subject category must start with ‘x-tao-scoring’, and end with the subject, which could be anything. Possible categories are x-tao-scoring-english, x-tao-scoring-math, or x-tao-scoring-biology, for example.

      TAO Grader only: Assigning scorers according to subject category is only one of two options. See the chapter Creating a scoring project in the section on TAO Grader for details on the assignment strategies available.

      

   5. View and score items configured for manual scoring. Once test takers complete an assessment, scorers will be able to view and score items for manual scoring from the session overview page. For more information, see documentation on TAO Grader.