TAO Grader is an add-on to the TAO product suite which provides manual scoring for open-ended questions. It is carried out on the fly: when the test-taker has finished a test, the item responses in need of manual scoring are sent to TAO Grader. Once the scores have been submitted by the manual scorers for all the responses of a particular test session (i.e. a delivery execution), they are returned to TAO, where they will be displayed on the TAO Results page for that test.

As soon as a test-taker has submitted a test and the automatic scoring of any relevant items has been carried out, the Results page for that TAO delivery will display the scores for those items. The score columns of any items flagged as needing manual scoring will, however, still be empty.

At this point, TAO hands over the remaining items to TAO Grader to be scored. The two platforms are fully integrated, so there is no need to extract the responses which need manual scoring from the CSV file of the results of a TAO delivery: this all takes place automatically.

Note: TAO Grader is only available for Ignite and Enterprise customers.

To ensure that test-takers’ responses are processed correctly, the relevant test items need to be configured for manual scoring when the item is authored. Follow the steps below to do this.

1. Label the item as one which requires manual scoring.

Click on Outcome Declarations in the Response Properties panel to the right of the item. The outcome(s) for your item will appear, as in the image below. Clicking on the edit icon to the right of an outcome will open the values associated with it. The first outcome listed for this item of the test (i.e. the first aspect of the response which needs to be scored) is WORDCHOICE.

The first value is Interpretation – the name of the outcome. This is the name which will be displayed to the person carrying out the manual scoring. In the example below, this is Word Choice.

In the  drop-down menu under External Scored, select Human.

This value indicates the type of scoring which is needed for this particular outcome (in this case, scoring by a human), thus the item will be sent to TAO Grader for manual scoring.

2. Add a description of the scoring criteria.

The manual scorer needs to know what scoring system to use for the item (or outcome) in question. This is shown in the form of a rubric which contains the scoring criteria you wish to be employed, and is accessed via a link to a hosted PDF file.

In the box labeled Long Interpretation, add a link to the hosted PDF file that contains the scoring rubric.  This will be accessed later by the scorer.

3. Define the score range.

Enter the range (e.g. 1-4) in the box labeled Value.

It is possible to define different ranges, as long as the following rules are respected: both the minimum and maximum should be defined, using only integers. The maximum should not be negative and should be greater than the minimum.

Note: In order for an item’s responses to be “eligible” for manual scoring, there must be a valid score range under ‘Value’, and ‘human’ must be selected in the ‘externalScored’ box.

Note: If an item has more than one outcome (i.e. if different aspects of a response are awarded separate scores, meaning that multiple scores are needed for one item), the scoring method must be configured for each one. This is described as ‘multi-trait scoring’ on the manual scoring platform. In the example on the image above, two other aspects of this test item are visible: Understanding and Mechanics.

4. Define the subject category of your item.

The subject category of an item needs to be defined. This is to ensure that it goes to an appropriate scorer in cases where scorers are assigned on the basis of the subject category of the items. (Assigning scorers according to subject category is only one of two options. See the chapter Creating a scoring project in the section on TAO Grader for details on the assignment strategies available.)

This is done in the properties of the test section in question (it can also be defined for individual items – i.e. each item can be classified differently, under item properties). See the image below for an example.

Click on the settings for the area of the test you wish to define (the cogs at the top, or the cogs next to the test-part, section or item, as desired). In the properties panel on the right, click on Categories. Enter the subject category you need for the current test, section, or item. Each subject category must start with ‘x-tao-scoring’, and end with the subject, which could be anything. Possible categories are x-tao-scoring-english, x-tao-scoring-math, or x-tao-scoring-biology, for example.

5. View the manually scored Items in TAO.

After the relevant items in the delivery have been manually scored in TAO Grader and the scores have been sent to TAO (as a rule, scores are sent to TAO when all responses have been scored for a given delivery execution), you can refresh your Results page in TAO.

The scores awarded manually will show in the item table – both the overall score for the item, and any scores given for particular aspects (outcomes) of the item. The overall score for the test can also now be calculated, using the scoring method which has been configured in TAO for that test.