Scoring Tests

Test scores are determined by a student’s performance in the various Items of a Test. Individual item scores can be tallied using different methods to produce the final test result. This chapter shows how to configure the test scoring rules, and compute the Outcome Declarations (i.e. define the scoring criteria).

For information on the scoring rules used for items, see Scoring items automatically and Scoring items manually.

The scoring method to be employed for a specific test is defined after that test has been created, before it can be used in an assessment. There are four options:

• calculate a score based on the ratio of correct to incorrect responses;
• generate a PASS or FAIL;
• apply a method which is defined externally (and imported);
• calculate no score at all

Follow the steps below to set up your chosen scoring method.

1. After you have created your test by selecting the items which are to go in it, hover over the cog wheels just below the Action Bar, at the top of the right-hand corner of the Canvas, and you will see the option Manage Test Properties. Click on this and the general properties of the test will be displayed in the right-hand panel, including the properties for scoring.

2. Click on the button Scoring.

There are four methods for scoring a test, or “outcome processing” as this is called in QTI. Enter your choice from the four options below in Outcome Processing:

• None: If this option is selected, any existing scoring rules will be removed. In this case, no test score will be generated. Scores for the individual items of the test can be extracted by clicking on Results, or the QTI Results API. Use this option if you do not need aggregated test scores, or want to do custom outcomes processing outside of TAO.
• Custom: This option is only applicable when a test with custom processing rules is imported – i.e. the processing rules are defined outside of TAO. A screen ‘Edit processing rule’ will come up when you select ‘custom’. Here, select your (external) scoring rule which states how you would like the test taker responses to the item to be processed. An example of an application for custom processing rules would be if you want to take multiple interactions (in the same item) into consideration. You may want to specify, for example, that if the test-taker completes interaction A or interaction B correctly – but not both – they are awarded half the score (half of the computed MAX_SCORE). You can customize the processing rules to do this.
• Total Score. If you select this as your chosen method, all scores from all parts of the test will be added together, and outcome variables for the test-taker’s score (SCORE_TOTAL), the maximum score of all items (SCORE_TOTAL_MAX) and the ratio of correct responses (SCORE_RATIO) will be generated.

• Cut Score: If you choose this option, a cut-off point for passing the test will be fixed. Enter the score which is to be used as the cut-off point in the Cut Score box. This is set as a ratio of the total score. A PASS variable is then generated, indicating if the student has passed the test. When “Category Score” is active (see below), PASS variables for all categories are generated, all using the same cut-off point.

If you choose either of the latter methods (Total Score or Cut Score), some additional information may be required, depending on how you want to configure the test.

3. Defining different categories for your test.

A check-box for Category Score will appear. If your test contains different categories, Category Score must be selected, so that the scores can be calculated for each category separately.

To assign your items to separate categories, click on the settings icon to the right of an item. In the properties panal on the right, enter the name of the category you wish to assign that item to in the category box.

4. Defining weights for the items in your test.

Different weights can be assigned to different items of the test, based on how much they should contribute to the final test score. To assign a weight, click on the settings icon to the right of the item for you wish to add a weight. The item properties panel will open on the right.

Under Weights, select a Weight identifier and a value for that item. The identifier refers to the weighting scheme you choose for the test. by defauolt this is called ‘WEIGHT’, but if you want to use different weighting schemes in different test scenarios, you will need to define another one.

The value refers to the weight you wish to assign to that item. For example, if a weight of 2 is entered, the score for that item is multiplied by 2 when calculating the final test score. If, for any item of the test, there is no such weight definition at the Item level, the scoring engine assumes the default value = 1.

Note: It is possible to define different weights for the same item (in different weighting schemes), and then choose between them for different test scenarios. This may be useful if, for example, a test covers different topics, and in one particular test a certain topic is considered more important than the others. To do this, click on ‘Add a new weight’ below the original weight. 

When item metadata is inserted, it is automatically transferred to the categories of the item. Categories can also be managed manually.

5. Click on the button Outcome Declarations.

The Outcome Declarations are directly related to the scoring rules you have chosen above; the list of generated outcome variables is as follows:

• If you selected None as the outcome processing method, all outcome variables are removed
• If you selected Total score as the outcome processing method, outcome variables are generated to calculate total scores
• If you selected Cut score, as the outcome processing method, outcome variables are generated to calculate total scores and variables which are checked against the cut-score which has been defined (typically pass/fail)
• If a WEIGHT has been applied, additional outcome variables are generated to compute weighted scores

Note: Outcomes are generated automatically, but in the few cases where this does not occur, the values can be generated using the ‘regenerate’ button.

The configuration of the test scoring rules is now complete.