Scoring test-takers’ responses
Items are assigned to scorers for scoring (or reviewing) by the Scoring Project Manager. It may be that all the items in a particular test for which manual scoring is required are assigned to one scorer, or only a subset of those items – this depends on the nature of the test and the assignment strategy. Scorers will see only those items which are assigned to them within a specific test.
Once you are on the page displaying your Scoring Tasks (see the chapter Navigation for scorers for information on how to navigate the platform as a scorer), follow the steps below to score the responses assigned to you.
1. Click on the START button to start scoring.
You will be taken to the first item of the test which needs to be scored.
The main part of the screen shows the question as it appeared to the test-taker, as well as the test-taker’s response. An example is given in the image below.
In the panel on the left, the name of the test and the current item appear at the top, beneath which are one or several score boxes.
2. Enter the appropriate scores in the box(es).
If there is only one box, a general score for the response is expected. Where ‘multi-trait’ scoring is used (i.e. where different aspects of a response are awarded separate scores – in TAO: multiple outcomes), there will be multiple boxes in the score panel. This is the case in the test item in the image, which includes aspects such as Word Choice and Understanding, etc.
Scoring ranges have been configured in TAO, and can vary. In the image above, a range of 1 to 4 has been used.
Clicking on ‘see criteria’ next to any of these score boxes will open a detailed description of how scores should be awarded for that particular box.
A highlighter is available in three different colors to facilitate the scoring process – you may wish to highlight certain types of error found in the response, for example, in order to calculate the score(s) for that item.
Depending on the type of interaction used in the item, some responses can be very long. In this cases, a scrollbar is provided, so that you can scroll up and down in the test-taker’s response.
3. Flag the item if it needs further attention.
If you are unsure how to score a response, you can leave the score box blank and add a note – a memo – in the box. (This might be something like ‘need to discuss this with the project manager’, for example.) The item can then be bookmarked by clicking on the bookmark icon, which is located to the right of the test information at the top of this panel.
4. Proceed to the next item.
At the bottom of the panel there is a See Overview button, with two arrows. To go to the next item which has been assigned to you within that test, click on the righthand arrow, and repeat the scoring procedure.
At this stage – until you submit your final scores – you can go back and forth between items using the arrows at the bottom of the panel, and change the scores as many times as you like.
5. Submit your scores.
Scores can be submitted for a test when scoring is completed for all items (and all bookmarked issues resolved).
To do this (or simply to check your progress), click on the See Overview button at the bottom of the panel. This gives item-by-item information, as shown in the image.
The panel on the left gives a list of the items of the test delivery which require manual scoring. The first one is displayed on the screen. The ALL column shows the scores awarded in the score boxes for that item. The INCOMPLETE column shows whether all the score boxes contain a score, and under BOOKMARKED they will be flagged if they need further attention. This information comes up for all the test-takers sitting the same test as you visit each item, so if you wish, you can score one item for all test-takers before proceeding to the next, as well as compare all test-taker scores for one particular item at a glance.
When scoring is complete and all bookmarked issues are resolved, the blue bar at the bottom of the screen will say: ‘All done, everything is scored!’. If you are happy with the scores you have given, click on SUBMIT THE SCORE, next to this message.
Note: Scores cannot be changed after they have been submitted: they are sent to TAO for further processing. As a rule, responses are sent to TAO when everything is scored for a given delivery execution (i.e.when scores have been submitted for all responses of a given test-taker for a given test).
When you click on SUBMIT THE SCORE, you will be taken back to the Scoring Projects screen for that project, which will now tell you that the project is 100% complete.
Note: If, for any reason, the scoring process is interrupted (for example if there is a system crash or a session timeout), the session will be restored, when you reopen TAO Grader, at the point where it was interrupted. In other words, nothing will be lost (as long as you reconnect with the same browser).
