Chapter 2. Techniques for testing

Process: Objective and subjective evaluation

If we view objectivity and subjectivity of evaluation along a continuum, we can represent various assessment and scoring methods along its length.

Test items that can be evaluated objectively have one right answer (or one correct response pattern, in the case of more complex item formats). Scorers do not need to exercise judgment in marking responses correct or incorrect. They generally mark a test by following an answer key. In some cases, objective tests are scored by scanning machines and computers. Objective tests are often constructed with selected-response item formats, such as multiple-choice, matching, and true-false. An advantage to including selected-response items in objectively scored tests is that the range of possible answers is limited to the options provided by the test writer—the test taker cannot supply alternative, acceptable responses.

Because much of what we assess in reading and listening comprehension measures is first interpreted by the test writer, some degree of subjectivity is present in objectively scored items. For that reason, assessments of the Interpretive mode, even those comprised of "one-right-answer" items, might not be placed all the way at the objective end of the continuum.

Evaluating responses objectively can be more difficult with even the simplest of constructed-response item formats. An answer key may specify the correct answer for a one word, gap-filling item, but there may in fact be multiple, acceptable alternative responses to that item that the teacher or test developer did not anticipate. In classroom testing situations, teachers may perceive some responses as equally or partially correct, and apply some subjective judgment in refining their scoring criteria as they mark tests. Informal scoring criteria for short-answer items probably work well for classroom testing as long as they are applied consistently and are defensible.

Just as there may be few truly objective measures of second language knowledge and skill, so too is it rare to find purely subjective evaluations of performance. Allowing the subjective impressions of scorers to determine learners' grades would not be acceptable to most students, their parents, or other stakeholders. We do not usually have to justify our opinion that a work of art is good or bad—we simply like it or we don't. Since our judgment has no significant consequences for the artist (unless we are art critics), a subjective evaluation is acceptable. It is also not a matter of concern that the many viewers of the artwork do not agree about its quality.

In assessment, we strive to ensure two types of reliability: inter-rater (raters agree with each other) and intra-rater (a rater gives the same score to a performance rated on separate occasions). The higher the stakes, the more reliable (consistent) judgments must be. Scoring criteria, in the form of rubrics, are generally used to guide raters to arrive at the same, or nearly the same, evaluation of a product. Thus, although it is common to refer to scoring which requires human judgment as subjective evaluation, in most cases we might place it near the midpoint on our objective-subjective continuum.

In rated assessments, the scoring criteria form an integral part of the evaluation. Specialists in language testing often identify three key components in performance assessment. These components are:

• Tasks that are effective in eliciting the performance to be assessed.
• Rating criteria to evaluate the quality of the performance. The criteria reflect the relative importance of various aspects of the performance, and are appropriate for the population being assessed.
• Raters that are trained to apply the criteria and can do so consistently.

RULES FOR WRITING MULTIPLE-CHOICE QUESTIONS

Multiple choice is a form of assessment in which respondents are asked to select the best possible answer (or answers) out of the choices from a list.

 1. Use Plausible Distractors (wrong-response options)

• Only list plausible distractors, even if the number of options per question

changes

• Write the options so they are homogeneous in content

• Use answers given in previous open-ended exams to provide realistic

distractors

2. Use a Question Format

• Experts encourage multiple-choice items to be prepared as questions (rather

than incomplete statements)

Incomplete Statement Format:

The capital of California is in

Direct Question Format:

In which of the following cities is the capital of California?

3. Emphasize Higher-Level Thinking

• Use memory-plus application questions. These questions require students to

recall principles, rules or facts in a real life context.

• The key to preparing memory-plus application questions is to place the

concept in a life situation or context that requires the student to first recall the

facts and then apply or transfer the application of those facts into a situation.

• Seek support from others who have experience writing higher-level thinking

multiple-choice questions.

4. Keep Option Lengths Similar

• Avoid making your correct answer the long or short answer

5. Balance the Placement of the Correct Answer

• Correct answers are usually the second and third option

6. Be Grammatically Correct

• Use simple, precise and unambiguous wording

• Students will be more likely to select the correct answer by finding the

grammatically correct option

7. Avoid Clues to the Correct Answer

• Avoid answering one question in the test by giving the answer somewhere

else in the test

• Have the test reviewed by someone who can find mistakes, clues,

grammar and punctuation problems before you administer the exam to

students

• Avoid extremes – never, always, only

• Avoid nonsense words and unreasonable statements

8. Avoid Negative Questions

• 31 of 35 testing experts recommend avoiding negative questions

• Students may be able to find an incorrect answer without knowing the

correct answer

9. Use Only One Correct Option (Or be sure the best option is clearly the best

option)

• The item should include one and only one correct or clearly best

answer

• With one correct answer, alternatives should be mutually exclusive

and not overlapping

• Using MC with questions containing more than one right answer

lowers discrimination between students

10. Give Clear Instructions

Such as:

Questions 1 - 10 are multiple-choice questions designed to assess your ability to

remember or recall basic and foundational pieces of knowledge related to this course.

Please read each question carefully before reading the answer options. When you

have a clear idea of the question, find your answer and mark your selection on the

answer sheet. Please do not make any marks on this exam.

Questions 11 – 20 are multiple-choice questions designed to assess your ability to

think critically about the subject. Please read each question carefully before reading

the answer options. Be aware that some questions may seem to have more than one

right answer, but you are to look for the one that makes the most sense and is the most

correct. When you have a clear idea of the question, find your answer and mark your

selection on the answer sheet. You may justify any answer you choose by writing

your justification on the blank paper provided.

11. Use Only a Single, Clearly-Defined Problem and Include the Main Idea in

the Question

• Students must know what the problem is without having to read the

response options

12. Avoid the “All the Above” Option

• Students merely need to recognize two correct options to get the

answer correct

13. Avoid the “None of the Above” Option

• You will never know if students know the correct answer

14. Don’t Use MC Questions When Other Item Types Are More Appropriate

• limited distractors or assessing problem-solving and creativity

The Matching Format

The matching test item format provides a way for learners to connect a word, sentence or phrase in one column to a corresponding word, sentence or phrase in a second column. The items in the first column are called premises and the answers in the second column are the responses. The convention is for learners to match the premise on the left with a given response on the right. By convention, the items in Column A are numbered and the items in Column B are labeled with capital letters.

 

DIFFERENCE BETWEEN COMPLETION AND SHORT ANSWER

EXAMPLE OF A COMPLETION QUESTION The first Prime Minister of Canada was _________________. EXAMPLE OF A SHORT ANSWER QUESTION 1.      Who was the first Prime Minister of Canada? _____________________________ KEY: Sir John A. Macdonald

• Completion = fill-in the blank
• Short Answer = answer the question
• when we say short answer, talking sentence or less
• more than one sentences or paragraph are short written response, because no longer objective items --> need more complex scoring scheme

ESSAY QUESTION

Writing an effective essay examination requires two important abilities: recalling information and organizing the information in order to draw relevant conclusions from it. While this process sounds simple, writing an effective essay examination under pressure in limited time can be a daunting task.

Common strategy terms for Essay writing are as follows:

1. Analyze: Divide an event, idea, or theory into its component elements, and examine each one in turn: Analyze Milton Friedman's theory of permanent income.
2. Compare and/or Contrast: Demonstrate similarities or dissimilarities between two or more events or topics: Compare the portrayal of women in Beloved with that in Their Eyes Were Watching God.
3. Define: Identify and state the essential traits or characteristics of something, differentiating it clearly from other things: Define Hegelian dialectic.
4. Describe: Tell about an event, person, or process in detail, creating a clear and vivid image of it: Describe the dress of a knight.
5. Evaluate: Assess the value or significance of the topic: Evaluate the contribution of black musicians to the development of an American musical tradition.
6. Explain: Make a topic as clear and understandable as possible by offering reasons, examples, and so on: Explain the functioning of the circulatory system.
7. Summarize: State the major points concisely and comprehensively: Summarize the major arguments against using animals in laboratory research.