The
process of developing certification examination
questions
It is a long and costly process
for ASQ that involves several hundred hours of
work by teams of certified professionals
attending various sessions. The key workshops
involved in the exam development process are
(a) Job Analysis Workshop,
(b) Test Specification Workshop,
(c) Item Writing,
(d) Item Review
(e) Exam Review. Participants for
every session are to read details of the process
and relevant details as preparation before every
workshop.
(a) Job Analysis
Workshop: This workshop of 12 Subject
Matter Experts (SMEs) results in setting the
foundation for the Job Analysis Survey. The
survey is mailed to about 2000 SMEs in that field
(irrespective of ASQ membership) to determine
what knowledge and skills are appropriate to test
for the specific certification exam.
(b) Test Specification
Workshop: The tasks and knowledge areas
obtained from job analysis are then assembled
into an outline format with topics and categories
that logically group to form Body of Knowledge
(BOK) for the specific certification category.
The SMEs then develop explanatory subtext that
clearly describes the particulars of each
subtopic. The number of questions and the time
limit for taking the exam are also determined at
this workshop.
(c) Item Writing
Workshop: In this workshop 24 SMEs who
are trained in item writing develop question
items in small groups. Each item (assessment
professionals use this term instead of question)
that is developed in this session must have the
following:
- Only 1 key (correct answer)
that must have team agreement.
- A reference to support the key.
- A rationale for the item which explains what
the item is testing and why the key is correct
and the other options are not.
- Classification to the most detailed level of
the BOK.
Item writing workshops typically
produce 200-300 items over a two day period. The
items produced are considered `raw’ and will be
reviewed by another group of SMEs at an Item
Review Workshop.
(d) Item Review Workshop: About 4 weeks before the meeting , the twelve
SMEs are sent the items to be reviewed in the
workshop. They answer the items as if they were
candidates sitting for the exam. While they
review the items they are expected to make
comments about them and submit those comments to
the ASQ test developer. The comments from all
SMEs are then compiled into one document, which
is used for item review workshop. The SMEs
attending this workshop are trained in a variety
of assessment issues. Their focus is to review
the items to ensure that there is only one key
and that the item is clearly written and
accurate. They also confirm the reference and
rationale for each item. Once all components of
the item are verified the item is approved and
can subsequently be selected for use in an exam.
(e) Exam-Review Workshop: Once the raw items are reviewed and
approved, the ASQ test developer creates a draft
exam in accordance with the test specifications,
and reviews it for overlap or cueing between
items. The12 SMEs attending this workshop take
the exam just as any candidate would. As in the
case of Item Review discussed above, they are
asked to comment on the items and send those
comments to ASQ Test Developer. At the workshop
the members are trained in a variety of
assessment issues and then asked to focus on
items identified as “problematic” from the
review prior to the workshop. Once all the issues
in the exam are resolved, the committee approves
the exam and it can then be used for test
administration.
The process of
determining a passing grade
Whenever an exam is administered
with a new body of knowledge, the exam is
subjected to a cut score process. A group of
12-14 certified professionals (other than those
who were involved in developing the questions or
reviewing the exam) is asked to rate each
question in the new exam in terms of difficulty
for a minimally qualified examinee or candidate.
In other words the team is asked "What % of
minimally qualified candidates for the exam will
get this question correct?"
The cut score study participants
consider many factors such as the basic or
advanced level of knowledge required to know the
right answer, any potential errors a candidate
can make because of the various choices given,
the way the question is presented (including the
use of English language styles and words), etc.
Whenever members of the cut score panel show an
expected performance score that is more than 25
points apart from other panel’s rating, the
difficulty factors for that question are debated
and discussed until a consensus is reached by the
group.
The final cut score developed by
the cut score study panel is presented to the ASQ
Certification Board as the recommended percent or
portion correct needed to pass the test. See
Appendix-B for a sample of the item statistics
template exam stats template used for discussions
during the cut score study and a sample summary
table of what the ASQ Certification Board sees
when determining the cut score of a new body of
knowledge.
What is described above is a very
brief description of the cut score process. I was
a member of the cut score study panel for the CQA
exam administered in June of this year, and this
is the process we used to set the cut for that
new body of knowledge. It is the same process ASQ
uses on all of its exams.
Note: The
procedures ASQ uses to develop tests and
establish pass point are in accordance with those
described in the ANSI 17024 standard; these
procedures are considered accepted practices as
outlined in the Standards for Educational and
Psychological Testing, which are developed by the
American Educational Research Association,
(AERA), the American Psychological Association
(APA), and the National Council on Measurement in
Education (NCME). |