1. Item Development procedures and qualifications of SME/item writers:
All individuals who write items must be certified in the area in which they are writing items. That is, all individuals writing items for the CCSP must have their CCSP certification and DACBSP item writers must have their DACBSP certification. We accept nominations from the ACBSP for SMEs (Subject Matter Experts) to write the items and the ACBSP asks for volunteers for SMEs in their materials advertising the annual symposium. Prior to writing any items, we have all SMEs fill out a form to provide their background and document their qualifications as SMEs and keep these on file. We hope to have committees of SMEs of between eight and twelve members. Workshops have typically taken place annually at the annual symposium of the ACBSP. SMEs are individuals who are certified in the area they are working in (i.e., SMEs working on the CCSP examination have their CCSP certification and SMEs working on the DACBSP examination have their DACBSP certification).
We have a written description of minimal competence for both CCSP and DACBSP certificants. We review this description prior to the workshop and ask for any additions or changes to the definition. We remind the SMEs that we are writing items to determine minimum competence and that they should be writing items for minimum competence.
Items may be written prior to the workshop or written in the workshop. If they are written in the workshop, we have the item writers write in small groups. All items, whether written prior to the workshop or in the workshop, must be approved by the entire item writing committee before they are added to the active item bank. If any one SME does not approve of an item, it is either reworked until all agree it is appropriate or the item is discarded.
All item writers are briefly provided training and guidelines in writing items. Confidentiality is reviewed and all SMEs sign a confidentiality statement agreeing to keep all information discussed in the workshop confidential.
2. Passing point determination procedures (cut-score reports) and equating procedures if examinations are equated:
The Angoff method of setting the cut score is used. The Angoff values used have been set in a workshop of SMEs. The requirements for SMEs who participate in the cut-score workshops are the same as the requirements of SMEs who write items. They must be certified by the test in which they are reviewing (CCSP reviewers must have their CCSP certification and DACBSP reviewers must have their DACBSP certification). In all workshops, we obtain a written description of their credentials and keep these on file. We hope to have committees of SMEs of between eight and twelve members.
We have developed a description of minimal competence for both CCSP and DACBSP certificants. We review this description prior to the workshop and ask for any additions or changes to the definition. We begin the cut-score workshop by briefly explaining the cut-score procedure and reminding candidates of the need for confidentiality. SMEs in the cut-score workshop also sign a confidentiality agreement. We ask the SMEs to imagine that there are 100 candidates who are all minimally competent. They are to provide an estimate of the number (out of 100) of minimally competent candidates who would answer each item correctly. For training, on the first 10 or so items (or until the SMEs have the idea) we provide feedback to them of the percentage of people who got the item being rated correct. We then rate the rest of the items. We provide feedback periodically on the actual performance of the item to keep the SMEs somewhat grounded.
If the ratings of any of the SMEs differ by less than 30 points from any of the other SMEs, we average all of their ratings and that becomes the Angoff value for the item. If the ratings of any of the SMEs differ by more than 30 points we ask them to discuss the item. After the discussion we ask them to re-rate the item. We accept their ratings then, averaging their ratings, even if they are still more than 30 points out.
To create the cut score we average the Angoff values for all the items on the test. We do not have Angoff values for all items so we use linear equating to adjust the cut score based on the number of items that have and do not have Angoff values and the difference in performance between them.
From this raw cut score, we subtract one and one-half Standard Error of Measure (SEM) from the Angoff derived cut score on the CCSP examination and two and one-half SMEs from the Angoff derived cut score on the DACBSP examinations. The SEMs to subtract were determined by comparing passing scores and points from the examinations before the Angoff values were used and after they were used. These cut scores have been used consistently for some time now and are supported by the ACBSP as the most appropriate cut scores.
3. Scoring process and sample score reports for both passing and failing candidates. Statistical analysis of examination results.
All scan sheets are scanned twice, then electronically compared to assure there are no scan errors. If there are differences between the two scannings, the answer sheets are visually inspected and corrections made to the electronic file. The scan files are visually inspected to see if there are any anomalies, such as gaps or items answered out of range. If the intention of the candidates can be discerned, the scan file is corrected. An example of an anomaly that can be corrected would be a blank space that occurs because the candidate did not completely erase an answer and the scanner interpreted the item as being given two answers.
The scan files are then scored using ITEMAN. All item statistics are observed and any item with questionable item statistics is flagged. We consider an item to have questionable item statistics when it has either a negative biserial correlation or incorrect alternatives are answered more often than the correct alternative. All items that are questionable are sent to an SME for review to determine if there is a problem with the item or not. Also, all candidate comment forms that have anything written on them are sent to the reviewing SME who considers the candidate comments when reviewing items. The candidate comment sheets are anonymous and candidates can discuss anything about the test administration, the test in general, or specific items. Any item that is felt to be questionable (the correct alternative may not be completely correct or incorrect alternatives may be partially or wholly correct) is removed from scoring.
All candidates whose scores are at or near the cut score have their answer sheets hand scored. We randomly hand score a small sample of other candidates to verify that the computer scoring is correct. The tests are scored with two completely separate computer programs and the results compared. ITEMAN is a mature program that offers consistent results. The programs that create the diagnostic profiles also create total scores for the candidates and the results of these two analyses are compared.
The review at the end of the test is the second review that each test goes through. All tests, in draft form, are reviewed by an SME prior to their being printed. We check for duplicate items in the draft and assure that there are no duplicate items prior to their being sent for review. If the SME reviewing the tests finds other duplicate items, we remove one and replace it with another item in the same category.
Note that the tests are assembled according to the test specification. The CCSP and DACBSP test specifications were reviewed in 2002.
Candidates who passed are sent a pass letter from Plut Examination Services (PES) that lists the passing point and their score. Candidates who do not pass are sent a pass letter that lists the passing point and their score along with a diagnostic profile. This diagnostic profile provides the total number of items in each category and the number of items that they correctly answered in each category.
PES provides feedback to the colleges about the performance of their candidates compared to the entire sample. PES also provides feedback about the performance of the examination to the ACBSP. Samples of the forms used are included with this letter.
All procedures are designed to assure that the examinations are valid and legally defensible. These procedures assure that the candidates who pass are at least minimally competent and those who do not pass have not demonstrated minimal competence. The procedures also assure that the candidates receive the score the deserved and that the tests are fair to all candidates, whether they pass or not.