In today’s environment, there is no guarantee of writing ability based on a level of education contained in a resume. Asking for previous writing samples has been shown to be unreliable and very limited in validity. The writing skills exercise method uses information specifically related to the position. The exercise is objectively scored without the scorer knowing the identity of the candidate. The exercise is designed to be used in conjunction with a structured behavior interview.
Determining the Level of Writing Ability for the Target Job
Your job analysis has determined that the position requires the ability to write. The question then becomes “What level of writing ability is necessary upon entry?” Listed below in ascending level of writing abilities are some examples:
- Simple Sentence
- Complex Sentence
- Form Letter
- Request Letter response (cut and paste or original)
- Complaint Letter response
- General Information Correspondence (letter or memo)
- News Letter Article
- Press Release
- Grant Proposals
Developing the Exercise
The writing ability exercise is one type of general exercise known as Work Sample Exercises. Work Sample Exercises are based on the premise that the best predictor of future behavior is observed behavior under similar situations. These exercises require the candidate to perform tasks that are similar to those that are performed on the job. Thus, the exercise must measure writing abilities necessary upon entry to the job. The subject matter expert (SME) is responsible for preparing the writing assignment and instructions for completing the assignment to the candidate. The SME then completes the writing assignment, which is used to develop the scoring key
Developing the Scoring Key
Based on the SME’s completed exercise the scoring key is developed primarily by using the SME’s judgment and the job analysis. The exercise is usually scored along two general criteria: 1) format and general use of English and grammar; and 2) exercise content. Using a Request Letter response (low to medium ability) as an example the following items could be scored according to criterion 1: salutation, acknowledgment of letter, letter closing, signature, grammar, sentence structure, spelling, and word usage. Using criterion 2: providing the requested materials and providing additional information could be scored. Rating scales are used to rate each of the sub-criteria to obtain a numeric score. The rating scales should be defined by benchmarks for each sub-criterion. Benchmarks are suggested answers to the questions that are linked to the rating scale. Benchmarks provide a frame of reference for assessing the candidate’s responses objectively and consistently. There should usually be at least three suggested answers for each sub-criterion: a superior, a satisfactory, and an unsatisfactory response to develop benchmarks, using the guide below, ask SMEs to create answers that fit the different levels of the rating scale.
5 What would one expect or want an outstanding candidate to give as the best possible answer?
3 What is an acceptable answer that one would expect a qualified candidate to give?
1 What would one expect as a poor answer from a candidate who has little or no knowledge or skill on this job requirement?
Written Exercise Administration (This is the most critical part of the exercise)
Depending on the length of the exercise, candidates will be scheduled before the interview. The written exercise administration will take place in a room or cube containing a computer with a word processing program (MS Word). Ideally the written exercise should be administered in a private and quiet room. Although the actual job may be performed in a noisy office, every effort should be made to provide the candidate with a location that is most conducive to completing the exercise.
An exercise administrator who is not part of the interview process is necessary to administer the exercise. The exercise administrator will assign a random ID number to the candidate when they arrive for the exercise. The candidate will be instructed to use the ID number at all times during the exercise. The exercise administrator will then read the specific instructions for the exercise to candidate. When the candidate has five minutes remaining to complete the exercise, the exercise administrator will give the candidate a five-minute warning. When the time limit has expired or the candidate finishes the exercise, the exercise administrator will collect the candidate’s exercise and instruction sheet. The exercise administrator will check the candidate’s exercise to assure only the exercise ID is used on the exercise. The exercise administrator will then make a photocopy of the original. After all the candidates have participated in the exercise, the administrator will retain the originals and provide the copies to the exercise scorer(s).
It is recommended that the Hiring Authority score all of the exercises using a score sheet based on the sub-criteria being scored. If a panel scores the exercise they should be trained. Each panel member shall receive a copy of each candidate’s written exercise and score them independently. After scoring all candidates, the panel should review their individual ratings as a group and make sure that there is a general consensus on each sub-criterion. For example, on a scale of 1 to 5, if one rater gave a 4, one a 3, and the other a 1 on the same sub-criteria, this might indicate some rater bias or misunderstanding of the candidate’s comments. All raters should be no more than one point away from all other raters’ scores. For example, it would be acceptable to have two 3s and a 2 given on the same question. If raters find that there is not a consensus on a sub-criterion, they should discuss the reasons for their ratings and attempt to reach a consensus. Once the all written exercise scores are final, the exercise administrator will inform the rater(s) of the names of the candidates. The results of the interview and exercise may be combined at this point.