Criteria Guidelines

CRITERIA

SCORING

GUIDANCE

1.   The goals of the exercise are clearly stated

1 = No

2 = Yes

Rater needs to determine whether the goals of the exercise are clearly stated in the exercise materials. Goals are sometimes also referred to as the “purpose” of the exercise.

DEFINITION: The goals of the exercise represent what the developers of the exercise are ultimately trying to achieve through the exercise (i.e., the overarching purpose, the “big picture”). The goals are general statements of anticipated outcomes and are usually, more global in scope than objectives and not expected to be measurable. Exercise goals usually fall into at least one of the following categories: training, problem solving, or evaluation.

2.   The objectives of the exercise are clearly stated

1 = No

2 = Yes

Rater needs to determine whether the objectives of the exercise are clearly stated in the exercise materials.

DEFINITION: Objectives of the exercise represent what the developers of the exercise expect the users to specifically accomplish as a result of having completed the exercise. Objectives are statements describing the results to be achieved and the manner in which these results will be achieved. Objectives help accomplish the goals of the exercise and can be process or outcome-oriented, depending on goals of exercise.

3.   Exercise objectives are appropriate given the goals of the exercise.

0 = Not enough information available to evaluate the criterion.

1 = None of the stated objectives are appropriate or none are stated.

2 = Some but not all of the stated objectives are appropriate.

3 = All of the stated objectives are appropriate.

Rater needs to determine whether the objectives correspond to the goals of the exercise. Objectives should be derived from the goals, and as such, are the written, active, and operational subdivisions of a goal.  If the goals of the exercise can be identified (even if they are not clearly stated), use that information to respond to this criterion.

FOR EXAMPLE: If the goal of the exercise is to evaluate X, the objectives might be: (1) To assess the proficiency of performing task Y; and (2) To evaluate the ability to conduct task Z.

SCORING:

• If you are unable to identify the goals, the exercise should be given a score of “0”.
• If no objectives are stated, the exercise should be given a score of “1”. 

4.   The exercise addresses each of its objectives.

0 = Not enough information available to evaluate the criterion.

1 = Exercise does not address any of the key objectives.

2 = Exercise addresses some but not all of the key objectives.

3 = Exercise addresses all of the key objectives.

Rater needs to determine whether the exercise is designed to test the stated objectives (i.e., are each of the objectives addressed during the exercise?). Do not use concluding statements made in the after action report as evidence that this criterion was met; the rater must come to his/her own conclusion based on the description of the exercise itself.

FOR EXAMPLE: If an exercise objective is to assess gaps in communications between public health officials and hospital emergency room providers, the exercise should be designed to specifically test the communication processes and procedures, and the events in the exercise scenario should be of such a scale that gaps in communication occur.

SCORING:

• If Criterion #2=1 and Criterion #3=0 then score Criterion #4=0
• If Criterion #2=1 and Criterion #3=1 then score Criterion #4=1.

5.   Exercise objectives are measurable within the context of the exercise.

1 = None of the objectives are measurable.

2 = Some but not all of the objectives are measurable.

3 = All of the objectives are measurable.

Rater needs to determine whether the objectives are well defined and measurable. The objectives should be specific to what will be done, who will do it, when it will be completed, how it will be evaluated, and by what measure.

NOTE: Some objectives can be appropriately measured in a yes/no scale if they focus on whether or not something happened while other objectives which focus on the extent to which something happened are more appropriately assessed with quantitative measures.

SCORING:

• If Criteria #1,2, 3 and/or 4 are scored poorly (0’s or 1’s), be more cautious about scoring this criterion.

6.   The scenario used in the exercise is appropriate given the goals and/or objectives of the exercise.

0 = Not enough information available to evaluate the criterion.

1 = None of the scenario is appropriate.

2 = Some but not all of the scenario is appropriate.

3 = All of the scenario is appropriate.

Rater needs to determine if the scenario is appropriate only in terms of what the exercise is trying to accomplish (i.e. its goals and objectives). Rater should also determine whether all of the different dimensions of the emergency event (e.g. how the emergency situation evolves throughout the exercise, whether it is consistent with known epidemiology) are at all possible given what is likely in real life. Rater should consider the method of introduction of the agent, official and public response, etc. Rater should also consider whether the timeframe covered by the scenario is appropriate given the exercise objectives.

DEFINITION: “Scenario” refers to the story describing the emergency event/situation at different stages. It is a hypothesized chain of events, which usually consists of an initial set of conditions and a time line of significant events that are presented to exercise participants to achieve the exercise objectives.

For more information about specific bioterrorism agents, diseases, and other threats see: http://www.bt.cdc.gov/

7.   The exercise scenario is internally consistent.

0 = Not enough information available to evaluate the criterion.

1 = No

2 = Yes

Rater needs to determine if the characteristics/parameters of the emergency event(s) described in the scenario are possible given other facts of the scenario and that the scenario is plausible.

FOR EXAMPLE: The public health resources described in the exercise as being needed should be consistent with the size and type of the emergency event described in the scenario.

8.   The exercise scenario is a realistic depiction of the capabilities and resources likely to be available to a participating health jurisdiction.

0 = Not enough information available to evaluate the criterion.

1 = No

2 = Yes

In determining if the exercise presents a realistic depiction of the capabilities and resources available to participants, the rater should consider the following: personnel, equipment, IT infrastructure, lab capacity, partner organizations, and procedures.

NOTE: This criterion is especially relevant for exercises in which the main goal is to test proficiency, since the capabilities and resources depicted in the exercise should reflect capabilities and resources that could be made available.

9.   The exercise documentation gives clear guidance as to who should participate in the exercise, and which other organizations or functions need to be simulated.

1 = No guidance is provided.

2 = Some guidance is provided.

3 = Clear guidance is provided.

Rater must consider whether the exercise materials identify the players who should be invited to participate in the exercise.

NOTE: The individuals/organizations mentioned in the exercise scenario do not necessarily correspond to the key individuals/organizations who should be invited to participate in the exercise (e.g., the National Guard is called in to maintain order in the exercise scenario, but only staff from public health departments are invited to participate in the exercise itself). . Also, it is entirely appropriate if some of the individuals/organizations are simulated in the scenario.

SCORING:

• If an exercise provides a list of agencies/organizations, fields of work, or subject area expertise to be represented at the exercise, the exercise should be given a score of “3”.
• If a list of agencies/staff who participated in the exercise is provided in the documentation, that would suffice as “some guidance,” and the exercise should be given a score of “2”.

10. The exercise is designed to engage all invited participants.

0 = Not enough information available to evaluate the criterion.

1 = Exercise is designed to engage none of the participants.

2 = Exercise is designed to engage some but not all of the participants.

3 = Exercise is designed to engage all of the participants.

Rater needs to look at the different components of the exercise and determine if all the participants have been given an appropriate role during the exercise and enough of a role within the exercise to merit their presence during the exercise. Rater should consider only the participants identified by the exercise developers; key participants that were excluded by the exercise developers should not be considered.

FOR EXAMPLE: If mental health professionals are invited to participate in the exercise, but there is no role for them (e.g., there is nothing in the exercise regarding the worried well), then their participation in the exercise may be a poor use of their time. Likewise, having police involved in an epidemiologic investigation may also not be an appropriate use of their time.

NOTE: Do not use concluding statements in the after action report that say “everyone was engaged” as evidence that this criterion was met. The rater must come to his/her own conclusion based on the description of the exercise components themselves.

SCORING:

• If Criterion #9=0 then Criterion #10=0.

11. Exercise guidance and materials are adequate to allow others to easily replicate the exercise.

1 = No written materials are provided or instructions are not available for evaluation.

2 = Materials are provided but are not clear and/or detailed enough.

3 = Clear materials are provided with satisfactory detail.

Rater needs to assess whether the materials developed for the exercise are adequate for others aside from the exercise developers to conduct the exercise. Rater should consider whether the instructions needed to facilitate the exercise are provided. Instructional materials may include: a facilitator’s guide, slide presentation, explicit instructions for engaging participants; and guidelines for how to make sure that the exercise continues to move forward, including a list of questions to pose to participants to guide them through the discussion if needed. Rater should also consider whether the information needed by participants is provided. These materials may include: a participant’s guide/handbook, instructions/guidelines to participants on how the exercise should proceed, and a complete list of relevant resource materials based on the exercise scenario and objectives.

FOR EXAMPLE: If an exercise involves a response to a smallpox outbreak, resource materials might include case definitions and CDC vaccination protocols. If an exercise involves testing the logistics of distributing the Strategic National Stockpile, materials that simulate Stockpile contents should be available.

12. The exercise is designed to result in action items.

1 = No, or not described in available exercise documentation.

2 = Yes

Rater needs to determine whether developing action items was an original part of the exercise design. In other words, developing action items is a structural element of the exercise design (planned a priori).

DEFINITION: “Action items” refer to next steps to be taken by the exercise users and/or participants to address preparedness issues based on the results of the exercise. Action items are also referred to as recommendations for improvement, an improvement plan, or  “next steps.”

SCORING:

• If there is no mention in the exercise materials that action items are to be developed, this criterion should given a score of “1”.

1 = No, or not described in available exercise documentation.

2 = Yes

Rater needs to determine whether getting feedback from participants was an original part of the exercise design. In other words, participant feedback is a structural element of the exercise (planned a priori). This could include a participant evaluation form that asks participants to rate various exercise components or an opportunity at the end of the exercise for participants to share their views about the exercise (often called a “hot wash” or “debriefing”).

The feedback should be provided after the exercise is completed, reflecting how well the exercise went or where it went wrong. The feedback could be specifically related to the structure of the exercise (e.g., not enough time provided to complete all activities specified in the exercise; key personnel were not included in the exercise, etc.) or what was learned from the exercise (e.g., we learned that we need a better process to vaccinate the population; we are not prepared to respond to an botulism outbreak, etc.).

NOTE: Do not use the fact that feedback is documented in the after action report as evidence that this criterion was met; the rater must come to his/her own conclusion based on the description of the exercise design itself. However, if the after action report does indicate that a feedback process was in place (i.e., feedback forms are described and summarized, etc.) and provides detail as to what feedback was solicited and when, that can sufficiently reflect that the exercise was designed to provide feedback.

14. The exercise, as designed, can be completed within the scheduled timeframe.

= No timeframe provided for completing the exercise OR unable to ascertain if timeframe is appropriate from available exercise documentation.

1 = Timeframe is not appropriate.

2 = Timeframe is appropriate.

Rater needs to decide whether, given the amount of time allocated to completing the exercise, there is enough time to complete all of the exercise components (i.e., if the exercise is meant to be completed in one day, can it in fact be completed in one day given its design/components).

NOTE: Many times, an exercise will state a timeframe for the exercise (e.g., “The exercise will take 6 hours to complete.”); provide a detailed schedule for the exercise; or a master event list that provides a step-by-step listing of the exercise activities. Use this information in determining whether or not you think the exercise can be completed within the scheduled timeframe.