Approach Used to Create the Database

The Public Health Preparedness Database was created by RAND researchers, under contract with the Department of Health and Human Services (HHS) Office of Public Health Emergency Preparedness (OPHEP).

Goal

The goal of the RAND project is to identify the best exercises to help local and state public health departments prepare for public health emergencies, including bioterrorism.

Evaluation Approach

We used three main steps in developing the database:

Identified and cataloged existing preparedness exercises that had sufficient documentation for evaluation.
Developed a set of criteria by which to rate the exercises.
Rated the exercises using these criteria.

We describe these steps in more detail below. For a complete description of our approach, you can read the report.

Step One: Identifying and Cataloguing Existing Preparedness Exercises that Had Sufficient Documentation

To develop a database of exercises, RAND researchers contacted bioterrorism coordinators throughout the United States, including those working for the Department of Homeland Security, the Department of Defense, the Environmental Protection Agency, the Department of Agriculture, state and local health departments, and private sector companies.

Through these contacts, we constructed a database of over 100 existing exercises. The exercises we collected included drills, orientations, tabletop exercises, functional exercises and full-scale exercises. We then made an effort to collect as much documentation as possible for each of the exercises, including exercise plans, after action reports, scenarios, facilitator and participant guides, presentation materials participant and observer evaluations, and observer instructions. For descriptions of these document types, click here. Exercises for which we had at least an exercise plan or after action report and one other document from the list above were selected for evaluation.

Step Two: Developing Criteria for Evaluating Exercise Design

We developed a set of criteria for rating preparedness exercises. To do this, we:

determined five “domains” of the exercise design that were most important to measure
developed one or more criteria for each domain.

The utility of the criteria was assessed in terms of:

Feasibility: whether the criteria are sufficiently variable to distinguish good exercises from poor ones;
Reliability: consistency among reviewers in applying the criteria; and
Validity: the reasonableness of the criteria, i.e., their freedom from bias.

We aimed to develop criteria that are broadly applicable, clearly defined, measurable, and capable of rating both substance and procedure. Our efforts produced 14 criteria, which fall into the following domains: 1) Goals and Objectives, 2) Scenario, 3) Participation, 4) Materials, and 5) Execution and Feedback.

The five domains are shown in the left column of the table below. Related evaluation criteria are shown in the right column. For a detailed explanation, read the report. Also, see the criteria with the scoring guidance.

Domains	Related Criteria and Scoring Range
Goals and Objectives. The goal represents what the developers of the exercise are ultimately trying to achieve through the exercise (the "big picture"). Objectives refer to what the developers of the exercise expect the users to specifically accomplish as a result of having completed the exercise.	1. The goals of the exercise are clearly stated (1-2). 2. The objectives of the exercise are clearly stated (1-2). 3. Exercise objectives are appropriate given the goals of the exercise (0-3). 4. The exercise addresses each of its objectives (0-3). 5. Exercise objectives are measurable within the context of the exercise (1-3).
Scenario. "Scenario" refers to the story describing the emergency event/situation at different stages in its evolution.	6. The scenario used in the exercise is appropriate given the goals and/or objectives of the exercise (0-3). 7. The exercise scenario is internally consistent (0-2). 8. The exercise scenario is a realistic depiction of the capabilities and resources likely to be available to a participating health jurisdiction (0-2).
Participation. Participation refers to information about what types of personnel should participate and which agencies (local, state, and federal) should be represented in the exercise.	9. The exercise documentation gives clear guidance as to who should participate in the exercise, and which other organizations or functions need to be simulated (1-3). 10. The exercise is designed to engage all invited participants (0-3).
Materials. Materials refer to a set of written documents available for conducting the exercise. These might include an exercise plan, a facilitator guide, a participant guide, a scenario, presentation materials, an after-action report, and participant and observer evaluation forms.	11. Exercise guidance and materials are adequate to allow others to easily replicate the exercise (1-3).
Execution and Feedback. Execution and Feedback refers to whether the exercise is designed to elicit action items for the organization and feedback about the exercise from participants, as well as whether the exercise can be completed within the scheduled timeframe.	12. The exercise is designed to result in action items (1-2). 13. The exercise is designed to solicit feedback from participants (1-2). 14. The exercise, as designed, can be completed within the scheduled timeframe (0-2).

Step Three: Using the Criteria to Rate the Exercises

We used the criteria to evaluate the exercises. Exercises were evaluated solely based on the written materials collected. Most of the exercises we evaluated have a scenario and an after-action report or exercise plan. A substantial percentage of these exercises also have presentation materials, participant guides, or facilitator guides.

Each exercise is reviewed by a minimum of three trained reviewers. The assigned reviewers read the available documentation and rated the exercises according to the fourteen criteria.

Reviewers also used the documentation to collect descriptive data including name of lead agency, geographic region, type of exercise, goals and objectives, primary agent used in the exercise, source of the disaster, and resources needed to implement the exercise.

The appropriateness of the exercise design was assessed in terms of the average score for each exercise overall, within each of the five domains, and for each individual criterion. Exercises were ranked based on three categories of relative performance: Excellent, Good, and Fair. In developing the database, we only selected those exercises that had an overall performance score of “Excellent” or “Good”.

Click here for additional information on the analysis and construction of measures, which can be found in the technical report, Enhancing Public Health Preparedness: Exercises, Exemplary Practices, and Lessons Learned.