Survey Development

A key feature of the Mosaic II study is an expanded effort to measure reform-oriented classroom practices. Multiple methods including surveys, logs, classroom artifacts, teacher interviews and observations were used to derive a measure of the extent to which teachers are incorporating standards-based reform principles into their curriculum and instruction. Of particular interest is the development of scenarios included in the teacher survey developed for each of the five grade-level-subject combinations. To gauge teacher reactions to situations related to actual topics being taught in the project sites, two scenarios were developed for each survey. Below we summarize the method used to develop the surveys, focusing particularly on the scenario development.

Advisory Committees.
First we assembled committees of expert advisors in mathematics and science to help us develop methods of measuring standards-based teaching in the two subjects. The committees met in the fall and winter at RAND to work on a description of standards-based instruction that could be used as a basis for instrument development. In addition, we set up an electronic mail alias to facilitate further communication about the framework following the meeting. We also held extended conference calls to review drafts and refine our thinking.

The committee members are listed here.

Elements of Standards-Based Teaching in Mathematics and Science.
The first documents to be produced were descriptions of the important elements of standards-based teaching in mathematics and science. The process began with RAND staff reviewing the national standards and related documents and extracting key features. These materials were given to the committees, which developed explicit frameworks for the project that emphasized how features would be manifested in practice. That is, the committees tried to develop both a taxonomy of standards-based instructional practices in each subject and operational definitions of what each element would look like in the classroom. These documents were used to identify elements of standards-based practice that could be measured using different methods, i.e., surveys, logs, observations, and interviews.

You may download the mathematics elements and the science elements in PDF format.

Teacher Surveys.
We drafted surveys for each grade-by-subject combination. For the first year of data collection the surveys addressed third-grade mathematics, third-grade science, sixth-grade mathematics, sixth-grade science, and seventh-grade mathematics. As the cohorts progressed in grade levels each year, the surveys were revised to address the appropriate grade level instruction. (Copies of the final year's surveys are available here.)

Scenarios. Each survey contains two scenarios tailored to the curriculum and grade level of the five sites. The scenarios include four applied instructional problems that are set in the context of one instructional unit. These problems contain descriptions of realistic classroom settings and events and ask teachers to indicate their likely actions in response to the situation. These items attempt to capture teachers' tendencies to teach using standards-based approaches, by using a format that is less likely to lead to response biases than traditional approaches, and that is grounded in an actual lesson rather than a decontextualized problem.

Each scenario begins with a Context section that establishes an overall curricular setting for the four applied instructional problems. The focus is a new instructional unit lasting approximately one week. To insure that the problems do not ask teachers about curriculum that is unfamiliar, the scenario focus was on part of the curriculum that is taught in the target site at the target grade level. The context section describes the mathematics or science concepts students are to learn, as well as the students' prior skills and knowledge relevant to the curriculum focus.

To insure that responses to the problems are not unduly affected by teachers' specific experiences with particular students, the focal unit was not drawn specifically from the existing curriculum. For example, if the sixth grade science curriculum includes the study of natural and man-made timekeeping devices, including water clocks, an appropriate context for a scenario might be a unit in which students develop time-keeping devices using sand.

After the context section, four applied instructional problems are presented. The problems ask teachers how they would (1) introduce a new unit, (2) respond to student errors, (3)reconcile competing student explanations, and (4) emphasize different learning objectives. Each of these applied instructional problems is followed by several options that specify different ways teachers can respond to the teaching situations. For each option there are four response categories ranging from very unlikely to very likely. The options are designed to reflect a range of responses from less reform-oriented to more reform-oriented.

Problem Situations. Below is a description of the problem situations to which the teacher is asked to respond, and examples of the range of response options, and an example of one problem. In the science surveys, Scenario I involved a curriculum unit in which students conduct a controlled experiment to explore a scientific relationship.

Applied Instructional Problem 1: Introducing the Unit
The first problem focuses on the manner in which the teacher would introduce a new unit. The problem statement describes the first experimental activity in the unit. The respondent is asked to rate various ways of introducing the new unit before embarking on that activity.

Options: The options range from less reform-oriented (e.g., explain exactly how students are supposed to carry out the experiment) to more reform-oriented (e.g., pose an engaging problem to motivate the experiment).

Below is an example of the problem and response options from the 5th grade science survey:

You are now ready to begin your lessons on pH tolerance. There are different ways a teacher might begin this series of lessons.

How likely are you to do each of the following as an initial activity in this series of lessons?

Applied Instructional Problem 2: Inconsistent Results, Because of Inadequate Procedural Controls
The second problem deals with procedural mistakes during an experimental investigation. The problem statement indicates that time has passed and the students are now conducting the investigation in pairs or groups. However, the members of one working group are conducting the experiment in different ways so they are obtaining different results or two groups have obtained inconsistent results. The respondent is asked to rate various ways of responding to the inconsistent results.

Options: The options range from less reform-oriented, e.g., tell the students how to do it correctly to a more reform-oriented response, e.g., present the inconsistent results as a problem for discussion by the class.

Applied Instructional Problem 3: Different explanations or interpretations
The third problem setting occurs later in the unit after the experimental activity is completed. Students have been asked to explain the results that were achieved, to draw an inference, or to extend the results to a different set of conditions. Two different responses are given, and both are partially correct or are plausible in the context of what has been observed. The respondent is asked to rate various responses to receiving two scientifically reasonable responses.

Options: The options range from less reform-oriented, e.g., accepting the correct aspects of both responses and moving on to another topic to more reform-oriented, e.g., discussing the differences and how to make sense of them.

Applied Instructional Problem 4: Instructional goals
The fourth problem asks the respondent which learning objectives would be emphasized if he or she were going to teach this unit to his or her current class. The respondent is asked to rate various learning objectives.

Options: The options range from less reform-oriented e.g., learning facts and following directions to more reform-oriented e.g., understanding relationships or concepts.

Parallel scenarios were developed for each subject-grade combination. The first scenario in each survey dealt with students conducting a controlled experiment to explore a scientific relationship. The second scenario in each survey dealt with involves students observing how elements of a system operate and drawing correlational relationships but not manipulating variables.