Development of the Mail Surveys

Two questionnaires were developed for the mail survey data collection, one for Part B and one for Part C. The questionnaires asked about state monitoring and improvement activities in 2004-05, which at the time, was the most recently completed monitoring cycle. With the exception of a few items at the beginning of the Part C questionnaire that helped establish the Part C context in each state, the Part B and Part C questionnaires essentially included the same content.

Both questionnaires were constructed using an iterative approach. A review of the Federal and state IDEA monitoring literature, preliminary site visits to two states, and discussions with the study's Advisory Panel provided the initial ideas for structuring the questionnaires and developing items.

Following revision of preliminary drafts of the questionnaires, the Advisory Panel provided written feedback, as well as feedback via conference calls. The Advisory Panel members have expertise in program and/or education evaluation; state monitoring practices; parent advocacy; state Part C and B administration; technical assistance; special education law, regulations, and policy; and early intervention and preschool special education. Panel members were asked to provide input regarding the relevance and clarity of the questions, item organization, clarity of instructions, availability of the information requested, adequacy and appropriateness of item response options, and time required to complete the questionnaire.

Following Advisory Panel input, the questionnaires were revised, and then Westat conducted a pilot test of the mail survey instruments in five states. Pilot states were selected according to criteria hypothesized to be related to state monitoring and improvement systems, such as the size of the special education child count and geographic location, in order to test the instruments in a variety of settings. Also, for the Part C questionnaire, we chose states with different Part C lead agencies (health, education, and other). Thus, two states completed the Part B questionnaire, two states completed the Part C questionnaire, and one state, where the state department of education was the Part C lead agency, completed both the Part B questionnaire and the Part C questionnaire.

During the pilot, states were asked to complete the questionnaire as if they were participating in the actual study. States were asked to use a form to provide feedback about problematic questions, the clarity of questions, availability of information requested, adequacy and appropriateness of item response options, and the time to complete the questionnaire. After piloting the questionnaires and reviewing the feedback forms, we contacted participants by A-2 telephone for additional information, if problematic questions were identified or other comments were provided. Based on information gathered through the pilot test, the questionnaires underwent another round of revisions, were reviewed internally, and then finalized. The final Part B and Part C questionnaires focused on the following topics (see appendices B and C for the complete questionnaires):

• Context for Monitoring and Improvement (Part C questionnaire only);
• Design for Monitoring and Improvement Activities;
• Data Collection and Analysis;
• Staffing and Training;
• Role of Stakeholders;
• Reporting;
• Process for State and Local Improvement; and
• History of Monitoring and Improvement.

Top