Co-Principal Investigators: Murray, Christy; Stevens, Elizabeth; Roberts, Greg

Purpose: This project developed, refined, and piloted a professional development model designed to support middle school teacher implementation of reading practices known to improve literacy for middle grades students.

Project Activities: The research team conducted an iterative development process in 2 phases over 2 years at 3 middle schools. Throughout the development process, researchers (1) engaged stakeholders and end-users via monthly school leadership team meetings, bi-monthly or post-iteration advisory council meetings about the components and structure of the professional development (PD) model; (2) identified and documented changes to the model, using formative data; (3) documented reasons for changes and the extent to which they may be conditioned on district, school, classroom, or student-level factors; and (4) specified a final model, including any variations due to contextual or student factors. In the final year of the project, the team tested the promise of the fully developed intervention via a pilot study. In this study, the team randomly assigned 6 schools to treatment or business-as-usual (BAU).

Key Outcomes: The main features of the intervention and findings of the project's pilot study are as follows:

Main features of the intervention: The Middle School Matters (MSM) intervention includes the following (Stevens, E. A., et al., 2022):

• Research-based reading practices implemented in combination with cognitive science practices across all content areas
• High-quality tools and resources for teachers that align with the research-based reading practices and include such tools as practice briefs, demonstration videos, and implementation toolkits
• Strategic planning process that includes (a) needs assessment, (b) goal setting, and (c) development of action steps so that all PD can be aligned with a common vision, existing initiatives, and a school's most pressing needs
• Ongoing and embedded professional learning that begins with initial PD and coupled with contextualized professional learning activities, including in-class demonstration, in-class observation and feedback, small group lesson planning, and follow-up PD sessions
• Follow-up opportunities that help schools monitor implementation via observations, coaching, and discussion of observation data to develop refinement PD sessions
• Scaffolding process to build capacity within the school leadership team

Main findings from the pilot study:

• Students in schools assigned to the PD condition significantly outperformed those in the BAU condition on a measure of main idea generation (ES=0.29) but not on measures of asking and answering questions (ES=0.11) and general reading comprehension (ES = –.0.09) (Stevens, Murray, Scammacca, Haager, & Vaughn, 2022)
• Schoolwide, targeted, and ongoing PD woven into content area instruction may hold promise for improving text comprehension (i.e., main idea generation) of middle grade students (Stevens, Murray, Scammacca, Haager, & Vaughn, 2022).
• Teachers indicate a desire to increase the amount of reading in their classes but cite challenges such as students' difficulty with reading and a preference for using other types of teaching activities. Teachers also cite the need for higher quality texts, and science and social studies teachers report dissatisfaction with and limited use of textbooks. Implications for practice include providing professional development focused on selecting appropriate texts for instruction, using text reading routines that enhance purposeful learning, and integrating reading practices into existing instructional delivery practices (Murray, Stevens, & Vaughn, 2021).

Structured Abstract

Setting: Participating schools were located in rural and near-urban settings in Texas.

Sample: Participants of the iterative development phase included 3 middle schools (with 57 ELA, science, and social studies teachers and their 1,860 students). Pilot study participants included 6 middle schools (with 131 English language arts, science, and social studies teachers and their 1,952 students that returned parent consent forms) across 2 school districts. The districts served approximately 20,000 and 25,500 students, respectively. Graduation rates were 90.3 percent to 97.4 percent. Both districts served approximately 51 percent economically disadvantaged students with 11 of students enrolled in special education. Students with limited English proficiency represented 16.6 percent and 22.3 percent of the population in each district. The racial/ethnicity make-up of the first district was 2.8 percent African American, 64.25 percent Hispanic, 29.1 percent White, 1 percent Asian, and 2.6 percent other races/ethnicities. The second district's make-up was 15.3 percent African American, 48.7 percent Hispanic, 23.1 percent White, 8 percent Asian, and 4.9 percent other races/ethnicities.

Intervention: Components of this year-long, campus-wide intervention included (1) a well-defined set of research-based practices, (2) high-quality resources that align with the research-based practices, (3) a strategic planning process, (4) ongoing and embedded professional learning, (5) follow-up, and (6) capacity building to support sustainability. Specifically, participating teachers received the following:

• An initial PD session on implementing get the gist (main idea) with content area texts in August and asking and answering questions with content area texts in January
•  Initial PD included a model lesson by the PD facilitator and lesson planning time for teachers. These two reading practices were chosen because they have an existing research base, are high-impact, and are easy to embed within existing lessons using content area texts.
• High-quality resources to facilitate implementation of get the gist and asking and answering questions, including paper and electronic copies of a practice guide, sample model lessons, and teacher and student materials, such as cue cards, learning logs, and posters
• The materials were customized for each content area of ELA, science, and social studies.
• Ongoing coaching, collective learning, and reflection via brief professional learning community (PLC) meetings every 2 weeks
• This included a model lesson delivered by the MSM Coach in at least one classroom of each content area while teachers observed, the use of fidelity checklists for each reading practice, implementation calendars, and PLC meeting agendas with goal setting activities.

Implementation of the reading practices and the ongoing coaching, collective learning, and reflection occurred over 10 weeks in the fall semester and 8 weeks in the spring semester.  During the second week of each semester's implementation, teachers delivered their own model lesson to students. Beginning in week 3, teachers were asked to embed the reading practices into their typical content area instruction at least two times per week.

Research Design and Methods: Iterative development occurred in 2 phases over 2 years at 3 middle schools. Throughout the development process, the researchers (1) engaged stakeholders and end-users via monthly school leadership team meetings, bi-monthly or post-iteration advisory council meetings about the components and structure of the PD model; (2) identified and documented changes to the model, using formative data; (3) documented reasons for changes and the extent to which they may be conditioned on district, school, classroom, or student-level factors; and (4) specified a final model, including any variations due to contextual or student factors. To examine and refine consistency of PD across sites in the final pilot study, the researchers also developed and finalized a fidelity checklist for the initial and follow-up PD sessions.

For the pilot study, the researchers conducted an underpowered cluster-randomized trial (CRT) in which they randomly assigned a new set of schools (n = 6) at the school level to treatment or business-as-usual (BAU). Prior to randomization, the six schools were matched into pairs to improve the likelihood of a balanced randomization with baseline equivalence on all pretest measures. Schools were paired on available school-level information on variables likely to affect baseline equivalence for reading comprehension, such as percentage of English learners and economically disadvantaged. Treatment schools implemented the target research-based reading practices for 10 weeks in the fall semester and 8 weeks in the spring semester. BAU schools did not participate in the treatment PD model. They received business-as-usual PD and delivered business-as-usual instruction. The researchers measured fidelity of implementation through online surveys administered every 2 weeks that asked teachers to report the number of times they implemented the practices within a 2-week period and self-report surveys that asked teachers to report their use of each practice before and after participating in the MSM PD model PD. Participating students at both treatment and BAU schools received pre- and post-tests (see Key Measures below).

Control Condition: In both the development phase and in the pilot study, schools assigned to the BAU condition received business-as-usual professional development and delivered business-as-usual instruction.

Key Measures: The researchers measured student reading comprehension using the Gates-MacGinitie Reading Test – Reading Comprehension subtest. They measured the pilot study students' application of the reading practices through the Strategy Use Measure – Middle School (SUM-MS), which the research team adapted from the elementary-grade version of the SUM (The Meadows Center for Preventing Educational Risk, 2008) during the development phase. Fidelity of implementation was measured through researcher-developed online surveys administered every 2 weeks that asked teachers to report the number of times they implemented the practices within a 2-week period. It was also measured in both treatment and BAU schools through researcher-developed self-report surveys that asked teachers to report about their use of get the gist and asking and answering questions last year (prior to implementation of the MSM PD model) and current (after implementation of the MSM PD model). Teachers completed these self-report surveys at the end of the 10-week implementation window in the fall and the 8-week implementation window in the spring. The measures were developed in the development phase and used in the pilot study.

Social validity was measured through researcher-developed online survey that asked teachers to rate statements about the usability of the practices, feasibility of the practices, and the impact of the practices on students' reading comprehension. The researchers also developed this measure in the development phase and used in the pilot study. A text-use questionnaire was developed by the researchers to gather self-report data from middle school teachers about their text use (i.e., opportunities for students to read and learn from text) during content-area instruction. The questionnaire contained 12 items on reading and text use during content-area instruction. The items asked about the amount of time students read in class, reading materials, reading formats, and challenges to using text reading more often.

Data Analytic Strategy: To examine the promise of the intervention for improving reading comprehension in middle school students in the pilot study, analysis of covariance (ANCOVA) models were estimated for each student outcome. The pretest for each outcome was included as a covariate. Student-level effect sizes and the associated standard errors were calculated using the formula for Hedges' g.

Products and Publications

ERIC Citations: Find available citations in ERIC for this award here.

Project website: https://GreatMiddleSchools.org Additional online resources and information:

• YouTube channel: https://www.youtube.com/@middleschoolmattersinstitu4831
• Additional products, such as the Middle School Matters Field Guide, practice briefs, infographics, and teacher and student materials, developed as part of this grant are available at www.GreatMiddleSchools.org.

Select Publications:

Capin, P., Hall, C., Stevens, E.A., Steinle, P., & Murray, C.S. (2022). Evidence-based reading instruction for secondary students with reading difficulties within multitiered systems of support. Teaching Exceptional Children. Advance online publication. doi.org/10.1177/00400599221079643

Murray, C.S., Stevens, E.A., Vaughn, S. (2021). Teachers' text use in middle school content-area classrooms. Reading and Writing. Advance online publication. doi.org/10.1007/s11145-021-10177-y

Stevens, E. A., & Austin, C. (2021). Structured reading comprehension intervention. In L. Spear-Swerling (Ed.), Structured Literacy Interventions for Children with Reading Difficulties (pp. 162–188). Guilford Press.

Stevens, E. A., Murray, C., Fishstrom, S. & Vaughn, S. (2020). Using question generation to improve reading comprehension for middle grade students. Journal of Adolescent and Adult Literacy, 64(3), 311–322. dx.doi.org/10.1002/jaal.1105

Stevens, E. A., Murray, C.S., Scammacca, N., Haager, & Vaughn, S. (2022). Middle School Matters: Examining the effects of a schoolwide professional development model to improve reading comprehension. Reading and Writing, 35(1), 1839–1864. doi.org/10.1007/s11145-022-10271-9

Stevens, E. A., & Vaughn, S. (2021). Using paraphrasing and text structure instruction to support main idea generation. Teaching Exceptional Children. Advance online publication. 53(4), 300–308. doi.org/10.1177/0040059920958738