Middle School Matters: Promoting Research- and Evidence-Based Practices to Support Reading Comprehension (MSMPREP)

Co-Principal Investigators: Murray, Christy; Stevens, Elizabeth; Roberts, Greg

Main features of the intervention: The Middle School Matters (MSM) intervention includes the following (Stevens, E. A., et al., 2022):

• Research-based reading practices implemented in combination with cognitive science practices across all content areas
• High-quality tools and resources for teachers that align with the research-based reading practices and include such tools as practice briefs, demonstration videos, and implementation toolkits
• Strategic planning process that includes (a) needs assessment, (b) goal setting, and (c) development of action steps so that all PD can be aligned with a common vision, existing initiatives, and a school's most pressing needs
• Ongoing and embedded professional learning that begins with initial PD and coupled with contextualized professional learning activities, including in-class demonstration, in-class observation and feedback, small group lesson planning, and follow-up PD sessions
• Follow-up opportunities that help schools monitor implementation via observations, coaching, and discussion of observation data to develop refinement PD sessions
• Scaffolding process to build capacity within the school leadership team

Main findings from the pilot study:

• Students in schools assigned to the PD condition significantly outperformed those in the BAU condition on a measure of main idea generation (ES=0.29) but not on measures of asking and answering questions (ES=0.11) and general reading comprehension (ES = –.0.09) (Stevens, Murray, Scammacca, Haager, & Vaughn, 2022)
• Schoolwide, targeted, and ongoing PD woven into content area instruction may hold promise for improving text comprehension (i.e., main idea generation) of middle grade students (Stevens, Murray, Scammacca, Haager, & Vaughn, 2022).
• Teachers indicate a desire to increase the amount of reading in their classes but cite challenges such as students' difficulty with reading and a preference for using other types of teaching activities. Teachers also cite the need for higher quality texts, and science and social studies teachers report dissatisfaction with and limited use of textbooks. Implications for practice include providing professional development focused on selecting appropriate texts for instruction, using text reading routines that enhance purposeful learning, and integrating reading practices into existing instructional delivery practices (Murray, Stevens, & Vaughn, 2021).

• An initial PD session on implementing get the gist (main idea) with content area texts in August and asking and answering questions with content area texts in January
•  Initial PD included a model lesson by the PD facilitator and lesson planning time for teachers. These two reading practices were chosen because they have an existing research base, are high-impact, and are easy to embed within existing lessons using content area texts.
• High-quality resources to facilitate implementation of get the gist and asking and answering questions, including paper and electronic copies of a practice guide, sample model lessons, and teacher and student materials, such as cue cards, learning logs, and posters
• The materials were customized for each content area of ELA, science, and social studies.
• Ongoing coaching, collective learning, and reflection via brief professional learning community (PLC) meetings every 2 weeks
• This included a model lesson delivered by the MSM Coach in at least one classroom of each content area while teachers observed, the use of fidelity checklists for each reading practice, implementation calendars, and PLC meeting agendas with goal setting activities.

Implementation of the reading practices and the ongoing coaching, collective learning, and reflection occurred over 10 weeks in the fall semester and 8 weeks in the spring semester.  During the second week of each semester's implementation, teachers delivered their own model lesson to students. Beginning in week 3, teachers were asked to embed the reading practices into their typical content area instruction at least two times per week.

For the pilot study, the researchers conducted an underpowered cluster-randomized trial (CRT) in which they randomly assigned a new set of schools (n = 6) at the school level to treatment or business-as-usual (BAU). Prior to randomization, the six schools were matched into pairs to improve the likelihood of a balanced randomization with baseline equivalence on all pretest measures. Schools were paired on available school-level information on variables likely to affect baseline equivalence for reading comprehension, such as percentage of English learners and economically disadvantaged. Treatment schools implemented the target research-based reading practices for 10 weeks in the fall semester and 8 weeks in the spring semester. BAU schools did not participate in the treatment PD model. They received business-as-usual PD and delivered business-as-usual instruction. The researchers measured fidelity of implementation through online surveys administered every 2 weeks that asked teachers to report the number of times they implemented the practices within a 2-week period and self-report surveys that asked teachers to report their use of each practice before and after participating in the MSM PD model PD. Participating students at both treatment and BAU schools received pre- and post-tests (see Key Measures below).

Social validity was measured through researcher-developed online survey that asked teachers to rate statements about the usability of the practices, feasibility of the practices, and the impact of the practices on students' reading comprehension. The researchers also developed this measure in the development phase and used in the pilot study. A text-use questionnaire was developed by the researchers to gather self-report data from middle school teachers about their text use (i.e., opportunities for students to read and learn from text) during content-area instruction. The questionnaire contained 12 items on reading and text use during content-area instruction. The items asked about the amount of time students read in class, reading materials, reading formats, and challenges to using text reading more often.