Cultivating lifelong learners is central to the mission of the Commonwealth of the Northern Mariana Islands (CNMI) Public School System (PSS). This focus includes providing students with a strong start for college, career, and life after graduation. Toward this end, the CNMI PSS is focusing on strengthening key priority areas, particularly early literacy. Early literacy emerged as a focus because fewer than half (44 percent) of its third graders placed at or above grade level on the Star Reading Assessment in 2023. 1, 2
The CNMI PSS is using several strategies to provide targeted support for PSS’s earliest learners, kindergarten through grade 3, to help them reach grade-level literacy and improve reading outcomes. Through the CNMI Data Wayfinding Partnership for the Improvement of Early Literacy, the Regional Education Laboratory (REL) Pacific is working with the CNMI PSS to identify potential methods or approaches to improve the identification and selection of elementary reading strategies, specifically Multi-Tiered System of Supports (MTSS) Tier 2 literacy interventions.3
Tier 2 interventions are a key component of a Multi-Tiered System of Supports (MTSS), which is a framework designed to provide different levels of academic and behavioral support based on student needs. Within MTSS, Tier 2 offers targeted, small-group interventions for students who need additional help beyond general classroom instruction (Tier 1) but do not yet require intensive, individualized support (Tier 3).
In one applied research project, REL Pacific conducted a systematic literature review of literacy interventions for K–3 students, analyzed the research studies, and presented the findings in an evidence and gap map (EGM), a resource that CNMI PSS educators and other educators can use to make evidence-based decisions. The full report is available on the IES website.
What is an evidence and gap map?
Instructional decisionmakers often face an overwhelming and confusing literacy landscape. An evidence and gap map (EGM) is a tool that can help educators navigate this abundance of information by presenting essential details about interventions in a quick-reference format. REL Pacific’s literature review of literacy interventions for K-3 students depicts the existing research on literacy interventions, including available evidence on the efficacy of the intervention as well as gaps in the research where there is no evidence, in a new EGM.
Understanding the Evidence and Gap Map for Tier 2 Literacy Interventions
An initial search of Tier 2 literacy interventions yielded 267 studies. Of these, REL Pacific identified 20 that met the What Works Clearinghouse (WWC) standards with or without reservations.4 WWC is a set of Institute of Education Sciences guidelines that is used to evaluate the quality and credibility of research study designs.
The REL Pacific EGM displays the literature review’s findings as follows:
- Name of the intervention.
- Eight literacy outcome domains.
- Strength of the evidence (Every Student Succeeds Act [ESSA] evidence tier) for the intervention.
- Improvement index (expected change in student performance) for students who receive the intervention.
The 20 studies included represented 18 Tier 2 interventions. REL Pacific organized these studies into an EGM that shows:
- The name of the intervention — see a full list in the report.
- The literacy outcome domain — addressed by the intervention and aligned to the eligible domains in the associated WWC Study Review Protocol5 (literary achievement, phonics and related alphabetics, reading comprehension, reading fluency, vocabulary, writing conventions, writing quality, and academic dispositions).
- The strength of the evidence — ratings defined by the Every Student Succeeds Act (ESSA) evidence tiers.6
- The improvement index7 — a measure that estimates the expected change in student performance (on a percentile scale) if they receive the intervention.
More information on interpreting and understanding the evidence rating and the improvement index can be found in the EGM below.
Interpreting an Evidence and Gap Map
Reading and using an EGM means first gaining familiarity with the elements of an EGM chart.
- The top row of the chart highlights the eight literacy outcome domains included in the WWC Study Review Protocol. These are the academic domains that WWC reviewers use when evaluating literacy interventions.
- The first column, Intervention, specifies the 18 Tier 2 literacy interventions from the 20 studies reviewed by the research team while subsequent columns provide the direction of effect and the improvement index as well as the evidence tier the study (or studies) of the intervention achieved.
| Intervention | Literary achievement | Phonics and related alphabetics | Reading comprehension | Reading fluency |
| Achieve3000 |
|
| 0U | +2M |
| Early Reading Intervention |
| -2U |
| -6U |
| Early Vocabulary Intervention |
|
|
|
|
| Guided Reading |
| +9U | +1U | +10U |
| i-Ready Reading | -1U |
|
|
|
| Istation | 0U |
|
|
|
| LetterWorks |
| +28P |
| +7U |
Figure 2. Excerpt from the REL Pacific evidence and gap map (EGM). The full EGM can be found on page 8 of the study report.
The table below describes the characteristics of each EGM cell, using examples for interpretation.
Table 1. Evidence and gap map cell characteristics.
Element | Definition | Example |
| Improvement Index: Represented by the number and +/- sign in each cell. | The improvement index is the average difference in percentile rank for a student who receives the intervention compared to a student who does not receive the intervention.
The +/- symbols represent the direction of the effect for the average student receiving the intervention, with + indicating a higher performance and - indicating a potentially lower performance.8 | Looking at the LetterWorks intervention, we see that it has a rating of +28 for the phonics and related alphabetics domain. For example, if the average student not receiving the LetterWorks intervention scored in the 50th percentile on this domain, the average student who received the intervention could have a 28-point increase in percentile rank, and therefore rank in the 78th percentile in the same domain. |
ESSA Evidence Tier (Effectiveness) Rating: Represented by the superscript letter and cell color. | Evidence is defined within the ESSA four tiers of evidence––strong evidence, moderate evidence, promising evidence, and demonstrates a rationale (uncertain)–which provide a framework that policymakers can use to better understand the rigor of the findings and how they might be relevant to their context.
The evidence ratings include:
Some factors that could reduce a study’s ESSA rating include the statistical significance of the findings, the number of students in the study (sample size), the characteristics of the students in the study (for example, grade level, gender, race), and the alignment of the outcomes with the study goals. | Considering the LetterWorks intervention, despite an anticipated 28-point gain in percentile rank for students who participate in the intervention, note that this intervention has a “P” for a “Promising” evidence rating. Although the intervention reflects a large effect on percentile rank, there were considerations with the study design that reduced the strength of the evidence. |
The EGM is meant to be a starting point in the decisionmaking process for choosing Tier 2 interventions that will support student achievement. To use the EGM to identify an intervention for your school/district, you can:
- Identify a literacy domain where student achievement data show there is need for improvement.
- Pinpoint interventions with an improvement index and an effectiveness rating for that domain.9
- Compare the improvement index and effectiveness ratings for each intervention in that domain and identify those that interest you.
- Gather additional information about the intervention from the intervention’s website, the study review on the WWC website, or the original research study. You can also research your own school context, to better understand how an intervention may fit the needs of your classroom(s), school(s), or district. Some things to consider include:
- What is the strength of this intervention’s impact?
- For whom is the intervention effective? Is the context (setting and student population) in which the intervention was studied similar to my school/district context?
- Does the intervention use strategies that are feasible to implement considering other ongoing literacy supports?
- What resources are necessary to implement this intervention? Do I have access to these resources?
Considering a Supplemental Matrix of Tier 2 Literacy Intervention Strategies
REL Pacific also developed a supplemental matrix of interventions based on the Michigan Multi-Tiered System of Supports (MiMTSS) framework. Developed by literacy experts in the field, MiMTSS includes seven key elements and associated strategies of Tier 2 interventions based on existing research. The supplemental matrix specifies whether each intervention includes one or more of the strategies within the MiMTSS Tier 2 intervention elements (see excerpt in the table below; the full resource is in table 2, p. 10, of the report).
Example: Using the Evidence and Gap Map and Supplemental Matrix of Tier 2 Literacy Intervention Strategies Together
A grade 3 teacher identifies a small group of students who need additional support with reading fluency based on their recent literacy screener results. The teacher wants to give these students more intentional instruction, outside of whole-class instruction, to target their reading fluency skills and monitor their progress.
Because there are no readily-available resources to address this need, the teacher first uses the EGM to identify interventions with strong improvement index scores in the reading fluency domain. Next they use the supplemental matrix to narrow their search to interventions that offer small-group instruction and progress-monitoring tools. The teacher identifies a specific intervention as a good option for their students’ reading needs and then reaches out to their instructional coach and/or administrator for next steps in the decision making process.
This resource can be used to narrow down the interventions identified in the EGM for strategies and implementation considerations that address specific needs in different district, school, or classroom contexts.
| Intervention | Explicit instruction | Small group instruction | Matching instruction to student needs | ||||
Structured curriculum | Gradual skill-building | Small group | Individual | Responsive and targeted | Leveled materials | Learning pathways | |
| Achieve3000 | X |
|
| X |
| X | X |
| Early Reading Interventiona | X | X | X |
|
|
|
|
| Early Vocabulary Intervention |
| X | X |
|
|
|
|
| Guided Reading |
| X | X |
| X | X |
|
| i-Ready Reading | X |
| X | X |
|
| X |
| Istation | X |
| X | X |
|
| X |
| LetterWorksb |
|
| X |
|
|
|
|
Figure 3: Excerpt of the supplemental matrix of intervention strategies. The full matrix can be found on page 10 of the study report.
Learn more about the CNMI partnership’s initiatives, including the following projects supported by REL Pacific, on the partnership page and in this blog.
1 National Assessment of Educational Progress (NAEP). (2024). Explore results for the 2024 NAEP Reading Assessment. U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. https://www.nationsreportcard.gov/reports/reading/2024/g4_8/
2 Commonwealth of the Northern Mariana Islands Public School System. Statewide Assessments. District Data. https://envisioncnmipss.org/district-data#ec0f42a9-f14a-4ccd-8a12-9066c6833166
3 Tier 2 interventions are a key component of a Multi-Tiered System of Supports, a framework designed to provide different levels of academic and behavioral support based on student needs. Tier 2 offers targeted, small-group interventions for students who need additional help beyond general classroom instruction (Tier 1) but do not yet require intensive, individualized support (Tier 3).
4 What Works Clearinghouse. (2022). What Works Clearinghouse Procedures and Standards Handbook, version 5.0. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/WWC/Docs/referenceresources/Final_WWC-HandbookVer5_0-0-508.pdf.
5 What Works Clearinghouse. (2023). Study Review Protocol, version 5.0. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/wwc/Docs/ReferenceResources/WWC-SRP50-508.pdf.
6 REL Midwest has published a handout and video about the ESSA tiers of evidence.
7 The improvement index is a statistical measure that estimates the expected change in student performance when receiving the intervention, compared to those who do not receive the intervention. It is typically reported on a percentile scale (ranging from -50 to +50). A positive (+) index means a student receiving the intervention is expected to score a certain amount of percentile points higher than a similar student who did not receive the intervention. A negative index (-) indicates that the intervention may not be effective or could even be associated with lower performance. See table 2 for an example of interpreting the improvement index in the EGM.
8 A large difference in percentile rank does not mean that an intervention will automatically receive a strong evidence (s) rating. This is because ESSA tier ratings depend on several factors from a study—including the statistical significance of findings and the strength of the study design—that serve as criteria for each rating.
9 Educators who use an EGM to inform decisions should keep in mind that the improvement index and ESSA tier ratings should be considered together. These features are meant to describe the strength of evidence for each intervention and encourage users to compare the interventions using additional information from the study, the school/district context, and implementation guidelines.
