Biliteracy Seals in a Large Urban District in New Mexico: Who Earns Them and How Do They Impact College Outcomes? REL 2023-140
Mihaly, Kata; Arellano, Brenda; Prier, Shannon (2022). Regional Educational Laboratory Southwest. Retrieved from: https://eric.ed.gov/?id=ED624239
-
examining6,801Students, grades12-PS
Single Study Review
Review Details
Reviewed: May 2024
- Single Study Review
- The WWC did not review the study because it is a descriptive study or correlational study without a comparison group, and it would not meet WWC standards.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Study sample characteristics were not reported.Practice Guide
Review Details
Reviewed: November 2023
- Practice Guide (findings for Biliteracy seals in New Mexico – Mihaly et al. (2022))
- Quasi-Experimental Design
- Meets WWC standards with reservations because it uses a quasi-experimental design in which the analytic intervention and comparison groups satisfy the baseline equivalence requirement.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
||
|---|---|---|---|---|---|---|---|---|---|
|
College Enrollment |
Biliteracy seals in New Mexico – Mihaly et al. (2022) vs. Business as usual |
1 Year |
Full sample;
|
84.90 |
71.60 |
Yes |
|
||
| Show Supplemental Findings | |||||||||
|
College Enrollment Full Time |
Biliteracy seals in New Mexico – Mihaly et al. (2022) vs. Business as usual |
1 Year |
Full sample;
|
84.30 |
68.30 |
Yes |
|
||
|
College Enrollment - 4 Year Institution |
Biliteracy seals in New Mexico – Mihaly et al. (2022) vs. Business as usual |
1 Year |
Full sample;
|
60.60 |
53.70 |
Yes |
|
||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
28% English language learners -
Female: 53%
Male: 47% -
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
New Mexico
-
Race Other or unknown 100% -
Ethnicity Hispanic 74% Not Hispanic or Latino 26% -
Eligible for Free and Reduced Price Lunch Free or reduced price lunch (FRPL) 30% No FRPL 70%
Study Details
Setting
The study took place in 13 New Mexico high schools and consisted of just 12th grade students across three cohorts from 2017/18–2019/20 graduating classes.
Study sample
A total of 13 high schools are included in the sample. A total of 522 intervention and 6279 matched comparison students across New Mexico make up the matched sample. The authors do not provide information on the numbers of teachers included in high schools. Approximately 28% of the students in both intervention and comparison conditions are classified as English learners at some point over the course of their educational career. Approximately 27% of students in the analytic sample speak Spanish at home, 30% are eligible for the National School lunch program and 74% identify as Hispanic. The sample was also 47% male.
Intervention Group
New Mexico awards biliteracy seals to high school graduates to recognize their proficiency in a non-English language. The current study focuses on the effect of earning a seal on college outcomes. Biliteracy seals provide a process for the New Mexico Public Education Department and school districts to encourage students to pursue proficiency in an additional non-English language and demonstrate their language skills to future employers as well as college admissions officers. Students could demonstrate these literacy skills across their high school course of study. Students complete the biliteracy seal through one of two common routes, the assessment and portfolio pathways. For the assessment pathway, students must earn a C or higher in four credits in the same non-English language course and then demonstrate proficiency on an assessment in the same language. For the portfolio pathway students must also earn a C or higher and then create a portfolio demonstrating their biliteracy skills by presenting their portfolio to a panel.
Comparison Group
The comparison condition consists of students who did not earn a biliteracy seal from either the state or their district. Students may have attempted but not earned a seal or have never attempted to earn a seal. The authors created the comparison condition by using an entropy balancing matching technique, outlined in Hainmueller, 2012; Zhao, 2019; Zhao & Percival, 2017.
Support for implementation
The authors do not mention any supports for implementation.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, statistical significance, and sample size of the findings within a domain, the WWC assigns effectiveness ratings as one of the following: Tier 1 (strong evidence), Tier 2 (moderate evidence), Tier 3 (promising evidence), uncertain effects, and negative effects. For more detail, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).