Experimental Evidence on the Effects (or Lack Thereof) of Informational Framing during the College Transition
Kramer, Jenna W. (2020). AERA Open, v6 n1 . Retrieved from: https://eric.ed.gov/?id=EJ1248501
-
examining9,200Students, grades12-PS
Practice Guide
Review Details
Reviewed: June 2023
- Practice Guide (findings for Text messages with clear directions on how to successfully complete college transition steps—Kramer (2020))
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Fall college enrollment |
Text messages with clear directions on how to successfully complete college transition steps—Kramer (2020) vs. Business as usual |
0 Months |
Text messages with clear directions treatment group;
|
77.80 |
78.70 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Continued spring college enrollment |
Text messages with clear directions on how to successfully complete college transition steps—Kramer (2020) vs. Business as usual |
3 Months |
Text messages with clear directions treatment group;
|
47.90 |
47.60 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 52%
Male: 48% -
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Tennessee
-
Race Asian 1% Black 21% Other or unknown 7% White 71% -
Ethnicity Hispanic 5% Not Hispanic or Latino 96% -
Eligible for Free and Reduced Price Lunch Other or unknown 100%
Study Details
Setting
The study took place in 452 high schools located in 84 Tennessee counties.
Study sample
The researchers randomly assigned 4,600 students each to the intervention and comparison groups. A total of 9,200 students who had been accepted to a college were included in the study. The full study included additional intervention conditions. Across all of the intervention and comparison conditions in the study, 52% of the students were female. Seventy-one percent were White, 21% were Black, 1% were Asian, and 7% did not report race. Five percent were Hispanic or Latino. Forty-six percent were first generation college students and 35% had no expected family contribution for college funding, based on income level. All students in the study had successfully applied and enrolled in the Tennessee Promise program, which provides students with financial assistance.
Intervention Group
Students in the intervention condition received a series of text messages to take clear, simple steps toward enrolling in college like taking 10 minutes to meet with their mentor on their enrollment progress. These text messages aimed to encourage students to take more college-going actions by simplifying the necessary steps. Text messages covered a range of topics, including enrollment, admissions, financial aid, scholarship eligibility, and retention. Messages were sent on regular intervals on the 1st and the 15th of each month. Text messages began in May 2017 and continued to be sent out through December 2017. Students received two text messages each month.
Comparison Group
Students in the comparison condition also received text messages encouraging them to complete necessary steps to enroll in college, but they did not focus on simple and quick actions students could take to continue the process. Comparison condition text messages covered the same range of topics as the intervention condition including enrollment, admissions, financial aid, scholarship eligibility, and retention. Messages were sent on regular intervals on the 1st and the 15th of each month. Text messages began in May 2017 and continued to be sent out through December 2017. Students received two text messages each month.
Support for implementation
The intervention was conducted in partnership with the nonprofit tnAchieves. The study did not provide information on support for implementation.
Practice Guide
Review Details
Reviewed: June 2023
- Practice Guide (findings for Text messages encouraging students to seek peer support in transitioning to college—Kramer (2020))
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Fall college enrollment |
Text messages encouraging students to seek peer support in transitioning to college—Kramer (2020) vs. Business as usual |
0 Months |
Peer support condition vs. business-as-usual condition;
|
78.20 |
78.70 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Continued spring college enrollment |
Text messages encouraging students to seek peer support in transitioning to college—Kramer (2020) vs. Business as usual |
3 Months |
Peer support condition vs. business-as-usual condition;
|
47.30 |
47.60 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 52%
Male: 48% -
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Tennessee
-
Race Asian 1% Black 21% Other or unknown 7% White 71% -
Ethnicity Hispanic 5% Not Hispanic or Latino 96% -
Eligible for Free and Reduced Price Lunch Other or unknown 100%
Study Details
Setting
The study took place in 452 high schools located in 84 Tennessee counties.
Study sample
The researchers randomly assigned 4,600 students each to the intervention and comparison groups. A total of 9,200 students who had been accepted to a college were included in the study. The full study included additional intervention conditions. Across all of the intervention and comparison conditions in the study, 52% of the students were female. Seventy-one percent were White, 21% were Black, 1% were Asian, and 7% did not report race. Five percent were Hispanic or Latino. Forty-six percent were first generation college students and 35% had no expected family contribution for college funding, based on income level. All students in the study had successfully applied and enrolled in the Tennessee Promise program, which provides students with financial assistance.
Intervention Group
Students in the intervention condition received a series of text messages encouraging them to work with peers who have similar backgrounds or friends to complete necessary steps to enroll in college. These text messages were intended to emphasize students’ sense of belongingness in the college entry process to encourage them to take more college-going actions. Text messages covered a range of topics, including enrollment, admissions, financial aid, scholarship eligibility, and retention. Messages were sent on regular intervals on the 1st and the 15th of each month. Text messages began in May 2017 and continued to be sent out through December 2017. Students received two text messages each month.
Comparison Group
Students in the comparison condition also received text messages encouraging them to complete necessary steps to enroll in college, but they did not direct them to work with peers and friends. Comparison condition text messages covered the same range of topics as the intervention condition including enrollment, admissions, financial aid, scholarship eligibility, and retention. Messages were sent on regular intervals on the 1st and the 15th of each month. Text messages began in May 2017 and continued to be sent out through December 2017. Students received two text messages each month.
Support for implementation
The intervention was conducted in partnership with the nonprofit tnAchieves. The study did not provide information on support for implementation.
Practice Guide
Review Details
Reviewed: June 2023
- Practice Guide (findings for Text messages that emphasize the risk of losing financial assistance and other benefits to motivate students to take steps to transition to college—Kramer (2020))
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Fall college enrollment |
Text messages that emphasize the risk of losing financial assistance and other benefits to motivate students to take steps to transition to college—Kramer (2020) vs. Business as usual |
0 Months |
Loss aversion treatment group;
|
78.10 |
78.80 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Continued spring college enrollment |
Text messages that emphasize the risk of losing financial assistance and other benefits to motivate students to take steps to transition to college—Kramer (2020) vs. Business as usual |
3 Months |
Loss aversion treatment group;
|
48.30 |
47.60 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 52%
Male: 48% -
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Tennessee
-
Race Asian 1% Black 21% Other or unknown 7% White 71% -
Ethnicity Hispanic 5% Not Hispanic or Latino 96% -
Eligible for Free and Reduced Price Lunch Other or unknown 100%
Study Details
Setting
The study took place in 452 high schools located in 84 Tennessee counties.
Study sample
The researchers randomly assigned 4,600 students each to the intervention and comparison groups. A total of 9,200 students who had been accepted to a college were included in the study. The full study included additional intervention conditions. Across all of the intervention and comparison conditions in the study, 52% of the students were female. Seventy-one percent were White, 21% were Black, 1% were Asian, and 7% did not report race. Five percent were Hispanic or Latino. Forty-six percent were first generation college students and 35% had no expected family contribution for college funding, based on income level. All students in the study had successfully applied and enrolled in the Tennessee Promise program, which provides students with financial assistance.
Intervention Group
Students in the intervention condition received a series of text messages encouraging them to complete necessary steps to enroll in college. The messages emphasized that by not taking those steps students would risk losing financial assistance, college course preferences, academic counseling, and other benefits. These text messages that emphasized the potential loss of postsecondary benefits were intended to motivate students to take more college-going actions. Text messages covered a range of topics, including enrollment, admissions, financial aid, scholarship eligibility, and retention. Messages were sent on regular intervals on the 1st and the 15th of each month. Text messages began in May 2017 and continued to be sent out through December 2017. Students received two text messages each month.
Comparison Group
Students in the comparison condition also received text messages encouraging them to complete necessary steps to enroll in college, but they did not emphasize the potential loss of postsecondary benefits. Comparison condition text messages covered the same range of topics as the intervention condition including enrollment, admissions, financial aid, scholarship eligibility, and retention. Messages were sent on regular intervals on the 1st and the 15th of each month. Text messages began in May 2017 and continued to be sent out through December 2017. Students received two text messages each month.
Support for implementation
The intervention was conducted in partnership with the nonprofit tnAchieves. The study did not provide information on support for implementation.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).