Miller, Sarah; Davison, Jenny; Yohanis, Jamie; Sloan, Seaneen; Gildea, Aideen; Thurston, Allen (2017). Education Endowment Foundation. Retrieved from: https://eric.ed.gov/?id=ED581121
Reviewed: March 2017
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Combination of standardized scores on Hodder Access Maths Test and General Certificate of Secondary Education Maths test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Full sample;
|
0.05 |
0.02 |
Yes |
|
|
|
| Show Supplemental Findings | |||||||||
|
Combination of standardized scores on Hodder Access Maths Test and General Certificate of Secondary Education Maths test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Low achievement;
|
-1.16 |
-1.27 |
Yes |
|
||
|
Combination of standardized scores on Hodder Access Maths Test and General Certificate of Secondary Education Maths test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Small schools;
|
-0.01 |
-0.15 |
Yes |
|
||
|
Combination of standardized scores on Hodder Access Maths Test and General Certificate of Secondary Education Maths test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Female;
|
0.04 |
-0.06 |
Yes |
|
||
|
Combination of standardized scores on Hodder Access Maths Test and General Certificate of Secondary Education Maths test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Key stage 3 (years 7 and 9);
|
0.08 |
0.00 |
Yes |
|
||
|
Combination of standardized scores on Hodder Access Maths Test and General Certificate of Secondary Education Maths test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Non-English learners;
|
0.12 |
0.04 |
Yes |
|
||
|
Combination of standardized scores on Hodder Access Maths Test and General Certificate of Secondary Education Maths test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
High achievement;
|
0.98 |
0.95 |
No |
-- | ||
|
Combination of standardized scores on Hodder Access Maths Test and General Certificate of Secondary Education Maths test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Male;
|
0.13 |
0.09 |
No |
-- | ||
|
Combination of standardized scores on Hodder Access Maths Test and General Certificate of Secondary Education Maths test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Free school meals;
|
-0.27 |
-0.28 |
No |
-- | ||
|
Combination of standardized scores on Hodder Access Maths Test and General Certificate of Secondary Education Maths test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Key stage 4 (year 11);
|
0.11 |
0.11 |
No |
-- | ||
|
Combination of standardized scores on Hodder Access Maths Test and General Certificate of Secondary Education Maths test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
English learners;
|
-0.17 |
-0.05 |
Yes |
|
||
|
Combination of standardized scores on Hodder Access Maths Test and General Certificate of Secondary Education Maths test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Large schools;
|
0.18 |
0.32 |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Combination of standardized scores on Hodder Access Reading Test and General Certificate of Secondary Education English test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Full sample;
|
0.05 |
0.06 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Combination of standardized scores on Hodder Access Reading Test and General Certificate of Secondary Education English test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Low achievement;
|
-0.91 |
-0.97 |
Yes |
|
||
|
Combination of standardized scores on Hodder Access Reading Test and General Certificate of Secondary Education English test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Key stage 3 (years 7 and 9);
|
0.08 |
0.03 |
No |
-- | ||
|
Combination of standardized scores on Hodder Access Reading Test and General Certificate of Secondary Education English test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Female;
|
0.14 |
0.08 |
No |
-- | ||
|
Combination of standardized scores on Hodder Access Reading Test and General Certificate of Secondary Education English test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Large schools;
|
0.26 |
0.22 |
Yes |
|
||
|
Combination of standardized scores on Hodder Access Reading Test and General Certificate of Secondary Education English test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Non-English learners;
|
0.15 |
0.11 |
No |
-- | ||
|
Combination of standardized scores on Hodder Access Reading Test and General Certificate of Secondary Education English test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Male;
|
0.01 |
0.01 |
No |
-- | ||
|
Combination of standardized scores on Hodder Access Reading Test and General Certificate of Secondary Education English test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Small schools;
|
-0.05 |
-0.03 |
No |
-- | ||
|
Combination of standardized scores on Hodder Access Reading Test and General Certificate of Secondary Education English test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Key stage 4 (year 11);
|
0.12 |
0.17 |
No |
-- | ||
|
Combination of standardized scores on Hodder Access Reading Test and General Certificate of Secondary Education English test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
High achievement;
|
0.87 |
0.91 |
No |
-- | ||
|
Combination of standardized scores on Hodder Access Reading Test and General Certificate of Secondary Education English test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Free school meals;
|
-0.29 |
-0.18 |
No |
-- | ||
|
Combination of standardized scores on Hodder Access Reading Test and General Certificate of Secondary Education English test |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
English learners;
|
-0.31 |
-0.21 |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Number of days present (standardized) |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Full sample;
|
0.04 |
-0.01 |
Yes |
|
|
|
| Show Supplemental Findings | |||||||||
|
Number of days present (standardized) |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Key stage 4 (year 11);
|
-0.05 |
-0.17 |
Yes |
|
||
|
Number of days present (standardized) |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Female;
|
-0.02 |
-0.04 |
No |
-- | ||
|
Number of days present (standardized) |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Non-English learners;
|
0.00 |
-0.03 |
No |
-- | ||
|
Number of days present (standardized) |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Male;
|
0.02 |
0.01 |
No |
-- | ||
|
Number of days present (standardized) |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
High achievement;
|
0.03 |
0.00 |
No |
-- | ||
|
Number of days present (standardized) |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Free school meals;
|
-0.42 |
-0.37 |
No |
-- | ||
|
Number of days present (standardized) |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
Key stage 3 (years 7 and 9);
|
0.03 |
0.05 |
No |
-- | ||
|
Number of days present (standardized) |
Parent Engagement Project (PEP) vs. Business as usual |
0 Days |
English learners;
|
0.03 |
0.08 |
No |
-- | ||
Characteristics of study sample as reported by study author.
| Other or unknown | 30% | |
| White | 70% |
The study was conducted in 36 geographically dispersed secondary schools in England. Each school contained students from Key Stages (KS) 3 and 4 of the English educational system. Within participating schools, students in Years 7, 9, and 11 were eligible for inclusion in the study. KS3 includes Years 7 and 9, and students in these years are typically between 11 and 14 years old. KS4 includes Year 11, and students in Year 11 are typically 15 to 16 years old.
The analytic sample included 29 schools and 58 KS groups (29 KS groups in the intervention condition and 29 KS groups in the comparison condition). For the absenteeism outcome, the analytic sample included 7,436 students in the intervention group and 7,919 in the comparison group. For the general mathematics achievement outcome, the analytic sample included 5,613 students in the intervention group and 5,977 students in the comparison group. For the general literacy achievement outcome, the analytic sample included 5,376 students in the intervention group and 6,037 students in the comparison group. In both the intervention and comparison groups, 55% of students were male. 14% of students in the intervention group and 18% of students in the comparison group were eligible for free school meals. 17% of students in the intervention group and 19% of students in the comparison group were English as an Additional Language (EAL) status. 18% of students in the intervention group and 20% of students in the comparison group were characterized as having Special Educational Needs (SEN). 73% of students in the intervention group and 67% of students in the comparison group were "White British." 57% of students in the intervention group and 77% of students in the comparison group were in KS3.
The intervention is the Parent Engagement Project (PEP), an intervention designed to improve student outcomes through increased parent engagement by text message. It is a year-long, school-level intervention. The intervention consists of text messages sent directly by schools to parents at specific times and intervals during the school year. The types of text messages sent included advance notice of tests and important deadlines (4 days and 1 day in advance); notice of missing homework; and summaries of daily lessons, with conversational prompts (rotated weekly across the subjects of math, reading, and science). The text messages were designed to be sent by schools via their existing school information management systems. The intervention was intended to send approximately 65 texts to parents throughout the year; the average number texts actually sent to each parent was 30, with a range from 15 to 77. An average of 22.19 texts were sent about upcoming assessments, 11.39 texts were send regarding missing homework, and 5.08 texts were sent with conversational prompts.
The comparison condition was "business as usual." Parents received standard school-level communication via existing school information management systems, email, telephone calls, letters, and student homework diaries/planners. The comparison condition did not receive any text messaging.
The intervention is intended to be used with existing school information management systems. Implementing schools and teachers received an initial training for a full day in September 2014 and ongoing support throughout the evaluation. An additional half-day of training was held in January 2015. Detailed instructions were provided on the content of the text messages. Each school appointed one staff member to be the project liaison officer, who was responsible for coordinating the text messages within the school. The study team also employed research assistants who were responsible for overseeing the implementation of PEP across schools. Each research assistant worked with a group of project liaison officers to facilitate consistent implementation across schools, including obtaining test dates and conversation prompts. More than 70% of school personnel reported positive reactions to the training received and being confident about implementing the intervention. Video recordings of the initial training were made available to all schools.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).
Resources
Newsflash
Contact Us
Facebook
Twitter
YouTube