The James Madison Legacy project: Evaluation report
Owen, D., Hartzell, K., & Sanchez, C. (2020). Georgetown University. https://cerl.georgetown.edu/james-madison-legacy-project-evaluation-report/.
-
examining38,074Students, grades6-12
Grant Competition
Review Details
Reviewed: December 2024
- Grant Competition (findings for James Madison Legacy Project (JMLP))
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a cluster randomized controlled trial with low cluster-level attrition and individual-level non-response.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Researcher-developed student civic knowledge test |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 2: Traditional vs Comparison - Middle School;
|
13.77 |
11.31 |
Yes |
|
|
|
|
Researcher-developed student civic knowledge test |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 3: Traditional vs Comparison - High School;
|
16.69 |
13.75 |
Yes |
|
|
|
|
Researcher-developed student civic knowledge test |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 3: Traditional vs Comparison - Middle School;
|
12.76 |
11.16 |
Yes |
|
|
|
|
Researcher-developed student civic knowledge test |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 2: Hybrid vs Comparison - Middle School;
|
13.06 |
11.31 |
Yes |
|
|
|
|
Researcher-developed student civic knowledge test |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 2: Hybrid vs Comparison - High School;
|
17.75 |
15.44 |
Yes |
|
|
|
|
Student Civic Knowledge Test - High School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Aggregated sample: Traditional vs Control - High School;
|
17.15 |
15.05 |
Yes |
|
|
|
|
Researcher-developed student civic knowledge test |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 1: Traditional vs Comparison - Middle School;
|
12.75 |
11.17 |
No |
-- | ||
|
Researcher-developed student civic knowledge test |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 1: Traditional vs Comparison - High School;
|
17.43 |
15.39 |
Yes |
|
|
|
|
Researcher-developed student civic knowledge test |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 3: Blended vs Comparison - High School;
|
15.87 |
13.75 |
Yes |
|
|
|
|
Researcher-developed student civic knowledge test |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 2: Traditional vs Comparison - High School;
|
17.28 |
15.44 |
Yes |
|
|
|
|
Student Civic Knowledge Test - Middle School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Aggregated sample: Traditional vs Control - Middle School;
|
12.66 |
11.53 |
Yes |
|
|
|
|
Researcher-developed student civic knowledge test |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 3: Blended vs Comparison - Middle School;
|
11.86 |
11.16 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Student Civic Knowledge Test - Middle School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 2: Traditional vs Control - Middle School;
|
13.77 |
11.91 |
Yes |
|
||
|
Student Civic Knowledge Test - High School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 3: Traditional vs Control - High School;
|
16.69 |
13.62 |
Yes |
|
||
|
Student Civic Knowledge Test - Middle School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 3: Traditional vs Control - Middle School;
|
12.76 |
10.98 |
Yes |
|
||
|
Student Civic Knowledge Test - Middle School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 2: Hybrid vs Control - Middle School;
|
13.06 |
11.91 |
Yes |
|
||
|
Student Civic Knowledge Test - High School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 2: Hybrid vs Control - High School;
|
17.75 |
16.17 |
Yes |
|
||
|
Student Civic Knowledge Test - High School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 3: Blended vs Control - High School;
|
15.87 |
13.62 |
Yes |
|
||
|
Student Civic Knowledge Test - Middle School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 1: Traditional vs Control - Middle School;
|
12.75 |
11.66 |
No |
-- | ||
|
Student Civic Knowledge Test - High School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 1: Traditional vs Control - High School;
|
17.43 |
15.44 |
Yes |
|
||
|
Student Civic Knowledge Test - High School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 2: Traditional vs Control - High School;
|
17.28 |
16.17 |
Yes |
|
||
|
Researcher-developed student civic knowledge test |
James Madison Legacy Project (JMLP) vs. Intervention |
0 Days |
Cohort 3: Traditional vs Blended - Middle School;
|
12.76 |
11.86 |
No |
-- | ||
|
Student Civic Knowledge Test - Middle School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Business as usual |
0 Days |
Cohort 3: Blended vs Control - Middle School;
|
11.86 |
10.98 |
No |
-- | ||
|
Researcher-developed student civic knowledge test |
James Madison Legacy Project (JMLP) vs. Intervention |
0 Days |
Cohort 2: Traditional vs Hybrid - Middle School;
|
13.77 |
13.06 |
No |
-- | ||
|
Researcher-developed student civic knowledge test |
James Madison Legacy Project (JMLP) vs. Intervention |
0 Days |
Cohort 3: Traditional vs Blended - High School;
|
16.69 |
15.87 |
No |
-- | ||
|
Student Civic Knowledge Test - High School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Intervention |
0 Days |
Cohort 2: Hybrid vs Traditional - High School;
|
17.75 |
16.79 |
No |
-- | ||
|
Researcher-developed student civic knowledge test |
James Madison Legacy Project (JMLP) vs. Intervention |
0 Days |
Cohort 2: Traditional vs Hybrid - High School;
|
17.28 |
17.75 |
No |
-- | ||
|
Student Civic Knowledge Test - High School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Intervention |
0 Days |
Cohort 3: Blended vs Traditional - High School;
|
15.87 |
16.99 |
No |
-- | ||
|
Student Civic Knowledge Test - Middle School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Intervention |
0 Days |
Cohort 2: Hybrid vs Traditional - Middle School;
|
13.06 |
13.20 |
No |
-- | ||
|
Student Civic Knowledge Test - Middle School (Owens et al., 2020) |
James Madison Legacy Project (JMLP) vs. Intervention |
0 Days |
Cohort 3: Blended vs Traditional- Middle School;
|
11.86 |
12.26 |
No |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 50%
Male: 50% -
Rural, Suburban, Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Alaska, Alabama, Arkansas, Arizona, California, Colorado, Connecticut, District of Columbia, Delaware, Florida, Georgia, Hawaii, Iowa, Idaho, Illinois, Indiana, Kansas, Kentucky, Louisiana, Massachusetts, Maryland, Maine, Michigan, Minnesota, Missouri, Montana, North Carolina, North Dakota, Nebraska, New Hampshire, New Jersey, New Mexico, Nevada, New York, Ohio, Oklahoma, Oregon, Pennsylvania, Rhode Island, South Carolina, South Dakota, Tennessee, Texas, Virginia, Vermont, Washington, Wisconsin, West Virginia, Wyoming
-
Race Asian 6% Black 12% Native American 2% Other or unknown 20% Pacific Islander 1% Two or more races 9% White 51% -
Ethnicity Hispanic 20% Other or unknown 80% -
Eligible for Free and Reduced Price Lunch Other or unknown 100%
Study Details
Setting
The study was conducted in eligible middle and high schools across 46 states and the District of Columbia, in urban, suburban, and rural areas. The schools served high-need students and offered civics, social studies, or American government courses.
Study sample
Schools in the first cohort were randomly assigned to the traditional James Madison Legacy Project (JMLP) model, where experts meet in-person with teachers or to the business-as-usual comparison condition. Schools in the second cohort were randomly assigned to the traditional JMLP model, a hybrid model (combination of in-person meetings and videos), or the comparison condition. Schools in the third cohort were randomly assigned to the traditional JMLP model, a blended model (videos only), or the comparison condition. The analytic sample sizes were as follows: (1) Middle schools in Cohort 1: traditional: 15 schools, 1,225 students; comparison: 12 schools, 499 students; (2) High schools in Cohort 1: traditional: 23 schools, 2,232 students; comparison: 22 schools, 1,254 students; (3) Middle schools in Cohort 2: traditional: 27 schools, 1,627 students; hybrid: 28 schools, 1,039 students; comparison: 25 schools, 1,314 students; (4) High schools in Cohort 2: traditional: 46 schools, 3,232 students; hybrid: 47 schools, 2,770 students; comparison: 43 schools, 3,202 students; (5) Middle schools in Cohort 3: traditional: 29 schools, 2,913 students; blended: 27 schools, 2,587 students; comparison: 26 schools, 1,049 students; and (6) High schools in Cohort 3: traditional: 45 schools, 5,473 students; blended: 46 schools, 4,926 students; comparison: 41 schools, 2,732 students. Averaging across cohort and grade level, the sample characteristics of the participating students are as follows. Half of the sample was male and the other half female. Just over half (51%) were White, with 12 percent Black, 6 percent Asian, 2 percent Native American/Alaskan Native, 1 percent Native Hawaiian/Pacific Islander, 9 percent multiple races, and the remainder unknown. Twenty percent of participants reported their ethnicity as Hispanic.
Intervention Group
The James Madison Legacy Project (JMLP) consisted of a civics curriculum paired with professional development (PD). The curriculum was developed by the Center for Civic Education, called "We the People: The Citizen and the Constitution (WTP)." Teachers were required to teach at least 40 hours of the WTP curriculum, and students participated in simulated congressional hearings. Three variations of the JMLP PD were evaluated in this report. Teachers who participated in the Traditional PD model met with experts in person and listened to lectures on civics content. The experts answered questions and facilitated discussion. Teachers who participated in the Hybrid PD model viewed interactive, online video interviews with experts who have experience teaching the WTP PD program. These videos presented content from a subset of the units, and experts lectured on the content from the remaining units in person. Teachers who participated in the Blended PD model received video-recorded lectures from all of the units. All intervention teachers were also asked to participate in the professional learning community and online discussions. Participating teachers were selected by school administrators based on their perceived disposition toward student success, their qualifications for teaching civics, and their commitment to the program's goals.
Comparison Group
The business-as-usual control group included schools that offered civics, social studies, or American government courses and teachers who had no prior experience with the WTP curriculum and did not receive any of the JMLP PD program.
Support for implementation
Support for the intervention consisted of a 4- to 5-day in-person regional summer institute for teachers in Cohorts 2 and 3. Teachers in Cohort 1 received shorter meetings throughout the academic year in lieu of the summer institute. and follow-up professional development for teachers. Teachers also participated in additional 1- to 2-day PD sessions throughout the academic year, mostly in person. Coordinators and mentor teachers attended 2-day preparatory meetings prior to the start of each cohort, to learn about the research aspects of the project Teachers also received a classroom set of WTP books and instructional resources, travel stipends, a substitute reimbursement, and a stipend.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).