Study: 89802

Department-funded evaluation

Review Details

Reviewed: September 2022

At least one finding shows promising evidence of effectiveness

At least one statistically significant positive finding

Meets WWC standards with reservations

For:

Department-funded evaluation (findings for Learning by Making)

Using:

Rating:

Meets WWC standards with reservations because it uses a cluster quasi-experimental design that provides evidence of effects on individuals by satisfying the baseline equivalence requirement for the individuals in the analytic intervention and comparison groups.

This review may not reflect the full body of research evidence for this intervention.

Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Findings

General Mathematics Achievement outcomes—Indeterminate effect found for the domain
Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement index	Evidence tier
LbyM Mathematics	Learning by Making vs. Business as usual	0 Days	Full sample; 150 students	50.73	46.21	No	--

General science achievement outcomes—Statistically significant positive effect found for the domain
Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement index	Evidence tier
LbyM Science	Learning by Making vs. Business as usual	0 Days	Full sample; 150 students	49.36	41.49	Yes	15

Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Sample Characteristics

Characteristics of study sample as reported by study author.

13% English language learners
Other or unknown: 100%
Rural
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
California

Race
Asian		1%
Black		1%
Native American		14%
Other or unknown		42%
White		42%

Ethnicity
Hispanic		41%
Not Hispanic or Latino		59%

Eligible for Free and Reduced Price Lunch
Free or reduced price lunch (FRPL)		61%
Other or unknown		39%

Study Details

Setting

The study of the LbyM STEM 1 project took place in the 2016-2017 academic year. High school students were recruited from high-need rural high schools in Mendocino County, California. These six high-need rural high schools in Mendocino County primarily serve families with high rates of family poverty and low rates of college readiness.

Study sample

Across participating schools, 41% of students were Hispanic or Latino, 1% were African American, 42% were white, 1% were Asian or Pacific Islander, 14% were Native American, and 1% were two or more races. Sixty-one percent of students were eligible for free or reduced price lunch, and 13% were English learners.

Intervention Group

The Learning by Making (LbyM) curriculum integrates the NGSS Scientific and Engineering Design Practices and CCSS-M Standards for Mathematical Practice into a modified project-based learning approach whereby the use of mathematics is integrated with science content and technology as students do experiments related to real-world problems. During LbyM investigations, students write Logo code to read the sensors and obtain data, building on basic coding to perform individualized experiments, create simulations, and explore models. Students also analyze and interpret data while using mathematics and engaging in computational thinking without the difficulties usually encountered in learning complex programming languages. LbyM is implemented via professional development for teachers and the Learning by Making curriculum, which aims to improve teachers’ instructional competence, students’ engagement, students’ interest and confidence in STEM, and teachers’ instructional practices. The intended results of these changes include greater student high school math and science achievement, increased student college qualification and STEM majors chosen, increased college acceptance and completion, and increased STEM professionals from under-represented and high-need populations.

Comparison Group

Comparison students used business-as-usual mathematics and science curricula. No additional information is provided by the evaluators on the curriculum or professional development that constitutes business as usual, but students were likely exposed to instruction and support services as they had in the past.

Support for implementation

No implementation support was described separate from the intervention components, which provided professional development and the Learning by Making curriculum to teachers.

Evaluation of Learning by Making i3 Project: STEM Success for Rural Schools