The objective of this guide is to provide teachers with specific recommendations that can be carried out in the classroom without requiring systemic change.

Science Teachers Learning through Lesson Analysis (STeLLA®) is a professional development program, developed by BSCS Science Learning, that aims to improve students’ science achievement by improving teachers’ science content knowledge and their abilities to (a) explain science concepts to students, (b) clearly identify to students the science concepts used in student learning activities, and (c) engage students in thinking about science.

Great Explorations in Math and Science^® (GEMS^®) Space Science Sequence is an instructional curriculum for grades 3–5 that covers fundamental concepts, including planetary sizes and distance, the earth’s shape and movement, gravity, and moon phases and eclipses. Part of the GEMS^® core curriculum, GEMS^® Space Science Sequence uses the solar system as the focal point for learning. The sequence uses models, hands-on investigations, peer-to-peer discussions, reflection, and informational student readings. Students complete four units, each lasting between four and nine sessions. Each unit builds upon knowledge from previous units and can be used independently or in conjunction with one another for an overall learning progression.

Technology Enhanced Elementary and Middle School Science (TEEMSS) is a physical science curriculum for grades 3–8 that uses computers, sensors, and interactive models to support investigations of real-world phenomena. Through 15 inquiry-based instructional units, students interact with computers, gather and analyze data, and formulate ideas for further exploration. This information is managed by software in a handheld computer and transmitted to other students and to the teacher. The program includes a web-based teacher-reporting tool that allows teachers to review student portfolios and gather student responses for assessment and class discussion.

The Leadership and Assistance for Science Education Reform (LASER) program is intended to build capacity for implementing inquiry-based science curricula in schools and districts. When participating in LASER, school or district teams attend leadership development institutes to plan the implementation of inquiry-based science curricula. These school or district teams receive support for key aspects of implementation such as professional development for teachers, access to instructional materials, and support for selecting appropriate assessments. LASER also helps schools and districts partner with scientists, science educators, and local business and community leaders who can promote and further support the implementation of inquiry-based science instruction.

Full Option Science System™ (FOSS) is a science curriculum for students in kindergarten to grade 8 with content in physical science, earth science, and life science. The curriculum consists of a series of 8- to 9-week modules in kindergarten to grade 5, and 9- or 18-week courses in grades 6 to 8.

Project-Based Inquiry Science™ is a science curriculum for students in grades 6–8 with approximately 13 independent instructional units, each covering a topic in life science, earth science, or physical science. Students work together in small groups to conduct hands-on explorations or investigations, read relevant informational texts, reflect on what they have learned, and apply new knowledge.

Great Explorations in Math and Science® (GEMS®) The Real Reasons for Seasons is a curriculum unit for grades 6–8 that focuses on the connections between the Sun and the Earth to teach students the scientific concepts behind the seasons. The unit utilizes models, hands-on investigations, peer-to-peer discussions, reflection, and informational student readings to help students understand science content and develop scientific investigation skills.

The LeTUS program is a three-year, project-based, technology-integrated middle school science curriculum for grades 6–8. The LeTUS program is composed of multiple units, each lasting between eight and ten weeks. Topics include global warming, water and air quality, force and motion, communicable diseases, and ecological systems. The units are designed around projects through which students learn science by conducting scientific investigations and using interactive computer software along with scientific visualization and graphing tools. Each unit stresses inquiry, student collaboration, and the use of computing and communications technologies. The sequence of units can be used in different ways, depending on standards and curriculum requirements; for example, teachers can use units at grade levels other than those suggested. Each unit can also be used independently when inserted into a different curricular context.

ARIES: Exploring Motion and Forces is a physical science curriculum for students in grades 5–8 that employs 18 inquiry-centered, hands-on lessons called “explorations.” The curriculum draws upon students’ curiosity to explore phenomena, allowing for a discovery-based learning process. Group-centered lab work is designed to help students build an understanding of inertia, friction, gravity, speed, and acceleration. Students examine their prior ideas about the phenomena, formulate questions, build and use an apparatus to observe natural phenomena, make predictions, and gather data through structured experiments. Exploring Motion and Forces is part of the ARIES sequence of eight physical science units. The ARIES sequences can be used together for an overall curriculum or independently.

ThinkerTools is a computer-based program that aims to develop students’ understanding of physics and scientific modeling. The program is composed of two curricula for middle school students, ThinkerTools Inquiry and Model-Enhanced ThinkerTools. ThinkerTools Inquiry allows students to explore the physics of motion and then asks them to apply that knowledge to solve real-world problems. In the Model-Enhanced ThinkerTools curriculum, students create computer models that express their own theories of force and motion.

Chemistry That Applies is an instructional unit designed to help students in grades 8–10 understand the law of conservation of matter. It consists of 24 lessons organized in four clusters. Working in groups, students explore four chemical reactions: burning, rusting, the decomposition of water, and the reaction of baking soda and vinegar. As part of the unit, students conduct experiments in which they cause these reactions to happen, obtain and record data in individual notebooks, analyze the data, and use evidence-based arguments to explain the data. The instructional unit engages the students in a structured sequence of hands-on laboratory investigations interwoven with other forms of instruction.

The Integrated Mathematics, Science, and Technology (IMaST) program is a 6th, 7th, and 8th grade curriculum that promotes both hands-on learning for students and teamwork among teachers from different disciplines. IMaST emphasizes learning based on constructivist theory and active student participation involving a hands-on approach. Coordination of a wide variety of activities helps students grasp the many natural interdisciplinary connections in the curriculum. A team of mathematics, science, and technology specialists, in collaboration with other field experts, did the research to create the curriculum, built on major themes that are presented in modules. Each theme develops the focus of all disciplines in relation to several key concepts that lead toward the same objective. IMaST has developed its activities using benchmarks, national standards, and state frameworks.

This study examines the efficacy, cost, and implementation of an integrated science and literacy curriculum for kindergarten. The study was conducted in a large urban district and included 1,589 students in 71 classrooms in 21 schools. The research includes a multi-site cluster-randomized controlled trial and mixed-methods cost and implementation studies. Analysis revealed significant impacts on comprehension, letter-naming fluency, and motivation to read. No main impacts were observed on decoding, word identification, or writing; however, exploratory analysis revealed that students whose teachers implemented the treatment with fidelity performed statistically significantly better in writing and decoding. The cost to produce the observed effects was estimated at $480 per student, two-thirds of which was borne by the school. Despite this cost, treatment classrooms achieved savings by using an average of three fewer instructional programs than control classrooms. Teachers reported positive effects from the integrated curriculum on student engagement, learning, and behavior.

Mission HydroSci (MHS) is a 3D game-based learning environment and curriculum that supports middle school student learning of water systems science and scientific argumentation. MHS is a rigorous, coherent and engaging 6 to 8-day curriculum with all learning activities and social interactions taking place in the virtual world and with teachers observing and supporting students through an online support system enhanced by analytics. MHS was evaluated in comparison to a high- quality alternative intervention developed by the Biological Sciences Curriculum Study (BSCS) using a stratified randomized block experimental design where 'classroom' was the unit of random assignment, stratified by teacher. The comparison curriculum is called Earth's Water Systems (EWS) and is provided online using the Canvas learning management platform. Three measurable outcomes: (1) content knowledge, (2) competency in scientific argumentation, and (3) affect for science and technology were used in the pre- post-comparison of MHS with EWS. The findings of this randomized experiment showed that MHS achieved roughly equivalent water systems learning outcomes and significantly higher development of argumentation competencies when compared to the EWS curriculum. The impacts of both MHS and the EWS curriculum on affect for science and technology were equivalent and slightly negative. A secondary exploratory quasi-experimental design (QED) analysis was conducted that found significant positive effects for MHS in comparison to EWS on water systems understandings and stronger detected effects for students' argumentation.

This article describes the effects of an analysis-of-practice professional development (PD) program on elementary school students' (Grades 4-6) science outcomes. The study design was a cluster-randomized trial with an analysis sample of 77 schools, 144 teachers and 2,823 students. Forty-two schools were randomly assigned to treatment, (88.5 hours) of integrated analysis-of-practice and content deepening PD (over the course of one year) while 35 schools were randomly assigned to receive an equal number of PD hours in science content deepening alone. Students' content knowledge, as measured by a project-specific test, was compared across treatment groups. The effect size for this comparison was 0.52 standard deviations in favor of students whose teachers participated in the PD that included analysis-of-practice. This effect compares favorably to that of other elementary school interventions whose effectiveness was studied with a narrowly focused outcome measure. Analysis of the demographics of the study schools suggests that the treatment effect could be relevant outside the local study context. Implications for future research include tests of mediation for teacher-level outcomes and efficacy tests of specific teaching strategies (intervention subcomponents).

With national focus on reading and math achievement, science and social studies have received less instructional time. Yet, accumulating evidence suggests that content knowledge is an important predictor of proficient reading. Starting with a design study, we developed content-area literacy instruction (CALI) as an individualized (or personalized) instructional program for kindergarteners through 4th graders to build science and social studies knowledge. We developed CALI to be implemented in general education classrooms, over multiple iterations (n = 230 students), using principles of design-based implementation research. The aims were to develop CALI as a usable and feasible instructional program that would, potentially, improve science and social studies knowledge, and could be implemented during the literacy block without negatively affecting students' reading gains (i.e., no opportunity cost). We then evaluated the efficacy of CALI in a randomized controlled field trial with 418 students in kindergarten through 4th grade. Results reveal that CALI demonstrates promise as a usable and feasible instructional individualized general education program, and is efficacious in improving social studies (d = 2.2) and science (d = 2.1) knowledge, with some evidence of improving oral and reading comprehension skills (d = 0.125).

This study examined the efficacy of a curriculum-based intervention for high school science students. Specifically, the intervention was two years of research-based, multidisciplinary curriculum materials for science supported by comprehensive professional development for teachers that focused on those materials. A modest positive effect was detected when comparing outcomes from this intervention to those of business-as-usual materials and professional development. However, this effect was typical for interventions at this grade span that are tested using a state achievement test. Tests of mediation suggest a large treatment effect on teachers and in turn a strong effect of teacher practice on student achievement--reinforcing the hypothesized key role of teacher practice. Tests of moderation indicate no significant treatment by demographic interactions.

This report presents the results of an experiment conducted in Alabama beginning in the 2006/07 school year, to determine the effectiveness of the Alabama Math, Science, and Technology Initiative (AMSTI), which aims to improve mathematics and science achievement in the state's K-12 schools. This study is the first randomized controlled trial testing the effectiveness of AMSTI in improving mathematics problem solving and science achievement in upper-elementary and middle schools. AMSTI is an initiative specific to Alabama and was developed and supported through state resources. An important finding is the positive and statistically significant effect of AMSTI on mathematics achievement as measured by the SAT 10 mathematics problem solving assessment administered by the state to students in grades 4-8. After one year in the program, student mathematics scores were higher than those of a control group that did not receive AMSTI by 0.05 standard deviation, equivalent to 2 percentile points. Nine of the 10 sensitivity analyses yielded effect estimates that were statistically significant at the 0.025 level, consistent with the main finding. The estimated effect of AMSTI on science achievement measured after one year was not statistically significant. Based on the SAT 10 science test administered by the state to students in grades 5 and 7, no difference between AMSTI and control schools could be discerned after one year. Changes in classroom instructional strategies, especially an emphasis on more active-learning strategies, are important to the AMSTI theory of action. Therefore, a secondary investigation of classroom practices was conducted, based on data from survey responses from teachers. For both mathematics and science, statistically significant differences were found between AMSTI and control teachers in the average reported time spent using the strategies. The effect of AMSTI on these instructional strategies was 0.47 standard deviation in mathematics and 0.32 standard deviation in science. Two years of AMSTI appeared to have a positive and statistically significant effect on achievement in mathematics problem solving, compared to no AMSTI. Two years of AMSTI appeared to have a positive and statistically significant effect on achievement in science. AMSTI appeared to have a positive and statistically significant effect on reading achievement as measured by the SAT 10 test of reading administered by the state to students in grades 4-8. AMSTI did not appear to have a statistically significant effect on teacher-reported content knowledge in mathematics or science after one year. AMSTI did not appear to have statistically significant differential effects on student achievement in mathematics problem solving or science based on racial/ethnic minority status, enrollment in the free or reduced-price lunch program, gender, or pretest level. Appended are: (1) Explanation of primary and secondary confirmatory outcome measures; (2) Explanation of exploratory research questions; (3) Selection and random assignment of schools; (4) Statistical power analysis; (5) Data collection procedures and timeline; (6) Description of program implementation data collected but not used in report; (7) Alabama Math, Science, and Technology Initiative (AMSTI) teacher survey #3; (8) Data cleaning and data file construction; (9) Attrition through study stages for samples used in the confirmatory analysis; (10) Description of degree rank; (11) Equivalence of Year 1 baseline and analyzed samples for confirmatory student-level and classroom practice outcomes; (12) Internal consistency and validity of active learning measures; (13) Number of students and teachers in schools in analytic samples used to analyze Year 1 confirmatory questions; (14) Attrition through study stages for samples used in Year 1 exploratory analysis; (15) Tests of equivalence for baseline and analytic samples for Year 1 exploratory outcomes; (16) Statistical power analyses for moderator analyses; (17) Derivation and motivation of the Bell-Bradley estimator when measuring estimated two-year effect of the Alabama Math, Science, and Technology Initiative (AMSTI); (18) Attrition through study stages for samples contributing to estimation of two-year effects; (19) Examination of equivalence in baseline and analytic samples used in the estimation of two-year effects; (20) Estimation model for two-year effects of the Alabama Math, Science, and Technology Initiative (AMSTI); (21) Topics and instructional methods used at the Alabama Math, Science, and Technology Initiative (AMSTI) summer institute; (22) Parameter estimates on probability scale for odds-ratio tests of differences between Alabama Math, Science, and Technology Initiative (AMSTI) and control conditions in Year 1 (associated with summer professional development and in-school support outcomes); (23) Descriptive statistics for variables that change to a binary scale used in the Alabama Math, Science, and Technology Initiative (AMSTI) and control conditions in Year 1; (24) Comparison of assumed parameter values and observed sample statistics for statistical power analysis after one year; (25) Parameter estimates for Stanford Achievement Test Tenth Edition (SAT 10) mathematics problem solving after one year; (26) Parameter estimates for Stanford Achievement Test Tenth Edition (SAT 10) science after one year; (27) Parameter estimates for active learning in mathematics after one year; (28) Parameter estimates for active learning in science after one year; (29) Sensitivity analyses of effect of the Alabama Math, Science, and Technology Initiative (AMSTI) on Stanford Achievement Test Tenth Edition (SAT 10) mathematics problem solving achievement after one year; (30) Sensitivity analyses of effect of the Alabama Math, Science, and Technology Initiative (AMSTI) on Stanford Achievement Test Tenth Edition (SAT 10) science achievement after one year; (31) Sensitivity analyses of effect of the Alabama Math, Science, and Technology Initiative (AMSTI) on active learning instructional strategies in mathematics classrooms after one year; (32) Sensitivity analyses of effect of the Alabama Math, Science, and Technology Initiative (AMSTI) on active learning instructional strategies in science classrooms after one year; (33) Tests for violations of factors associated with assumption of equal first year effects on students in Alabama Math, Science, and Technology Initiative (AMSTI) and control schools; (34) Post hoc adjustment to standard error for estimate of two-year effect of the Alabama Math, Science, and Technology Initiative (AMSTI) on mathematics achievement after two years; (35) Parameter estimates for effect of the Alabama Math, Science, and Technology Initiative (AMSTI) after two years; (36) Parameter estimates for effect of the Alabama Math, Science, and Technology Initiative (AMSTI) on student reading achievement after one year; (37) Parameter estimates for teacher content and student engagement after one year; (38) Estimates of effects for terms involving the indicator of treatment status in the analysis of the moderating effect of the three-level pretest variable; (39) Parameter estimates for the analysis of the moderating effect of racial/ethnic minority status on the impact of the Alabama Math, Science, and Technology Initiative (AMSTI) on reading after one year; (40) Parameter estimates for analysis of average effect of the Alabama Math, Science, and Technology Initiative (AMSTI) on reading by racial/ethnic minority students after one year; and (41) Parameter estimates for effect of the Alabama Math, Science, and Technology Initiative (AMSTI) on reading for White students after one year. (Contains 26 figures, 136 tables, 1 box and 130 footnotes.)

To identify links among professional development, teacher knowledge, practice, and student achievement, researchers have called for study designs that allow causal inferences and that examine relationships among features of interventions and multiple outcomes. In a randomized experiment implemented in six states with over 270 elementary teachers and 7,000 students, this project compared three related but systematically varied teacher interventions--"Teaching Cases, Looking at Student Work, and Metacognitive Analysis"--along with no-treatment controls. The three courses contained identical science content components, but differed in the ways they incorporated analysis of learner thinking and of teaching, making it possible to measure effects of these features on teacher and student outcomes. Interventions were delivered by staff developers trained to lead the teacher courses in their regions. Each course improved teachers' and students' scores on selected-response science tests well beyond those of controls, and effects were maintained a year later. Student achievement also improved significantly for English language learners in both the study year and follow-up, and treatment effects did not differ based on sex or race/ethnicity. However, only Teaching Cases and Looking at Student Work courses improved the accuracy and completeness of students' written justifications of test answers in the follow-up, and only Teaching Cases had sustained effects on teachers' written justifications. Thus, the content component in common across the three courses had powerful effects on teachers' and students' ability to choose correct test answers, but their ability to explain why answers were correct only improved when the professional development incorporated analysis of student conceptual understandings and implications for instruction; metacognitive analysis of teachers' own learning did not improve student justifications either year. Findings suggest investing in professional development that integrates content learning with analysis of student learning and teaching rather than advanced content or teacher metacognition alone. (Contains 1 figure and 4 tables.)

This project entailed a three-year efficacy evaluation of the Computer and Team Assisted Mathematical Acceleration (CATAMA) Lab developed by the Center for Social Organization of Schools at Johns Hopkins University. The CATAMA Lab was proposed as an immediate and practical approach to addressing the different types of math deficits held by students at urban high-poverty schools. The Lab required only 1 teacher per school reducing staff and professional development requirements. It used multiple instructional techniques (including individualized computer instruction, direct instruction, pair and team learning, and individual instruction) to teach math concepts and skills. By taking the place of an elective it allowed students to continue with their on-grade math class. For a more detailed description of the Lab see Appendix 2. The original goal of the project was to establish the Lab at three urban schools serving high-poverty high-minority middle grade students (grades 5-8). Students underperforming in mathematics (as established by district standardized tests) were to take a trimester course of study in the Lab to increase their knowledge of math concepts and skills taught by a regular math teacher receiving extensive ongoing professional development. Students were to take the Lab as an elective course while continuing with their regular math class. From each school's pool of students eligible to participate, students were to be randomly assigned to take the Lab. An implementation analysis was to measure the teaching of the concepts and skills to be taught in the Lab. To evaluate the impact of the intervention, students' math achievement, as measured by standardized math tests, was to be compared to eligible students not assigned to the Lab. This report discusses the project in three sections: (1) A comparison of the actual project with the planned project; (2) The descriptive results from the project; (a) Description of the sample; (b) Description of implementation of the CATAMA Lab; and (3) The evaluative results from the project.

Because standardized tests in science are not given to PreK-3 students in Ohio, this report examined the longitudinal effects of learning from a teacher who had participated in the NURTURES professional development program. Specifically, it looked at the effects on students' mathematics and reading learning in grades 2-5 and science learning in 5th grade in 2017. Students who were in 5th grade at that time could have had a NURTURES-trained teacher at any time between kindergarten and 3rd grade. Thus, the study followed students up to 5 years after having a NURTURES teacher. The sample included the population of students enrolled in the 41 elementary schools in the Toledo Public School District. Students who never learned from a teacher who participated in NURTURES served as the control group. The data came from the 2017 administration of the Ohio Measure of Academic Progress (MAP) (NWEA, 2019) for mathematics and reading and the Ohio Achievement Test in Science for science (Ohio Achievement Assessment, 2015). The total number of students from these schools who took the May 2017 reading MAP was 6759 and the total number who took mathematics was 6703. The number of those students who had at least one NURTURES teacher was 2801 (41.4%) for reading and 2707 (41.6%) for mathematics. Analysis of the reading scores showed 2.14 advantage points for NURTURES students as compared to the average non-intervention student to an annual growth rate of 7.02 units (p < .001). The treatment effect size (Hedges' g) was 0.12. For mathematics there were 1.55 advantage points to an annual growth rate estimated to be 8.17 units (p < .001) as compared to the average non-intervention student. The treatment effect size (Hedges' g) was 0.09. Analysis of the 5th grade Ohio Achievement Science Subtest showed that students associated with at least one NURTURES-trained teacher was modeled to have a 5.86 advantage points as compared to the average non-intervention student. The treatment effect size (Hedges' g) was 0.08, which is to be interpreted as a treatment group having, on average, 0.08 higher scores in standard deviation units as compared to the scores of the control cohort. When compared with our earlier evaluation report (2016; revised in 2018), we see that students who had a NURTURES-trained teacher, on average, continued to show greater gains compared to students who did not. In addition, the achievement gaps between non-minority and minority students in reading and mathematics were reduced when the minority students had a NURTURES-trained teacher and the non-minority students did not. In science, the impact of the intervention roughly compensated for the attainment gap between boys and girls and partially ameliorated the gap between minority and non-minority children's scores associated with these demographic factors. [Published May 6, 2020 with minor revisions based in WWC inquiries published January 5, 2024.]

Playground Physics is a technology-based application and accompanying curriculum designed by New York Hall of Science (NYSCI) to support middle school students' science engagement and learning of force, energy, and motion. The program includes professional development, the Playground Physics app, and a curriculum aligned with New York State Learning Standards, Common Core State Standards, and Next Generation Science Standards. The iOS app allows students to record and review videos through three "lenses": (1) motion; (2) force (Newton's third law); and (3) energy, and the curriculum integrates informal and formal, inquiry-based learning strategies to promote greater student knowledge and understanding of physics. The program was designed to be implemented in a formal school setting during the regular school day. This report describes the results of an experimental study of the Playground Physics program's impact on learning of physics concepts, student engagement, and science-related attitudes. Sixty New York City middle grade teachers were randomly assigned to treatment or control conditions. Treatment teachers were asked to participate in Playground Physics professional development and use Playground Physics as part of their physics instruction during the 2015-16 academic year; control teachers were asked to use their regular instruction. In total, 15 teachers left the study. The final sample included student data from 24 treatment teachers and 21 control teachers. The following are appended: (1) Playground Physics Curriculum Activities; (2) Student Outcome Measures; (3) Teacher Survey; (4) Impact Analysis Technical Approach; (5) Output from Statistical Models; (6) Knowledge Assessment Responses and Standards Alignment; (7) 2014-15 Fidelity of Implementation Analysis; and (8) Supplemental Analysis.

The National Math + Science Initiative's (NMSI's) College Readiness Program (CRP) is an established program whose goal is to promote science, technology, engineering, and mathematics education in high schools to improve students' readiness for college. It provides teacher, student, and school supports to promote high school students' success in mathematics, science, and English Advanced Placement (AP) courses, with a focus on students who are traditionally underrepresented in the targeted AP courses. Through a federal Investing in Innovation Fund (i3) validation grant awarded to NMSI in 2011, CRP was implemented in a total of 58 high schools in two states--Colorado and Indiana--beginning in the 2012-13 school year. American Institutes for Research (AIR) conducted an independent evaluation of the impacts of CRP on students' AP outcomes in these schools for the three cohorts of schools that adopted the program in sequential years, using a comparative interrupted time series (CITS) design that matched comparison schools to program schools in the two states. Overall, schools implementing CRP demonstrated significantly larger increases in the share of students taking and passing AP tests in targeted areas relative to comparison schools in each of the three cohorts of schools, and the gains in CRP schools were sustained over time. Fidelity of program implementation was evaluated using a fidelity matrix approach required as part of the National Evaluation of the i3 program, which showed that not all elements of the program were implemented with high fidelity. Teachers and students were not always able to attend all meetings, and schools did not always meet negotiated enrollment targets. Teacher survey data indicated that teachers found the professional development activities provided by CRP to be the most helpful support they received under CRP, and students reported that the tutoring and special study sessions were the most helpful. Although the program provided financial incentives to both teachers and students that were tied to student performance on AP tests, these incentives were considered the least important element of the program by both teachers and students

The purpose of this study was to evaluate the Literacy-Infused Science Using Technology Innovation Opportunity (LISTO) validation project (Valid 45). LISTO was funded by the Investing in Innovation (i3) Fund and involved a multi-year intervention that provided virtual professional development and coaching, and literacy-infused science curricula to fifth-grade science teachers who taught predominantly low-income students and in predominantly rural public schools in Texas. The overarching goal of LISTO is to validate, via a 5-year longitudinal randomized controlled trial (RCT) study, literacy-infused science (LIS) instructional and curricular innovations to increase instructional capacity of teachers and to improve students' science and reading/writing literacy achievement in rural/non-rural schools for economically challenged (EC), inclusive of English language learners (ELL) students. Outcomes collected in the 2017-18 school year were considered to be exploratory, given the timing of Hurricane Harvey, which impacted Texas in August of 2017. Outcomes in the 2018-19 school year served as the confirmatory contrasts. LISTO resulted in increased teacher capacity to implement research-based strategies while teaching science content, yet this improvement did not necessarily translate into improved student achievement in science or reading. The LISTO professional development and coaching covered pedagogical strategies for teaching science, including those that have been shown to improve literacy and be particularly effective for ELs. There was a negative impact on students' science achievement in both 2017-18 (ES = -0.10) and in 2018--19 (ES = -0.13). There was a negative program impact on students' science interest (ES = -0.14), as measured by a survey, in 2017-18, and no impact in 2018-19. These quantitative findings were in conflict with qualitative data collected from LISTO teachers, who indicated that the program led to improvements in both science vocabulary and engagement and self-efficacy in science for students. LISTO had positive effects on teacher practices for a subsample of teachers, specifically on increased delivery of research-based instruction to teach science content as rated on a rubric by external reviewers (ES = +1.12). LISTO appeared to improve instructional practices for a sample of teachers who implemented the program for two years with complete data but did not positively impact student or teacher outcomes more broadly. However, results should be cautiously interpreted due to limitations of delayed and incomplete implementation in the first year of the project due to Hurricane Harvey. Encouragingly, teachers' overall positive reactions to the program suggest its potential to improve student affect and learning, but more extensive implementation experience by teachers and multi-year exposure by students starting from early grades may be needed to yield measurable benefits. Clearly, such focuses emerge as a highly recommended topic for future research.

This paper presents the findings from a randomized control trial study of reading/literacy-integrated science inquiry intervention after 1 year of implementation and the treatment effect on 5th-grade low-socio-economic African-American and Hispanic students' achievement in science and English reading. A total of 94 treatment students and 194 comparison students from four randomized intermediate schools participated in the current project. The intervention consisted of ongoing professional development and specific instructional science lessons with inquiry-based learning, direct and explicit vocabulary instruction, and integration of reading and writing. Results suggested that (a) there was a significantly positive treatment effect as reflected in students' higher performance in district-wide curriculum-based tests of science and reading and standardized tests of science, reading, and English reading fluency; (b) males and females did not differ significantly from participating in science inquiry instruction; (c) African-American students had lower chance of sufficiently mastering the science concepts and achieving above the state standards when compared with Hispanic students across gender and condition, and (d) below-poverty African-American females are the most vulnerable group in science learning. Our study confirmed that even a modest amount of literacy integration in inquiry-based science instruction can promote students' science and reading achievement. Therefore, we call for more experimental research that focus on the quality of literacy-integrated science instruction from which middle grade students, particularly low-socio-economic status students, can benefit.

This research is part of a larger, IES-funded study titled: "Measuring the Efficacy and Student Achievement of Research-based Instructional Materials in High School Multidisciplinary Science" (Award # R305K060142). The larger study seeks to use a cluster-randomized trial design, with schools as the unit of assignment, to make causal inferences about the effect of treatment on both students and teachers. The research described in this report addresses the following research question associated with path "a" in Figure 1: (1) What is the mean difference in teacher outcome (i.e., instruction) across the treatment groups? (a) What is the effect size (practical significance)? (b) Is the difference statistically significant at the alpha = 0.05 level?; and (2) If practically or statistically significant differences in instruction exist across treatment groups, to what extent can the differences be attributed to the treatment (instructional materials and PD)? The research takes place in both suburban and rural high schools in the state of Washington. In particular, the suburban schools are clustered near Seattle/Tacoma and the rural schools are clustered near Yakima. The data from this analysis suggest that the PD treatment was more effective in fostering reform-oriented science instruction, on average, than was the extant PD experienced by the business-as-usual comparison group. This difference was both statistically and practically significant. Applying this result to the authors' hypothesis of mediation, they now have confidence that one of the causal paths (path a) that are necessary to argue mediation is trustworthy. Further study of path b is necessary to understand whether instruction is serving as a mediator of the treatment effect. That said, there is evidence in the literature suggesting that the possibility of a significant b path is quite real. For example, Hedges and Hedberg (2007) found that in school-level interventions, a considerable amount of the variance in outcomes was attributable to teacher and /or classroom effects. Threats to internal validity that are noteworthy include limitations in the authors' confidence that the post-intervention differences in RTOP scores were not pre-existing (i.e., not attributable to the treatment). Unfortunately, they did not have a baseline RTOP measure that could have served as a covariate in the main effect analysis of treatment. Use of such a covariate would have likely provided a more precise estimate of the treatment effect. Further, because the comparison group received business-as-usual PD, this experience was highly variable across teachers. The research team has only cursory knowledge of the nature and duration of extant PD experienced by the comparison group. As such, there is limited clarity in the PD experiences to which the treatment is being compared. In the context of an efficacy trial, external validity (i.e., generalizability) of findings is not paramount. However, it should be noted again that the authors' sampling approach was not random. Therefore, they are cautious not to suggest that their treatment effect estimates would generalize far beyond their sample of rural and suburban schools in Washington state. (Contains 1 figure and 6 tables.)

Our research project was guided by the assumption that students who learn to understand phenomena in everyday terms prior to being taught scientific language will develop improved understanding of new concepts. We used web-based software to teach students using a "content-first" approach that allowed students to transition from everyday understanding of phenomena to the use of scientific language. This study involved 49 minority students who were randomly assigned into two groups for analysis: a treatment group (taught with everyday language prior to using scientific language) and a control group (taught with scientific language). Using a pre-post-test control group design, we assessed students' conceptual and linguistic understanding of photosynthesis. The results of this study indicated that students taught with the "content-first" approach developed significantly improved understanding when compared to students taught in traditional ways. (Contains 8 tables and 2 figures.)

The Technology Enhanced Elementary and Middle School Science II project (TEEMSS), funded by the National Science Foundation, produced 15 inquiry-based instructional science units for teaching in grades 3-8. Each unit uses computers and probeware to support students' investigations of real-world phenomena using probes (e.g., for temperature or pressure) or, in one case, virtual environments based on mathematical models. TEEMSS units were used in more than 100 classrooms by over 60 teachers and thousands of students. This paper reports on cases in which groups of teachers taught science topics without TEEMSS materials in school year 2004-2005 and then the same teachers taught those topics using TEEMSS materials in 2005-2006. There are eight TEEMSS units for which such comparison data are available. Students showed significant learning gains for all eight. In four cases (sound and electricity, both for grades 3-4; temperature, grades 5-6; and motion, grades 7-8) there were significant differences in science learning favoring the students who used the TEEMSS materials. The effect sizes are 0.58, 0.94, 1.54, and 0.49, respectively. For the other four units there were no significant differences in science learning between TEEMSS and non-TEEMSS students. We discuss the implications of these results for science education.

This study investigated the effectiveness of the Model of Reading Engagement (MORE), a content literacy intervention, on first graders' science domain knowledge, reading engagement, and reading comprehension. The MORE intervention emphasizes the role of domain knowledge and reading engagement in supporting reading comprehension. MORE lessons included a 10-day thematic unit that provided a framework for students to connect new learning to a meaningful schema (i.e., Arctic animal survival) and to pursue mastery goals for acquiring domain knowledge. A total of 38 first-grade classrooms (N = 674 students) within 10 elementary schools were randomly assigned to (a) MORE at school (MS), (b) MORE at home, (MS-H), in which the MS condition included at-home reading, or (c) typical instruction. Since there were minimal differences in procedures between the MS and MS-H conditions, the main analyses combined the two treatment groups. Findings from hierarchical linear models revealed that the MORE intervention had a positive and significant effect on science domain knowledge, as measured by vocabulary knowledge depth (effect size [ES] = 0.30), listening comprehension (ES = 0.40), and argumentative writing (ES = 0.24). The MORE intervention effects on reading engagement as measured by situational interest, reading motivation, and task orientations were not statistically significant. However, the intervention had a significant, positive effect on a distal measure of reading comprehension (ES = 0.11), and there was no evidence of Treatment × Aptitude interaction effects. Content literacy can facilitate first graders' acquisition of science domain knowledge and reading comprehension without contributing to Matthew effects.

Science education has experienced a significant transition over the last decade, catalyzed by a re-envisioning of what students should know and be able to do in science. That re-envisioning culminated in the release of the Next Generation Science Standards (NGSS) in 2013. The new standards set off a chain reaction of standards adoption and implementation across states, districts, and schools, including steps taken toward transforming science professional learning, instruction, curriculum, and assessment. It was in this dynamic context that Empirical Education conducted an impact evaluation, as part of an Investing in Innovation (i3) grant, of WestEd's Making Sense of SCIENCE project, a teacher professional learning model aimed at raising students' science achievement through improving science instruction. Under this grant, WestEd and Empirical Education also partnered with Heller Research Associates (HRA) to conduct an implementation study and a scale-up study of Making Sense of SCIENCE. The impact evaluation, which is the focus of this report, was a two-year cluster-randomized control trial (RCT) that took place in California and Wisconsin across seven school districts and 66 elementary schools in the 2016-17 and 2017-18 school years. The study randomized schools to either receive the Making Sense of SCIENCE professional learning or to the business-as-usual ("control") group, which received the professional learning (delayed-treatment) after the study ended. The study found that Making Sense of SCIENCE had a positive impact on teacher content knowledge (effect size = 0.56, p = 0.006) and a positive impact on a holistic scale of teacher pedagogical content knowledge (effect size = 0.41, p = 0.026). The study also yielded positive and significant impacts on the amount of time teachers spent on science instruction (effect size = 0.40, p = 0.015) and on the emphasis that teachers placed on NGSS-aligned instructional practices, with statistically significant effect sizes ranging from 0.40 to 0.49. This suggestive evidence that Making Sense of SCIENCE changes classroom science learning experiences in ways that align with expectations in NGSS, which is a hypothesized precursor to measuring impacts on student achievement, deserves notice. In regard to student science achievement, the study did not find statistically significant results for the full sample of students (effect size = 0.06, p = 0.494), or for the sample of students in the bottom third of incoming math and English Language Arts achievement, with effect sizes of 0.22 (p = 0.099) and 0.073 (p = 0.567), respectively. Notably, with the exception of one negative effect size, additional analyses on student achievement using different measures and samples yielded positive, but not statistically significant, effect sizes ranging from 0.02 to 0.12. Evaluators offer three potential contributors to the findings of limited impact on student science achievement. First, given the timing of the study in relation to the release of the NGSS in 2013, finding a suitable NGSS-aligned student science assessment was a challenge. Second, most study schools and districts had not yet adopted NGSS-aligned curricula and did not have access to NGSS-aligned curriculum resources. Third, the sample of teachers was unstable across the two years, with the percentage of teachers leaving the school congruous to the percentage observed at the national level. [For the appendices, see ED609254; for the research summary, see ED609256.]

This randomized controlled trial examined effects of the MyTeachingPartner-Math/Science intervention on the quality and quantity of teachers' mathematics and science instruction, and children's mathematics and science outcomes in 140 pre-kindergarten classrooms. Teachers participated in the intervention for two years with consecutive cohorts of children. Results from Year 1 are considered experimental, however due to high levels of attrition, results from Year 2 are considered quasi-experimental. Across both years, intervention teachers exhibited higher quality and quantity of instruction. In Year 1, there were no significant effects of the intervention on children's outcomes. In Year 2, children in intervention classrooms made greater gains in teachers' ratings of mathematics and science skills and performed better on a spring assessment of science skills. These results have implications for designing and evaluating professional development aimed at supporting children's mathematics and science knowledge and skills.

Three central Virginia school districts and engineering education researchers at the University of Virginia were awarded an Investing in Innovation development grant to design, implement, test, and nationally disseminate a project-based engineering curriculum for middle school students. Referred to as invention kits, the curriculum is developed to teach key science and engineering principles and related skills to Grades 7 or 8 students by constructing modern interpretations of 19th-century inventions that sparked industrial activity within society: the solenoid, the linear motor, and the linear generator. As part of the external evaluation, American Institutes for Research (AIR) conducted an impact study to assess the invention kits' effect on students' engineering and physical science knowledge, as well as students' interest and confidence in STEM learning. The study used a quasi-experimental comparison group design investigating differences in student pre- and-posttests during the 2017-18 school year. Students in four schools across the three districts used a set of three invention kits in their engineering electives, as compared with students taking engineering electives in three schools within one district that had business-as-usual engineering curriculum. AIR studied implementation of the kits by collecting data reported by teachers on student use of kit components, interviews with teachers on how kits were incorporated into their engineering elective curriculum and adapted for use with their students, and observations of kits in use during site visits. The research team did not find a statistically significant difference between the physical science and engineering assessment scores of students who used the kits and comparison students. The research team did not find a statistically significant difference between the measures of STEM interest and confidence of students who used the kits and comparison students. Teachers and students in two of the four schools in the treatment group implemented the three invention kits with fidelity (defined as with at least 75% of their students, using at least 60% of kit components). One of the three kits (Solenoid) was implemented with fidelity by all four participating schools.

The Science Notebook in a Universal Design for Learning Environment (SNUDLE) is a digital notebook that uses the Universal Design for Learning framework to support active science learning among elementary school students, particularly those who struggle with reading and writing or are unmotivated to learn science. Preliminary findings from the first of a two-year randomized control trial suggest no significant impact on motivation or academic achievement in science among the full sample of fourth graders receiving the SNUDLE intervention. Moderator analysis indicates significant positive interaction effects of the intervention on motivation in science and science content assessments among students with learning disabilities.

The National Math + Science Initiative's (NMSI's) College Readiness Program (CRP) is an established program whose goal is to promote science, technology, engineering, and mathematics education in high schools to improve students' readiness for college. The program provides teacher, student, and school supports to promote high school students' success in mathematics, science, and English Advanced Placement (AP) courses, with a focus on students who are traditionally underrepresented in the targeted AP courses. Through a scale-up grant awarded to NMSI by the Investing in Innovation (i3) program, the CRP was implemented in 28 schools in the 2016-2017 school year. CRESST conducted an independent evaluation of the impact of the CRP on students' AP outcomes using a randomized cluster trial with 28 CRP schools and 24 control schools in 10 states. The evaluation of the CRP consisted of two parts: (1) assessment of the program's impact on selected student AP exam outcomes and (2) assessment of the fidelity of implementation of the CRP. Program impact was evaluated using a 2-level hierarchical generalized linear model (HGLM) with students nested within schools The descriptive statistics showed that a higher percetange of students in the treatment schools took at least one AP course (30.7%) compared to those in the control schools (26.4%) by approximately 4.3%, however the difference was not statistically significant. In addition, students in the treatment schools were not more likely to achieve a score of 3 or higher, when compared to the delayed treatment schools. We further examined the effectiveness of the CRP using the prior year's school-level performance on the AP exam as a covariate. As with the above findings, the results indicated the probability of a student taking at least one AP course or scoring 3 or higher on at least one AP exam is not statistically different between students in the treatment schools and those in the control treatment schools. Fidelity of implementation was evaluated using a fidelity matrix approach (required as part of the evaluation of the i3 program), which showed that not all elements of the program were implemented with high fidelity. Overall results, however, indicated that 23 schools out of 28 treatment schools (82.1%) achieved 80% or better implementation fidelity, for an average fidelity score of 89.5%. Seven schools achieved a perfect 100% fidelity score. Looking at the different indicator groups (school, teacher and student), we found that all school support measures across all schools were implemented with fidelity. In over 80% of schools, not all teachers fulfilled their requirements for attending all training sessions, and so this component was not implemented with fidelity. Stipends and teacher awards were paid as expected as were student award payments. Teacher survey data indicated that teachers found the training and professional development activities provided by the CRP to be the most beneficial program supports relating to helping increase student achievement in AP courses. Teacher incentives were chosen as the least important program component relating to increasing student performance by 16% of teachers and student incentives by 12% of teachers. Teachers did, however, view the student incentives as an important program component to encourage enrollment in AP courses. Likewise, students rated the financial incentives on average as somewhat important in encouraging them to participate in AP courses.

Strong correlational evidence suggests that involvement in the arts improves students' academic outcomes and memory of learning events (e.g., Peppler et al., 2014; Robinson, 2013; Scripps & Paradis, 2014). It is unclear, however, whether the improved outcomes are the result of general exposure to the arts, arts integrated into content instruction, the use of effective instructional practices, or a combination of these factors. Moreover, as a growing number of studies suggest that arts-integrated pedagogy enhances learning, few empirical studies have explicitly examined the direct effect of an arts-integrated curriculum on learning and specifically on students' memory for non-arts academic content. Thus, this study sought to determine the effects of arts-integrated lessons on long-term memory for science content. We hypothesized that embedding arts-based activities into conventionally taught lessons would produce learning outcomes as good as or better than traditional instruction. This paper describes the results of a randomized control trial that measured retention of science content using arts-integrated science units and matched units employing convention science instruction. The study was conducted in 16 fifth-grade classrooms in an urban mid-Atlantic school district.

In August 2010, the Smithsonian Science Education Center (SSEC) received a grant of more than $25 million from the U.S. Department of Education's Investing in Innovation (i3) program for a five-year study to validate its Leadership Assistance for Science Education Reform (LASER) model in three very diverse regions of the United States: rural North Carolina, northern New Mexico, and the Houston Independent School District (HISD). This current report focuses on the confirmatory and exploratory research questions submitted to i3 for the two studies conducted for the LASER i3 validation grant, providing clarifying detail related to methodology and instrumentation. The studies were conducted to answer two confirmatory research questions and two exploratory research questions.

This five-year evaluation examined the effectiveness of a promising middle-school mathematics intervention funded through an Investing in Innovation (i3) development grant. Evaluation objectives were to: (1) study the impact of an intervention aimed at increasing the academic achievement of students in Algebra I--a gate-keeping course--as measured by student performance on an end-of-year state test in mathematics; and (2) better understand the relationship between intervention impact and implementation fidelity, as measured by levels of compliance by teachers with the study protocol. The intervention was piloted in Year 2 of the grant (2011-12 school year) that was followed by a two-year [randomized control trial] RCT in grant years 3 (2012-13 school year) and 4 (2013-14 school year). Data collected in the RCT years were focused on impact and exploratory analyses, respectively. For the RCT component, 70 Grade 8 Algebra I teachers were recruited from 15 school districts across California. Randomization, conducted by WestEd in spring 2012, was conducted at the teacher level. Students were assigned to classrooms without knowledge of the group membership of teachers (treatment vs. control), using each district's routine placement policies. Fidelity of implementation study was monitored by collecting systematically information from teachers assigned to the treatment condition throughout the course of the study. The contrast of interest was performance on a standardized Algebra I test by students assigned to classrooms taught by treatment teachers compared to performance by students assigned to classrooms taught by control teachers. The final analytic sample for the 2012-13 cohort included 1,384 students assigned to 28 treatment teachers and 1,088 students assigned to 27 control teachers. None of the contrasts showed a statistically significant difference at the 0.05 level. Students who were assigned to classrooms taught by treatment teachers did not perform differently in relation to those assigned to classrooms taught by control teachers. Overall findings from the implementation study indicated that great variability emerged in the ways in which teachers implemented the intervention. The threshold for fidelity was reached with only one component (Instructional Unit #1) of the four studied (three instructional units and professional coaching). The following appendices are included: (1) Logic Model: SLOPE (DEV11) v.13; (2) Teacher Background Survey; (3) Interpreting Intervention Impact through the Lens of Implementation Fidelity: Findings from a Federally Funded Evaluation--Paper Presented at the Annual Meeting of the American Educational Research Association (Chicago, Illinois, April 19, 2015); (4) Implementation Survey for Air Traffic Control; (5) 2012-2013 Measuring Fidelity of Implementation for Algebra I Drop-in Units: DEV11 (SLOPE); (6) Teacher-Level Participation in i3 SLOPE Evaluation (2011-2014); and (7) Findings from Evaluator Study of Implementation: Implementation Year 1.

Previous research has linked inquiry-based science instruction (i.e., science instruction that engages students in doing science rather than just learning about science) with greater gains in student learning than text-book based methods (Vanosdall, Klentschy, Hedges & Weisbaum, 2007; Banilower, 2007; Ferguson 2009; Bredderman, 1983; Shymansky, Hedges, & Woodworth, 1990). The LASER model, being validated in the current study, has already been the subject of a number of case studies (RMC Research Corporation, 2010; Horizon Research, 2010; Vanosdall et al., 2007). However, experimental studies of the type that might establish a causal link between program implementation, student science learning, and other valued outcomes have yet to be conducted. Only a handful of studies have involved random assignment, and most of these have involved random assignment of students in a relatively small number of classrooms (see Furtak et al. 2009). With support from the U.S. Department of Education's Investing in Innovation Fund (i3), the current validation study of the LASER Model encompasses approximately 60,000 students, 1,900 teachers, and over 140 district administrators and principals. The efficacy of the LASER program has important implications for both research and practice when working with high-poverty schools and districts, who have limited resources and time available for science interventions. LASER's initial success with early learners also demonstrates its potential for reducing the development of chronic, long-term deficiencies and academic problems. One table and one figure are appended.

"MyTeachingPartner--Math/Science" ("MTP-MS") is a system of two curricula (math and science) plus teacher supports designed to improve the quality of instructional interactions in pre-kindergarten classrooms and to scaffold children's development in mathematics and science. The program includes year-long curricula in these domains, and a teacher support system (web-based supports and in-person workshops) designed to foster high-quality curricular implementation. This study examined the impacts of the intervention on the development of mathematics and science skills of 444 children during pre-kindergarten, via school-level random assignment to two intervention conditions ("Basic: MTP-M/S" mathematics and science curricula, and "Plus: MTP-M/S" mathematics and science curricula plus related teacher support system) and a Business-as-Usual control condition ("BaU"). There were intervention effects for children's knowledge and skills in geometry and measurement as well as number sense and place value: Children in "Plus" classrooms made greater gains in geometry and measurement, compared with those in "BaU" classrooms. Children in "Plus" classrooms also performed better on the number sense and place value assessment than did those in "Basic" or "BaU" classrooms. We describe the implications of these results for supporting the development of children's knowledge and skills in early childhood and for developing and providing teachers with professional development to support these outcomes.

"MyTeachingPartner--Math/Science" ("MTP-MS") is a system of two curricula (math and science) plus teacher supports designed to improve the quality of instructional interactions in pre-kindergarten classrooms and to scaffold children's development in mathematics and science. The program includes year-long curricula in these domains, and a teacher support system (web-based supports and in-person workshops) designed to foster high-quality curricular implementation. This study examined the impacts of the intervention on the development of mathematics and science skills of 444 children during pre-kindergarten, via school-level random assignment to two intervention conditions ("Basic: MTP-M/S" mathematics and science curricula, and "Plus: MTP-M/S" mathematics and science curricula plus related teacher support system) and a Business-as-Usual control condition ("BaU"). There were intervention effects for children's knowledge and skills in geometry and measurement as well as number sense and place value: Children in "Plus" classrooms made greater gains in geometry and measurement, compared with those in "BaU" classrooms. Children in "Plus" classrooms also performed better on the number sense and place value assessment than did those in "Basic" or "BaU" classrooms. We describe the implications of these results for supporting the development of children's knowledge and skills in early childhood and for developing and providing teachers with professional development to support these outcomes.

This study evaluated an approach to professional development for middle school science teachers by closely examining one grade 8 course that embodies that approach. Using a cluster-randomized experimental design, the study tested the effectiveness of the Making Sense of SCIENCE[TM] professional development course on force and motion (Daehler, Shinohara, and Folsom 2011) by comparing outcomes for students of teachers who took the course with outcomes for students of control group of teachers who received only the typical professional development offered in their schools and districts. The study estimated impacts on student science achievement for all grade 8 students in the study sample as well as for the subsample of English language learners. It also estimated impacts on teacher science and pedagogical knowledge. Results for the primary confirmatory analyses indicate that after adjusting for multiple comparisons, there were no statistically significant differences between the test results on science content of students in intervention group classrooms and students in control group classrooms. Intervention group students in neither the full sample (effect size = 0.11) nor the English language learner subsample (effect size = 0.31) scored significantly higher on the ATLAST Test of Force and Motion than did their control group counterparts. Similarly, intervention group students in neither the full sample (effect size = 0.03) nor the English language learner subsample (effect size =0 .09) scored higher on the physical science reporting clusters of the California Standards Test than did their control group counterparts. Results for the intermediate confirmatory analyses indicate that after adjusting for multiple comparisons, teachers who received the professional development course outscored their control group counterparts on the ATLAST Test of Force and Motion for Teachers (effect size = 0.38), as well as on their ratings of confidence in their ability to teach force and motion (effect size = 0.49). With one exception, the study findings were not sensitive to variations in specification of the estimation models. The exception is that, for teacher content knowledge, inclusion of the pretest in the impact analysis model (basic model plus pretest) decreased the point estimate from 9.8 to 6.1 and the effect size from 0.61 to 0.38. In exploratory analyses, the study investigated whether there were differential impacts on student and teacher content knowledge outcomes across the six research sites. The estimated impacts were most pronounced at two of the six sites. For the full sample of students, point estimates for student and teacher content knowledge of force and motion followed exactly the same rank order at all sites. There are three main limitations of this study. First, there was high sample attrition: 48 of the 181 teachers who were randomly assigned to intervention and control groups left the study before data collection was completed. However, there is no evidence that attrition resulted in significant differences at the baseline between the intervention and control samples used in the analysis. Second, the study did not include analyses of classroom implementation of course-related practices. As a result it is not possible to infer whether the lack of student effects is due to a failure of treatment group teachers to modify classroom practices or a failure of modified practices to affect student outcomes. Third, the findings are based on volunteer teachers and students whose parents provided consent. It is possible that the findings would have been different had teachers been required to participate in the intervention, and all students been tested. Appended are: (1) Study power estimates; (2) Procedure for assigning blocks for recruited sample and final analytic sample; (3) Teacher agreement to protect the study; (4) Teacher survey responses related to contamination across groups; (5) Parent consent form; (6) California content standards in physical science reporting clusters; (7) Student data obtained from district administrative records; (8) Survey items used to measure teacher confidence; (9) Course session video recording protocol; (10) Course session attendance sheet; (11) Student test administration instructions for proctors; (12) Teacher test administration instructions for site coordinators; (13) Baseline equivalence of teacher demographics in intervention and control group samples; (14) Class selection worksheet; (15) Sensitivity analysis for nesting of students within teachers or classes within teachers; (16) Impact estimation methods; (17) Missing item--level data; (18) Schedule and content goals of Making Sense of SCIENCE[TM] professional development course on force and motion; and (19) Sensitivity analyses based on different models and analytic samples. (Contains 51 tables, 4 figures and 8 footnotes.)