Inside IES Research

In recent months, several federal reports and resources related to English learner (EL) learning and education related to science, technology, engineering, and mathematics (STEM) have been released.

First, the Office of English Language Acquisition (OELA) released its third “data story” about ELs in US schools. This story, which builds on two previously released stories about the characteristics and educational experiences of ELs, focuses specifically on ELs’ NAEP performance and high school graduation rates. Through interactive infographics (many of which are built on data from the National Center for Education Statistics), the story shows that higher percentages of ELs are proficient in math than in reading, but that nearly half of all states experienced declines in the number of ELs who scored proficient in math between 2009 and 2017. The story also shows that graduation rates for ELs improved by 10 percentage points between 2010-11 and 2015-16 (from 57 percent to 67 percent), but still fall well below the rates for non-ELs (84 percent). While interesting and informative, the data story also underscores the necessity of research and development to produce better resources and information to support EL learning.

In that vein, the National Academies of Sciences, Engineering, and Medicine released English Learners in STEM Subjects: Transforming Classrooms, Schools, and Lives. This report examines what we know about ELs’ learning, teaching, and assessment in STEM subjects and provides guidance on how to improve STEM learning outcomes for these students. It reflects the consensus of a committee of EL experts that was chaired by NCER and NCSER grantee Dr. David Francis and included past grantees Dr. Okhee Lee and Dr. Mary Schleppegrell alongside a dozen other experts in EL education, STEM education, and teaching. One of the report’s central conclusions is that ELs develop proficiency in both STEM subjects and language when their classroom teachers provide them with opportunities for meaningful interaction and actively support both content and language learning. Given that many STEM teachers do not receive preparation to teach in this way, the report provides several recommendations to improve pre-service and in-service training. It also includes recommendations for how developers and publishers might produce better instructional materials and assessments to help both teachers and EL students. 

Efforts of both types – instructional preparation and development of new materials – may be further supported by two new toolkits released by the Office of Education Technology. The toolkits are designed for educators and developers, and each is organized around five specific guiding principles to help the targeted group approach education technology with ELs’ unique needs in mind. The principles for developers emphasize the importance of thinking ahead about EL needs for those who wish to make products for this population. Meanwhile, the educator principles center on issues of awareness, and encourage teachers to learn more about the features, platforms, and resources that are available for ELs in the world of education technology. The principles also complement one another – for example, developers are encouraged to offer instruction-focused professional development, and educators are encouraged to seek out the same.

Brought together, these resources provide a snapshot of ELs’ mathematics achievement, a summary of research evidence about learning and instruction for ELs in STEM, and a set of principles to guide instruction and development efforts in the technology space moving forward. They also make a clear case for continued investment in R&D efforts to support STEM learning for both EL students and their teachers. Since 2010, the National Center for Education Research has invested nearly $20 million across 13 research and researcher-practioner partnership grants that have focused on STEM learning and ELs. Several such grants are coming to a close in the 2019 fiscal year; watch this space for future blog posts about the products and findings from these projects.

Since 2014, IES has funded the Center for the Analysis of Postsecondary Readiness (CAPR) to answer questions about the rapidly evolving landscape of developmental education at community colleges and open-access four-year institutions. CAPR is providing new insights into how colleges are reforming developmental education and how their reforms are impacting student outcomes through three major studies:

• A survey and interviews about developmental education practices and reform initiatives
• An evaluation of the use of multiple measures for assessing college readiness
• An evaluation of math pathways.

Preliminary results from these studies indicate that some reforms help more students finish their developmental requirements and go on to do well in college-level math and English.

National Study of Developmental Education Policies and Practices

CAPR has documented widespread reform in developmental education at two- and four-year colleges through a national survey and interviews on developmental education practices and reforms. Early results from the survey show that colleges are moving away from relying solely on standardized tests for placing students into developmental courses. Colleges are also using new approaches to delivering developmental education including shortening developmental sequences by compressing or combining courses, using technology to deliver self-paced instruction, and placing developmental students into college-level courses with extra supports, often called corequisite remediation.

Developmental Math Instructional Methods in Public Two-Year Colleges (Percentages of Colleges Implementing Specific Reform Strategies)

Notes: Percentages among two-year public colleges that reported offering developmental courses. Colleges were counted as using an instructional method if they used it in at least two course sections. Categories are not mutually exclusive.

Evaluation of Developmental Math Pathways and Student Outcomes

CAPR has teamed up with the Charles A. Dana Center at the University of Texas at Austin to evaluate the Dana Center Mathematics Pathways (DCMP) curriculum at four community colleges in Texas. The math pathways model tailors math courses to particular majors, with a statistics pathway for social science majors, a quantitative reasoning pathway for humanities majors, and an algebra-to-calculus pathway for STEM majors. DCMP originally compressed developmental math into one semester, though now the Dana Center is recommending corequisite models. Instructors seek to engage students by delving deeply into math concepts, focusing on real-world problems, and having students work together to develop solutions.

Interim results show that larger percentages of students assigned to DCMP (versus the traditional developmental sequence) enrolled in and passed developmental math. More of the DCMP students also took and passed college-level math, fulfilling an important graduation requirement. After three semesters, 25 percent of program group students passed a college-level math course, compared with 17 percent of students assigned to traditional remediation.

Evaluation of Alternative Placement Systems and Student Outcomes (aka Multiple Measures)

CAPR is also studying the impact of using a combination of measures—such as high school GPA, years out of high school, and placement test scores—to predict whether students belong in developmental or college-level courses. Early results from the multiple measures study show that, in English and to a lesser extent in math, the multiple measures algorithms placed more students into college-level courses, and more students passed those courses (compared to students placed with a single test score).

College-Level English Course Placement, Enrollment, and Completion in CAPR’s Multiple Measures Study (Percentages Compared Across Placement Conditions)

College-Level Math Course Placement and Completion in CAPR’s Multiple Measures Study

Looking Ahead to the Future of Developmental Education

These early results from CAPR’s evaluations of multiple measures and math pathways suggest that those reforms are likely to be important pieces of future developmental education systems. CAPR will release final results from its three studies in 2019 and 2020.

Guest blog by Nikki Edgecombe and Alexander Mayer

Nikki Edgecombe is the principal investigator of the Center for the Analysis of Postsecondary Readiness, an IES-funded center led by the Community College Research Center (CCRC) and MDRC, and a senior research scientist at CCRC. Alexander Mayer is the co-principal investigator of CAPR and deputy director of postsecondary education at MDRC.

Just over five years ago the Institute of Education Sciences (IES) and the National Science Foundation (NSF) released the Common Guidelines for Education Research and Development. The Guidelines provided the expected purposes, justifications, and contributions of various types of research aimed at improving our knowledge of interventions and strategies for improving teaching and learning.  Since 2013, there has been increased attention to replication and reproducibility studies and their role in building the evidence base. In response to this interest and the importance of this work, the two organizations jointly issued the new Companion Guidelines on Replication and Reproducibility in Education Research to supplement the Common Guidelines for Education Research and Development. The companion document provides guidance on the steps researchers can take to promote corroboration, ensure the integrity of research, and extend the evidence base.

The Companion Guidelines identify principles to help education stakeholders design and report reproducibility and replication studies. These principles are consistent with and draw from guidelines provided by scientific and professional organizations, advisory committees, and have emerged in consultation with the field (e.g., Dettmer, Taylor, and Chhin, 2017; Subcommittee on Replicability and Science, 2015). The principles address three main areas – (1) replication and reproducibility at the proposal stage, (2) promoting transparency and openness in designing studies, and (3) considerations in the reporting of results. 

Although the importance of reproducibility and replication studies for advancing scientific knowledge has been widely acknowledged, there are several challenges for researchers in our field, including actual or perceived disincentives (e.g., publication bias; reputation and career advancement norms; emphases on novel, potentially transformative lines of inquiry), implementation difficulties (especially for direct replications), and complexities of interpreting results (e.g., lack of consensus on what it means to “replicate” findings, low statistical power for replications). Grant funding agencies such as IES and NSF as well as education researchers have a role to play in addressing these challenges, promoting reproducibility and replication studies, and ultimately moving the field forward.

Why focus on replication and reproducibility?

The original Common Guidelines document did not substantively address issues pertaining to replication and reproducibility of research.  Given the interest in and importance of this work, IES and NSF are providing additional clarity to the field in terms of common definitions and principles around replication and reproducibility.

Who is the audience for the Companion Guidelines on Replication and Reproducibility? 

The primary audience for this document is education researchers; however, education research funding agencies and reviewers of grant applications are additional audiences for this document.

How should this document be used by researchers intending to apply for grants to conduct a reproducibility or replication study?

This document is meant to highlight the importance of replication and reproducibility studies and to offer guidelines to education stakeholders for thinking about and promoting reproducibility and replication in education research. It does not supersede the guidance provided in the requests for applications provided by IES and NSF. 

What are the guiding principles for proposing replication and reproducibility studies?

The overarching principles at the grant proposal stage are as follows:

1. Clarify how reproducibility or replication studies would build on prior studies and contribute to the knowledge base.
2. Clearly specify any variations from prior studies and the rationale for such variations.
3. Ensure objectivity (e.g., by conducting an independent investigation, or by putting safeguards in place if the original investigator(s) is involved).

In addition to these principles, the document also lays out principles for promoting transparency, open science, and reporting results.

Read the full Companion Guidelines here.

I came to the Institute of Education Sciences (IES) in 2002 to build connections between education and the basic science of learning and development. The weak links between these two fields were surprising to me, given how foundational such science is to the very purpose of education.

IES had just launched the Cognition and Student Learning program[1], and researchers were invited to submit applications to examine whether principles of learning established in basic science were robust when examined in education settings.  Six years later, we launched the Social and Behavioral Context to Support Academic Learning to understand the ways in which the social environment of classrooms and school affected learning. Together, IES has invested over $445M, an investment that has contributed substantially to our foundational knowledge of teaching and learning.

I was surprised by this recent blog by Bob Pianta and Tara Hofkens. While they acknowledge the research that IES has supported to transform education practice, they did not seem to realize our substantial, ongoing investments in the basic science of teaching and learning—both in and out of classrooms.

In part, this may reflect their perception of what types of work we support under our Exploration goal – which is not limited to “scouring databases” but instead involves all types of research, including small-scale experiments and longitudinal studies. These projects generate foundational knowledge about what factors are associated with learning outcomes and can potentially be changed through education. In fact, the questions that Pianta and Hofkens want answered by the basic science of education are the same questions that some IES grantees have been examining over the course of the last 15 years. 

Here are just a few examples.

1. What factors regulate children's attention in a classroom setting? Anna Fisher and her team found that cluttered classroom walls in kindergarten led to greater distraction and less learning – a finding that captured the imagination of the nation and the nation’s educators.
2. What roles do the capabilities of peers play in advancing children's cognitive capabilities? A new study led by Adrienne Nishina is examining how student’s ability to think about situations from different perspectives is related to their day-to-day interactions with peers from diverse backgrounds.
3. What factors promote or inhibit teachers' responses to children's perceived misbehavior? Teachers’ expertise and teachers’ emotional competencies are two factors that IES-funded researchers have found to relate to their responses to children’s behavior.
4. What role do social and emotional experiences and affective processes play in fostering learning? Shannon Suldo and her team find that the coping strategies that high school students choose to manage their responses to stressors are linked to learning outcomes.
5. What are the components of school climate that matter the most for different forms of student success? Two recent projects, one in Cleveland, and one in Virginia, are using survey data to explore the relationship between school climate, social behavioral competencies and academic outcomes. The teams are also exploring how those relationships vary within student subgroups.

Funding the basic science of teaching and learning—in and out of classrooms—has been and will continue to be a cornerstone of the work that IES funds. The IES investment in this area is broad, and is shared in books such as Make It Stick: The Science of Successful Learning, Becoming Brilliant: What Science Tells us About Raising Successful Children, and Educator Stress: An Occupational Health Perspective. 

Importantly, IES is not the only funder in this area. The National Science Foundation invests substantially in their Science of Learning portfolio, the McDonnell Foundation’s Understanding Human Cognition portfolio includes an explicit request for projects at the intersection of cognition and education, and the Child Development and Behavior Branch of the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) supports a variety of relevant research programs.  I agree that we need systematic investment in the basic science of teaching and learning. But we must build on what we have already learned.  

We are grateful that Pianta and Hofkens recognize the importance of investing in this area. Perhaps the fact that they did not acknowledge the substantial investments and contributions IES has made in exploring the important questions they pose is an IES problem. While we have invested heavily in the science of learning, we have skimped on brand development and self-promoting. If someone as central to the field such as Pianta, who has received several IES grants, including research training grants, doesn’t know what IES has done, that is a red flag that we will need to attend to.

In the meantime, we hope that this brief glimpse into our investment to date has illustrated some of the questions that the basic science of teaching and learning within education can answer. More importantly here’s where you can seek funding for this type of work.

Elizabeth Albro

Commissioner, National Center for Education Research

November 8^th, 2018 is National STEM Day! Today is a great day to talk to learners of all ages and abilities about Science, Technology, Engineering and Mathematics (STEM). The Institute of Education Sciences (IES) has some great resources for exploring STEM learning - visit our new STEM Topic page to learn more. Through research grants from the National Center for Education Research (NCER) and the National Center for Special Education Research (NCSER), and innovations developed as part of the Small Business Innovation Research (SBIR) program, IES has supported the development and testing of many programs, practices, and policies to improve student outcomes in STEM. 

Below, we provide links to a few projects and activities for instructors and learners to explore on National STEM Day, but remember, STEM Day can be every day!

• NumberShire is a mobile and desktop game-based mathematics intervention funded with several grants from NCSER and SBIR that  builds understanding of whole number concepts among early elementary students with or at risk for learning disabilities (video demonstration).
• Improving Children's Understanding of Equivalence (ICUE) supplements teachers’ existing mathematics instruction and helps students develop understanding of mathematical equivalence. Developed with support from NCER, ICUE is currently being evaluated in second grade classrooms. ICUE includes teacher manuals, student workbooks, manipulatives, assessment items, and a 2-hour professional development workshop to provide teachers with information about how to implement the intervention (video demonstration).
• Two innovative education technology products developed with funding from the SBIR program are intended to transform chemistry instruction and learning. Happy Atoms is a physical hand-held magnetic molecular modeling set with a companion digital app that can recognize student created models and provide feedback and information to enrich learning.  HoloLab Champions uses an immersive virtual reality (VR) game environment within which high school students perform chemistry experiments.
• Combined Cognitive and Motivational Supports for STEM Learning is a supplemental Blackboard module for postsecondary introductory biology courses developed with support from NCER. This module leverages short cognitive and motivational interventions that show promise for engaging students and improving outcomes, and is available from IDEALS.

Christina Chhin is the program officer for the Science, Technology, Engineering, and Mathematics (STEM) Education research topic within the National Center for Education Research, Sarah Braisel is the program officer for the Science, Technology, Engineering, and Mathematics (STEM) Education research topic within the National Center for Special Education Research, and Ed Metz is the program officer for both our Small Business Innovation Research program as well as our Education Technology research topic within the National Center for Education Research.