Inside IES Research

Notes from NCER & NCSER

Equity Through Innovation: New Models, Methods, and Instruments to Measure What Matters for Diverse Learners

In today’s diverse classrooms, it is both challenging and critical to gather accurate and meaningful information about student knowledge and skills. Certain populations present unique challenges in this regard – for example, English learners (ELs) often struggle on assessments delivered in English. On “typical” classroom and state assessments, it can be difficult to parse how much of an EL student’s performance stems from content knowledge, and how much from language learner status. This lack of clarity makes it harder to make informed decisions about what students need instructionally, and often results in ELs being excluded from challenging (or even typical) coursework.

Over the past several years, NCER has invested in several grants to design innovative assessments that will collect and deliver better information about what ELs know and can do across the PK-12 spectrum. This work is producing some exciting results and products.

  • Jason Anthony and his colleagues at the University of South Florida have developed the School Readiness Curriculum Based Measurement System (SR-CBMS), a collection of measures for English- and Spanish-speaking 3- to 5-year-old children. Over the course of two back-to-back Measurement projects, Dr. Anthony’s team co-developed and co-normed item banks in English and Spanish in 13 different domains covering language, math, and science. The assessments are intended for a variety of uses, including screening, benchmarking, progress monitoring, and evaluation. The team used item development and evaluation procedures designed to assure that both the English and Spanish tests are sociolinguistically appropriate for both monolingual and bilingual speakers.

 

  • Daryl Greenfield and his team at the University of Miami created Enfoque en Ciencia, a computerized-adaptive test (CAT) designed to assess Latino preschoolers’ science knowledge and skills. Enfoque en Ciencia is built on 400 Spanish-language items that cover three science content domains and eight science practices. The items were independently translated into four major Spanish dialects and reviewed by a team of bilingual experts and early childhood researchers to create a consensus translation that would be appropriate for 3 to 5 year olds. The assessment is delivered via touch screen and is equated with an English-language version of the same test, Lens on Science.

  • A University of Houston team led by David Francis is engaged in a project to study the factors that affect assessment of vocabulary knowledge among ELs in unintended ways. Using a variety of psychometric methods, this team explores data from the Word Generation Academic Vocabulary Test to identify features that affect item difficulty and explore whether these features operate similarly for current, former, as well as students who have never been classified as ELs. The team will also preview a set of test recommendations for improving the accuracy and reliability of extant vocabulary assessments.

 

  • Researchers led by Rebecca Kopriva at the University of Wisconsin recently completed work on a set of technology-based, classroom-embedded formative assessments intended to support and encourage teachers to teach more complex math and science to ELs. The assessments use multiple methods to reduce the overall language load typically associated with challenging content in middle school math and science. The tools use auto-scoring techniques and are capable of providing immediate feedback to students and teachers in the form of specific, individualized, data-driven guidance to improve instruction for ELs.

 

By leveraging technology, developing new item formats and scoring models, and expanding the linguistic repertoire students may access, these teams have found ways to allow ELs – and all students – to show what really matters: their academic content knowledge and skills.

 

Written by Molly Faulkner-Bond (former NCER program officer).

 

IES Funds First Large-Scale Evaluation Study of Public Preschool Montessori

The Montessori method of education was developed over 100 years ago by Dr. Maria Montessori. This “whole child” approach centers around the theory that children are capable of initiating learning in a thoughtfully prepared environment that develops children’s physical, social, emotional, and cognitive growth. Core components of Montessori education are mixed age classrooms in three-year groupings (e.g., 3-6 year olds, 6-9, 9-12, etc.), a carefully prepared environment filled with appropriate materials and lessons, student freedom to select lessons and activities each day, and daily uninterrupted 3-hour work blocks.

   

According to the National Center for Montessori in the Public Sector (NCMPS), there are currently over 5,000 Montessori schools in the U.S., 500 of which are public schools and over 150 of which serve public preschool and kindergarten students.  Despite its growing popularity in public preschools and Head Start schools, no large-scale evaluation of the efficacy of the Montessori model on children’s academic, social, and emotional skills has been conducted. 

This year, IES funded the first such study. A project team led by Dr. Ann-Marie Faria and Ms. Karen Manship (American Institutes for Research) and Dr. Angeline Lillard (University of Virginia) will study more than 650 children for three years, beginning with their entry at age 3 into preschool. Importantly, this study relies on individual random student assignment via lottery entry to compare preschool students who enroll in Montessori at age 3 to those who are assigned to a waitlist control group (and thus are in other settings such as public PreK, daycare, or a home setting). Data will be collected in diverse urban and suburban school districts across the country, including Houston (TX), Hartford and New Haven (CT), and Washington, DC.

Researchers will examine the impact of preschool Montessori education on children’s academic, social, and emotional skills, as well as kindergarten readiness skills. The research team will also conduct a cost effectiveness study of the public Montessori preschool model, and will examine the effect of fidelity of implementation of Montessori on student outcomes. Collectively, the findings from this study will provide valuable evidence of the efficacy of Montessori preschool education. Ultimately, the researchers plan to disseminate their findings to educators, parents, and policymakers through research briefs, infographics, blog posts, and webinars.

 

By Amanda M. Dettmer, PhD, American Psychological Association Executive Branch Science Fellow/ AAAS Science & Technology Policy Fellowship

Photo credit: Marilyn Horan, Carroll Creek Montessori Public Charter School

    

IES Expands Research in Social Emotional Learning

Social and emotional learning (SEL) is a key ingredient of high-quality education care, is important for both educators and children, and has been associated with children’s concurrent and later academic and social success.

Over a decade ago, Yale University’s Center for Emotional Intelligence developed and began testing RULER, an SEL program geared toward children and educators (i.e., school leaders, teachers, and staff). RULER stands for five key social and emotional skills: Recognizing emotions in self and others, Understanding the causes of emotions in self and others, Labeling and talking about emotions, Expressing emotions across situations, and Regulating emotions effectively. For children and the key adults in their lives, RULER combines a whole-school professional development approach with a skill-building curriculum targeting educator and student social and emotional skills, school and classroom climate, and educator and student well-being. RULER is currently offered for pre-k–12 and out-of-school-time settings.

IES has supported the development and testing of RULER programs since 2012. The first IES award supported the modification of existing components of the RULER K-8th grade intervention and creation of new developmentally appropriate content for preschool settings. RULER is currently implemented in over 200 early childhood school- and home-based programs across the country and nearly 2,000 K-12 schools nationwide. Although RULER’s evidence-base has been growing over the years, RULER has not been systematically studied in large-scale, randomized controlled trials in preschool settings nor has it undergone an external evaluation in the later grades.

That is about to change: this year, IES awarded two grants to study the effects of the RULER programs. One will study the efficacy of whole-school RULER implementation for preschool students (under the Early Learning Programs and Policies program), and the other will do so for grades K-6 (under the Social and Behavioral Context for Academic Learning program).

The Preschool RULER grant (PI: Craig Bailey, PhD) will assess school readiness in children aged 3-5, as well as outcomes at the teacher/classroom and school leader/school levels. The researchers will study 72 early childhood centers, including public, private, and Head Start programs from urban areas in Connecticut, using a multisite, cluster-randomized control trial design. Altogether, approximately 216 classrooms, 1,800 staff, and 2,160 children will participate. Children, educators, and school leaders will be assessed for social and emotional skills, and educators/leaders will be assessed for emotionally intelligent pedagogy and leadership. Children will also be assessed for their approaches to learning, pre-literacy, and pre-math skills. This study will provide evidence about the efficacy of RULER in preschool settings and contribute to our understanding of high quality early childhood interventions that promote social emotional learning.

 

The other grant, for RULER in grades K-6 (PI: Jason Downer, PhD), will be the first large-scale external evaluation of RULER. The study will take place in 60 urban and suburban public elementary schools, including 420 teachers and 2,520 K-6 students in Virginia. Key outcomes for this study will include school climate assessments (assessed by teacher and principal reports), teacher well-being (assessed by self-report), and four student outcomes: social-emotional skills, behavior, academic engagement and academic achievement (assessed by standardized assessments, tests, and attendance records). Ultimately, this study will describe RULER’s effects on school climate, teacher well-being, classroom climate, and student outcomes.

By Amanda M. Dettmer, AAAS Science & Technology Policy Fellow Sponsored by the American Psychological Association Executive Branch Science Fellowship

Photo credits: Yale Center for Emotional Intelligence

Computerized Preschool Language Assessment Extends to Toddlers

Identifying young children with language delays can improve later outcomes

Language is a core ability that children must master for success both in and out of the classroom. Extensive studies have shown that many tasks, including math, depend on linguistic skill, and that early language skills are predictive of school readiness and academic success. Being able to quickly identify children at early ages with language delays is crucial for targeting effective interventions.

Enter the QUILS.

In 2011, the National Center for Education Research (NCER) at IES funded a 4-year grant to Dr. Roberta Golinkoff (University of Delaware) and Drs. Kathy Hirsh-Pasek (Temple University) and Jill de Villiers (Smith College) to develop a valid and reliable computer-based language assessment for preschoolers aged 3-5 years old. The resulting product was the Quick Interactive Language Screener (QUILS), a computerized tool to measure vocabulary, syntax, and language acquisition skills. The assessment ultimately measures what a child knows about language and how a child learns, and automatically provides results and reports to the teacher.

The preschool version of QUILS is now being used by early childhood educators, administrators, reading specialists, speech-language pathologists, and other early childhood professionals working with young children to identify language delays. The QUILS is also being utilized in other learning domains. For example, a new study relied on the QUILS, among other measures, to examine links between approaches to learning and science readiness in over 300 Head Start students aged 3 to 5 years.

QUILS is now being revised for use with toddlers. In 2016, the National Center for Special Education Research (NCSER) funded a 3-year study to revise the QUILS for use with children aged 24-36 months. The researchers have been testing the tool in both laboratory and natural (child care centers, homes, and Early Head Start programs) settings to determine which assessment items to use in the toddler version of QUILS. Ultimately, these researchers aim to develop a valid and reliable assessment to identify children with language delays so that appropriate interventions can begin early.

By Amanda M. Dettmer, AAAS Science & Technology Policy Fellow Sponsored by the American Psychological Association Executive Branch Science Fellowship

Taking Education Research Out of the Lab and Into the Real World

The Institute of Education Sciences is committed to supporting research that develops and tests solutions to the challenges facing education in the United States and working to share the results of that research with a broad audience of policymakers and practitioners. In this blog post, Katherine Pears (pictured right), a principal investigator from the Oregon Social Learning Center, shares how an IES-funded grant supported not only the development and testing of an intervention, but its implementation in classrooms.

Sometimes when I tell people that I am a research scientist, they say "Ok, but what do you do?" What they are really asking is whether the work I am doing is helping students and families in actual schools and community. It’s a good question and the short answer is “Yes!” but moving programs from the research lab to children, families and schools takes time and funding. Here is how it worked with my program.

I wanted to help children at risk for school failure to start kindergarten with skills to help them to do better. I teamed up with Dr. Phil Fisher and others, and we built on years of research and program development at the Oregon Social Learning Center (OSLC) to create a program called Kids in Transition to School (KITS). This is a summer program that continues into the first few weeks of school and focuses not only on letters and numbers, but also on teaching children how to get along with others and how to learn (such as focusing their attention). We also train the KITS teachers to use positive teaching strategies and help parents learn the same techniques to increase the chances that children will succeed.

However, before a school will put a program like KITS in place, they need to have some proof that it will actually work. That is why we sought external funding to evaluate the program.  With funding from the Institute of Education Sciences (IES), as well as the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) and the National Institute on Drug Abuse (NIDA), we were able to test KITS with three different groups of children at high risk for difficulties in school: children in foster care, children with developmental disabilities and behavior problems, and children from low-income neighborhoods. 

Across these studies, KITS led to positive outcomes for children and the parents. Funding from IES helped us show that the KITS Program worked with different groups of children and that it was a good candidate for use in school settings. In many cases, this is where the process of moving programs from research to practice can get bogged down due to a gap between researchers and the people in the education systems that could use the programs. IES is strongly committed to bridging that gap by supporting partnerships between researchers and practitioners. In our IES-funded study, we partnered with two of our local school districts and United Way of Lane County (UWLC). Working with the schools and community agencies helped us to both test KITS and develop plans for how the districts could keep the program going after the funding for the research study ended.

Our partnership with UWLC also enabled us to start a new project to bring KITS to more schools. With funding from the Corporation for National and Community Service (CNCS) Social Innovation Fund, UWLC offered grants to education foundations in Lane County, Oregon to implement KITS in the local school districts. The Social Innovation Fund allows promising programs to be brought to scale (i.e., made available to large numbers of people) by providing funding for community agencies to start and develop plans for sustaining these programs. In 2015, the KITS Program had about 30 educators serving about 120 children and families in 6 school districts in Lane County, Oregon. By summer 2017, two years into the grant, the KITS Program had trained 125 educators to serve approximately 435 children and families in 13 school districts in the county. That’s more than a threefold increase in the number of children and families served, as well as the number of teachers trained in the KITS Program.

We will continue to work with partners to evaluate the effects of the KITS program and make it available to more school districts. However, without the funding from IES and other federal agencies, we would not have been able to both test the program and partner with schools to make KITS a part of their everyday practice.