Inside IES Research

IES funded the Expanding the Evidence Base for Career and Technical Education (CTE) Research Network (CTERN) in FY 2018 in order to increase the quality and rigor of CTE research, specifically by (1) coordinating IES-funded researchers studying CTE using causal designs and (2) training new researchers in causal methods to address CTE-related research questions. In this guest blog, the Network Lead’s PI, Katherine Hughes, and Training Lead, Jill Walston, from the American Institutes for Research (AIR), discuss the evolution of the institute across four years of training supported by the grant and what they learned about the components of effective training, in the hopes of sharing lessons learned for future IES-funded trainings.

About the Summer Training Institute

Each summer since 2020, CTERN has held summer training institute on causal research methods in CTE.  Across four summers, we had 81 trainees, including junior faculty, researchers in state or university research offices or institutes, doctoral students, and researchers in non-profit organizations. During the institutes, we had expert CTE researchers and national and state CTE leaders deliver presentations about CTE history, policies, theories, and recent research.

The major focus of the training was on research designs and statistical methods for conducting research that evaluates the causal impact of CTE policies and practices on student outcomes. The participants learned about conducting randomized controlled trials—considered the gold standard for causal research—as well as two quasi-experimental approaches, regression discontinuity and comparative interrupted time series designs. After presentations about the approaches, students worked with data in small groups to complete data analysis assignments designed to provide practical experience with the kinds of data and analyses common in CTE research. The small groups had dedicated time to meet with one of the instructors to discuss their analyses and interpret findings together. The combination of presentations and practical applications of data analysis with real data, and time in small groups for troubleshooting and discussion with CTE researchers, made for a rich experience that students found engaging and effective. The students received an IES certificate of course completion to mark their accomplishment.

Making Continuous Improvements Based on Lessons Learned

We had a continuous improvement mindset for our summer institute. After each week-long session was completed, the CTE research network director, training coordinator, and instructors met to review their perceptions of the training and most importantly the feedback students provided at the end of the week. We applied the lessons learned to make improvements to the agenda, communications, and student grouping approaches to the plans for the following summer.

Over the course of the four years of the summer institute training, we made a number of adjustments in response to feedback.

• We continued to offer the institute virtually. The institute was originally intended to be held in person; an earlier blog describes our necessary pivot to the online format. While we could have safely changed to an in-person institute in 2022 and 2023, feedback from our students showed that the virtual institute was more accessible to a geographically diverse group. Many trainees said they would not have even applied to the institute if they would have had to travel, even with a stipend to help cover those costs.
• We added more time for the students to get to know one another with virtual happy hours. Compared to in-person trainings, virtual trainings lack those natural opportunities for informal communications between students and with instructors that can foster engagement, trust, and joint purpose. While we couldn’t replicate in-person networking opportunities, we were able to improve the experience for the students by being intentional with informal gatherings.
• We expanded the time for the small groups to meet with their instructors. Students reported that this office hour time was very valuable for their understanding of the material and in interpreting the output of the analyses they ran. We extended this time to optimize opportunities for discussion and problem solving around their data analysis assignments.   
• We made improvements to the data assignment guidance documents. In the first year, students reported that they spent more time on figuring out initial tasks with the data which left less time for running analyses and interpreting their output. We modified our guidance documents that accompanied the assignments to spell out more explicitly some of the initial steps to shorten the time students spent on set-up and maximize their time doing the important work of coding for the analyses and examining output. We also provided links to resources about the statistical packages used by the students for those that needed time to brush up on their skills before the training began.
• We doubled down on efforts to stay connected with the trainees and supported ways to have them stay connected to each other. For example, we let them know when CTERN’s researchers are presenting at conferences and invite them to connect with us and each other at these conferences. We’re now organizing a LinkedIn group to try to develop a community for our training alumni.

Our summer training institutes were a great success. We look forward to continuing this opportunity for researchers into the future, with a new version to be offered in the summer of 2025 by the CTE Research Network 2.0.

Jill Walston, Ph.D., is a principal researcher at the American Institutes for Research with more than 20 years of experience conducting quantitative research, developing assessments and surveys, and providing technical support to researchers and practitioners to apply rigorous research and measurement practices. Dr. Walston is the lead for training initiatives for the IES-funded Career and Technical Education Research Network.

Katherine Hughes, Ph.D., is a principal researcher at the American Institutes for Research and the principal investigator and director of the CTE Research Network and CTE Research Network 2.0. Dr. Hughes’ work focuses on career and technical education in high schools and community colleges, college readiness, and the high school-to-college transition.

This blog was produced by Corinne Alfeld (Corinne.Alfeld@ed.gov), a Program Officer in the National Center for Education Research (NCER).

IES widely supports and disseminates high-quality research focusing on science, technology, engineering, and mathematics (STEM) through NCER and NCSER. To celebrate National STEM Day on November 8 and every day, we highlight some of the work that NCER and NCSER have supported over the years in the various STEM areas, as well as opportunities for funding future work. Additional information about IES’s investment in STEM education can also be found on our STEM topic page.

Science

• Researchers developed ChemVLab+ an online chemistry intervention that allows high school students to perform experiments and analyze data in a flexible, multimedia virtual chemistry lab environment. The online modules promote conceptual understanding and science inquiry skills aligned to the Next Generation Science Standards. The chemistry activities are freely available on the project website.
• Researchers are developing Words as Tools, an intervention for emergent bilingual adolescents that is designed for use in English as a second language classes to promote development of metalinguistic awareness with science vocabulary. The lessons, being developed with a lens of culturally sustaining pedagogy, are intended to help build knowledge of essential science words as well as how words work in science.
• Researchers are evaluating the efficacy of an integrated science and literacy curriculum (ISLC) designed to engage first grade students in scientific investigations at a level appropriate for young learners. ISLC addresses the challenges of language and literacy development by ensuring that the language of science is brought forward and explicitly addressed in an integrated approach.
• Through Project MELVA-S, researchers are developing an online formative assessment that measures the science vocabulary knowledge of Latinx bilingual students with different levels of English and Spanish language proficiencies. Results from the assessment can be used to monitor the progress of individual students, help teachers differentiate language and vocabulary instruction, and provide additional science vocabulary supports.

Technology

• Using The Foos by codeSpark, researchers are exploring computational thinking processes in grades 1 and 3 through a series of classroom-based studies.
• Researchers are evaluating the efficacy of the CAL-KIBO curriculum, an educational robotics program designed for use with early elementary school-aged students to examine its impact on computational thinking, fluid reasoning, and math achievement.
• Researchers are systematically investigating how specific features of immersive virtual reality (IVR) can be used to improve student outcomes in science learning. In particular, the researchers are exploring how visual and auditory IVR design features can enhance affective state and cognitive processing in general and for specific subgroups of learners.
• Researchers are developing and testing TaylorAI, an artificial intelligence formative feedback and assessment system for hands-on science investigations to help build student competence as they engage in laboratory activities.
• In partnership with the National Science Foundation, IES is co-funding two National Artificial Intelligence (AI) Institutes. Under NCER, the Institute for Inclusive and Intelligent Technologies for Education (INVITE) is developing artificial intelligence (AI) tools and approaches to support behavioral and affective skills (for example, persistence, academic resilience, and collaboration) to improve learning in STEM education. Under NCSER, the AI Institute for Exceptional Education (AI4ExceptionalEd) is using multiple cutting-edge AI methodologies to create the technology to assist speech-language pathologists with identifying students in need of speech and language services and delivering individualized interventions.

Engineering

• Researchers are developing an innovative teacher professional learning intervention called Elevating Engineering with Multilingual Learners that is intended to help grade 3-5 teachers develop the knowledge and skills they need to effectively teach engineering to English learners and all students through culturally and linguistically responsive pedagogies and engineering instruction.
• Product developers and researchers are developing and testing NEWTON-AR, an augmented reality (AR) application-based engineering, computer science, and STEM puzzle game for children in kindergarten to grade 3. Intended for use in classrooms, after-school programs, and at home, NEWTON-AR will combine AR, engineering, simulation, making, and programming into a sandbox game where students create, modify, simulate, prototype, and test contraptions to solve puzzle challenges.

Mathematics

• Researchers have developed and tested for efficacy of Fusion, a first-grade intervention aimed at developing understanding of whole numbers for students at risk for mathematics learning disabilities. It is designed as a program for schools using a multi-tiered approach to instruction that provides increasingly intense levels of instruction based on the results of frequent progress monitoring of students.
• Researchers tested for efficacy of Pirate Math Equation Quest, a word problem-solving intervention for third grade students with mathematics difficulties, including students with or at risk for mathematics learning disabilities.
• Researchers assessed the efficacy of Interleaved mathematics practice, an intervention that rearranges math practice problems so that 1) different kinds of math problems are mixed together, which improves learning, and 2) problems of the same kind are distributed across multiple assignments, which improves retention. A new systematic replication study is also now underway to further examine the efficacy of interleaved mathematics practice.
• Researchers have conducted several impact studies (one conducted with grade 7 students in Maine and replication study conducted in North Carolina) of ASSISTments, a free web-based program that provides immediate feedback to students and teachers on homework. ASSISTments can be used with any commercial or locally developed math curriculum, and teachers can assign "mastery" problem sets that organize practice to facilitate the achievement of proficiency.  

STEM Education Research Funding Opportunities

Research grant funding opportunities focusing on STEM education can be found across several programs and competitions. Currently, there are several active funding opportunities where training or research with a STEM education focus would fit:  

• Early Career Development and Mentoring Program for Education Research – Contact: Jennifer.Schellinger@ed.gov; Deadline: January 11, 2024
• Early Career Development and Mentoring Program for Faculty at Minority-Serving Institutions (MSIs) – Contact: Katina.Stapleton@ed.gov; Deadline: January 11, 2024
• Extending the Reach of the Career and Technical Education (CTE) Research Network – Contact: Corinne.Alfeld@ed.gov;  Deadline: January 11, 2024
• Digital Learning Platforms to Enable Efficient Education Research Network – Contact: Erin.Higgins@ed.gov; Deadline: January 11, 2024  

More information on these fundings opportunities can also be found at: https://ies.ed.gov/funding/

This blog was written by Sarah Brasiel (sarah.brasiel@ed.gov), program officer at NCSER and Christina Chhin (christina.chhin@ed.gov), program officer at NCER.

The Department of Education’s Small Business Innovation Research Program (SBIR), which IES administers, funds the research, development, and evaluation of new, commercially viable education technology products. In this guest blog, Abi Olukeye of Smart Girl HQ discusses the inspiration behind her recently completed SBIR project, Dear Smart Girl, and the importance of helping girls envision themselves in STEM Careers.

What is Dear Smart Girl?

Our Dear Smart Girl platform is a learning experience that combines online interactive game-based learning curriculum with offline engaging activity kits and personalized STEM learning recommendations to enable elementary-aged girls achieve STEM career literacy by age 12. Our Dear Smart Girl platform is the only STEM career education platform on the market with an ecosystem of products with a research-driven design featuring age-appropriate, experienced-based informal learning content designed to facilitate STEM discovery, self-efficacy, and fluency for young female-identifying learners.

Through our Dear Smart Girl digital platform, we transform the way girls perceive and engage in STEM learning pathways by providing an innovative multi-stage learning experience.

Learning begins in the whimsical town of Ingenia, where learners are immersed into a digital world that is designed to capture aesthetics and themes that resonate with female-identifying students. Learners begin by selecting a storyline, each of which is associated with a STEM career and features a project-based problem-solving activity. Along the way, students gain new vocabulary and fluency with the subject area being exposed to them. In stage two, learners take their new skills offline and work to complete the real-life version of their game project using our complementary Dear Smart Girl project kit. These two stages of learning combine powerfully to strategically introduce, challenge, and engage young girls in STEM career exploration that builds their confidence and literacy in STEM pathways.

What inspired you to create the Dear Smart Girl platform?

The idea was born out of personal need. I started observing that at about age 3 my daughters were often describing toys and activities as either “boy things” or “girl things.” I was really stunned to see how early biases develop and felt strongly about finding ways to balance out their views. My first instincts were to find more toys and activities that would appeal to them and activities that would expose them broadly. And I fully anticipated that a quick internet search would surface plenty of options. I was so wrong. Not only were there limited options, most of what I found did not appeal to my daughters in terms of type of activity and aesthetics.

Reflecting on my own experience as someone who has a STEM degree and was, at the time, working at a global manufacturing firm leading technical projects, I decided to dive into the research about girls and STEM. I discovered that although women participate equally in the labor force, they only make up 28% of the STEM workforce. In addition, early adolescence tends to be when girls lean away from STEM at a higher rate than their male peers. That inspired me to work with other talented and passionate people to build products and facilitate experiences and help young female-identifying learning achieve STEM career literacy by age 12.

What are the types of STEM careers featured in Dear Smart Girl and why did you choose them?

Our pilot career module is an electrical engineering module, but over the course of the next two years, we are working to add five more game modules featuring chemical science, mechanical engineering, biology, software development and product designer careers. We select careers based on science standards being taught in 4th–7th grade. Our goal is to take topical themes and relate them to their real-world applications while also putting them in the context of the career domains that features the scientific concept and related skills. We also working to align to the National Career Clusters framework, which supports Career Technical Education (CTE) programs. 

What elements of Dear Smart Girl are uniquely tailored to female-identifying students?

We tailor our product to help sustain interest in STEM Career pathways in the following ways.

• We are intentional about selecting and mapping careers in ways that show real-world relevance. Anecdotally, we find that when learners are excited about what they are building, they are more engaged and motivated to learn the skills needed to accomplish their goal.
• We optimize our projects to create an experience that is a perfect blend of learning a new technical skill and creative design. According to Microsoft research on closing the STEM gap, 91% of girls describe themselves as creatives. When girls learn about how real-world STEM jobs can be used to help the world, their perception of the creativity and positive impact of STEM careers can more than double.
• We use beautiful illustrations, colors, and imagery to creative engaging worlds and digital environment rich with representation for diverse female-identifying students.

What advice can you give technology developers who focus on female-identifying students?
For developers working on products designed for female-identifying students, I recommend a collaborative development approach. We co-create every career module we work on with students to get feedback early and often. While it is easy to fall for stereotypical storylines, female-identifying students have diverse needs, interests, and learning styles that should be celebrated with well-designed learning platforms.

What are the next steps for Dear Smart Girl?

We are so excited to have been recently award a phase 2 award to expand and commercialize our career exploration modules. Over next two years, we will develop and launch five additional career modules, expand educator tools, build in extension activities, and launch to CTE programs across the country.

Abi Olukeye is the founder and CEO of Smart Girl HQ, a company dedicated to closing the gender gap for females in the STEM pipeline by increasing the number of positive experiences young girls have with STEM early in their learning journey. Her vision is to create an ecosystem of products that when used together are a powerful catalyst for sustaining long-term engagement in STEM for young girls. Her work has been supported by National Science Foundation and the Department of Education through Small Business Innovation Research (SBIR) Awards, the NC Idea Foundation, and the Vela Education Fund. Abi is the chair of the board of CSEdResearch.org and a past member of the Computer Science K12 Standards Committee for North Carolina. She holds a bachelor’s degree in computer science from Virginia Tech and MBA from Indiana University. She and her husband live in Charlotte, NC and have two young daughters who greatly inspire her work.

This blog was produced by Katina Stapleton (Katina.Stapleton@ed.gov), co-chair of the IES Diversity Council.

In February 2023, NCSER hosted a technical working group (TWG) on the Special Education Teacher Workforce to help identify ways research can be used to better prepare, support, and retain an effective K-12 special education teacher workforce. During this meeting, a group of experts on the K-12 special education teacher workforce identified critical problems facing the special education teacher workforce, discussed areas where more research is needed, and highlighted existing data that could be leveraged to better understand the dynamics of and potential solutions to these problems.

TWG members highlighted the lack of collegial and leadership support as one contributing factor to burnout and attrition. Special education teachers often report feeling like they are the only ones in the school taking responsibility and advocating for students with disabilities. This is compounded by the fact that general education teachers and administrators often receive very little training on how to support these students. As such, TWG members highlighted the importance of supportive and collaborative relationships with paraprofessionals, other teachers, and leaders. Several NCSER-funded studies have explored these types of collaborative relationships or developed programs to foster them through mentoring or co-teaching. We summarize some examples of this type of NCSER-funded research below.

To better understand how working conditions, including support from colleagues, affect special education teacher instruction and student reading outcomes, Elizabeth Bettini from Boston University led a research project comprised of several mixed-methods studies. A key finding was that special education teachers who had teaching partners were better able to provide effective instruction because partners can manage significant behavior, which allows teachers to focus on instruction. This type of support was also found to be essential for inclusion, as special educators without sufficient paraprofessional staff struggle to move students who need behavioral supports into general education classes. The PI is currently building upon this research in a new project that is developing a measure, ReSpECT (Revealing Special Educators' Conditions for Teaching), of special education teacher working conditions.

To promote positive outcomes and retention among new special education teachers, Kristi Morin at Lehigh University is leading Project STAY. The purpose of this project is to develop an induction program for teachers of students with autism who are in the first 3 years of their career. In addition to ongoing training, the program includes mentorship from experienced teachers and participation in a network of novice teachers as ways to provide new teachers with instructional and social/emotional support. While this is an ongoing project, IES looks forward to the impact this research will have on new special educators.

To improve collaboration between special education teachers and content-area teachers in addressing literacy needs, Jade Wexler from the University of Maryland, College Park developed CALI (Content-Area Literacy Instruction) professional development. The program is designed to improve literacy instruction in co-taught content area classes by providing teachers with an instructional framework, a planning process to clarify teacher roles, and technical assistance for applying the framework and planning process to their practice. Results of the pilot study revealed that the program led to beneficial outcomes for teachers and students. The project also resulted in resources for teachers, including downloadable CALI materials and a special issue of Intervention in School and Clinic with guidance on how to implement evidence-based literacy practices in content-area classes.

While IES-funded researchers have been hard at work investigating ways to foster productive collaboration and studying its outcomes, there are still many issues affecting the special education teacher workforce that need further study. To address this, the Special Education Research and Development Center Program is accepting grant applications to establish a new K-12 Special Education Teacher Workforce Center, with a deadline of January 11, 2024. The new R&D Center will (1) conduct research on the special education teacher pipeline and the role of specific programs and policies in shaping the special education teacher workforce; (2) provide national leadership to build researcher capacity, improve data collection on the special education teacher workforce, and disseminate findings; and (3) engage in supplemental, just-in-time research and/or national leadership activities based on emerging needs in the field.

This blog was written by Shanna Bodenhamer, virtual student federal service intern at  NCSER and doctoral candidate at Texas A&M University, and Katherine Taylor (Katherine.Taylor@ed,gov), program officer for the projects featured in this article and the contact for the FY 2024 Special Education Research and Development Center Program.

In honor of Native American Heritage Month, IES is highlighting the National Indian Education Study (NIES) conducted by the National Center for Education Statistics (NCES) in partnership with the Office of Indian Education (OIE). Dr. Meredith Larson, who has been with the National Center for Education Research (NCER) since 2010 interviewed Dr. Jamie Deaton about NIES. Dr. Deaton has worked at NCES since 2009 and became the NIES Project Director in April 2010.

What is NIES, and how is it similar or different from other NAEP studies?

NIES describes the condition of education for American Indian and Alaska Native (AI/AN) students in the United States. Since 2005, NCES has administered it in conjunction with the National Assessment of Educational Progress (NAEP) state-level assessments in mathematics and reading at grades 4 and 8. The very large NAEP sample allows us to report data for AI/AN students nationally and for various subgroups of AI/AN students. In NIES, students first take either the NAEP mathematics or reading assessment, followed by a NAEP survey questionnaire, and then an NIES survey questionnaire (which emphasizes Native language and culture). Both NAEP and NIES survey questionnaires are also administered to the teachers and school administrators of AI/AN students. You can learn more about the survey design here.

What are some examples of how have policymakers, practitioners, or researchers used it?

NIES data has been used in Congressional testimony and at the state level. For example, NIES data has been included in past testimonies to the House of Representatives Subcommittee on Interior, Environment and Related Agencies; the House of Representatives Subcommittee on Early Childhood, Elementary, and Secondary Education; and the Senate Committee on Indian Affairs. At the state level, Oregon used NIES data to support a successful request to its state legislature to approve a full-time Indian Education Specialist within the Oregon Department of Education.

We also want to ensure that a variety of educational leaders—especially Native leaders—are aware of the study and can access the results and products. In addition to the online reports, we also produce hard copies to ensure results get to those without easy access to online documents. We help distribute these widely via a Native-owned NIES contractor (currently Tribal Tech, LLC) to Tribal colleges and universities, AI/AN studies programs at colleges and universities, all federal and state recognized tribes, AI/AN focused media, research centers, and other related AI/AN non-profits.

In addition, we want to get the results in the hands of school leaders. For example, all Bureau of Indian Education (BIE) schools serving grades 4 and 8 are in the NIES sample, and all of these schools receive hard copies of NIES reports.

What makes working on NIES study interesting to you?

Building partnerships with Native leaders both within and outside the federal government has been really rewarding. We administer NIES on behalf of the U.S. Department of Education’s Office of Indian Education (OIE) which provides funding for the study; integrates NIES data collection with its work when possible; and serves as a strong partner, advocate, and disseminator for NIES results. NIES is not only conducted in conjunction with the NAEP program but also conducted in conjunction with OIE’s work. Over the years, I have regularly presented to OIE grantees, and this has been a wonderful forum to share more about the study and also draw connections to and learn more about grantee-related work.

For NIES to be successful, it needs to be guided not just by assessment experts, but by Native experts. To this end, NCES established the NIES Technical Review Panel (TRP) made up of individuals with expertise in matters related to the education of AI/AN students. Members oversee the development of the NIES survey questionnaires and guide the planning, drafting, and revision of NIES publications with their ongoing expert consultation. In conjunction with the release of the last two NIES reports, the TRP has also authored a companion document, called  Setting the Context, that provides perspective on how this study fits into the larger sphere of education for AI/AN students. Tribal Tech recruits BIE schools for the study, disseminates study results at conferences focused on AI/AN education (for example, National Indian Education Association Annual Convention & Trade Show), and has established a long-term partnership with a Native-owned printing company (Sault Printing Company Inc.) that helps produce and disseminate NIES-related documents.

What excites you about NIES?

I’m very excited about last year’s release of the 2019 NIES Qualitative Data Companion as a public use data file (available in Excel files on the NIES main page). The data release marked the first time in NIES program history that qualitative survey questions, collected since 2005, became publicly available. Prior to the release, we ensured that all student and teacher responses were reviewed and edited to remove the presence of names or addresses and any other Personal Identifiable Information (PII). My hope is that having an established process for releasing this type of data will be beneficial to other IES data collections. Members of the TRP deserve a lot of credit for continuously advocating for this data. Had the TRP not done so, there was a real possibility that we would have dropped these qualitative questions from future data collections. Instead, we now have a model to follow for getting this data out to the public. We are also working on releasing previous NIES Qualitative Data Companions from earlier NIES administrations too.

From a research perspective, what do these qualitative data provide?

Researchers without a restricted-use data (RUD) license now can access this robust dataset. We recognize that many of our stakeholders live in remote areas and/or have other barriers to accessing the RUD (for example, those not affiliated with an institution). I think for doctoral candidates these data provide an opportunity for a dissertation with data already gathered and accessible for analysis.

There are many different angles to approach this open-ended data. For example, the final question on the NIES student survey is “What else would you like to say about yourself, your school, or about American Indian or Alaska Native people?” I’m curious what researchers would find as key differences when comparing grade 4 responses to grade 8. What are some themes and patterns in student respondents? What is the breakdown between responses that pertain to Native language and culture, and how does this differ across school types, such as public schools run by the states and schools operated through funding from the Bureau of Indian Education?  

Are there other resources for researchers interested in NIES data?

Another public use tool is the NIES Data Explorer available at NDE Core Web (nationsreportcard.gov). This explorer includes a wealth of data from all previous NIES administrations. If you go to the NDE Core Web page, you will find other relevant data explorers available including the Main NAEP, High School Transcript Study, and Long-Term Trend.

This blog was produced by Meredith Larson (Meredith.Larson@ed.gov), research analyst and program officer for postsecondary and adult education, NCER. Individuals or organizations interested in learning about field-initiated research or training grant opportunities to conduct work relevant to Native American/Alaska Native prekindergarten through postsecondary and adult education may contact her for initial technical assistance.