Inside IES Research

Notes from NCER & NCSER

Cost Analysis in Practice (CAP) Project Provides Guidance and Assistance

In 2020, as part of a wider IES investment in resources around cost, IES funded the Cost Analysis in Practice (CAP) Project, a 3-year initiative to support researchers and practitioners who are planning or conducting a cost analysis of educational programs and practices. The CAP Project Help Desk provides free on-demand tools, guidance, and technical assistance, such as support with a cost analysis plan for a grant proposal. After inquiries are submitted to the Help Desk, a member of the CAP Project Team reaches out within two business days. Below is a list of resources that you can access to get more information about cost analysis.

 

STAGES FOR CONDUCTING A COST ANALYSIS

 

CAP Project Resources

Cost Analysis Standards and Guidelines 1.0: Practical guidelines for designing and executing cost analyses of educational programs.

IES 2021 RFAs Cost Analysis Requirements: Chart summarizing the CAP Project’s interpretation of the IES 2021 RFAs cost analysis requirements.

Cost Analysis Plan Checklist: Checklist for comprehensive cost analysis plans of educational programs and interventions.

Introduction to Cost Analysis: Video (17 mins).

 

General Cost Analysis Resources

The Critical Importance of Costs for Education Decisions: Background on cost analysis methods and applications.

Cost Analysis: A Starter Kit: An introduction to cost analysis concepts and steps.

CostOut®: Free IES-funded software to facilitate calculation of costs once you have your ingredients list, includes database of prices.

DecisionMaker®: Free software to facilitate evidence-based decision- making using a cost-utility framework.

Cost-Effectiveness Analysis of Early Reading Programs: A Demonstration With Recommendations for Future Research: Open access journal article.

 

*More resources available here.


The content for this blog has been adapted from the Cost Analysis in Practice Project informational flyer (CAP Project, 2020) provided by the CAP Project Team. To contact the CAP Help Desk for assistance, please go to https://capproject.org/. You can also find them on Twitter @The_CAP_Project.

IES Grantees Receive SPR Awards

Three IES-funded investigators were presented with awards from the Society for Prevention Research (SPR) last week. We recognize and applaud these investigators (pictured above from left to right) Elizabeth Stormshak, Dorothy Espelage, and Patrick Tolan.

Dr. Elizabeth Stormshak, Philip H. Knight Chair and Department Head for Counseling Psychology and Human Services at University of Oregon, received the Translational Science Award from SPR. This award is given to an individual or a team of individuals in recognition for contributions to the field of prevention science through translational research. Dr. Stormshak’s research focuses on understanding risk factors in early and middle childhood associated with the development of problem behavior in late adolescence, including substance use and delinquency. She also studies the process of disseminating evidence-based interventions into real world community settings. She has been the Principal Investigator on multiple IES grants, including a 2018 NCSER-funded project to examine the long-term efficacy of the Kindergarten Family Check-Up (FCU), a school-based, family-centered intervention intended to prevent student social and behavioral problems. This grant is a follow-up and extension to her recently completed randomized controlled trial of Kindergarten FCU, funded by NCER, which found positive impacts on student behavior and academic outcomes during and up to 3 years after the transition to kindergarten. The primary aims of the newer grant are to determine the long-term impact of receiving the original kindergarten intervention and the effects of a middle school booster session of FCU on students' behavior and academic outcomes. Dr. Stormshak is also the director of a NCSER-funded postdoctoral training program focused on the prevention of school-based social and behavioral problems.

Dr. Dorothy Espelage, William C. Friday Distinguished Professor of Education at the University of North Carolina, Chapel Hill, received SPR’s Prevention Science Award. This award is given for the application of scientific methods to develop and test prevention strategies. Dr. Espelage is a leading expert on school safety and has led multiple research studies on school-based violence and bullying. She has been involved in several IES grants. Most recently, she is serving as the Principal Investigator of a 2019 NCSER-funded project to develop and test a professional development program aimed at enhancing elementary school teachers' knowledge and skills for identifying, mitigating, and preventing bullying among students with and without disabilities. Dr. Espelage also led an exploratory study funded by NCER to better understand how teacher practices influence elementary school students’ interpersonal relationships in the classroom and related behavioral outcomes. 

Dr. Patrick H. Tolan, Charles S. Robb Professor of Education at the University of Virginia in the Curry School of Education and in the Department of Psychiatry and Neurobehavioral Sciences in the School of Medicine, and Director Emeritus of the YouthNex Center in the UVA Center to Promote Effective Youth Development, received the Advances in Culture and Diversity in Prevention Science Award as one of three founding members of the Boys of Color Collaborative. This award is given for contributions to the field of prevention science in the area of community and culture. Dr. Tolan’s research career spans 34 years with a focus on program evaluation for promoting positive youth development and preventing youth violence. Dr. Tolan is currently the Principal Investigator of a 2019 Follow-Up study of an IES efficacy study of the integration of two prevention programs, Good Behavior Game and My Teaching Partner (GBG+MTP), for teachers who have recently entered the teaching profession.

Congratulations to the award recipients!

This blog was co-authored by Jackie Buckley (NCSER), Katie Taylor (NCSER), Emily Doolittle (NCER), and Amy Sussman (NCSER).

 

Using Mistakes as a Vehicle for Learning in Mathematics: From Research to Practice at Scale in Education

Every student makes mistakes. But not every student is given the opportunity to learn from mistakes. Left unaddressed, the mathematical misconceptions that underlie many mistakes can keep students from progressing in mathematics.

 

At the request of districts in the Minority Student Achievement Network (MSAN), a Strategic Education Research Partnership (SERP) team was convened in 2007 to address a widening achievement gap in Algebra I. The team was charged with identifying an intervention strategy, subject to several district constraints:

  1. The solution would need to be applied to all students in the regular classroom to avoid the stereotype threat associated with separating students based on performance and to protect the intervention from budget cuts that target supplemental, after-school, and summer programs first.
  2. A new curriculum was off the table because it would create upheaval for a time and would be followed by a decline in student performance during the period of adjustment.
  3. Extensive teacher training was considered undesirable because it would be costly and because algebra teachers consider themselves more expert in mathematics teaching than central office staff who would be requiring the training.

 

Julie Booth joined the partnership, and with funding from IES, led the iterative development and testing of worked example assignments that, with the input of teachers and administrators, fit within the routines of the classroom. The result—AlgebraByExample—consists of 42 uniquely designed assignments that address misconceptions, harness the power of explanation, and use mistakes as a vehicle for learning.

Typical math assignments require students to solve problems on their own. If a student’s work is incorrect, the student may never focus on what went wrong. ByExample assignments also give students problems to solve, but they first provide a solution to a similar problem that is marked right or wrong. Students are prompted with questions that target common misconceptions and errors before solving a similar problem on their own. Each assignment contains several strategically designed item pairs:

 

 

Designed in collaboration with teachers from districts in several states, the assignments can be easily incorporated into any Algebra I curriculum and teachers can choose in what way and in what order to use them. The assignments were tested in randomized trials in classrooms in eight districts with more than 6,000 students. Not only did students using AlgebraByExample improve an average of 7 percentage points on an assessment of standardized test items, students at the lower end of the distribution improved the most. The PDF downloads of the assignments are freely available for anyone to use.

The success of AlgebraByExample  led to  further IES funding of MathByExample for Grades 4 and 5 and GeometryByExample for high school geometry .

 

Resources:

AlgebraByExample website

MathByExample website

Booth et al, 2015

NSF Stem for All Video Submission 2019

 

Interview with Dr. Suzanne Donovan (SERP), Dr. Julie Booth (Temple University), and Allie Huyghe (SERP), the developers of the ByExample interventions.

 

 

Was it part of the original plan to develop an intervention that could one day be used at scale in schools?

Yes. SERP partnerships begin with problems of practice nominated by district partners, but the partnership agreement distinguishes SERP from a consultant. The intention from the start is to frame the problem and design a solution that can be used at scale. SERP has developed in-house, user-centered design expertise so that resources (such as the ByExample products) developed through partnerships meet the needs of teachers and students. Products scale when they improve the experience of teachers and students. Both the model and the internal design capacity allow SERP to move from problem framing through research, development, and dissemination of a product with IES grant funding.

 

Describe the initial research and development that occurred.

Dr. Julie Booth drafted initial assignments drawing on the mathematics misconceptions literature. SERP held regular partnership meetings with teachers and administrators at which assignments were reviewed and additional misconceptions were nominated for attention in the assignments. Administrators agreed to randomization of the assignments across classrooms and within-teacher. Assignments were first tested in individual topic blocks and revised in accordance with student performance data, observations, and teacher feedback. A year-long pilot study was then conducted using the full set of assignments.

 

Beyond IES or ED grants, what additional funding was needed to develop the intervention?

For the ByExample work, additional funding was provided by the Goldman Sachs Foundation in the initial phase to support partnership formation, problem framing, and the solution generation. IES grants funded the research and development, along with initial dissemination activities to make the materials available to the public. We were also able to develop an online platform to allow for digital use with the IES grant funds.

 

What model was used for dissemination and sustainability?

The assignments are available as free downloads on SERP’s website, and as printed workbooks through SERP’s partner print-on-demand company. They have been publicized through online communications, journal articles, presentations at conferences of various types, social media, and word of mouth. There will be a small fee for use of the digital platform to support its maintenance, but the PDFs will remain as free downloads. We have been able to sustain the collaboration of the partnership team by responding to requests from educators to expand the approach to other grade levels and submitting additional proposals to IES that have been awarded.

 

What advice would you provide to researchers who are looking to move their research from the lab to market? What steps should they take? What resources should they look for?

First, I would note that it is difficult to persuade educators to use a product that solves a problem they don’t believe they have. Listen to educators and apply research expertise to address the challenges that they experience on a day-to-day basis. Design for ease of use by teachers. No matter how good your strategy or marketing is, if it’s too much work for an already busy teacher to use, you may get uptake by a few committed teachers, but not at scale. Finally, pay attention to where teachers get their information. For AlgebraByExample, we got a big boost from the Marshall Report, produced by a teacher for other teachers to call attention to usable research.  

 

In one sentence, what would you say is most needed for gaining traction and wide scale use by educators?

Design for the routines of the classroom.

 


Suzanne Donovan, PhD, is the founding Executive Director of the SERP Institute, an education research, development, and implementation organization incubated at the National Academies. SERP leads collaborations of educators, researchers, and designers to generate research-based, scalable, and sustainable solutions to critical problems of practice. 

Julie Booth, PhD, is a Professor of STEM Education and Psychology and the Deputy Dean of Academic and Faculty Affairs at Temple University’s College of Education and Human Development. Her work focuses on translating between cognitive science and education to better understand students’ learning and improve instruction, primarily in mathematics education. She is currently an Executive Editor for the Journal of Experimental Education.

Allie Huyghe is the Assistant Director of the SERP Institute, where she manages several projects, including the IES-funded MathbyExample and GeometryByExample projects. She is also intricately involved with other SERP areas of work, participating in the design of materials from early development through release to the public.

 

This interview was produced by Christina Chhin (Christina.Chhin@ed.gov) and Edward Metz (Edward.Metz@ed.gov) of the Institute of Education Sciences. This is the fifth in an ongoing series of blog posts examining moving from university research to practice at scale in education.​

 

 

Equity: Alignment of Mission and Methods

Editor's Note: The following post was originally posted on the IES-funded CTE Research Network. The grantee has given us permission to post it on the IES blog.

 

 

Funded in 2018 by the Institute of Education Sciences (IES), the Career and Technical Education (CTE) Research Network aims to conduct and promote high-quality casual studies examining the impact of career and technical education. Aligned with the theme of the January 2020 IES Principal Investigators Meeting – Closing the Gaps for All Learners – the Network’s activities include working to deepen the field’s understanding of issues of equity and inequity in CTE research and evaluation.

The importance of understanding equity in CTE research

The Wisconsin Department of Public Instruction defines equity in the following way:

“Every student has access to the educational resources and rigor they need at the right moment in their education across race, gender, ethnicity, language, disability, sexual orientation, family background, and/or family income.”

An explicit focus on equity in CTE is particularly important considering that in the not so distant past, vocational education (a precursor to the term career and technical education, or CTE) often served as the track for youth deemed “unable to learn” or “not college material.” In many cases, vocational education was used to systematically relegate students—many of whom were low-income, Black or African American, Latinx, or American Indian—into low-wage jobs that offered limited opportunities for growth.

Today, the focus of CTE has expanded to include fields in science, technology, engineering and mathematics (STEM) and represents for many young people an opportunity to graduate high school and enter postsecondary education or the labor market with highly valued skills and certifications in numerous fields. As CTE has evolved, participation has become associated with a variety of positive outcomes. For example, researchers have found that CTE course taking is associated with higher high school graduation and postsecondary enrollment rates, higher labor market earnings, and better overall student outcomes.

While these positive CTE outcomes are promising, there is more to understand about the causal outcomes associated with CTE participation, especially among subgroups of students based on race, gender, socioeconomic level, and ability status. IES and the CTE Research Network are committed to deepening the field’s understanding of equity and inequity in CTE studies. Along with acknowledging the pernicious ways in which vocational education has historically been used to discriminate against some students and disaggregating outcome data by student subpopulation (an emphasis in recent Perkins V legislation), the network concludes that at a minimum, engaging in equity-minded research and evaluation requires:

  • Establishing diverse research teams: Research has shown that diversity on teams yields greater innovation, more productivity, and better financial results (Levine, 2020). With these benefits in mind, it is important to be intentional in creating diverse research teams that can bring new perspectives, voices, and approaches to studies that aim to identify, analyze and interpret equity data.
  • Adopting an equity mindset in research and evaluation: To inform the field’s understanding of how CTE may promote or inhibit equitable student outcomes, researchers must commit to recognizing their own biases and examining how those biases may influence their research designs and analyses. An equity mindset also requires capturing and analyzing patterns of inequities that appear in administrative and implementation data.
  • Exploring intersectionality: Adopting an equity mindset—as important for research as is using valid and reliable measures—also requires conducting analyses of CTE outcomes that go beyond merely examining differences between subpopulations. Rather, analyses should also examine intersectionality within subpopulations (for example, by gender and race), which affords the field a more nuanced understanding of how outcomes for members of the same subpopulation may vary by other dimensions of identity (such as gender or ability status). Such analyses can help the field understand what works and for whom—information that can help drive policy and practice.
  • Addressing the systems, policies, and procedures that promote inequities: Inequities do not exist in a vacuum. Thus, it is important to contextualize causal CTE studies, acknowledging how systems, policies, and procedures may create barriers to success for some students. Analyses that take an ecosystems approach—focusing on how the social, economic, and geographic environment shapes outcomes—provide valuable insight into the nature of inequities that exist and how these inequities might be overcome. Equally important is to identify the possible or probable causes of inequities to understand how race, gender, and other variables influence students’ experiences in CTE. Analyses must also extend beyond merely identifying average effect sizes to investigating variation in treatment experiences by subpopulations, an approach that provides valuable insights into how young people in different subpopulations fare relative to their peers in specific contexts. Using data and analysis in this way can provide the evidence needed to support policy recommendations aimed at closing equity gaps and creating the conditions that all students need to transition successfully into adulthood.
  • Engaging the communities that participate in our studies: Because evidence is critical for making data-driven decisions, it is important when designing causal studies to include the participating communities and other stakeholders in the knowledge generation and interpretation processes. These communities and stakeholders can also play an important role in informing researchers’ understanding of the specific causes of inequities identified in study findings. Research should be an inclusive process—the communities being studied and those directly affected by research findings should be included in the planning, implementation, and interpretation of research.
  • Asking what more is needed to promote equity: Embracing equity as a measure of success in education research will take time and will require a significant shift in the way research is conceptualized, designed, and conducted. However, to promote a more just society, it is imperative that researchers keep equity at the center of their work.

Although the CTE Research Network is funded to conduct causal studies, which can play a role in identifying inequities, we realize that other research methods also play a role in deepening the field’s understanding of such inequities. For example, qualitative and implementation research can be used to gain important insight into the contextual factors that shape or reinforce inequities and can also be used to engage stakeholders as informants on the topic. Therefore, building the field’s knowledge of these issues will require employing a range of data collection efforts.

In the meantime, the CTE Research Network is taking the following action steps to continue to advance our equity-minded approach to CTE research:

  • Developing a set of equity questions to consistently consider during network convenings
  • Elevating issues of equity in all network presentations
  • Sharing resources on equity to help network members think critically about how best to bring an equity lens to bear on research and evaluation studies
  • Creating and promoting opportunities to help diversify researchers engaged in causal CTE research

As a network, we believe these research practices will shine a light on (in)equity in CTE. Where inequities exist, we hope our work will inform education policymaking that aims not only to close existing equity gaps but also to prevent the perpetuation of inequities in CTE. We invite other researchers to join us in this effort by taking similar action steps as part of their own research and evaluation endeavors. The following resources can inform researchers’ understanding of equity issues in general and in CTE studies in particular:

 

References

Andrews, K., Parekh, J., & Peckoo, S. (2019). How to embed a racial and ethnic equity perspective in research: Practical guidance for the research process. Washington, DC: Child Trends.

Dougherty, S. M. (2016). Career and technical education in high school: Does it improve student outcomes? Washington, DC: Thomas B. Fordham Institute. https://eric.ed.gov/?id=ED570132

Hemelt, S. W., Lenard, M. A., & Paeplow, C. G. (2017). Building better bridges to life after high school: Experimental evidence on contemporary career academies. Washington, DC: National Center for Analysis of Longitudinal Data in Education Research. https://eric.ed.gov/?id=ED572934

Hodge, E., Dougherty, S., & Burris, C. (2020). Tracking and the future of career and technical education: How efforts to connect school and work can avoid the past mistakes of vocational education. Boulder, CO: National Education Policy Center. Retrieved from http://nepc.colorado.edu/publication/cte

Kemple, J. (2008). Career academies: Long-term impacts on work, education, and transitions to adulthood. New York: MDRC. Retrieved from https://www.mdrc.org/publication/career-academies-long-term-impacts-work-education-and-transitions-adulthood

Rosen, R., & Molina, F. (2019). Practitioner perspectives on equity in career and technical education. New York: MDRC. https://eric.ed.gov/?id=ED596458


Written by Equity in CTE Workgroup, on behalf of the CTE Research Network

This is the fifth in a series of blog posts that stems from the 2020 Annual Principal Investigators Meeting. The theme of the meeting was Closing the Gaps for All Learners and focused on IES’s objective to support research that improves equity in access to education and education outcomes. Other posts in this series include Addressing Persistent Disparities in Education Through IES ResearchWhy I Want to Become an Education ResearcherDiversify Education Sciences? Yes, We Can!, and Closing the Opportunity Gap Through Instructional Alternatives to Exclusionary Discipline.

 

 

IES is Investing $20 Million to Improve Opportunities and Achievement for English Learners in Secondary School Settings

 

IES supports Research and Development Centers (R&D Centers) to conduct focused, scientific research on key education issues that face our nation. One of the most pressing issues currently is a need for a large-scale, coordinated research effort to improve opportunities and achievement for English Learners (ELs) in secondary school settings. These students face numerous barriers and challenges in accessing education opportunities, as they simultaneously develop English language proficiency and subject-matter knowledge. This has resulted in large, persistent differences in academic achievement outcomes, for example, between ELs and non-ELs on the 2017 National Assessment of Educational Progress (NAEP) in reading and math. These differences have consequences for how prepared (or unprepared) students will be to access opportunities in the workforce and postsecondary education.

The National Center for Education Research (NCER), within IES, is funding two national R&D centers on the topic of “Improving Opportunities and Achievements for English Learners in Secondary School Settings” to contribute to the solution of this complex education problem. These R&D centers aim to identify systemic—including policies, rules, and processes—and instructional influences that affect ELs’ access to the general curriculum and its relation to their outcomes. The centers will carry out complementary projects and engage in research, development, evaluation, and national leadership activities. These activities will facilitate meaningful EL participation in the general curriculum and generate insights about levers that have the potential to improve access and education outcomes for ELs in secondary school settings.

Each of the centers is conducting two lines of research, in which they will:

  • Identify and describe the policies and system-level practices that are associated with secondary ELs’ access to the general curriculum and their relation to education outcomes
  • Identify approaches to improving secondary ELs’ access to and ability to learn from instruction in general education courses where English is the language of instruction.

 

The R&D Centers

Transdisciplinary Approaches to Improving Opportunities and Outcomes for English Learners: Using Engagement, Team-Based Learning, and Formative Assessment to Develop Content and Language Proficiency is headed by Dr. David Francis at the University of Houston. This center will undertake a five-year, focused program of research aimed at identifying and removing barriers related to school-level practices influenced by policies that constrain course-taking, as well as developing and testing interventions that leverage transdisciplinary approaches to improve instruction for ELs in grades 6 and 9 science and social studies. These approaches include—

  • Foregrounding content to build language through content instruction
  • Using activities that are engaging and meaningful to students while involving students in the practices of the discipline
  • Organizing learning in heterogeneous teams (Team-Based Learning) to promote collaboration, discussion, and social motivation
  • Integrating formative assessment to improve teachers’ and students’ understanding of students’ development.

The center will also engage in a systematic program of outreach and dissemination at national, state, regional, and local levels, to help connect research findings to policy and practice.

The National Research and Development Center to Improve Education for Secondary English Learners is headed by Aída Walqui at WestEd. The center’s five-year program of research focuses on understanding policies and systems-level practices that are associated with ELs’ access to the general curriculum and how curriculum resources can strengthen the learning opportunities and experiences of both teachers and ELs as they engage in disciplinary practices. The center will—

  • Examine course-taking patterns across multiple states and explore how content-area course access is related to student outcomes
  • Describe models for co-teaching across the country and its implementation in diverse settings
  • Iteratively design and develop curricular materials in English language arts (ELA) and a math summer bridge course for grade 8 students to help teachers shift their pedagogical practices.

In addition, the center will engage in national leadership, capacity-building, and outreach activities to connect research findings to policy and practice and offer policymakers and practitioners compelling and actionable information.

 

Through the Education Research and Development Center Program, NCER’s goal is for the two centers to collectively and individually create innovative solutions, contribute to knowledge and theory, and provide reliable information about how to improve educational opportunities and achievement for ELs in secondary school settings. The body of research conducted by the two centers will identify and address the policies and system-level practices across the US that act as barriers to accessing rigorous courses. Their work will also inform instruction in four critical content areas of secondary instruction—science, social studies, ELA, and math—in several states, including Texas, Massachusetts, New York, and California.

 

For more information about IES’s support on the topic of English Learners, please see the program page; for current funding opportunities for research and training more generally, please see here. Stay tuned for updates on Twitter (@IESResearch) and Facebook as IES continues to support research to improve educational outcomes for English Learners.


Written by Helyn Kim (Helyn.Kim@ed.gov), Program Officer for English Learners at the National Center for Education Research.