IES Blog

Institute of Education Sciences

Introducing REL 2022

As I write this, my colleagues and I at the Regional Educational Laboratory (REL) Program are thinking about a single number: 535. No, we’re not concerned about 535 because it represents the number of voting members of Congress, though that would be a good guess. We’re also not thinking about Interstate 535, the “2.78-mile-long Auxiliary Interstate Highway spur of I-35 in the U.S. states of Minnesota and Wisconsin,” though now I’m intensely interested in why it might be that, at least according to Wikipedia, this road is “known locally as the ‘Can of Worms’ interchange.” Instead, my colleagues and I are excited about 535 because it represents the number of days between now and the start of the next cycle of the REL program, affectionately known as REL 2022.

Over a year ago, we began a process that culminates in the awarding of contracts to run each of our regional labs. We are excited to share our preliminary thoughts about the contours of REL 2022 through a Request for Information, or RFI, which we have posted hereI hope you will take time to read the RFI. If you have questions or suggestions after doing so, I hope you are moved to comment. Details on how to offer your feedback can be found in the RFI.

Importantly, we aren’t proposing to radically restructure the REL program. Instead, we are retooling some existing expectations and adding a few new features. Below, I’ve highlighted a few proposed changes that merit special attention.

The purpose of RELs is to improve student outcomes. Not to put too fine a point on it, but everything that takes place in REL 2022 should be in service of improving student outcomes. This does not mean that every REL project will, by itself, have a directly observable impact on achievement. But the work of any given REL, in concert with the efforts of those with whom it works, should be trained on a singular focus: bettering the lives of the students through education. There is no other, better, or higher calling.

We accomplish our purpose by working in partnership with stakeholders to support their use of evidence-based practices. Evidence-based practice is “baked in” to the statute that authorizes the REL program, and the importance of building and using evidence in education—and government more generally—is reiterated throughout federal law. (See, for example, the Every Student Succeeds Act of 2015 and the Foundations for Evidence-based Policymaking Act of 2018.) However, our emphasis on evidence isn’t rooted in a statutory imperative. Instead, it’s based on a set of core beliefs about our work: that researchers and educators can strengthen education via the rigorous application of the scientific method; that resources, including money and time, are constrained and that efforts with demonstrated effectiveness should be prioritized; and that each and every student deserves the best of “what works” in education, no matter their circumstance.

Nothing changes if nothing changes. In the REL 2022 cycle, we are explicitly asking RELs to think of themselves as “change agents.” This expectation is, I believe, entirely new to the REL Program and is likely to be uncomfortable to some. For that reason, it is helpful to be clear about what we’re expecting and why. Here goes.

I daresay that, no matter how proud they might be of their students and their educators, there is not a state chief, a district superintendent, or building principal who would report they are serving each of their students as well as they wish they could. (If you’re the one who does, please stop reading this blog and call me. I want to share your successes!) Each of those leaders has something they want to do better on behalf of their students and are contemplating, if not actively pursuing, change. It is our hope that RELs can join them in making change, with evidence in hand and research tools at the ready. REL reports, resources, and trainings are not ends unto themselves. They are means to enable the change efforts of local, state, and regional education leaders, working on behalf of students to improve important outcomes.

RELs work in partnership. Education research and technical assistance must be done in partnership with those it is meant to inform. Absent that, it is likely to fail to achieve its goals. At best, potentially positive impacts will be blunted. At worst, harm will be done. There’s a simple solution: collaboration that authentically engages stakeholders in all phases of project design and execution. That isn’t, I realize, as simple to do as it is to write.

As vendors consider the REL 2022 cycle, we ask that they keep two things in mind about what we’ve traditionally called partnerships. First, there are no necessary restrictions on who RELs can partner with when working with stakeholders to achieve stakeholder goals. Does it make sense to partner across levels of education within a state? Do it. Is there a state or national advocacy association that would accelerate a partner’s progress? Engage it. Is there are role for business or industry? Leverage it. A second and closely related concept is that there are no restrictions on partnerships’ functional forms. In general, it does not matter one whit to IES whether you prefer NICs, DBIR, or any other particular form of research partnership. What does? That RELs build projects in partnership—however and with whomever—intentionally, with the goal of supporting partners’ change efforts to achieve the goals they have identified.

We encourage deeper, not broader, work. We believe RELs are more likely to achieve success when they focus partnerships on clearly defined problems of policy or practice in specific geographies. A “Six-State Research Alliance on High School Graduation” can do important and meaningful work—but the process of agreeing on the work to be done and the targets to be met, seeing that work through to completion, and then achieving pre-specified goals is likely to be exceptionally difficult. The “South-Central Kansas Partnership for Kindergarten Readiness” or the “Maricopa County Alliance for Reducing Chronic Absenteeism in High Schools” may be more likely to achieve impact. This is not to say that lessons learned locally should not be shared regionally or nationally, or that groups with common interests might not form “communities of practice” or other networks for the purpose of sharing information or building connection. Rather, we ask RELs be strategic in scoping their highest-intensity work.

We define success as achieving measurable stakeholder goals. Evaluating the impact of research and technical assistance projects is notoriously hard. Often, program managers and the evaluators with whom they work are forced to satisfice, relying upon end-user self-reports of the quality, relevance, and usefulness of a provider’s work. Counts of outputs, such as report downloads and attendees served, are particularly common metrics reported in evaluation studies. Satisfaction is the coin of the realm. Lest I be accused of throwing stones inside my own glass house, let me be clear that we currently use these very measures to characterize the effectiveness of the current REL program.

In REL 2022, it is our intention to shift focus beyond outputs to emphasize outcomes. We will ask RELs to demonstrate, on a regular basis, that they are making progress toward the goals stakeholders set for important student outcomes at the outset of their work, with the acknowledgment that outputs are often critical to achieving a long-term goal and that satisfaction can be an important leading indicator. In 2027, the mark of success won’t be a glowing narrative from a state superintendent or school superintendent about the REL cycle just passed. Instead, it’ll be seeing that the quantifiable goals those leaders set for their work with the REL program were achieved.   

Putting RELs’ capacity for rigorous R&D to work. Finally, there is one manifestly new requirement for RELs as part of the 2022 cycle, one that I am particularly excited about because it brings together the best of two NCEE programs: the RELs and the What Works Clearinghouse™ (WWC). As part of the 2022 cycle, each REL will be required to develop—and then evaluate—a comprehensive toolkit based on a WWC Practice Guide, helping educators instantiate evidence-based practices in the classroom. RELs already have experience taking the content from Practice Guides and transforming them into tools for educators. Two examples include Professional Learning Community guides for both foundational reading and English learners. Similarly, North Carolina State University’s Friday Institute has looked to Practice Guides for inspiration to develop massive open online courses (MOOCs), including foundational reading and fractions. None have been evaluated for efficacy. Of course, the development and testing of these new toolkits will follow the expectations set above, including the expectation that strong and inclusive partnerships are at the root of all high-leverage work.

My NCEE colleagues and I are excited about the possibilities that REL 2022 represents. The REL program has a proud history and a strong track record of service to local, state, and regional stakeholders. We hope that, as you review the REL 2022 RFI, you’ll find the next iteration of the program continues in that tradition. As always, I welcome your feedback.

Matthew Soldner

Commissioner, National Center for Education Evaluation and Regional Assistance

 

NHES Data Files Provide Researchers Supplemental Information on Survey Respondents’ Communities

Increasingly, researchers are merging survey data with data from external sources, such as administrative data or different surveys, to enhance analyses. Combining data across sources increases the usefulness of the data while minimizing the burden on survey respondents.

In September, the National Household Education Surveys Program (NHES) released restricted-use supplemental geocode data files that use sample respondents’ addresses to integrate the 2016 NHES Parent and Family Involvement in Education (PFI), Early Childhood Program Participation (ECPP), and Adult Training and Education (ATES) survey data with data from other collections. The supplemental geocode files include additional geographic identifiers, characteristics of respondents’ neighborhoods and local labor markets, radius-based measures of household proximity to job search assistance and educational opportunities, and, for surveys focused on children, school district identifiers based on home addresses and school district characteristics.

The new data can complement researchers’ analyses of data from all three surveys. Researchers can expand their analyses of school choice and access to K–12 schooling options using the PFI survey data. Those interested in analyses of decisions about children’s early education can use the ECPP survey data to look at the availability of Head Start programs, preschools in private schools near children’s homes, and the prevalence of prekindergarten programs in local school districts. Researchers interested in nondegree credential attainment and training for work can use data from the ATES to find information on local labor markets and the number of American Job Centers near respondents’ homes.

The NHES:2016 restricted-use supplemental geocode files are available to restricted-use license holders to be used in conjunction with the NHES:2016 survey data files. To access the full set of NHES:2016 geocode supplemental restricted-use data files, apply for a restricted-use license. You can also browse the list of variables in the supplemental geocode files.

 

By Emily Isenberg and Sarah Grady, NCES

Equity: Alignment of Mission and Methods

Editor's Note: The following post was originally posted on the IES-funded CTE Research Network. The grantee has given us permission to post it on the IES blog.

Funded in 2018 by the Institute of Education Sciences (IES), the Career and Technical Education (CTE) Research Network aims to conduct and promote high-quality casual studies examining the impact of career and technical education. Aligned with the theme of the January 2020 IES Principal Investigators Meeting – Closing the Gaps for All Learners – the Network’s activities include working to deepen the field’s understanding of issues of equity and inequity in CTE research and evaluation.

 

The importance of understanding equity in CTE research

The Wisconsin Department of Public Instruction defines equity in the following way:

“Every student has access to the educational resources and rigor they need at the right moment in their education across race, gender, ethnicity, language, disability, sexual orientation, family background, and/or family income.”

An explicit focus on equity in CTE is particularly important considering that in the not so distant past, vocational education (a precursor to the term career and technical education, or CTE) often served as the track for youth deemed “unable to learn” or “not college material.” In many cases, vocational education was used to systematically relegate students—many of whom were low-income, Black or African American, Latinx, or American Indian—into low-wage jobs that offered limited opportunities for growth.

Today, the focus of CTE has expanded to include fields in science, technology, engineering and mathematics (STEM) and represents for many young people an opportunity to graduate high school and enter postsecondary education or the labor market with highly valued skills and certifications in numerous fields. As CTE has evolved, participation has become associated with a variety of positive outcomes. For example, researchers have found that CTE course taking is associated with higher high school graduation and postsecondary enrollment rates, higher labor market earnings, and better overall student outcomes.

While these positive CTE outcomes are promising, there is more to understand about the causal outcomes associated with CTE participation, especially among subgroups of students based on race, gender, socioeconomic level, and ability status. IES and the CTE Research Network are committed to deepening the field’s understanding of equity and inequity in CTE studies. Along with acknowledging the pernicious ways in which vocational education has historically been used to discriminate against some students and disaggregating outcome data by student subpopulation (an emphasis in recent Perkins V legislation), the network concludes that at a minimum, engaging in equity-minded research and evaluation requires:

  • Establishing diverse research teams: Research has shown that diversity on teams yields greater innovation, more productivity, and better financial results (Levine, 2020). With these benefits in mind, it is important to be intentional in creating diverse research teams that can bring new perspectives, voices, and approaches to studies that aim to identify, analyze and interpret equity data.
  • Adopting an equity mindset in research and evaluation: To inform the field’s understanding of how CTE may promote or inhibit equitable student outcomes, researchers must commit to recognizing their own biases and examining how those biases may influence their research designs and analyses. An equity mindset also requires capturing and analyzing patterns of inequities that appear in administrative and implementation data.
  • Exploring intersectionality: Adopting an equity mindset—as important for research as is using valid and reliable measures—also requires conducting analyses of CTE outcomes that go beyond merely examining differences between subpopulations. Rather, analyses should also examine intersectionality within subpopulations (for example, by gender and race), which affords the field a more nuanced understanding of how outcomes for members of the same subpopulation may vary by other dimensions of identity (such as gender or ability status). Such analyses can help the field understand what works and for whom—information that can help drive policy and practice.
  • Addressing the systems, policies, and procedures that promote inequities: Inequities do not exist in a vacuum. Thus, it is important to contextualize causal CTE studies, acknowledging how systems, policies, and procedures may create barriers to success for some students. Analyses that take an ecosystems approach—focusing on how the social, economic, and geographic environment shapes outcomes—provide valuable insight into the nature of inequities that exist and how these inequities might be overcome. Equally important is to identify the possible or probable causes of inequities to understand how race, gender, and other variables influence students’ experiences in CTE. Analyses must also extend beyond merely identifying average effect sizes to investigating variation in treatment experiences by subpopulations, an approach that provides valuable insights into how young people in different subpopulations fare relative to their peers in specific contexts. Using data and analysis in this way can provide the evidence needed to support policy recommendations aimed at closing equity gaps and creating the conditions that all students need to transition successfully into adulthood.
  • Engaging the communities that participate in our studies: Because evidence is critical for making data-driven decisions, it is important when designing causal studies to include the participating communities and other stakeholders in the knowledge generation and interpretation processes. These communities and stakeholders can also play an important role in informing researchers’ understanding of the specific causes of inequities identified in study findings. Research should be an inclusive process—the communities being studied and those directly affected by research findings should be included in the planning, implementation, and interpretation of research.
  • Asking what more is needed to promote equity: Embracing equity as a measure of success in education research will take time and will require a significant shift in the way research is conceptualized, designed, and conducted. However, to promote a more just society, it is imperative that researchers keep equity at the center of their work.

Although the CTE Research Network is funded to conduct causal studies, which can play a role in identifying inequities, we realize that other research methods also play a role in deepening the field’s understanding of such inequities. For example, qualitative and implementation research can be used to gain important insight into the contextual factors that shape or reinforce inequities and can also be used to engage stakeholders as informants on the topic. Therefore, building the field’s knowledge of these issues will require employing a range of data collection efforts.

In the meantime, the CTE Research Network is taking the following action steps to continue to advance our equity-minded approach to CTE research:

  • Developing a set of equity questions to consistently consider during network convenings
  • Elevating issues of equity in all network presentations
  • Sharing resources on equity to help network members think critically about how best to bring an equity lens to bear on research and evaluation studies
  • Creating and promoting opportunities to help diversify researchers engaged in causal CTE research

As a network, we believe these research practices will shine a light on (in)equity in CTE. Where inequities exist, we hope our work will inform education policymaking that aims not only to close existing equity gaps but also to prevent the perpetuation of inequities in CTE. We invite other researchers to join us in this effort by taking similar action steps as part of their own research and evaluation endeavors. The following resources can inform researchers’ understanding of equity issues in general and in CTE studies in particular:

 

References

Andrews, K., Parekh, J., & Peckoo, S. (2019). How to embed a racial and ethnic equity perspective in research: Practical guidance for the research process. Washington, DC: Child Trends.

Dougherty, S. M. (2016). Career and technical education in high school: Does it improve student outcomes? Washington, DC: Thomas B. Fordham Institute. https://eric.ed.gov/?id=ED570132

Hemelt, S. W., Lenard, M. A., & Paeplow, C. G. (2017). Building better bridges to life after high school: Experimental evidence on contemporary career academies. Washington, DC: National Center for Analysis of Longitudinal Data in Education Research. https://eric.ed.gov/?id=ED572934

Hodge, E., Dougherty, S., & Burris, C. (2020). Tracking and the future of career and technical education: How efforts to connect school and work can avoid the past mistakes of vocational education. Boulder, CO: National Education Policy Center. Retrieved from http://nepc.colorado.edu/publication/cte

Kemple, J. (2008). Career academies: Long-term impacts on work, education, and transitions to adulthood. New York: MDRC. Retrieved from https://www.mdrc.org/publication/career-academies-long-term-impacts-work-education-and-transitions-adulthood

Rosen, R., & Molina, F. (2019). Practitioner perspectives on equity in career and technical education. New York: MDRC. https://eric.ed.gov/?id=ED596458


Written by Equity in CTE Workgroup, on behalf of the CTE Research Network

This is the fifth in a series of blog posts that stems from the 2020 Annual Principal Investigators Meeting. The theme of the meeting was Closing the Gaps for All Learners and focused on IES’s objective to support research that improves equity in access to education and education outcomes. Other posts in this series include Addressing Persistent Disparities in Education Through IES ResearchWhy I Want to Become an Education ResearcherDiversify Education Sciences? Yes, We Can!, and Closing the Opportunity Gap Through Instructional Alternatives to Exclusionary Discipline.

 

 

IES is Investing $20 Million to Improve Opportunities and Achievement for English Learners in Secondary School Settings

IES supports Research and Development Centers (R&D Centers) to conduct focused, scientific research on key education issues that face our nation. One of the most pressing issues currently is a need for a large-scale, coordinated research effort to improve opportunities and achievement for English Learners (ELs) in secondary school settings. These students face numerous barriers and challenges in accessing education opportunities, as they simultaneously develop English language proficiency and subject-matter knowledge. This has resulted in large, persistent differences in academic achievement outcomes, for example, between ELs and non-ELs on the 2017 National Assessment of Educational Progress (NAEP) in reading and math. These differences have consequences for how prepared (or unprepared) students will be to access opportunities in the workforce and postsecondary education.

The National Center for Education Research (NCER), within IES, is funding two national R&D centers on the topic of “Improving Opportunities and Achievements for English Learners in Secondary School Settings” to contribute to the solution of this complex education problem. These R&D centers aim to identify systemic—including policies, rules, and processes—and instructional influences that affect ELs’ access to the general curriculum and its relation to their outcomes. The centers will carry out complementary projects and engage in research, development, evaluation, and national leadership activities. These activities will facilitate meaningful EL participation in the general curriculum and generate insights about levers that have the potential to improve access and education outcomes for ELs in secondary school settings.

Each of the centers is conducting two lines of research, in which they will:

  • Identify and describe the policies and system-level practices that are associated with secondary ELs’ access to the general curriculum and their relation to education outcomes
  • Identify approaches to improving secondary ELs’ access to and ability to learn from instruction in general education courses where English is the language of instruction.

 

The R&D Centers

Transdisciplinary Approaches to Improving Opportunities and Outcomes for English Learners: Using Engagement, Team-Based Learning, and Formative Assessment to Develop Content and Language Proficiency is headed by Dr. David Francis at the University of Houston. This center will undertake a five-year, focused program of research aimed at identifying and removing barriers related to school-level practices influenced by policies that constrain course-taking, as well as developing and testing interventions that leverage transdisciplinary approaches to improve instruction for ELs in grades 6 and 9 science and social studies. These approaches include—

  • Foregrounding content to build language through content instruction
  • Using activities that are engaging and meaningful to students while involving students in the practices of the discipline
  • Organizing learning in heterogeneous teams (Team-Based Learning) to promote collaboration, discussion, and social motivation
  • Integrating formative assessment to improve teachers’ and students’ understanding of students’ development.

The center will also engage in a systematic program of outreach and dissemination at national, state, regional, and local levels, to help connect research findings to policy and practice.

The National Research and Development Center to Improve Education for Secondary English Learners is headed by Aída Walqui at WestEd. The center’s five-year program of research focuses on understanding policies and systems-level practices that are associated with ELs’ access to the general curriculum and how curriculum resources can strengthen the learning opportunities and experiences of both teachers and ELs as they engage in disciplinary practices. The center will—

  • Examine course-taking patterns across multiple states and explore how content-area course access is related to student outcomes
  • Describe models for co-teaching across the country and its implementation in diverse settings
  • Iteratively design and develop curricular materials in English language arts (ELA) and a math summer bridge course for grade 8 students to help teachers shift their pedagogical practices.

In addition, the center will engage in national leadership, capacity-building, and outreach activities to connect research findings to policy and practice and offer policymakers and practitioners compelling and actionable information.

 

Through the Education Research and Development Center Program, NCER’s goal is for the two centers to collectively and individually create innovative solutions, contribute to knowledge and theory, and provide reliable information about how to improve educational opportunities and achievement for ELs in secondary school settings. The body of research conducted by the two centers will identify and address the policies and system-level practices across the US that act as barriers to accessing rigorous courses. Their work will also inform instruction in four critical content areas of secondary instruction—science, social studies, ELA, and math—in several states, including Texas, Massachusetts, New York, and California.

 

For more information about IES’s support on the topic of English Learners, please see the program page; for current funding opportunities for research and training more generally, please see here. Stay tuned for updates on Twitter (@IESResearch) and Facebook as IES continues to support research to improve educational outcomes for English Learners.


Written by Helyn Kim (Helyn.Kim@ed.gov), Program Officer for English Learners at the National Center for Education Research.

 

IES Makes Two New SBIR Awards for the Full-Scale Development of Web-based Tools to Inform Decision Making by Postsecondary Students

The Institute of Education Sciences (IES) has announced two new awards to technology firms to fully develop web-based tools that inform student decision making in postsecondary education. The projects will focus on generating a measure of the return of investment (ROI) for different educational training programs and careers, so that high school and college students have access to data-driven information to guide their decisions.

The awards were made through a special topic offered by the U.S. Department of Education/IES Small Business Innovation Research (known as ED/IES SBIR) program, which funds the research and development of commercially viable education technology. (For information on the 22 awards made through the IES 2020 standard solicitation, read here.)

 

Background and Awards

While websites like College Scorecard and CareerOneStop provide information to explore training programs in colleges and occupations of interest, there is no tool that helps students understand the costs and benefits of individual postsecondary programs in an integrated, customizable, and user-friendly manner.  

A 2019 Phase I SBIR special topic solicitation requested proposals from small businesses to develop new ROI tools that would improve program completion rates, employment and earnings, and satisfaction with college or career, all while minimizing education-related debt. In 2019, two Phase I awards (see here and here) were made to firms to develop prototype ROI tools. The 2020 Phase II SBIR special topic solicitation requested proposals from the two 2019 Phase I awardees with plans for the full-scale development of the ROI tools, as well as an interface to present information to students and users.  Both firms received 2020 Phase II awards through this special topic:

 

  • Illinois-based BrightHive, Inc. is fully developing an open-source toolkit that states and networks can use to calculate metrics that support and inform decision-making by postsecondary students. The Training, Education, and Apprenticeship Program Outcomes Toolkit (TEAPOT) will include several components to support implementation, including data requirements, a field guide with instructional materials and technical tools to help partners to securely and ethically combine wage and education data, a resource hub containing expert guidance and recommendations for deploying the ROI metrics in different contexts and apps, and a reference interface that allows states and networks to present ROI data as part of their career guidance applications.

 

  • Virginia-based Vantage Point Consultants is fully developing the Return on College (ROC) tool to support students, families, and counselors with understanding of the lifetime costs and opportunity tradeoffs associated of different postsecondary degree programs.  In one version of ROC, students will be able to search colleges nationwide based on geographic location and occupational goals, with personalized output on projected debt, loan repayment, disposable income, graduation risk, and the ROI focal points of these decisions. In the other version of ROC, prospective students visiting a college website will receive data driven information on potential careers, salaries, price, and debt expected for degrees offered by a school.

 

After development is complete in early 2022, researchers will analyze whether the tools function as intended and are feasible for students to use. Research will also test if the tools show promise for producing a meaningful and accurate measure of ROI. Both firms will also focus on preparing for the wide-scale distribution of the tools to end users, as well as plans for sustaining the tools over time.

Stay tuned for updates on Twitter (@IESResearch) as IES projects drive innovative forms of technology.


Written by Edward Metz (Edward.Metz@ed.gov), Program Manager, ED/IES SBIR