Inside IES Research

Notes from NCER & NCSER

Career Pathways Research at HHS: Lessons and Opportunities for Education Research

Over the past few years, staff from IES and the Office of Planning, Research, and Evaluation (OPRE) in the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS), have been learning from and supporting one another’s work. Our offices have a shared interest in understanding and improving outcome for adults in postsecondary career pathway programs and for creating a strong evidence base.

We have even funded projects that dovetail nicely. For example, IES funded a development project focusing on Year Up, and OPRE included Year Up programs in their Pathways for Advancing Careers and Education (PACE) Study.

For IES researchers not already aware of OPRE’s research, we would like to highlight three things that may be particularly relevant to the work IES hopes to support.

A Growing Portfolio of Career Pathway Research: For over a decade, OPRE has created a robust research portfolio longitudinal, rigorous experimental career pathways program research.  These career pathway programs provide postsecondary education and training through a series of manageable steps leading to successively higher credentials and employment opportunities. In particular, OPRE has supported the Pathways for Advancing Careers and Education (PACE) Study and the rigorous evaluation of the Health Profession Opportunity Grants (HPOG) Program. You can find reports from these and other self-sufficiency, welfare, and employment activities in the OPRE Resource Library.

Available Data and Funding Opportunity: Later this summer, OPRE will be archiving data from PACE and HPOG at the University of Michigan’s Inter-University Consortium on Political and Social Research (ICPSR) at the Institute for Social Research. These data will be available as restricted-use files for secondary data analysis. To encourage research, OPRE announced a funding opportunity Career Pathways Secondary Data Analysis Grants to support secondary analysis these data. Applications are due August 16, 2019.

Understanding Programs’ Motivations for Participating in Research: Getting education programs (schools, universities, etc.) to join multi-year, randomized controlled trials is difficult. Programs are wary of random assignment or finding null or negative effects. Yet, the programs that participated in the PACE Study were overall quite supportive. A recent report “We Get a Chance to Show Impact”, Program Staff Reflect on Participating in a Rigorous, Multi-site Evaluation documents the hurdles and benefits of participation from a program’s point of view. These programs’ insights are particularly useful for any researcher hoping to form partnerships with education settings.

To learn more about the ongoing career pathways research at OPRE and their findings, please visit https://www.acf.hhs.gov/opre/research/project/career-pathways-research-portfolio.

IES Makes Two New Awards for the Development of Web-based Tools to Inform Decision Making by Postsecondary Students

In June, the Institute of Education Sciences (IES) announced two new awards to technology firms to develop web-based tools that inform student decision making in postsecondary education. The projects will focus on generating a measure of the return of investment (ROI) for different educational training programs and careers so that high school and college students have access to data-driven information to guide their decisions.

The awards were made through a special topic offered by the IES Small Business Innovation Research (known as ED/IES SBIR) program, which funds the research and development of commercially viable education technology. (For information on the 21 awards made through the IES 2019 standard solicitation, read here.)

Background and Awards

While websites like College Scorecard and CareerOneStop provide information to explore training programs in colleges and occupations of interest, there is no tool that helps students understand the costs and benefits of individual postsecondary programs in an integrated, customizable, and user-friendly manner.  

The special topic SBIR solicitation requested proposals from small businesses to develop new ROI tools that would combine information on fees, time to complete, and projected earnings so that students can easily compare college and career pathways. The IES-funded ROI tools aim to improve student program completion rates, with higher employment and earnings, less education-related debt, and more satisfaction with their selected paths. The special topic SBIR solicitation offered up to $200,000 for firms to develop and evaluate a prototype of their ROI tool. 

Two awards were made through this special topic:

  • Illinois-based BrightHive, Inc. is developing a prototype of the Training, Education, and Apprenticeship Program Outcomes Toolkit (TEAPOT). Designed to inform student training and educational decision making over a variety of potential pathways, TEAPOT will improve the flow and accuracy of data resulting in improved estimates of the ROI for different postsecondary education pathways.  The team will develop a data interoperability system and simplified toolkit for states and local postsecondary and workforce development organizations. The toolkit will provide more high quality, consistent, and granular information on postsecondary outcomes. The prototype will calculate ROI using student information, programmatic information (with an emphasis on net program costs to allow for variations by program type at the same institution), and access to wage and employment data sets.
  • Virginia-based Vantage Point Consultants is developing a prototype of a user-contextualized ROI tool that prospective students will use to make meaning of lifetime costs and opportunity tradeoffs associated with different degree programs offered by postsecondary institutions. The ROI tool will incorporate information on student goals and academic, professional, and personal characteristics.  The prototype will include an interface to present information to guide decision making based on an ROI calculation that discounts earning cash-flows under current and future state career and education assumptions, while subtracting college cost. In the first phase of work, the project will use information from data partners including Burning Glass Technologies and from public sources at the Department of Labor and Department of Education.

After developing prototypes, researchers will analyze whether the tools function as intended and are feasible for students to use. Research will also test if the tool shows promise for producing a meaningful and accurate measure of ROI.  Both firms are eligible to apply for additional funding to complete the full-scale development of the ROI tool, including developing an interface to improve user experience and conducting additional validation research.

Stay tuned for updates on Twitter (@IESResearch) as IES projects drive innovative forms of technology.

Written by Edward Metz, program manager, ED/IES SBIR

CAPR: Answers to Pressing Questions in Developmental Education

Since 2014, IES has funded the Center for the Analysis of Postsecondary Readiness (CAPR) to answer questions about the rapidly evolving landscape of developmental education at community colleges and open-access four-year institutions. CAPR is providing new insights into how colleges are reforming developmental education and how their reforms are impacting student outcomes through three major studies:

  • A survey and interviews about developmental education practices and reform initiatives
  • An evaluation of the use of multiple measures for assessing college readiness
  • An evaluation of math pathways.

Preliminary results from these studies indicate that some reforms help more students finish their developmental requirements and go on to do well in college-level math and English.

National Study of Developmental Education Policies and Practices

CAPR has documented widespread reform in developmental education at two- and four-year colleges through a national survey and interviews on developmental education practices and reforms. Early results from the survey show that colleges are moving away from relying solely on standardized tests for placing students into developmental courses. Colleges are also using new approaches to delivering developmental education including shortening developmental sequences by compressing or combining courses, using technology to deliver self-paced instruction, and placing developmental students into college-level courses with extra supports, often called corequisite remediation.

Developmental Math Instructional Methods in Public Two-Year Colleges (Percentages of Colleges Implementing Specific Reform Strategies)

Notes: Percentages among two-year public colleges that reported offering developmental courses. Colleges were counted as using an instructional method if they used it in at least two course sections. Categories are not mutually exclusive.

Evaluation of Developmental Math Pathways and Student Outcomes

CAPR has teamed up with the Charles A. Dana Center at the University of Texas at Austin to evaluate the Dana Center Mathematics Pathways (DCMP) curriculum at four community colleges in Texas. The math pathways model tailors math courses to particular majors, with a statistics pathway for social science majors, a quantitative reasoning pathway for humanities majors, and an algebra-to-calculus pathway for STEM majors. DCMP originally compressed developmental math into one semester, though now the Dana Center is recommending corequisite models. Instructors seek to engage students by delving deeply into math concepts, focusing on real-world problems, and having students work together to develop solutions.

Interim results show that larger percentages of students assigned to DCMP (versus the traditional developmental sequence) enrolled in and passed developmental math. More of the DCMP students also took and passed college-level math, fulfilling an important graduation requirement. After three semesters, 25 percent of program group students passed a college-level math course, compared with 17 percent of students assigned to traditional remediation.

Evaluation of Alternative Placement Systems and Student Outcomes (aka Multiple Measures)

CAPR is also studying the impact of using a combination of measures—such as high school GPA, years out of high school, and placement test scores—to predict whether students belong in developmental or college-level courses. Early results from the multiple measures study show that, in English and to a lesser extent in math, the multiple measures algorithms placed more students into college-level courses, and more students passed those courses (compared to students placed with a single test score).

 

College-Level English Course Placement, Enrollment, and Completion in CAPR’s Multiple Measures Study (Percentages Compared Across Placement Conditions)

 

College-Level Math Course Placement and Completion in CAPR’s Multiple Measures Study

Looking Ahead to the Future of Developmental Education

These early results from CAPR’s evaluations of multiple measures and math pathways suggest that those reforms are likely to be important pieces of future developmental education systems. CAPR will release final results from its three studies in 2019 and 2020.

Guest blog by Nikki Edgecombe and Alexander Mayer

Nikki Edgecombe is the principal investigator of the Center for the Analysis of Postsecondary Readiness, an IES-funded center led by the Community College Research Center (CCRC) and MDRC, and a senior research scientist at CCRC. Alexander Mayer is the co-principal investigator of CAPR and deputy director of postsecondary education at MDRC.

Evaluating Oregon’s Adult Basic Skills Transition Planning Process: An Interview with Judith Alamprese

In her 2017 IES grant, Judith Alamprese (Abt Associates) is collaborating with the state of Oregon’s Office Community College and Workforce Development to evaluate a program that aims to help adults earn a GED® and transition into postsecondary education. This project is funded under the Low-Cost, Short-Duration Evaluation of Education Interventions competition, which supports research that aims to produce meaningful results for local and/or state education agencies quickly. Program Officer, Meredith Larson, interviewed Ms. Alamprese about this current work, how it came into being, and what it might mean for Oregon and adult education more broadly.

Tell us about your area of research and why it’s important to Oregon.

My current research is focused on determining effective interventions for assisting low-skilled adults establish and succeed in a career pathway. Oregon was one of the first states to implement a statewide initiative for transitioning adult basic skills learners (henceforth adult learners) to further education and work, and this project expands Oregon’s activities to support adult learners’ success.  

What is your current project studying?

The Transition Planning Process (TPP) project is a collaboration between Oregon’s Office of Community College and Workforce Development (CCWD) and Abt Associates (Abt). We are using a randomized controlled trial (RCT) to test whether text messaging helps adult learners earn a GED® and transition to postsecondary education and training.

What is TPP, and why is this approach innovative?

TPP is a text messaging intervention in which transition facilitators who work with the adult learners send text messages to the learners to help keep them on track to complete their GED® and enroll in postsecondary courses. The intervention is a supplement to the facilitators’ other transition activities to prepare learners for next steps in education and work.

TPP has a standardized list of text messages to prompt learners to take the GED® tests, set college goals, access information on college planning and other college preparation activities. Facilitators can send texts customized to programs’ specific transition activities.

CCWD chose text messaging because it appeared to be a low-cost approach that could support existing transition activities and provide a boost to ABS learners. The TPP project is an exciting opportunity to determine whether texting can be effective with ABS learners, and may be a promising approach for encouraging specific behaviors in learners preparing to go to college.

How did this project come into being?

The TPP project grew out of Abt’s and CCWD’s work together on an IES Researcher-Practitioner Partnership grant. This grant was a longitudinal study of Oregon’s Pathways for Adult Basic Skills Transition to Postsecondary Education and Work initiative. The findings from Abt’s analyses of adult learners’ GED® attainment and postsecondary participation prompted Oregon to want to try some additional strategies to encourage ABS learners to earn a secondary credential and enroll in postsecondary courses.

What is the current status of the project?

The study is underway, and transition facilitators are providing text messages to encourage adult learners to initiate and complete GED® testing, determine next steps, and begin the postsecondary planning process. The facilitators have found that while many treatment group learners respond to the texts, some learners have chosen to increase their face-to-face interaction with their facilitators. The facilitators report that texting is an efficient way to reinforce learners and check on their progress.

Why is this work important?

This research is particularly important because it is a rigorous test of an intervention that could be beneficial to adult basic skill learners nationwide and could leverage such programs’ existing activities in transitioning learners from basic skills programs to further education and training. We will learn more about the types of information and support that are most persuasive in helping learners succeed.

Teaching 21st Century Skills to Community College Students: An Innovative Approach Under Development in California

As part of our series recognizing Career and Technical Education (CTE) month, we interviewed Mary Visher, Senior Associate at MDRC, about her recently funded study, Teaching and Learning 21st Century Skills in Community Colleges: A Study of the New World of Work Program (NWoW). This project is developing and testing an innovative program aimed at teaching 21st century skills to community college students in CTE courses. Researchers, policymakers, and practitioners are working closely together to improve upon NWoW, which is already in use in several community colleges in California.

What are 21st century skills, and why are they important?

There is growing consensus among researchers, practitioners, and employers that 21st century skills–e.g., adaptability, analytic mindset, collaboration, and communication–are essential for success in both school and in the modern workplace. We believe that postsecondary programs need to incorporate strategies to ensure that students graduate with these skills.

Why are you (personally) excited about this work?

About two years ago, I was interviewing community college students in California about their experiences when a young man walked into the room. The first thing I noticed were his tattoos, which covered every inch of his exposed skin. The next thing I noticed was how he strode over to me, stretched out his hand to shake mine with a firm grip, looked me in the eye, smiled warmly and introduced himself. Having interviewed hundreds of community college students for other research projects, I had rarely – if ever – encountered this level of self-assurance, respectful courtesy, and ability to immediately adapt and respond to an unfamiliar person with an unfamiliar purpose.

The young man told me that he had been incarcerated as a teen for gang involvement, and, after being discharged from prison, he couldn’t find work. With few other options, he enrolled in the diesel technology program at his local community college, but this was not an ordinary CTE program. It included NWoW.

Through NWoW, he learned 21st century skills in the context of learning diesel technology skills and had worksite experiences to practice both. He told me that this part of his education “changed his life.” He did so well in a job interview at a food processing equipment manufacturer that he was not only hired, but quickly promoted to a management position. At the time we met, he was to receive company training and another promotion, but he still planned on earning his certificate. After that, he planned on applying to a state university to pursue a BA or a master’s degree.

The young man credited his professional success to NWoW, where he learned behaviors and skills no one else had taught him. It is exciting to be a part of developing and testing a program that may affect the lives of adult students in such ways.

How did NWoW come into being?

In 2015, faculty at a community college in California noticed the deficiencies in students’ soft skills and developed a 21st century skills curriculum to use in their classes. They added a work-based learning and an assessment component a short time later. Soon thereafter, the California Community Colleges Chancellor’s Office noticed their work and gave them support to further develop the program and take it to more colleges. 

What are the core components of the NWoW program?

NWoW is designed to promote growth in 10 skills and has 3 core components, all 3 of which will undergo an iterative development process in the next 2 years:

  1. A curriculum designed to be embedded in CTE community college courses;
  2. A work-based learning component to allow students practice the skills in an authentic work setting; and
  3. An assessment/credentialing component allowing students to earn a “digital badge” in each of the 10 skills.

What is your research goal?

Our goal is to work with instructors (including the original program designers), employers, and other partners to further refine and enhance this program. Moreover, we hope to address important questions in the field about whether or how such skills can be taught and learned in the classroom, how to measure them, and how to signal competency to employers and others. 

Who else is involved?

MDRC is working with the NWoW team and its partners, including the California Community Colleges Chancellor’s Office, the Foundation for California Community Colleges, LinkedIn, and badgr. The development work is ongoing with three community colleges, and we will then test the improved version in a new set of colleges.

By Mary Visher, Senior Associate at MDR​C, interviewed by Meredith Larson, NCER