Inside IES Research

Notes from NCER & NCSER

IES Announces Forthcoming Funding Opportunity For the R&D of an “ROI Tool” to Inform Students’ Postsecondary Education and Career Decision Making

Students with electronic devices sitting against a wall.

Overview

On or about February 15, 2019, the Small Business Innovation Research Program at the US Department of Education’s Institute of Education Sciences (ED/IES SBIR) anticipates releasing a Special Topic Solicitation #91990019R0016 in Postsecondary Education. The solicitation will be announced through an IES Newsflash and will be posted here. It will request Phase I proposals for awards of up to $200,000 for 8 months to develop a prototype of a "ROI tool.” The tool will be designed to measure the costs versus benefits (the return on investment) of different postsecondary education and training programs to help students make well-informed choices about options to pursue after they complete high school.

Applicants must be a for-profit business 500 employees or less, and U.S. owned and operated. Applicants may partner with entities or organizations working on related initiatives in the field of postsecondary education, or may subcontract to non-profit researchers or individuals with specialized expertise as needed. The due date for submission for proposals will likely be on or about April 15, 2019, with awards in mid-June, and projects beginning shortly thereafter. All Phase I awardees will be eligible to apply for a Phase II award in 2020, for $900,000 for full scale development and research to test and validate the ROI tool.

Background

While many websites provide ways for students to explore colleges or careers and occupations of interest (e.g., such as College Scorecard and CareerOneStop), there is currently no tool that helps students understand the costs and benefits of individual postsecondary programs in an integrated, customizable, and user-friendly manner.  An ROI tool would likely combine information on individual program’s tuition and fees, time needed to complete, and expected earnings. Because these characteristics can vary significantly across programs and institutions, creating a single estimated measure of ROI would allow students to more easily compare postsecondary program options. If it helps students make better choices, it could lead to improved program completion rates, higher levels of employment and earnings, less education-related debt, and more satisfaction with their selected education and career paths. 

The ED/IES SBIR Special Topic intends to fund up to five (5) Phase I projects to (a) develop and research a prototype of an ROI tool, and (b) conduct planning and concept testing for a fully developed ROI tool that provides a user-friendly experience for students. The prototype of the ROI tool developed in Phase I shall integrate with one or more existing technology systems, data sets, data standards, or resources (such as CareerOneStop or College Scorecard), and add new data elements provided by an end-user.  After a successful Phase I project, it is anticipated that small businesses that win Phase II awards will complete the full-scale development of the ROI tool that was started in Phase I, including developing an interface to improve the experience of students using the ROI tool.

Because data for ROI at the program level may only be available from some states, regions, or sets of institutions at this time, it is expected that the scope of the ROI tool developed in Phase I & II would be limited and would not be an attempt to calculate ROI for every program and institution in the country. Applicants must propose a project scope that appropriately reflects the datasets that are to be integrated within the new ROI tool, and the amount of funding and time allotted for development and research of the SBIR awards in Phase I and II.  Small businesses that are interested in this solicitation must have expertise with related efforts in the field to enhance student choices by linking education and workforce information.

Potential applicants may submit questions to ED’s Contracting Specialist Kevin.Wade@ed.gov. All questions and responses will be posted publically on the same website where the solicitation is posted as Amendments to the Solicitation.

 

CAPR: Answers to Pressing Questions in Developmental Education

Since 2014, IES has funded the Center for the Analysis of Postsecondary Readiness (CAPR) to answer questions about the rapidly evolving landscape of developmental education at community colleges and open-access four-year institutions. CAPR is providing new insights into how colleges are reforming developmental education and how their reforms are impacting student outcomes through three major studies:

  • A survey and interviews about developmental education practices and reform initiatives
  • An evaluation of the use of multiple measures for assessing college readiness
  • An evaluation of math pathways.

Preliminary results from these studies indicate that some reforms help more students finish their developmental requirements and go on to do well in college-level math and English.

National Study of Developmental Education Policies and Practices

CAPR has documented widespread reform in developmental education at two- and four-year colleges through a national survey and interviews on developmental education practices and reforms. Early results from the survey show that colleges are moving away from relying solely on standardized tests for placing students into developmental courses. Colleges are also using new approaches to delivering developmental education including shortening developmental sequences by compressing or combining courses, using technology to deliver self-paced instruction, and placing developmental students into college-level courses with extra supports, often called corequisite remediation.

Developmental Math Instructional Methods in Public Two-Year Colleges (Percentages of Colleges Implementing Specific Reform Strategies)

Notes: Percentages among two-year public colleges that reported offering developmental courses. Colleges were counted as using an instructional method if they used it in at least two course sections. Categories are not mutually exclusive.

Evaluation of Developmental Math Pathways and Student Outcomes

CAPR has teamed up with the Charles A. Dana Center at the University of Texas at Austin to evaluate the Dana Center Mathematics Pathways (DCMP) curriculum at four community colleges in Texas. The math pathways model tailors math courses to particular majors, with a statistics pathway for social science majors, a quantitative reasoning pathway for humanities majors, and an algebra-to-calculus pathway for STEM majors. DCMP originally compressed developmental math into one semester, though now the Dana Center is recommending corequisite models. Instructors seek to engage students by delving deeply into math concepts, focusing on real-world problems, and having students work together to develop solutions.

Interim results show that larger percentages of students assigned to DCMP (versus the traditional developmental sequence) enrolled in and passed developmental math. More of the DCMP students also took and passed college-level math, fulfilling an important graduation requirement. After three semesters, 25 percent of program group students passed a college-level math course, compared with 17 percent of students assigned to traditional remediation.

Evaluation of Alternative Placement Systems and Student Outcomes (aka Multiple Measures)

CAPR is also studying the impact of using a combination of measures—such as high school GPA, years out of high school, and placement test scores—to predict whether students belong in developmental or college-level courses. Early results from the multiple measures study show that, in English and to a lesser extent in math, the multiple measures algorithms placed more students into college-level courses, and more students passed those courses (compared to students placed with a single test score).

 

College-Level English Course Placement, Enrollment, and Completion in CAPR’s Multiple Measures Study (Percentages Compared Across Placement Conditions)

 

College-Level Math Course Placement and Completion in CAPR’s Multiple Measures Study

Looking Ahead to the Future of Developmental Education

These early results from CAPR’s evaluations of multiple measures and math pathways suggest that those reforms are likely to be important pieces of future developmental education systems. CAPR will release final results from its three studies in 2019 and 2020.

Guest blog by Nikki Edgecombe and Alexander Mayer

Nikki Edgecombe is the principal investigator of the Center for the Analysis of Postsecondary Readiness, an IES-funded center led by the Community College Research Center (CCRC) and MDRC, and a senior research scientist at CCRC. Alexander Mayer is the co-principal investigator of CAPR and deputy director of postsecondary education at MDRC.

Teaching 21st Century Skills to Community College Students: An Innovative Approach Under Development in California

As part of our series recognizing Career and Technical Education (CTE) month, we interviewed Mary Visher, Senior Associate at MDRC, about her recently funded study, Teaching and Learning 21st Century Skills in Community Colleges: A Study of the New World of Work Program (NWoW). This project is developing and testing an innovative program aimed at teaching 21st century skills to community college students in CTE courses. Researchers, policymakers, and practitioners are working closely together to improve upon NWoW, which is already in use in several community colleges in California.

What are 21st century skills, and why are they important?

There is growing consensus among researchers, practitioners, and employers that 21st century skills–e.g., adaptability, analytic mindset, collaboration, and communication–are essential for success in both school and in the modern workplace. We believe that postsecondary programs need to incorporate strategies to ensure that students graduate with these skills.

Why are you (personally) excited about this work?

About two years ago, I was interviewing community college students in California about their experiences when a young man walked into the room. The first thing I noticed were his tattoos, which covered every inch of his exposed skin. The next thing I noticed was how he strode over to me, stretched out his hand to shake mine with a firm grip, looked me in the eye, smiled warmly and introduced himself. Having interviewed hundreds of community college students for other research projects, I had rarely – if ever – encountered this level of self-assurance, respectful courtesy, and ability to immediately adapt and respond to an unfamiliar person with an unfamiliar purpose.

The young man told me that he had been incarcerated as a teen for gang involvement, and, after being discharged from prison, he couldn’t find work. With few other options, he enrolled in the diesel technology program at his local community college, but this was not an ordinary CTE program. It included NWoW.

Through NWoW, he learned 21st century skills in the context of learning diesel technology skills and had worksite experiences to practice both. He told me that this part of his education “changed his life.” He did so well in a job interview at a food processing equipment manufacturer that he was not only hired, but quickly promoted to a management position. At the time we met, he was to receive company training and another promotion, but he still planned on earning his certificate. After that, he planned on applying to a state university to pursue a BA or a master’s degree.

The young man credited his professional success to NWoW, where he learned behaviors and skills no one else had taught him. It is exciting to be a part of developing and testing a program that may affect the lives of adult students in such ways.

How did NWoW come into being?

In 2015, faculty at a community college in California noticed the deficiencies in students’ soft skills and developed a 21st century skills curriculum to use in their classes. They added a work-based learning and an assessment component a short time later. Soon thereafter, the California Community Colleges Chancellor’s Office noticed their work and gave them support to further develop the program and take it to more colleges. 

What are the core components of the NWoW program?

NWoW is designed to promote growth in 10 skills and has 3 core components, all 3 of which will undergo an iterative development process in the next 2 years:

  1. A curriculum designed to be embedded in CTE community college courses;
  2. A work-based learning component to allow students practice the skills in an authentic work setting; and
  3. An assessment/credentialing component allowing students to earn a “digital badge” in each of the 10 skills.

What is your research goal?

Our goal is to work with instructors (including the original program designers), employers, and other partners to further refine and enhance this program. Moreover, we hope to address important questions in the field about whether or how such skills can be taught and learned in the classroom, how to measure them, and how to signal competency to employers and others. 

Who else is involved?

MDRC is working with the NWoW team and its partners, including the California Community Colleges Chancellor’s Office, the Foundation for California Community Colleges, LinkedIn, and badgr. The development work is ongoing with three community colleges, and we will then test the improved version in a new set of colleges.

By Mary Visher, Senior Associate at MDR​C, interviewed by Meredith Larson, NCER

 

What Are the Payoffs to College Degrees, Credentials, and Credits?

The Center for Analysis of Postsecondary Education and Employment (CAPSEE) is an IES-funded Research and Development Center that seeks to advance knowledge regarding the link between postsecondary education and the labor market. CAPSEE was funded through a 2011 grant from the National Center for Education Research (NCER) and is in the process of completing its work. CAPSEE will hold a final conference to discuss its findings on April 6 & 7 in Washington, DC.

Recently, Tom Bailey (pictured), Director of the Community College Research Center, Columbia University, Teachers College, and the Principal Investigator for CAPSEE, answered questions from James Benson, the NCER Program Officer for the R & D center.

Can you describe some of the original goals of CAPSEE?

We were especially interested in the economic benefits of a college education for community college students, including those who complete awards (Associate’s degrees or certificates) and those who do not, as well as those who transfer to four-year colleges. We were also interested in differences in earnings by field of study. When we started CAPSEE in 2012 there were a lot of studies that used survey datasets to look, in general, at the returns to completing a Bachelor’s degree. The CAPSEE approach was to use large-scale statewide databases and follow college students over time, to look in detail at their earnings before, during, and after college.

In addition, CAPSEE researchers sought to examine two key policy issues. One was how financial aid and working while enrolled affect students’ performance in college and their labor market outcomes. The other was whether for-profit colleges help students get better jobs.

You have synthesized findings from analyses in six states. What are your main findings?

We found that, in general, Associate’s degrees have good returns in the labor market; they’re a good investment for the individual and for society. However, there is quite a bit of variation in returns by program. For students in Associate degree programs primarily designed to prepare them for transfer to a four-year college, if they don’t transfer, their degrees will not be worth very much. But when they complete vocational degrees, especially in health-related fields, the earnings gains are usually strong and persistent (and robust to how we estimated them). Also, we did a lot of research on certificates, credentials that many see as the best fit for students on the margin of going to college. We found benefits to students who completed certificates, again especially in fields that directly relate to an occupation or industry. And finally, we examined outcomes for students who enrolled and took courses without attaining a degree or certificate. We found that their after-college earnings increased in proportion to the number of credits they earned.

"The fundamental policy implication is that college is a good investment."

What do you see as the key policy implications of these findings?

The fundamental policy implication is that college is a good investment. This merits emphasis because there are repeated critiques of college in terms of how much it costs and how much debt students accumulate. That said policymakers do need to think about the value of each postsecondary program. Even within the same institution, programs have very different outcomes. Yet on average, attending college for longer and attaining more credits has beneficial effects. Policymakers should see this evidence as supporting public and private investments in college.

What did you discover about the relationships between financial aid, college outcomes, and labor market outcomes?

In an era of tight public resources, the effectiveness of financial aid policy is a crucial issue. Financial aid does help students persist in college, but one way to promote greater effectiveness is through academic performance standards for students receiving federal financial aid. These have existed in the federal need-based aid programs for nearly 40 years, in the form of Satisfactory Academic Progress (SAP) requirements. These have not received much attention. Our research on SAP suggests such policies have heterogeneous effects on students in the short term: they increase the likelihood that some students will drop out, but appear to motivate higher grades for students who remain enrolled. After three years, however, the negative effects dominate. Though it has little benefit for students in the long term, SAP policy appears to increase the efficiency of aid expenditures because it discourages students who have lower-than-average course completion rates from persisting. But the policy also appears to exacerbate inequality in higher education by pushing out low-performing, low-income students faster than their equally low-performing, higher-income peers.

Many students work while in college.  Does this seem to help or hurt students in the long run?

Our research found that for Federal Work-Study (FWS) participants who would have worked even in the absence of the program, FWS reduces hours worked and improves academic outcomes but has little effect on post-college employment outcomes. For students who would not have worked, the effects are reversed: the program has little effect on graduation, but a positive effect on post-college employment.  Results are more positive for participants at public institutions, who tend to be lower income than participants at private institutions. Our findings suggest that better targeting to low-income and lower-scoring students could improve FWS outcomes. This is consistent with much of the CAPSEE research—you need more detail and specificity to really understand the relationship between education and employment and earnings.

What did you learn about credentials from for-profit institutions?

Our findings on students at for-profit colleges were quite pessimistic. Although enrollment in for-profit colleges grew significantly after 2000, the sector has been declining during the last two years, as evidence on inferior outcomes – particularly with regard to student debt – emerged. In general, our researchers found that for-profit students have worse labor market outcomes than comparable community college students although in some cases the difference is not statistically significant. Our evidence suggests that these colleges need to be monitored to ensure they are delivering a high-quality, efficient education.

You are holding the final CAPSEE conference in April. What do you hope people will get out of it?

At the conference, we will focus on several important and controversial policy questions related to higher education: 

  • Have changes in tuition and the labor market created conditions in which college is not worth it for some students, contributing to an unsupportable increase in student debt? 
  • Has higher education contributed to inequality rather than promoting economic mobility? 
  • Is continued public funding of college a worthwhile investment? 
  • Should public funding be used only for some programs of study?  
  • What are the arguments for and against making community college free?
  • Can changes in the operations and functioning of colleges change the return on investment from a college education for both the individual and society?
  • How important should information on earnings outcomes be for accreditation decisions and/or for eligibility of students to receive financial aid? 

At the conference participants will have the opportunity to discuss and learn about these issues drawing on five years of CAPSEE research as well as input from other experts.

Students with Disabilities and Postsecondary Success: An Interview with Lynn Newman, Ed.D. and Joseph Madaus, Ph.D.

By Meredith Larson, NCER Program Officer                                                                                     

Although more students with disabilities are pursuing postsec ondary education, completion rates for this group of students have not changed very much in recent years. In a two-year study funded through an IES grant, Lynn Newman, of SRI International, and Joseph Madaus, director of the Center on Postsecondary Education and Disability at the University of Connecticut, have examined the impact that supports and accommodations have had on the postsecondary success of students with disabilities.

             

 

At the heart of their study is the National Longitudinal Transition Study-2 , the largest and richest data set available to address the postsecondary experiences and outcomes of youth with disabilities. It is the only dataset that can address those topics for postsecondary students with disabilities nationally, independent of students’ decisions to disclose a disability to their postsecondary school.

Below are excerpts from an email interview with the researchers. 

 

What motivated your study, and what questions are you grappling with?

Although postsecondary enrollment rates for students with disabilities have increased dramatically for youth in all disability categories over the past two decades, postsecondary completion rates for students with disabilities have remained stagnant over time. These students continue to be less likely to graduate from postsecondary school than their general population peers.

This led us to ask

  • What is the link between receipt of postsecondary supports and accommodations – both those available because of a disability and those available to the general student body – and postsecondary persistence and completion for students with disabilities?
  • What factors are associated with requesting/receiving postsecondary supports and accommodations?

What are your major findings?

First and foremost, students with disabilities who received supports, particularly supports available to the full student body (such as tutoring and access to writing and study centers), are more likely to persist in and complete their postsecondary programs. This finding applies to students with disabilities enrolled at both 2-year and 4-year colleges.

However, we didn’t find a significant relationship between receipt of disability-specific supports and accommodations (such as test accommodations, readers, interpreters) and postsecondary persistence or completion for the full population of students with disabilities. We found that the link between supports/accommodations and outcomes differs by disability category. For example, students who were deaf or hard of hearing and received disability-specific accommodations and supports were more likely to persist in or complete postsecondary education than were those who had not received these types of help.  

Does your research suggest why some students seek out or use supports more than others?

Fewer than half of those with disabilities in postsecondary institutions accessed the types of supports available to the general student body, and less than one-quarter received disability-specific help during postsecondary school. Students who received transition planning education in high school and those whose transition plans specified needed postsecondary supports and accommodations were significantly more likely to access both generally-available and disability-specific supports in postsecondary school, particularly at 2-year institutions.

If you could tell each of your target audiences what your research means for them in practical terms, what would you say?

Students and families: By accessing supports and help at postsecondary institutions,  you increase your odds for postsecondary success. If you are uncomfortable sharing information about your disability, which is required to receive disability-specific supports, you should, at least, access the types of supports available to the general student body, such as tutoring and writing centers.

High school staff: Help students avail themselves of supports at the postsecondary level through transition planning. Transition planning education and transition plans that specify postsecondary accommodation needs significantly affect whether students seek postsecondary supports. Clearly, the transition education and planning you can do matters. However, as many as one third to one half of high school students with disabilities do not receive such transition planning services.

Postsecondary staff: Keep in mind that only 35% of students with disabilities who received services in high school disclosed their disability to their postsecondary institutions, so you probably have more students with disabilities on your campus than you may be aware of. Because receipt of postsecondary supports (especially general supports available to all students) are particularly beneficial to students with disabilities, we encourage active and broad outreach about these supports to the entire student body, rather than focusing on just the few students who have chosen to disclose their disability.

In addition, we encourage professional development for postsecondary staff, particularly those involved in providing generally available supports, to help them better recognize and support students with disabilities.

Researchers: Consider the representativeness of your samples of postsecondary students with disabilities. If respondents are identified through self-disclosure of a disability, your sample probably has a large amount of underreported students with disabilities overall. Your sample is also likely to be biased, in that students with more visible disabilities are much more likely to disclose their disability than are those in the higher incidence disability categories, such as learning disabilities.

What might some next research steps be?

Given our findings, we believe there are many opportunities for research related to postsecondary education for students with disabilities.  For example, researchers could study questions about the characteristics, content, extent, and timing of effective postsecondary supports and accommodations. The field would also benefit from additional knowledge about effective high school transition planning education and answers to questions about the characteristics and structures of high schools and postsecondary schools that offer effective supports and accommodations and transition planning education. 

Questions? Comments? Please send them to IESResearch@ed.gov.