About this practice guide

The Institute of Education Sciences (IES) publishes practice guides in education to provide educators with the best available evidence and expertise on current challenges in education. The What Works Clearinghouse (WWC) develops practice guides in conjunction with an expert panel, combining the panel’s expertise with the findings of existing rigorous research to produce specific recommendations for addressing these challenges. The WWC and the panel rate the strength of the research evidence supporting each of their recommendations. See Appendix A for a full description of practice guides.

The goal of this practice guide is to offer specific, evidence-based recommendations for college and university faculty, administrators, and advisors working to improve the success of students academically underprepared for college. Each recommendation includes an overview of the practice, a summary of evidence used in support of the evidence rating, guidance on how to carry out the recommendation, and suggested approaches to overcome potential roadblocks. Each recommendation includes an implementation checklist, as guidance for getting started with implementing the recommendation.

Practice guides published by IES are available on the WWC website at http://whatworks.ed.gov

This report was prepared for the National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, under the WWC contract to Development Services Group, Inc. (Contract ED-IES-12-C-0084).

Disclaimer

The opinions and positions expressed in this practice guide are those of the authors and do not necessarily represent the opinions and positions of the IES or the U.S. Department of Education. Educators, colleges, universities, and organizations should review the practice guide and apply its recommendations according to their specific needs and goals. It is important to recognize that the practice guide represents the judgments of the review panel regarding what constitutes sensible practice, based on the research that was available within the publication timeframe established in the review protocol. This practice guide should be used as a tool to assist in decisionmaking rather than as a "cookbook" or "recipe" that contains instructions to be strictly followed. Any references within the document to specific education products are illustrative and do not imply endorsement of these products to the exclusion of other products that are not referenced.

Acknowledgements

Development Services Group (DSG) and the Expert Panel would like to thank its team of abstract screeners, full-text study screeners, and WWC certified reviewers for all the dedicated effort to screen thousands of studies and review more than 100 studies. DSG is grateful to Sandra Jo Wilson, Bethany Gleason, Emily Tanner-Smith, and Alicia Hymel of the Vanderbilt University Peabody Research Institute team for their leadership in guiding the systematic reviews of evidence for this practice guide. We thank Lisa Hoover for her creative and professional work on the graphics included in this guide. Susanne Miller provided dedicated and valued support throughout the project by coordinating assignments for study reviews and managing author queries. DSG’s editors, Matthew Malamud, Michael Hopps, and Marla Fogelman, contributed valuable editorial support throughout the project. Most importantly, we are grateful to the researchers, practitioners, and educational leaders who have contributed to the scholarship and practical guidance shared in this guide.

Strategies for Postsecondary Students in Developmental Education − A Practice Guide for College and University Administrators, Advisors and Faculty

November 2016

Panel

Thomas Bailey (Chair)

TEACHERS COLLEGE, COLUMBIA UNIVERSITY

Joanne Bashford

MIAMI DADE COLLEGE

Angela Boatman

VANDERBILT UNIVERSITY

John Squires

SOUTHERN REGIONAL EDUCATION BOARD

Michael Weiss

MDRC

Staff

William Doyle

VANDERBILT UNIVERSITY

Jeffrey C. Valentine

UNIVERSITY OF LOUISVILLE

Robin LaSota

Joshua R. Polanin

Joshua R. Polanin

Elizabeth Spinney

Wesley Wilson

Martha Yeide

Sarah H. Young

DEVELOPMENT SERVICES GROUP, INC.

Project Officers

Vanessa Anderson

Jonathan Jacobson

Diana McCallum

Christopher Weiss

INSTITUTE OF EDUCATION SCIENCES

NCEE 2017-4011

U.S. DEPARTMENT OF EDUCATION

U.S. Department of Education

John B. King, Jr.

Secretary of Education

Institute of Education Sciences

Ruth Neild

Deputy Director for Policy and Research, Delegated Duties of the Director

National Center for Education Evaluation and Regional Assistance

Joy Lesnick

Acting Commissioner

November 2016

This report is in the public domain. Although permission to reprint this publication is not necessary, the recommended citation is below:

Recommended Citation: Bailey, T., Bashford, J., Boatman, A., Squires, J., Weiss, M., Doyle, W., Valentine, J. C., LaSota, R., Polanin, J. R., Spinney, E., Wilson, W., Yeide, M., & Young, S. H. (2016). Strategies for postsecondary students in developmental education – A practice guide for college and university administrators, advisors, and faculty. Washington, DC: Institute of Education Sciences, What Works Clearinghouse.

The citation for this WWC practice guide begins with the panel chair, followed by the rest of the panelists in alphabetical order, and followed by the methodological and content leads for the practice guide, and the staff writers, in alphabetical order.

This report is available on the IES website at http://whatworks.ed.gov.

Alternate formats

On request, this publication can be made available in alternate formats, such as Braille, large print, or CD. For more information, contact the Alternate Format Center at (202) 260-0852 or (202) 260-0818.

Table of Contents

Strategies for Postsecondary Students in Developmental Education − A Practice Guide for College and University Administrators, Advisors and Faculty

Contents

Introduction

Recommendation 1. Use multiple measures to assess postsecondary readiness and place students

Recommendation 2. Require or incentivize regular participation in enhanced advising activities

Recommendation 3. Offer students performance-based monetary incentives.

Recommendation 4. Compress or mainstream developmental education with course redesign

Recommendation 5. Teach students how to become self-regulated learners

Recommendation 6. Implement comprehensive, integrated, and long-lasting support programs

Appendix A. Postscript from the Institute of Education Sciences

Appendix B. About the authors

Appendix C. Disclosure of potential conflicts of interest

Appendix D. Evidence summary of studies in support of ratings

References

Endnotes

List of Tables

Table A. Institute of Education Sciences levels of evidence for WWC practice guides

Table B. Levels of evidence for each practice guide recommendation

List of Figures

Figure A. Prisma flow diagram for study identification, screening, eligibility, and inclusion status/ratings

Figure 1.1. Types of measures to assess college readiness and place students into courses

Figure 2.1. Sample enhanced advising planning tool

Figure 3.1. How to budget for performance-based monetary incentives

Figure 4.1. Student success through accelerated coursework

Figure 4.2. Inventory of strategies currently in place for accelerating students’ progress through developmental education

Figure 4.3. Planning issues and considerations in adopting compressed and mainstreaming developmental education models

Figure 5.1. Overview of the self-regulated learning process – Roles for instructors and students

Figure 5.2. Integration of self-regulated learning throughout a course

Figure 5.3. Sample self-regulated learning quiz in mathematics

Figure 5.4. Sample quiz reflection and revision sheet in mathematics

Figure 5.5. Paired discussion among students – evaluation of the quiz format

Figure 5.6. Guided self-reflection about the course

Figure 5.7. Discussion questions for applying self-regulated learning skills to other courses

Figure 6.1. Postsecondary impacts of Accelerated Study in Associate Programs

Figure 6.2. Illustrative example of comprehensive, integrated, and long-lasting supports

Introduction

Introduction to the Practice Guide on Strategies for Students in Developmental Education

Approximately one-third of students entering higher education do so without the necessary skills in at least one subject to enroll and succeed in college-level work.1 Underprepared college students are less likely than their peers to complete their educational objectives. As an example, 1992’s high school graduates who first entered community college and enrolled in at least one developmental education course had lower degree attainment rates 8 years after high school than first-time community college entrants who did not enroll in any developmental education courses (28% vs. 36%). For students first entering 4-year institutions, the picture was even bleaker, with students enrolling in at least one developmental course earning a degree at a much lower rate (52%) than first-time entrants to 4-year institutions not enrolling in developmental courses (78%).2

Colleges and universities have developed and adopted various strategies to help underprepared students acquire the skills and knowledge needed to move into college-level courses.3 These policies and practices have been referred to as “developmental education,” “remedial education,” or “college-readiness programs.”4 While all of these labels are currently used in postsecondary education and have been debated, state policymakers, higher education administrators, and faculty recognize that these policies and strategies can help create conditions that give underprepared students the best chance of realizing their educational goals.

This practice guide brings together two distinct sets of evidence-based practices: those that focus on instructional interventions, and those that focus on structural interventions. More specifically, two of the six recommended practices focus on student and faculty interactions in the classroom. These interventions involve curricular decisions that affect the content and pacing of classroom instruction. The second set includes four practical recommendations focused on structure and largely involve systems for determining which types of supports will be offered or mandated for students in, or at risk of requiring, developmental education. Together, these six practices can help create a learning environment that maximizes the chances that students will succeed in their coursework and chosen degree program. These practices can be implemented widely, regardless of existing policies on developmental or remedial education. However, without the support of policymakers and administrators, they will be difficult to implement at the institutional level.

Audience and institutions of interest

Administrators and faculty at institutions of higher education should find these practical recommendations helpful in meeting the needs of students who are underprepared for college-level work. Audiences that may be particularly interested in the instructional practices include program coordinators or directors of developmental education, curriculum design professionals, and faculty who teach developmental education. Certain interest groups, such as philanthropic leaders and associations engaged in developmental education reform, will find the evidence regarding instructional interventions for developmental education useful. Academic leaders in state and community colleges will be interested in interventions that mitigate the impact of lack of readiness for college-level coursework.

Audiences particularly interested in the structural interventions may include higher education administrators—especially those in financial aid offices, student counseling divisions, academic advisors, admissions offices, articulation officers at community colleges and universities, and liaisons who collaborate with high school systems to foster college readiness. Administrative leaders—including presidents, deans, provosts, and department chairs—can find recommendations that shape practices at the institutional, college, or departmental level. Test developers may find the information on assessment and placement helpful as they work to create new assessments and refine existing ones.

While the majority of students in need of additional developmental education in higher education are enrolled in community colleges, four-year institutions also serve students with developmental education needs.5 Even though the majority of the research with students in developmental education has been conducted in community colleges, this guide includes evidence conducted at four-year institutions that met our review criteria. Faculty, institutional leaders, and advisors from both sectors are encouraged to extrapolate from the recommendations and evidence presented for their own institutional contexts. The intended audience for this practice guide is all postsecondary institutions that serve students who are placed in, have been recommended to, or are at-risk of being placed into developmental education.

Scope of the practice guide

The goal of this practice guide is to present rigorously and systematically reviewed evidence in support of practical recommendations for improving outcomes for students at-risk for, or placed into, developmental education. The practice guide primarily focuses on interventions or practices aimed at improving students’ progress through developmental education, credit accumulation and persistence, academic achievement, and degree attainment.

To determine the scope of the practice guide, expert researchers and practitioners initially considered more than twenty practices, and narrowed the focus to eight practice areas, based on available evidence and expert opinion. The eight practices identified were: 1) comprehensive and integrated support programs; 2) contextualized instruction; 3) early assessment programs for at-risk high school students; 4) enhanced and early alert advising; 5) performance-based monetary incentives; 6) practices to accelerate, compress, or mainstream developmental education; 7) practices to modify information used to make placement decisions; and 8) practices to teach metacognition, productive persistence, and college success skills. This introduction includes a description of how the assembled team of expert researchers and practitioners reviewed the evidence on these practices to develop and refine their final recommendations.

Based upon review of the evidence, the expert panel decided not to include practice recommendations for early assessment programs for at-risk high school students, and contextualized instruction (without compression or acceleration) in the practice guide. The expert panel also found little evidence to support broad practices to teach metacognition, productive persistence, and college success skills. With evidence on positive effects of teaching self-regulated learning to students in developmental education, the expert panel refined this practice recommendation to teaching self-regulated learning.

Even though the panel did not include early assessment programs for at-risk high school students as one of its recommendations, one of the primary ways to help students succeed in higher education is engage students in early assessment and implement college-readiness interventions, so that they do not need developmental education courses in the first place. Several states have initiated programs to assess students during their junior year of high school and offer courses for underprepared students during their senior year.6 These programs involve measuring high school students’ readiness for college-level coursework, which can inform students and their families about the need for skill building in math, reading, or writing, and help them avoid formal placement in developmental education.7 While there are few studies of these relatively new programs, the expert panel convened by the What Works Clearinghouse (WWC) to review the evidence of these interventions believes that these early assessment and college-readiness interventions show promise, especially when implemented statewide. Due to the limited available evidence, however, early assessment and college-readiness interventions are not included as one of the six recommendations in this practice guide.

Once students arrive on a college campus, a primary challenge that practitioners face in improving outcomes for underprepared students is identifying those students who may need extra assistance to be college-ready. Unfortunately, many policies divide students into two broad categories: students who are ready for college-level work and those who are not.8 Within those two broad categories, states and postsecondary institutions have established multiple levels of developmental education progression, and college-level coursework (typically denoted in 100-level, 200-level, etc. courses). Since readiness for college-level work exists on a continuum, with most students displaying different degrees of readiness for different subjects, the expert panel recommends tailoring interventions for students at different degrees of readiness for college-level work.

A second challenge is that there is no single “magic bullet.” Many interventions may be more effective in combination than if implemented alone. For example, the evidence suggests that the bundling of interventions is associated with relatively strong effects in developmental students’ outcomes.9 These bundles of interventions can include some combination of full-time enrollment, enhanced advising, tutoring, accelerated coursework, a cohort model, or student financial incentives, among others.

Finally, the quality of instruction that occurs in developmental education likely plays a large role in students’ eventual success in postsecondary education. While this guide includes evidence-based teaching practices, such as self-regulated learning, there are other instructional practices not included that likely would also improve learning and result in higher rates of success for developmental students. Evaluations of many of these practices fell outside the scope of this guide, often because their efficacy has not been assessed rigorously using developmental students.

Summary of recommendations

1.Recommendation 1: Use multiple measures to assess postsecondary readiness and place students.

2.Recommendation 2: Require or incentivize regular participation in enhanced advising activities.

3.Recommendation 3: Offer students performance-based monetary incentives.

4.Recommendation 4: Compress or mainstream developmental education with course redesign.

5.Recommendation 5: Teach students how to become self-regulated learners.

6.Recommendation 6: Implement comprehensive, integrated, and long-lasting support programs.

Development of the practice guide

An expert panel guided the development of the practice guide. The panel included: Dr. Thomas Bailey, director, Community College Research Center and George and Abby O'Neill Professor of Economics and Education, Teachers College, Columbia University; Dr. Joanne Bashford, president, InterAmerican Campus, Miami Dade College; Dr. Angela Boatman, assistant professor of Public Policy and Higher Education, Vanderbilt University; John Squires, director of High School to College Readiness, Southern Regional Education Board; and Dr. Michael Weiss, senior associate, MDRC. The panel represented perspectives of practitioner leaders working on reforms in developmental education and the expertise of senior scholars in this field.

Once the expert panel was confirmed, practice guide development began with the Panel’s identification of 21 initial practices to improve outcomes for students in developmental education, in collaboration with the senior content expert (a member of the staff leadership team). In the first meeting, panelists received a briefing book on these 21 collaboratively-identified practices for supporting the success of underprepared students. The briefing book provided practice descriptions, contexts of implementation for the practice, and brief summaries of evidence for 21 interventions targeted at improving outcomes for students placed into, and at-risk of placement in, developmental education. Based on discussion of briefing materials on these 21 practices, the expert panel agreed upon an initial set of recommended practices for systematic search and review of the evidence. The initial list of practices provided a beginning foundation to define the scope of the practice guide. In their deliberations, panel members raised discussion about other practices, and freely deliberated as they narrowed a final list of practices for systematic search and review of evidence.

Panel members were asked to consider three questions about each possible practice:

1.In your professional opinion, is this practice likely to help underprepared students succeed? In short, does it work?

2.Is there likely to be a substantial evidence base to support the effectiveness of this practice?

3.Can this practice be replicated in a wide variety of locales, or is it highly specific to the institution or setting in which it has been developed?

The result of this meeting was a set of practices that panel members believed were promising enough to warrant a close investigation. The WWC developed a protocol that described how the review of these practices would be carried out, and this protocol included a systematic literature search strategy for each recommended practice. From this systematic search of literature reported between January 1, 1995, and August 31, 2015, 25,697 studies were identified from abstracts in the EBSCO and ProQuest research databases, and 95 additional studies were identified from supplemental searches of relevant websites, literature reviews, and reference harvesting from other studies.

Based upon the review protocol, studies were screened for relevance (e.g., whether they investigated the effects of an intervention and whether the sample consisted exclusively or largely of students in developmental education), and the full text of each study judged to be possibly relevant (based on information in the title and abstract) was obtained.

Trained and certified reviewers then examined these studies against the standards of the WWC, using the What Works Clearinghouse Procedures and Standards Handbook, Version 3.0. Among the 48 eligible reports and publications, 10 studies met WWC standards without reservations or met WWC standards with reservations.10

The panel then met to discuss the results of the literature search. Based on the findings and review of the evidence, the panel then made its final determination regarding the level of evidence for each practice. The panel was supported in its work by the reviews conducted by WWC staff, but the panel members made the final determination for each type of evidence.

In making its determination, the panel used the levels of evidence criteria, which are set by WWC policy, in Table A in this Introduction.

The full text of 439 studies was screened for relevance. While 68 studies met criteria for review in eight original practices, 48 studies were found to be eligible for review under the protocol, and aligned with the final six practice recommendations. As part of their deliberations, the panel considered evidence from 20 studies which were originally rated in the areas of early assessment and intervention, contextualized instruction, and metacognition. After review of the evidence, these 20 studies no longer aligned with the panel’s recommendations.

Figure A. Prisma flow diagram for study identification, screening, eligibility, and inclusion status/ratings

Table A. Institute of Education Sciences levels of evidence for WWC practice guides

Table B summarizes the recommended levels of evidence for each practice. The expert panel assessed the levels of evidence based on the IES criteria outlined in Table A.

Criteria for review of the evidence

The review protocol for the practice guide provides detail on the product scope, study review criteria, procedures for systematic search, and methods used for study review. For example, the review protocol for the practice guide established criteria regarding the study populations that would be considered eligible for review. In order for a study to be eligible for review, and used for a strong, moderate, or minimal evidence rating, a study must have included postsecondary students in the United States or Canada (including students who have not yet started their college careers), at least two-thirds of whom are in, have been recommended for, or are at risk for being placed into developmental education.

To be eligible for review and used for a moderate or minimal evidence rating, a study must have included postsecondary students in the United States or Canada (including students who have not yet started their college careers), at least 40 percent of whom are in, have been recommended for, or are at risk for being placed into developmental education.

The review protocol also established outcomes relevant to the review, briefly defined below. Within each outcome area, study reviewers follow guidance regarding which outcome measures are to be considered primary, and which are considered supplemental.

- College access and enrollment refers to the process of applying to, actually enrolling in, and attending a postsecondary institution. Examples of ways that enrollment might be operationally defined in studies include (a) applied vs. did not apply to college, (b) number of applications completed, (c) attended vs. did not attend a postsecondary institution or enrolled vs. not enrolled in the first semester or year of college, (d) selectivity of enrollment institution, (e) full time vs. part time enrollment, and (f) 4-year college or institution vs. 2-year college or institution vs. non-enrollment. Actual enrollment is the preferred measure.

- Progress through developmental education refers to the process of completing required developmental coursework. Examples of ways that progress through developmental education might be operationally defined include a) completed versus did not complete developmental education coursework, b) completed versus did not complete first college-level course in which remediation was needed, and c) grades earned in developmental courses. Passing college-level courses in the area of required developmental education is the preferred measure.

- Credit accumulation and persistence refers to progress toward the completion of a degree, certificate, or program. As mentioned under “outcome period relevance,” the primary focus for this outcome domain is upon the longest time period observed for the outcome (or outcomes) in this domain. Examples of ways that credit accumulation might be operationally defined in studies include a) number of credits earned toward degree completion, b) proportion of degree-bearing versus non-degree-bearing credits earned, c) ratio of credits earned to credits attempted, and d) enrollment persistence. If a study assesses credit accumulation and enrollment persistence, the former is the preferred measure.

- Academic achievement refers to measures which assess the extent to which students adequately complete expected coursework. Examples of ways that academic achievement might be operationally defined in studies include grade-point average, departmental final exams, and the ratio of courses passed versus failed.

- Transfer to a four-year institution refers to students’ transition to a bachelor’s degree granting program or institution, typically from a 2-year postsecondary institution that does not have any baccalaureate-degree granting programs (or may have few, selected ones). National Student Clearinghouse records are the preferred data source (which may be integrated into state longitudinal data systems).

- Attainment refers to the completion of a degree, certificate, or program. Examples of ways attainment might be operationally defined in a study include certificate completion rates and degree completion rates. Official school records are the preferred data source.

- Labor market refers to outcomes related to employment after the postsecondary experience. Examples of ways that labor market outcomes might be operationally defined in studies include (a) employed vs. not, (b) employed full-time vs. employed part-time, (c) employed in field of study vs. not, and (d) income earned. Continuous measures of actual income are preferred.

How to use this practice guide

Each section of this guide contains a recommended practice in which the practice itself and examples of the practice in action are described. The evidence base for the practice is reviewed, and recommendations for effectively implementing the practice are listed. When recommendations have two or more studies of similar interventions that assess effects on outcomes in the same outcome domain, meta-analyses were conducted (see the WWC’s Procedures and Standards Handbook, version 3.0, for more information). Individual effect sizes for studies used as evidence in support of a recommendation are also presented in Appendix D.

Note that this practice guide does not provide step-by-step instructions on how to fully implement each practice, but does provide preliminary guidance on how to implement the recommendation, as well as common roadblocks and suggested approaches for overcoming them.

Leaders on college campuses are subject to a host of constraints that could make implementing some of these recommendations difficult. These constraints come from budgets, available staff resources, or from system or state policies. With the exception of Recommendation 6, the practices identified here can be implemented in isolation, working within the constraints for a given institution. Recommendations 1–5 do not depend in a cumulative sense on having implemented another practice beforehand. In contrast, implementing comprehensive supports does imply a set of interdependent policies and practices which must be put in place at the same time.

Recommendation 1: Use multiple measures to assess postsecondary readiness and place students.

Most open-access institutions require incoming students to take brief standardized assessments in math, reading, and writing. The results of these assessments are used to place students in either developmental or college-level courses. However, few would argue that a single placement test is a perfect measure of college readiness. In fact, several studies have highlighted concerns about misplacement rates.14 For example, Scott–Clayton (2012) found that 24%–33% of students were misplaced when they matriculated. It was more common for students to be placed in developmental courses when they could have succeeded in college-level courses than the reverse. Since nearly 40% of community college students enrolled in developmental education fail to progress to credit-bearing coursework,15 misplacement into developmental education is particularly troubling.

A basic principle of psychological measurement is that when a construct like college readiness is measured imperfectly, one way to improve measurement is to assess the construct in multiple ways. Therefore, one way to improve the measurement of college readiness (and therefore to reduce misplacement) is to use multiple measures—such as high school GPA, the number of years since high school graduation or equivalent, the number of courses taken in the subject (e.g., English or math), and the highest level taken in the subject (e.g., Algebra I or Algebra II)—to inform placement decisions.

Figure 1.1. illustrates types of measures to assess postsecondary readiness and place students. In this regard, it is notable that the state of California mandates the use of multiple measures for placement into developmental education (Title 5 §55502, California Code of Regulations),16 and companies that produce the placement tests recognize that adding information beyond the test itself can improve placement decisions.17

Summary of evidence: Minimal Evidence

Experimental studies of the use of multiple measures for course placement investigate the effectiveness of different placement criteria (e.g., placement test alone versus placement test plus high school GPA) on students’ outcomes (such as credit accumulation and grades).

The WWC identified two studies that investigated the effectiveness of alternative or multiple measures for placement.18 Neither study met WWC group design standards. One study was a randomized controlled trial^19 for which the WWC could assess neither how much attrition the study experienced nor the extent to which the groups were similar at baseline. Similarly, in another study,20 the WWC was unable to assess the extent to which the groups were similar at baseline. Nevertheless, the panel believes that these studies offer useful insight into this recommendation. The studies are further discussed in Appendix D Table 1.1.

Two additional non-experimental studies are highlighted in Appendix D Table 1.2.21 Both of these studies suggest that adding information about student high school experiences (in particular, high school GPA) might help improve placement decisions. In addition to multiple measures, these studies offer some evidence that alternative measures of high school preparation (compared to scores on the institution’s standardized placement test) can also be effective in reducing misplacement.

How to carry out the recommendation

Figure 1.1. Types of measures to assess college readiness and place students into courses

Several prominent groups have provided guidance on using multiple measures to place students in developmental education.22 Below are the implementation considerations that the panel believes to be relevant.

1.Explore potential additional measures that faculty and administration believe might inform placement decisions. Investigate the feasibility of each of these, and retain those that can be measured reliably and with minimal additional cost (possible examples include high school GPA, the highest math course taken, the number of years since graduation, high school graduation status, and scores on college entrance examinations). Then, develop one or more placement rubrics based on the selected measures, and use administrative data to investigate the effect of the different rubrics on placement decisions. Use this information to propose a specific multiple measures policy.

2.Use a pilot period to study the placement rates and success rates of students involving different placement methods at your institution. For example, use existing administrative data at your institution to develop one or more alternative placement rubrics. Then, randomly assign students to be placed using the new multiple measures rubric(s) or your college’s current placement method. After random assignment, determine if student success rates varied as a function of placement method. For example, you might examine the course pass rate for students placed into college algebra using your current placement method compared to the course pass rate for students placed into college algebra using an alternative method. Finally, use the findings to modify the proposed policy as needed. The Research and Planning (RP) Group’s Multiple Measures Assessment Project offers some resources for setting up pilot studies, including a “Getting Started Guide.”23 As needed, partner with independent researchers from universities and contract research firms for additional capacity and expertise on the design and conduct of these studies.

3.After implementation, investigate the effects of the multiple measures policy on desired student outcomes over time. Continue to monitor the effects of the placement policy, and use new information to adapt, change, or expand implementation of multiple measures policies and practices.

4.Consider how college-readiness policies and practices in your state and local area impact the parameters of decision-making and implementation of multiple measures. For example, as states implement new college-readiness assessments associated with implementation of the Common Core State Standards, validation studies of these assessments then may offer information to guide the potential effectiveness of using them as a placement measure.

Potential roadblocks and suggested approaches

Roadblock 1.1. Using multiple measures for placement is resource-intensive.

Suggested Approach 1.1.a. Scale up a pilot effort, with support from leadership for larger-scale implementation, to gradually influence changes in placement practices. Perhaps the most significant barrier is the lack of resources to pay for a more comprehensive assessment process. For example, one faculty member at an institution that was incorporating multiple measures estimated that, to make recommendations, each student file required a 10-minute review, totaling about 40 hours for 250 students.24

One strategy for initiating time- and labor-intensive systemic change is to begin by convening a work group to lead small-scale piloting and plan for gradual expansion. For example, the University of Wisconsin-Marathon County began using multiple measures to place students with certain risk factors and then expanded to a larger population.25 Similarly, Long Beach City College (LBCC) first offered alternative measures for placement in fall 2012 with 933 students from the Long Beach Unified School District, expanded to two additional large districts the next fall, and two more the year after.26 Piloting also enabled gradual adjustments, refinements, and adaptations across colleges in California and in Wisconsin’s university system.

Moreover, the pilot efforts from LBCC helped to guide expansion of the multiple measures implementation across colleges in California. LBCC conducted and shared its pilot research with other colleges as part of the Student Transcript-Enhanced Placement Study, which found that high school grades were better predictors than placement tests (and other measures) of students’ passing their first attempted courses in English and math.27 Subsequently, a consortium of California colleges (California Partnership for Achieving Student Success) have implemented and studied the effects of using multiple measures for students’ academic outcomes.28

To maximize successful, long-term implementation of new multiple measures practices and policies that begin with a pilot effort, leadership strategies should be inclusive, collaborative, and adaptive. To translate pilot efforts to larger-scale change, leaders must also consider how to best leverage and maximize scarce resources in each step, while investing strategically in implementation of new placement measures.

Suggested Approach 1.1.b. Consider additional measures already collected by your institution or those that are easily assessed. Certain measures (such as college entrance tests, high school GPA, and the number of years since high school graduation) may already be collected by your institution and, therefore, represent low-cost information that can be used to potentially improve placement decisions.

Roadblock 1.2. Lack of support from key institutional staff can inhibit robust implementation.

Suggested Approach 1.2. Key institutional staff need to be motivated and engaged for effective reform to occur.29 Broad sharing of information about the severe error rates associated with the use of single-assessment measures for placement in courses may motivate and engage faculty and staff to reform institutional practices.

In addition to sharing research about the limitations of single-test placement, faculty and administrators can also provide evidence regarding the advantages of using multiple measures to increase probability of more accurate placement for a greater number of students, without negatively affecting student success rates. Duffy et al. (2014) reported that a multiple-measures working group in North Carolina met with faculty on several occasions to explain the research conducted by the Community College Research Center on this topic as a way of supporting faculty in adopting new placement practices and using multiple measures. When available, sharing the results of institutional research from pilot studies can garner support among faculty and staff.

Roadblock 1.3. Data availability and quality may be varied, depending on the specific assessment measures to be used.

Suggested Approach 1.3. Access available data for the targeted student population for the multiple measures to be used and, as a secondary strategy, consider the use of students’ self-report data. Access to high school transcript data by many open-access institutions has increased with the continued development of P–20 longitudinal, educational data systems in states. However, some institutions may still have limited access to transcript data, or find the data to be of low quality for their student population. In the absence of official records or other desired data, using students’ self-report data should be considered.

In cases where transcript data is not easily shared or accessible, student self-report data may offer some insight to the student’s academic preparedness, even if that information is relatively unreliable. According to Research for Action (2014), some colleges in California use measures, such as student surveys and interviews, to gather information about highest math course completed and grades in last English and math class completed. Several testing companies have also developed and evaluated assessments to measure students’ persistence, dependability, motivation, and ability to work with others. ACT, Inc., developed a standardized measure of student motivation, social engagement, and self-regulation, aligned with its college readiness standards. The assessment, Engage, is a 30-minute, multiple choice test designed to measure psychosocial attributes and study skills associated with college readiness. Little or no evidence has been reported to show the effectiveness of using these non-cognitive measures for students’ course placement.

While some researchers have identified concerns with students inflating their responses to socially desired qualities, Schmitt and colleagues (2011) found that students’ inflation on responses did not “impact the variability or validity of responses” (p. 12). Because research findings on this issue vary, the expert panel recommends cautiously monitoring response quality from self-report data in high-stakes situations.

Roadblock 1.4. Developing new algorithms for placement, merging data sources, and supporting use of new data for placement requires technical expertise that many open-access institutions do not currently have.

Suggested Approach 1.4. Faculty and staff who have strong interest and technical aptitude can be supported through professional development, encouragement, and compensated time for leading and performing the work involved. Resources and available expertise for implementation of new measures for placement will vary by state and institution. As mentioned previously, for example, the RP Group in California provides resources to help institutions get started with multiple measures implementation, and other states have developed similar expertise. Institutions should think creatively and strategically about how to develop internal expertise while investing in external advice on the development of new placement policies, integration of data sources, and analysis of the results from pilot studies.

Recommendation 1 implementation checklist

Recommendation 2: Require or incentivize regular participation in enhanced advising activities.

Advising, guidance, and counseling services have been developed to help students at all levels make choices about academic majors, understand the relationship between school and subsequent employment, and address a variety of academic and personal issues.31 Some colleges have created more intensive advising experiences, often called “enhanced advising”32 or “intrusive advising.”33 Enhanced advising replaces the quick, transactional structure of traditional advising (e.g., a focus on class schedules, degree requirements, and financial aid procedures) with a more holistic structure in which advisors ask deeper questions and engage more with students to help them succeed.34 Mentoring programs that aim to build relationships between students and knowledgeable adults on goal-oriented academic planning may also be considered as enhanced advising.

Researchers have estimated typical advisor-to-student ratios for large, community colleges and regional open-access institutions to be about 600:1.35 In a study of the Opening Doors program in Ohio, implementation of enhanced advising meant a reduction of the advisor-to-student ratio to 150:1, compared to the typical 1,000:1 in that college. Scrivener and Weiss (2009) also emphasized the importance of addressing any underlying structural deficiencies that limit more personalized, holistic advising practices.

To address costs associated with reducing advisor-to-student ratios, postsecondary institutions have adopted a range of technology-mediated advising tools and systems as a way of efficiently advising large numbers of students. Advising technologies may empower students to more actively engage with advisors, and provide tools to support effective “advising-as-teaching” approaches, but not supplant direct, interactive contact with advisors.36 In a study of community college students’ attitudes towards technology-mediated advising systems, Kalamkarian and Karp (2015) found that students seek an interactive approach with advisors, particularly for multi-semester course planning and other complex, cognitive tasks.

Summary of evidence: Moderate Evidence

The panel defined the level of evidence for this recommendation as moderate.

This recommendation is supported by three studies that met standards without reservations (Cousert, 1999; Scrivener & Weiss, 2009; Visher, Butcher, & Cerna, 2010). Cousert (1999) and Visher et al., (2010) investigated the impact of one-semester mentoring programs, and Scrivener and Weiss (2009) evaluated effects of an incentivized advising program offered over a two-semester period. The large majority (79%) of the three study samples (totaling 3,563 students) consisted of students enrolled in developmental education courses within the study period.

Credit accumulation was assessed in all three studies.37 Meta-analytic results indicated a small, statistically significant effect in favor of the group of students who received enhanced advising on college-level credit accumulation (g = +0.11, p < .001). To contextualize this effect, if comparison group students earned an average of 6.0 credits during the program semester, then the enhanced advising group would earn about 6.7 credits on average. This means that for approximately every 100 students who receive enhanced advising, 23 students will complete one additional 3-credit-hour course.

Academic achievement was assessed in two of the three studies. Meta-analytic results indicated a non-statistically significant effect for academic achievement (g = +0.01, p = .73). Progress through developmental education was assessed in two of the three

studies;38 the effect was positive, but not statistically significant (g = +0.07, p = .18).

Other effects assessed in these studies tended to be small and not statistically significant. For example, Scrivener and Weiss (2009) found a small but non-statistically significant positive effect on initial enrollment and a small but non-statistically significant negative effect on degree attainment (3% attainment in the comparison group versus 2% in the advising group, from the first program semester through third post-program semester). Appendix D Table 2 has more detail about these studies.

How to carry out the recommendation

An important aspect of enhanced advising is a dedicated and lasting interaction between an advisor and a student. The purpose is to interact personally with the student to foster learning, encourage course completion, and decrease institutional barriers that limit or prevent student participation in the intellectual and social life of the college.

1.Recruit and train advisors. It is crucial to encourage and gain the participation of a critical mass of advisors. The advisors may be college counselors,39 volunteers from college staff, or professors.40 Strategies for increasing the number of advisors may involve a combination of negotiating with existing faculty and staff to take responsibility for enhanced advising roles, along with hiring of additional professional advisors.

The program should include advisor training, and specific enhanced advising techniques should be discussed in the training. The National Academic Advising Association offered essential steps and components for developing the content of effective advisor training which include the college’s concept and definition of the advising process, the informational aspects of advising, and the relationship skills involved in supporting students’ decision-making and planning.41 Researchers have also found that training faculty and staff in how to conduct “advising-as-teaching” is a promising way to engage with students, and utilize technological advising tools as supports to facilitate this advising instruction.42

Furthermore, training should establish expectations for the frequency and intensity of contact with students, and provide supports for accomplishing this. Visher and colleagues (2010), as an example, suggested providing a variety of programming, including a training manual to advisors with student services information and frequently asked questions, a monthly “brown-bag” training or refresher sessions, incorporating team-building exercises between faculty/staff-student pairs, or a final semester gathering to recognize the hard work and dedication of the advisors and students.

2.Require or incentivize in-person advising meetings. While students may favor using technology-mediated tools for simple, administrative tasks, they generally find in-person meetings to be more effective for support and encouragement, and for productively engaging in goal-setting, complex educational planning, and problem-solving.43 Figure 2.1. offers a sample planning tool used to facilitate enhanced advising services at Ivy Tech Community College in Cousert (1999). While students were not required to participate in mentoring as part of the Cousert (1999) study, Ivy Tech Community College requires all new students to meet with an academic advisor before registering for classes. The Beacon Mentoring Program similarly offered support for accessing enhanced advising in the context of students’ required mathematics classes.

Incentivizing participation in in-person advising, such as by providing a small stipend contingent on attendance, is another way to bolster attendance. For example, Ohio’s Opening Doors program offered students $150 per semester for attending at least two in-person advising sessions.44 It may be helpful to prioritize supports for at-risk students, who are especially susceptible to discontinuing due to difficulties navigating the enrollment process.45

3.Require or incentivize students to meet with advisors frequently and over a long(er) period of time. One advisory meeting is generally not enough time to advise students, particularly for multi-semester course planning and career guidance. In the Opening Doors Program in Ohio, students had an average of 8 contacts with their advisors during the first and second semesters.46 Offering regular meetings, before, during, and at the end of semesters is one way to encourage frequent meetings. In addition, advisors may meet with groups of students, such as during a certain class or at a structured time, to facilitate the communication of information more efficiently. As reported in Visher et al. (2010), the Beacon Mentoring Program encouraged advisors to plan short information sessions to be presented during a developmental English course, which occurred 2–3 times per semester. Advisors also met with students throughout the semester. The Opening Doors Program encouraged advisors to meet with students at least twice per semester for two semesters in a row.47

4.Use early alert systems to identify the students who most need enhanced advising. Early alert and intervention systems are designed to identify and support students at risk of attrition and improve their retention, persistence, and success.48 Early alert systems, or early warning systems, identify students in need of an intervention, which often includes enhanced advising. Though no studies on early alert systems met WWC standards for inclusion in this practice guide, Zachry Rutschow and Schneider (2011), suggested that these systems can be effective if they bring students in contact with needed services (though note that mere alerts, without subsequent resource mobilization, are unlikely to be effective).

Figure 2.1. Sample enhanced advising planning tool

Potential roadblocks and suggested approaches

Roadblock 2.1. Decreasing the student-to-advisor ratio means finding and training more advisors.

Suggested Approach 2.1. Recruit advisors from various sources in addition to hiring trained specialists. For example, the Beacon Mentoring program recruited advisors not only from the counseling and advising department, but also from student life, research/technology, special academic/career programs, assessment and placement, admissions and records, outreach and orientation, administration, and financial aid. Faculty provided advisors with in-class time to make at least three presentations per semester, and also gave advisors data on students’ attendance and grades so that advisors could intervene with struggling students.50

The student-to-advisor ratio is high at most community colleges, and many students may then make decisions that are poorly suited to their needs, academic goals, and career objectives. In relation to a typical student-to-advisor ratio of about 600:1 in community colleges, the National Academic Advising Association recommends decreasing the advisor load as low as possible (without recommending a specific ratio; see Robbins, 2013). In the Opening Doors Demonstration in Ohio,51 the student-to-advisor ratio was 81:1 at Owens Community College and 157:1 at Lorain Community College. In this program, the reductions in student-to-advisor ratio were achieved by hiring additional counseling staff in the student services divisions of the colleges.52

To decrease the advisor load, trained college counselors and advisors should not be the only ones to provide assistance to students. Well-trained college faculty and staff should also provide advising and mentoring to at-risk students (for an example of how faculty and staff were used in this way, see Visher et al., 2010). While faculty and staff who volunteered to serve as mentors in the Beacon Mentoring program did so as part of their job, engaging additional faculty in staff in new advising roles may require additional incentives or contract requirements. Importantly, decreasing institutional barriers can be accomplished by simply discussing challenges in addressing the procedural aspects of college. For example, staff or faculty with experience determining college courses can be helpful to students who have difficulty making these decisions.

Roadblock 2.2. It may be difficult to get students to attend advising sessions.

Suggested Approach 2.2. Due to scheduling issues, community college students may find it difficult (and some may be unwilling) to participate in one-on-one advising sessions. In addition, the advising sessions may be difficult for students to set up or keep track of. Be attentive to potential scheduling issues when establishing the schedule for enhanced advising sessions. Scheduling of enhanced advising sessions should consider students’ work and course schedules. Also, some enhanced advising activities may be appropriate for class time (especially on the first day of classes; see, for example, Visher et al., 2010), and some may appropriately take advantage of video conferencing services, such as Skype and FaceTime. Finally, consider setting up an electronic reminder system (e.g., via email or text message).

Roadblock 2.3. Advisors may not always be fully equipped to deal with the range of issues presented by students. The range of issues faced by at-risk students is large and may require a skill set similar to that needed by counselors. College advisors may not be well-equipped to handle such problems and may inadvertently allow students to return to recurring behaviors.

Suggested Approach 2.3. Train and provide guidance to advisors regarding the range of services available on campus. Scrivener and Weiss (2009) provided a specific training module along with certification, while Visher et al. (2010) suggested providing a training manual to advisors with student services information and frequently asked questions, offering monthly “brown-bag” training or refresher sessions, and utilizing team-building exercises between faculty/staff-student pairs. The nature of this training should ensure that advisors understand the range of services available on campus.

Roadblock 2.4. Coordination and communication across student services is ineffective or underdeveloped. Advisors, professors, tutors, and staff sometimes fail to communicate or follow-up to all invested parties regarding students’ needs, in relation to their academic and career goals.

Suggested Approach 2.4. Train advisors to work with students and other resource providers to connect students with multiple solutions and supports as needed so that they may be successful with short- and long-term goals agreed upon in an action plan. Cousert (1999) outlined the approach used by Ivy Tech State College in central Indiana, whereby faculty mentors work with students on an individualized action plan. The mentors document referrals and resources for the students to access in the plan and then follow-up with the students to determine their results in accessing those supports. The action plan documents students’ concerns, actions to be taken, and timeline for completing the actions. At South Texas College, the Beacon Mentoring Program provided students with knowledge and support of campus resources, such as tutoring, advising and counseling services, financial aid, and priority registration. To provide support before they failed or dropped out of the class, mentors communicated with math instructors about struggling students and also instilled a climate where they were the “go-to” person on campus for students’ questions.53 Depending on the type of advising structures, communication mechanisms, and resources available, campuses can be resourceful and strategic about how to personalize and maximize students’ knowledge and access of needed campus resources.

Recommendation 2 implementation checklist

Recommendation 3: Offer students performance-based monetary incentives.

Performance-based incentives are monetary awards that are disbursed to students based on meeting specific academic benchmarks. These awards supplement students’ financial aid packages, which may be based on need (e.g., Pell grants) or past achievement (e.g., state merit aid grants). In certain instances, the performance-based awards may reduce the number of loan dollars taken and grant dollars awarded, but they usually result in a net financial gain for students (see Patel & Valenzuela, 2013). The short-term goal of such initiatives is to encourage students to perform better in (and successfully complete) their classes. A longer-term goal is to support students’ progress through developmental and degree requirements to increase degree attainment.54

Incentive programs can vary in several ways, including the amount of the incentive and the number of semesters for which students are eligible for it. In the studies summarized below, the incentive amounts ranged from about $600 to $1,500 per semester, and students were typically eligible for 2 or 3 semesters. Generally, students are allowed to use the awards for any purpose (that is, not necessarily for educational expenses; Scrivener & Coghlan, 2011). Incentives are usually distributed at predetermined time points throughout the semester, rather than in one lump-sum payment. The rationale behind this disbursement structure is that it encourages students to meet program goals (e.g., maintaining a minimum level of enrollment).

Summary of evidence: Moderate Evidence

The panel judged the level of evidence supporting this recommendation to be moderate.

Six randomized controlled trials, which involved nearly 8,000 participants, examined the effects of performance-based monetary incentives. About two-thirds (67%) of the students across these six study samples enrolled in developmental education courses within the study period. Three of these studies examined the effects of performance-based monetary incentives alone or with minimal additional services^55 and three studies examined performance-based incentives with additional services (i.e., Miller et al., 2011, coupled incentives with enhanced advising; Patel & Valenzuela, 2013, coupled incentives with enhanced advising, tutoring, workshops, and other supports; Sommo et al., 2014, combined incentives with supplemental math lab instruction).

Offering incentives only. Three randomized controlled trials were identified that investigated the impact of performance-based incentives alone or offered monitoring advising support in addition. All three studies met WWC group design standards without reservations. In two of these studies,56 most students (85 percent, across the two study samples) were in developmental education. In the other study,57 about 40% of students in the study sample took at least one developmental education course. (See Appendix D Table 3.1 for further details.)

Across these three studies, incentives were associated with a small but statistically significant impact on the first instance of enrollment after randomization to condition (g = +0.08, p < .01). To contextualize this effect, if the base rate of enrollment in the first academic term was 94% in the comparison group, about 94.6% of students in the intervention group would have initially enrolled. Two studies^58 assessed the effect of performance-based incentives on academic achievement; this meta-analytic effect was positive, but not statistically significant (g = +0.09, p = .09).

All three studies evaluated the effects of providing performance-based incentives on college-level credits earned, and the meta-analytic results indicated a positive and statistically significant increase (g = +0.10, p < .001) in credits earned (after 7 or 8 semesters, depending on the study). To contextualize this effect, if the typical comparison student earned 14 college-level credits during the first year, the typical intervention student would have earned about 15.2 credits. This suggests that for every three students in the intervention condition, one additional student would complete an additional 3-credit-hour course over 7–8 semesters.

Finally, two studies assessed the effect of performance-based incentives on degree attainment.59 The meta-analytic results indicated a non-statistically significant positive increase (g = +0.06, p = .07).

Combining incentives with other supports. Combining incentives with other supports. Similar to the meta-analytic effects observed in incentives-only studies, providing performance-based incentives with additional supports also had a small but statistically significant impact on enrollment (g = +0.08, p < .03) across the three studies.60 Contextualizing this effect leads to the same conclusion as the enrollment analysis above in the incentives-only grouping: if the base rate of enrollment was 94% in the comparison group, about 94.7% of students in the intervention group would have enrolled.

Sommo et al. (2014) was the only one of these studies to examine student progress through developmental education (i.e., later earning college-level credit in their area of remediation). In this study, developmental math students in the intervention condition were more likely than developmental math students in the comparison group to pass college-level math (32% versus 26%, p < .05).

Academic achievement was measured by two studies in this group.61 In both cases, academic achievement was measured by assessing the proportion of students who earned a C average or better across their coursework. Meta-analytic results indicated a positive and statistically significant intervention effect (g = +0.12, p < .01). To help understand this effect, if 50% of students in the comparison group earned at least a C average or better, on average, 54% of students in the intervention group would earn at least a C average or above.

All three studies assessed the effect of performance-based incentives with additional supports on credit accumulation, but the meta-analytic effect was not significant (g = +0.03, p = .30).

Finally, two studies assessed the effect of performance-based incentives on degree attainment. The meta-analytic results indicated a positive, statistically significant impact (g = +0.08, p < .02). To contextualize this effect, if 33% of students in the comparison group graduate, 36.1% of students in the intervention group would graduate. (See Appendix D Table 3.2 for further details.)

How to carry out the recommendation

The primary purpose of this intervention is to support students by tying monetary incentives, which do not replace financial aid but are given in addition to it, to achieving academic success. See Figure 3.1. for some tips on how to budget for performance-based monetary incentives.

1.Design and structure payments of students’ performance-based incentives to yield students a net financial gain. Since many financial aid packages are tied to a student’s level of unmet need, an important consideration is the impact that this monetary incentive will have on the overall level of aid. Mayer, Patel, and Gutierrez (2015a) suggested that financial aid offices work with students and administrators to ensure that any incentives provide a net financial gain to participating students.

2.Distribute payments incrementally, according to specific desired performance benchmarks. To encourage and motivate students to reach academic milestones, distribute payments throughout the semester or year. Midterms and final exams are good candidates for disbursement periods.

3.Target incentives to low-income students. The evidence suggests that providing monetary incentives has positive effects for low-income students and various at-risk student populations (such as Latino male students with delayed entry to college). Programs should consider prioritizing these students.

4.Provide student support services, in conjunction with performance-based monetary incentives. Many students in developmental education may find it difficult to make time for additional campus supports without a specific incentive—given their work, family, and academic demands. Monetary awards can help incentivize students to take advantage of support services (e.g., academic advising, tutoring, mentoring) that will help them succeed. For example, the Adelante Performance Award Program linked payments to utilizing support services and academic milestones, thereby achieving high student attendance for supports, such as program orientations, advising sessions, tutoring services, and workshops.62 These supports are academically useful for students, and the personal contact made during these interactions may also help students feel motivated, encouraged, and accomplished when they succeed.

As a useful resource, MDRC published a technical assistance guide for developing and implementing performance-based scholarships which highlights key strategies for assembling key stakeholders, funding your program, creating a budget and partnerships, as well as aspects of important elements of designing, implementing, and monitoring the program.63

Figure 3.1. How to budget for performance-based monetary incentives

Potential roadblocks and suggested approaches

Roadblock 3.1. Performance-based incentives are expensive to offer.

Suggested Approach 3.1. Consider costs in relation to potential enrollment increases and successes from the program. Although the initial financial commitment to these programs may result in difficult allocation decisions, institutions may benefit from increases in student enrollment over time if enough students are successful. Furthermore, Patel and Rudd’s (2012) analysis suggests that the cost to implement such programs might decrease as a function of the number of students enrolled (i.e., administrative costs per student are lower when more students are enrolled in the incentive program). Further, Mayer and colleagues (2015b) conducted cost-effectiveness analyses of six performance-based monetary incentives programs. In 2014 dollars, the average direct cost of a performance-based scholarship program was $2,345 per student for the program’s duration. Mayer and colleagues (2015b) reported that about 72% of the direct cost was direct payments to students, about one-fourth (26%) was program administration costs, and 2.5% of the cost was additional student services. They found that while the performance-based monetary incentives required substantial financial investment, the average cost per degree was about the same for students who received the intervention compared to students who did not have the opportunity to earn performance-based monetary incentives.

In addition, the costs of the performance-based monetary incentives will likely be a small proportion of students’ total financial benefits package. For example, in Ohio, students received an average of $765 in performance-based incentives, compared with $7,947 in total financial aid in the same year.65 Even though proportionately the performance-based monetary incentives are a small investment, the opportunity to earn performance-based monetary incentives appears to result in small, positive effects—both short- and long-term.

Roadblock 3.2. Institutions lack staff members who can be dedicated to running these programs.

Suggested Approach 3.2. These programs entail new processes that require dedicated staff to oversee them to be effective. If possible, create a new position to oversee the program or utilize “student success coordinators” where available. The duties of this person or persons should include monitoring transcripts to determine whether students had met the award criteria and distributing payments to students. This position could be housed in the financial aid office or the student services office.66 If resources are not available to hire dedicated staff, other individuals within the school could also serve as monitors (e.g., counselors) and payment conduits (financial aid officers or departmental staff).

Roadblock 3.3. Additional monetary awards may decrease the financial aid packages that students might otherwise receive.

Suggested Approach 3.3. Because many low-income students have unmet financial need after financial aid and expected family contributions are calculated, performance-based incentives can be a source to fill this gap rather than extra work hours, taking fewer courses, or taking out more loans. After financial aid is calculated, students without remaining unmet need may not be able to receive money from incentives due to federal, state, and institutional policies that restrict funding beyond students’ level of unmet need. One way to address this issue is to target students with larger amounts of unmet financial need. Another strategy that institutions can use to ensure a net financial gain for students is to reduce loans when a student earns a performance-based monetary incentive. In the case of students who do not earn the anticipated money from incentives to fulfill their unmet need, financial aid directors should make sure that students have access to loans to fill the gap. Financial aid offices should develop and apply procedures for not reducing students’ grant aid in the event of aid overages.67

Because students’ financial aid awards could be reduced if they receive additional funding through performance-based incentives, administrators and financial aid officers should work together to ensure students experience a net increase in aid. For example, the nonprofit education research organization MDRC worked with Hillsborough Community College (Tampa, FL) financial aid staff to ensure that the Mathematics Access Performance Scholarship recipients did not experience a negative impact on their financial aid packages, using strategies described above.68

Recommendation 3 implementation checklist

Recommendation 4: Compress or mainstream developmental education with course redesign.

Many colleges have adopted an accelerated developmental coursework sequence in which students can more quickly complete developmental education courses and enroll in college-level courses.70 These reforms are referred to interchangeably in the literature as “intensive,” “compressed,” “condensed,” or “time-shortened” models.71 And, while the specific steps employed in the models may vary,72 the intent is always to reduce the length of time students spend in developmental education with the aim of improved student success rates.

Participation in accelerated developmental experiences can minimize the negative effects of being placed into developmental education. Edgecombe (2011) noted that if students can register for more than one sequential course in a semester, they are more likely to enroll in the second course, thereby improving retention. Accelerated courses that mainstream developmental education students into college-level work with contextualization or supplemental instruction also help students achieve the goals and outcomes of the college-level course assignments. Further, Jaggars et al. (2015) noted that acceleration may promote persistence and academic success because the reduced time in developmental education also reduces the opportunity for external factors, such as work or family responsibilities, to hinder students’ success.

The following three types of accelerated instructional models have promising evidentiary support in the research literature.

Compressed courses with curricular redesign. This approach involves offering developmental courses in a compressed schedule so students may either complete more levels of coursework in a single term or a term of coursework in less than the traditional 16-week semester. These courses and models also adapt curricular, instructional, and assessment practices as needed while supporting students’ mastery of skills and content. Faculty adjust the pacing, workload, and instructional strategies to cover similar curricular content in a shorter timeframe. Examples include the City University of New York’s (CUNY) accelerated courses,73 Chabot College’s (California) one-semester English redesign, Community College of Denver’s FastStart program in math, and Broward (Florida) College’s math redesign.76

Mainstreaming with supplemental support. Mainstreaming involves placing developmental students into college-level courses alongside their non-developmental education peers. In doing so, students underprepared for college-level work may succeed in college-level classes with the addition of supplemental support services. Supplemental supports may include companion classes (whereby the course instructor also provides additional instruction to students who are identified as needing developmental education), lab sessions, integrated tutorial support, and additional class sessions.77 One example is the Accelerated Learning Program (ALP) at the Community College of Baltimore County.78 Another example is the New Mathways Project provides a 1-year statistical reasoning pathway for students in social sciences or a 1-year quantitative reasoning pathway for students in science, technology, engineering, and math fields paired with a college-level student success course. This helps students develop strategies for knowledge development, self-regulation, motivation, and learning.79

Mainstreaming with contextualization to students’ academic goals and interests. This model simultaneously provides students with college-level content instruction and integration of developmental education skill-building, tailored to students’ career, technical, or academic interests. For example, the Integrated Basic Education and Skills Training program (I-BEST) geared toward basic skills students interested in career and technical education majors, linked developmental education curricula with college-level curricula in a co-teaching model, focused on students’ occupational or disciplinary interests, and engaged local employers as partners.80 Statway, is geared toward students interested in humanities and social science majors who place into developmental math courses. The program involves mainstreaming these students into a 1-year college-level statistics course to provide them with foundational skills in statistics, data analysis, and causal reasoning.81

Summary of evidence: Minimal Evidence

The panel assigned a rating of minimal evidence to this recommendation.

One study that evaluated the effects of a compressed, redesigned developmental writing course^82 met WWC standards with reservations. The researchers found that students in an accelerated developmental writing course sequence were more likely than students in the traditional, longer sequence to pass college-level English, to have earned more college-level credits after 3 years, and to have earned a degree after 5 years. All of these effects were statistically significant. This study reported the effects of compression of developmental course material from 8–12 credit hours to 6–7 credit hours at CUNY and provides promising evidence to support accelerated writing instruction.83 The following three additional studies did not meet WWC group design standards, but the panel believes that they provide credible evidence to support this recommendation. See Appendix D Table 4 for further details about all four studies.

Mainstreaming with supplemental support: Additional evidence from the ALP. Cho et al. (2012) examined the effectiveness of the ALP at the Community College of Baltimore County. This study did not meet WWC group design standards because the WWC could not assess baseline equivalence between intervention and comparison groups. However, the researchers used a propensity-score matched sample and controlled for a number of observed student characteristics in the analyses to account for influences of factors, such as gender, age, race, and measures of student academic background (college placement scores in reading, English, and math, median household income, several financial aid indicators, and employed campus and cohort fixed effects). Based on an analysis of several cohorts of students followed for at least 1 academic year, the authors concluded that ALP students outperformed non-ALP students on college-level English course completion rates and persistence to the next year (Table 2, p. 7). While about 75% of ALP students completed first college-level English by the next year, only 39% of non-ALP students completed the first college-level English course in that timeframe. The rate of persistence was also higher for ALP students: 64% of ALP participants compared to 48% of non-ALP participants persisted to the next year. The effects for both of these outcomes were statistically significant.

Mainstreaming with contextualization: Additional evidence from Washington’s I-BEST program. Zeidenberg, Cho, and Jenkins (2010) assessed the effects of the I-BEST intervention on students enrolled in the 2006/07 and 2007/08 academic years.84 In the I-BEST program (a mainstreaming and contextualized instruction intervention), college-level occupational courses with integrated basic skills content are co-taught by a basic skills instructor and an occupational instructor. The occupational coursework was available in a variety of fields, such as health care, office work, and skilled trades. This quasi-experimental study did not meet WWC standards because the WWC could not assess the baseline equivalence of the intervention and comparison groups. In the propensity-score matched sample, all students in both groups were in adult basic education and workforce programs (i.e., all students took at least one career or technical education course). Adult basic education serves students at the lowest levels of developmental education coursework. Even though the study did not meet WWC evidence standards, regression analyses controlled for a number of factors, including students’ basic skills test scores, English proficiency, socioeconomic status, race, age, and gender. The researchers found substantial differences between college credits earned by I-BEST students and the non-I-BEST comparison group (Table 6, p. 14). I-BEST students earned a mean of about 25 college credits; students in the comparison group earned only 17. I-BEST students also had a substantially higher rate of receiving a degree or certificate: 51% of I-BEST participants earned a degree or certificate by Spring 2009 while just 14% of students in the comparison group earned a degree or certificate. The effects for both of these outcomes were statistically significant.

Compressed courses with redesign: Additional evidence from Chabot College’s integrated reading and writing course. Edgecombe et al. (2014) investigated the effectiveness of a 1-semester accelerated developmental education course in English compared with an equivalent two-course sequence. The accelerated course model also included a redesigned curriculum and pedagogical changes that attempted to strengthen the integration of developmental reading and writing instruction. This study did not meet WWC standards because the baseline equivalence of the intervention and comparison groups could not be assessed. However, regression analyses controlled for a number of factors, including students’ placement scores, socioeconomic background, gender, age, marital status, racial and ethnic background, educational goals, graduation from high school, enrollment status, and prior participation in a learning community. Based on the 5-year outcomes, the researchers found that 60% of accelerated developmental English course-takers passed the first college-level English course (i.e., completed and received a grade of C or higher) compared with just 40% of students who took the longer English developmental course sequence. Furthermore, students who participated in the 1-semester developmental English course earned an average of 36 college-level credits while students who took the traditional developmental English course sequence only earned an average of 29 college-level credits. Finally, while only 18% of non-accelerated developmental English course-takers graduated from college, 25% of accelerated English course-takers graduated from college. All of these effects were statistically significant.

How to carry out the recommendation

Many individuals working together contribute to successful outcomes achieved by underprepared college students through mainstreaming or compression of developmental education courses. Administrators, professional developers, faculty,87 articulation officers, advisors, and students each have important roles to play in ensuring efficient transitions for developmental education students to succeed in college-level courses. See Figure 4.1. for an illustration of this concept.

1.Obtain faculty and leadership support for rethinking course structures and policies to efficiently support students in mastering developmental education skills and earning college credits. Support from faculty and leadership is vital to developing and implementing an effective accelerated course model, especially if the large majority of students are to be affected by the changes. Particularly for mainstreaming models, colleges must establish collaborative structures and processes for instructors to communicate with one another and ensure that students are mutually supported as they complete college-level coursework. To begin, assess the existence and effectiveness of current strategies for accelerating students’ progress through developmental education (See Figure 4.2.). During planning, anticipate challenges and strategize collaboratively to achieve optimal success (See Figure 4.3.).

Figure 4.1. Student success through accelerated coursework

Figure 4.2. Inventory of strategies currently in place for accelerating students’ progress through developmental education

Figure 4.3. Planning issues and considerations in adopting compressed and mainstreaming developmental education models

2.Provide systematic professional development and support for faculty to implement new accelerated course models. New models of acceleration generally require systematic professional development for faculty and, depending on the type of acceleration, faculty needs for professional development will vary. Principles of faculty professional development should address:

- How to maintain rigorous college-level content and skill development while meeting needs of students who are not college-ready in all areas.

- Ways to best use the institution’s resources and the expertise of various instructors to compress, mainstream, or modularize course content.

- Models for integrating accelerated course models in mathematics, reading, and English, or contextualizing courses to students’ academic and career interests.

- If computer-based tools and programs are being adopted, training and support for effectively using these tools to support accelerated course models should be provided.

3.Offer support to faculty for differentiating instruction to students of varied levels of academic preparedness in their classes. In mainstreaming models, faculty who teach introductory college-level courses may not be accustomed to differentiating instruction for students without college-level proficiency. Or, they may not have experience tightly coordinating their curriculum and instructional delivery with other instructors (e.g., those who provide supplemental instructional support). Developmental education instructors need to become familiar with the rigor of the college course content and assessments and determine how to best support students’ success in mastering the college-level course material.

4.Mobilize and communicate targeted supports for students’ specific instructional needs. When accelerating students’ progress in developmental coursework, students who are not yet college-ready may require targeted supports.88 For example, students may need help sustaining motivation and integrating the skills and knowledge learned from module to module when using a modularization approach. Specific suggestions include:

- Decide on the best structure, staffing, and allocation of credit for supplemental instruction, modularized content, and redesigned courses to offer necessary targeted supports to students of varying skill levels.

- Use communication supports—such as text-based reminders about upcoming deadlines and encouragement to keep working—to keep students motivated, on track, and primed to use existing campus resources.

Potential roadblocks and suggested approaches

Roadblock 4.1. Students may not take advantage of accelerated course options and their benefits.

Suggested Approach 4.1. Faculty should work with academic advisors or mentors to ensure the appropriate students register for accelerated courses. Underprepared students may be hesitant to sign up for an accelerated course in a topic area in which they consider themselves weak or have been told they need remediation.89 Faculty and support staff should develop and distribute materials to inform students of available accelerated course sequencing.90 This practice would likely be more effective when implemented in concert with enhanced advising.

Roadblock 4.2. Adjunct faculty who teach developmental education students are often disconnected from instructional decision-making, professional development opportunities, and institutional supports.

Suggested Approach 4.2. Most developmental education courses are taught by adjunct faculty,91 who have more limited contact with students outside of class, are less aware of administrative changes, and are less involved in collective faculty decision-making, than full-time faculty and staff.92 Without intervention, an over-reliance on adjunct faculty to teach developmental education courses can limit the capacity of institutions to create effective accelerated instruction models. When considering the implementation of accelerated instructional models, adjunct faculty need institutional support, professional development, and engagement with curriculum development. As discussed by Kosiewicz and colleagues (2016), the involvement and support of adjunct faculty in the design and implementation of the accelerated curriculum will strengthen their investment and capacity to deliver the results anticipated from the new approach.

Roadblock 4.3. Faculty members are accustomed to the status quo of existing developmental education sequences, structures, and readiness levels of students in their classes.

Suggested Approach 4.3. Colleges should provide faculty with professional development, opportunities to collaboratively plan with other faculty on curriculum development, and incentives for faculty collaboration.93 Faculty who currently teach college-level courses may resist adaptations needed for mainstreaming developmental education students in their classes. Smooth and successful implementation of the changes requires deliberate efforts to cultivate champions and support from all stakeholders, which takes time and persistence. Faculty with expertise in teaching college-level content and faculty skilled in teaching developmental skills will need to partner to maintain instructional rigor for college-level mastery while differentiating instruction to meet students where they are. Communication and collaborative decision-making across departments and/or course levels will be necessary.

Roadblock 4.4. Existing state, college, and regional accreditation policies may not be well-aligned with chosen accelerated course models.

Suggested Approach 4.4. College-level leaders, state policymakers, regional accreditors, and discipline-specific associations should discuss adjustments to any necessary policies and procedures that would inhibit students in receiving appropriate credit for redesigned courses. Transfer and articulation policies may not currently support the models identified in this guide, and colleges may need to work proactively to ensure that new course models are recognized and properly credited when students transfer. Bracco and colleagues (2015) report upon North Carolina’s creation of the NC Ready for Success Steering Committee as one example of how multiple sectors worked together to better align policies and practices to smooth transitions for students’ success.

Recommendation 4 implementation checklist

Recommendation 5: Teach students how to become self-regulated learners.

Traditional academic instruction emphasizes learning content. However, many students, including those in developmental education, arrive on college campuses without knowing much about how they learn and which study strategies might work best for them. Teaching students to be more aware of their own learning processes through encouraging monitoring and reflection of their learning can help them become better learners.95

Teaching students to be self-regulated learners focuses students on the parts of the learning process over which they have control. Teaching students how to learn more effectively can be embedded into existing subject-matter coursework. Typically, teaching students to become self-regulated learners involves demonstrating how to (a) approach a task, (b) implement that approach or strategy, (c) evaluate how well the approach or strategy worked, and (d) decide what to do next.96

Young and Ley (2003, 2001) describe four instructional principles and activities to support underprepared students to become better self-regulated learners: 1) prepare and structure the learning environment to help students cope with distractions, 2) organize and transform instructional materials to support self-regulated learning, 3) work with students to keep records and monitor progress, and 4) work with students to evaluate performance against a standard.

Summary of evidence: Minimal Evidence

The panel judged the level of evidence supporting students in regulating their own learning as minimal.

One study met WWC standards without reservations.97 Zimmerman et al. (2011) observed statistically significant effects on three measures of progress in developmental education, including the best measure: the proportion of students who passed the college-level course in which remediation was needed (in this case, math). In the intervention group, 25% of students passed-college level math; only 13% of students in the comparison group did. See Appendix D Table 5 for more detail on this study.

How to carry out the recommendation

Again, teaching students how to become self-regulated learners involves introducing students to the process of (a) approaching a task, (b) implementing that approach or strategy, (c) evaluating how well the approach or strategy worked, and (d) deciding what to do next. Figure 5.1. provides an overview of the self-regulated learning process for instructors and students.

1.Encourage students to improve their estimation of their current capabilities. When students plan which strategies to use in a learning task, they have to make an accurate judgment of their capabilities and resources. Completing tasks in which students have been previously successful will likely take less forethought, because they already know strategies that work in those learning situations. When students encounter new learning tasks or situations, they will have to use their own judgment of how they performed previously on similar tasks and which strategies might be relevant. However, struggling learners are often poor judges of their own knowledge and capabilities.99 Monitoring and reflecting on the learning process allows students to practice judging their abilities, performing a task, and getting feedback to improve their judgment of their own knowledge and capabilities. For example, instructors can prompt students to reflect on what they think they know and can do while doing an academic task. After students receive feedback on the task, instructors can ask students to reflect on whether they were as able as they thought. Zimmerman et al. (2011) observed that after students in developmental education courses practiced this type of structured reflection, they were better judges of their own abilities than students who did not regularly participate in the reflection.

2.Integrate monitoring and reflecting on the learning process with coursework or course content. Students can learn to be better learners while learning course content. Instructors can structure their courses to support this by integrating into the course expectations and assignments student planning, reflection, and evaluation of the learning process.100 Instructors can prompt students at the start or end of assignments to aid students in monitoring and reflecting on their learning. These prompts can be steps that students have to go through to complete the assignment or process questions to direct students to evaluate their learning approach and determine what to do on similar tasks in the future. For example, for more general studying behaviors, instructors can ask questions, such as: “How do you take notes in class? Do you review your notes? When? How? Do you stop periodically and check to see if you are understanding the material?”101

The intervention in Zimmerman et al. (2011) gave students a chance to learn from their errors on quizzes by completing a self-reflection form in which they had to compare their estimates of ability with their quiz score, explain why they got the quiz item wrong, select a different strategy to try in the future, state their confidence that that strategy would be effective, and complete a similar problem to the one they answered incorrectly on the quiz. Teaching students to improve their learning process can also occur in a standalone course that makes linkages to content in courses that students are concurrently attending. See Figures 5.2 through 5.7 for examples.

3.Present students with examples and models of how to monitor and reflect in the course subject matter. Instructors can model monitoring and reflecting through making their thinking explicit in classroom discussions and lectures so that students have a model for how to approach an academic task. For example, an instructor might explain to students how to determine what is important in a reading passage. Or, instructors can ask students to be explicit about their monitoring and reflecting in class discussions. Student responses in discussion or homework problems can be used in the same way as instructor modeling of monitoring and reflecting.102 In addition, these examples may be more accessible for students because they may be more similar to their own thinking. For example, in Zimmerman et al. (2011), “the instructor encourage[d] students to go to the board to demonstrate problem solving and error detection strategies, and . . . the instructor encourages students to verbalize error detection and problem solving strategies while working through practice problems” (p. 146).

Potential roadblocks and suggested approaches

Roadblock 5.1. Some faculty may resist teaching developmental education students to regulate their learning because of scarce time and resources in the classroom.

Suggested Approach 5.1. Assure faculty that teaching students to be better self-regulated learners leads them to be more effective and efficient learners in the long term. Hudesman et al. (2013a) noted that new instructors are often resistant because of the extra time required to implement self-regulated learning programs. Experienced instructors may also have these concerns, particularly since adding more content or lessons to a course will necessitate the use of class time. However, as suggested by Hudesman et al. (2013a), after initial lags in learning, students may actually learn more efficiently once they become more effective self-regulated learners. Also, once students become more self-directed, the need for faculty intervention may fade.103 Hudesman et al. (2013a) also found that instructors may find it useful to automate administration and scoring of quizzes with tablet PCs to create time efficiencies and engage students with online feedback on quizzes and assignments.

Roadblock 5.2. Building students’ learning skills may not fit into faculty goals for a given course.

Suggested Approach 5.2. Involve faculty as meaningful partners in integrating practices to teach self-regulated learning. There are many reasons why faculty members may not see building learning skills as appropriate for their course. For example, they may assume that students already have these skills or that they should be learning them elsewhere. Given that the specific strategies chosen are likely less important than finding strategies that faculty can implement well and believe will be effective, there is room for faculty choice in the specific strategies implemented, which may facilitate faculty investment.

Roadblock 5.3. Students may resist monitoring and evaluating their learning process because they feel it is unfamiliar, cumbersome, time-consuming, irrelevant, or useless.

Suggested Approach 5.3. Discuss the relevance of reflecting on students’ own learning processes and give students time and space to internalize these practices. As postsecondary learners, students in developmental coursework have a long history of learning which, oftentimes, was less productive than it could have been. Instructors may find that students’ engagement in self-regulated learning processes in academic coursework increases after engaging students in applying these techniques to career objectives and personal goals such as improved relationships, health and athletics, weight loss, and others.

Students may be set in their ways about what works for their own learning. Or, students may not have considered or used more effective strategies that require them to monitor and reflect on their learning process. Also, thinking about the learning process while they are learning may feel more difficult to students, and the pace of learning can seem slower. Instructors can present self-regulated learning instruction as a way for students to become better learners, which will help them in their current and future classes and careers. Instructors can discuss how they use these strategies in their own learning and how other students have used these strategies to become more successful learners. Instructors can also model strategies in the classroom.

If instructors can give students time, space, and instruction to support them in becoming self-regulated learners, instructors may overcome the initial negative reaction to the novelty of the practices. For example, Hudesman et al. (2013b) noted that students, after being exposed to a self-regulated learning program, said the program created a “positive educational experience” regardless of whether they passed the course.104

Roadblock 5.4. Instructors may not be skilled self-regulated learners themselves.

Suggested Approach 5.4. Provide instructors with materials and training so they can put the skills they are teaching developmental education students into practice. Instructors may not use these practices regularly in their own learning. Or, they may be unfamiliar with terminology or steps for implementation. Like students, instructors can become more skilled in self-regulated learning through direct instruction and practice. In a supplemental report to Zimmerman et al. (2011), Hudesman and colleagues (2014) described how instructors in the intervention attended a 2-day onsite training program to learn how to implement self-regulated learning practices in their classes.105 Over the semester and after the training program ended, instructors completed logs covering classroom experiences and how they applied self-regulated learning. The logs gave instructors practice participating in the self-regulated learning process, because they asked instructors to reflect on their goals for the class period and evaluate how effective their strategies were implemented.106 Instructors received feedback on their logs from program staff.

Figure 5.1. Overview of the self-regulated learning process^107

Figure 5.2. Overview of the self-regulated learning process

Figure 5.3. Sample self-regulated learning quiz in mathematics

Figure 5.4. Sample quiz reflection and revision sheet in mathematics

Figure 5.5. Paired discussion among students—evaluation of the quiz format

Figure 5.6. Guided self-reflection about the course

Figure 5.7. Discussion questions for applying self regulated learning skills to other courses

Recommendation 5 implementation checklist

Recommendation 6: Implement comprehensive, integrated, and long-lasting support programs.

Some institutions have implemented comprehensive and integrated support programs that incorporate a variety of components with the goal of improving student educational outcomes. While many colleges offer multiple supports to their students, what differentiates this practice from business as usual is the intentional focus on integrating these supports and incentivizing participation in the long term. One example is the City University of New York’s (CUNY) Accelerated Study in Associate Programs (ASAP). ASAP provides a “comprehensive array of services and supports over a three-year period to help more students graduate and to help them graduate sooner.”114

Summary of evidence: Moderate Evidence

The panel has assigned a moderate evidence rating for this practice.

The rating is based on the expert judgment of the panel and on one large randomized controlled trial of 896 low-income students with developmental education needs^115 that meets WWC standards without reservations.

This study revealed statistically significant effects on its primary outcomes; many of these effect sizes were large in magnitude. Specifically, students who participated in ASAP were more likely to complete their developmental education requirements (75% completion rate in the ASAP group versus 57% in the comparison group); earn better grades (about 69% of the ASAP group and 60% of the comparison group earned at least a 2.0 GPA); earn more college credits (after 6 semesters, the ASAP group accumulated 43 credits and the comparison group earned 35 credits); earn a degree (usually an associate’s degree) at the study’s longest follow-up date (by the sixth semester, 40% of the ASAP group and 22% of the comparison group had earned a degree); and enroll at a four-year college within 3 years of entering the program (transfer rates were 25% in the ASAP group and 17% in the comparison group). Figure 6.1. and Appendix D Table 6 summarize these impacts.

Of the interventions studied for this guide, ASAP was the largest in terms of scale and scope. As a result of the expansive scope of the ASAP intervention, the panel observed greater improvement across more outcomes than in any of the other practices in this guide. While other practices received a moderate rating from the panel, this practice—implementing comprehensive, integrated, and long-lasting support programs—showed the largest increases in effects on college success outcomes, including degree attainment.

Figure 6.1. Postsecondary impacts of Accelerated Study in Associate Programs

Figure 6.2. Illustrative example of comprehensive, integrated, and long-lasting supports

How to carry out the recommendation

Many students in developmental education face a range of challenges.117 To ensure that they can succeed, institutional leaders should consider programs that combine student services, specific requirements, scheduling that encourages student cohorts, and financial supports. The impact of combining these supports appears to be more than simply providing additional help and instead creates a clear path to students’ success by systemically eliminating institutional barriers that create obstacles to students’ progress and attainment.

The ASAP intervention is one example of how to carry out comprehensive, integrated support programs for developmental education students.

The primary purpose of ASAP was to accelerate progress through college so that students could graduate in 3 years. ASAP required full-time attendance and provided students with services (such as comprehensive advisement, tutoring, and career assistance) and financial support (such as tuition waivers, free textbooks, and free public transportation passes). To ensure student schedules that facilitated accelerated study, ASAP offered and encouraged students to take blocked or linked courses for the first year and a student success course for the first few semesters. ASAP students were able to register for courses early, which helped them to design a schedule suitable to their needs. ASAP students were encouraged to take their developmental courses early and to graduate within 3 years. To successfully implement all of the programmatic components, a strong, collaborative organizational structure was developed. Figure 6.2. provides an overview of the components of this program.

1.Develop a strong, collaborative organizational structure that uses data to inform practice. Attention to the oversight and coordination of a comprehensive and integrated support program is critical. A large number of people and units will need to be involved in fairly constant communication for a program like this to be successful. Leadership at different levels is also necessary. For example, ASAP was implemented as a partnership between the CUNY Office of Academic Affairs and three participating community colleges. This required multiple layers of organizational structure and high levels of collaboration. The Office of Academic Affairs provided fiscal and programmatic oversight while the individual colleges established ASAP teams that operated the programs at their sites and provided direct services to the students. The central office staff and college ASAP directors met monthly and communicated frequently between meetings.118

It is also important to use data to inform program structure, monitor program implementation and usage, and evaluate effectiveness. ASAP implementers used data to:

- Develop the program structure based on motivating factors and needs of the student population. A comprehensive, integrated, long-lasting support program may look quite different from one location to another because students in different colleges are motivated by different incentives and have different needs. For example, in New York City, many students consider MetroCards to be valuable, given the expense, efficiency, and popularity of public transportation. Thus, MetroCards were given as an incentive to fulfill program requirements, such as going to advising appointments. In other locations, gift cards for gasoline may be a comparable incentive.

- Track the students as they go through the program and meet the various requirements. Students are required to enroll full time and attend advising, tutoring, career services, and the ASAP seminar. These activities need to be tracked to identify which students are eligible to receive their monetary incentives and remain in the program. To keep track of this information, each ASAP college recorded information about contact with participating students in a centralized data management system. Contacts with advising and career services were logged monthly, while ASAP seminar participation and tutoring visits were tracked weekly. CUNY central office evaluation staff analyzed program data monthly, and shared reports with the colleges. Key program staff and administrators then used the data to manage and modify the program, so that program benchmarks would be met. If data at a college showed, for example, that students were not meeting the target to meet with their advisor twice a month, the CUNY central staff and the college’s program director would enact solutions to resolve this issue.

- Assess program effectiveness. The central office conducted periodic surveys of ASAP students and held student focus groups about their perceptions of and satisfaction with the program. An impact evaluation was also conducted.

2.Clearly communicate expectations to students. By way of example, ASAP communicated its goals of full-time enrollment, taking developmental courses early, and graduating quickly through written materials, program orientation, and discussions with advisers.119 The messaging began immediately as students entered ASAP and continued throughout the program. The primary messages students received were to:

- Enroll full time and take intersession courses. Full-time enrollment was required for the ASAP intervention in each fall and spring semester that students were enrolled in the program. Advisors also encouraged students to take courses in the 6-week intercessions. Although it is still unknown which program components resulted in ASAP’s positive effects, the researchers speculated that the full-time enrollment requirements, coupled with the multiple supports to facilitate that enrollment, appear to be “central to the program’s success”.120 121 Colleges may also be able to be flexible when defining “full time.” For example, if full time is defined as 12 credit hours per semester for an undergraduate student, it could be redefined to 24 credit hours in an academic year (perhaps with a minimum of six credit hours in the fall and spring semesters). Doing so would allow students taking courses during intersessions and in the summer to count these toward their full time enrollment status, thereby both encouraging a more continuous enrollment pattern and providing some degree of flexibility for students who cannot manage four courses at a time.

- Take developmental courses early. Although ASAP did not incorporate an accelerated developmental coursework sequence, it promoted taking developmental courses early, mostly through conversations about course planning and registration with advisors. Developmental courses for ASAP students were often part of the linked or blocked courses offered to students during the first 2 semesters.122

- Graduate quickly. ASAP promoted a messaging strategy that students “should and can graduate within three years”.123 This message was promoted during program orientation, through written materials, and during advising sessions from the beginning through the end of the program. Taking advantage of winter and summer intersessions also helped students to graduate within three years.

3.Design course enrollment strategically. In ASAP, program developers designed course enrollment structures to group ASAP students together and allow ASAP students to maximize the likelihood of maintaining a full-time schedule. This included:

- Blocked or linked courses. ASAP students may enroll in linked courses in their first year. These linked courses include two or more classes grouped together with seats reserved for ASAP students.124 The purpose of linking was to create a learning community for students, which are based on the belief that active learning in a community based setting can improve academic outcomes by increasing academic and social integration.125

- Student success course. ASAP Students enroll in an ASAP seminar as a cohort during their first few semesters.126 This seminar was a noncredit course covering topics such as problem solving, stress management, study skills, and campus information. The seminar also helped with career and transfer planning and facilitated discussions on topics such as leadership and community involvement.

- Early registration. ASAP students can register for courses early,127 which gives students who work full time (or have other out-of-school responsibilities) the best chance to create a course schedule that fits their lives.

4.Decide which student support services will be provided and how to ensure that students participate. There are a number of different types of student support services that can be offered to developmental education students. In ASAP, participation in enhanced advising, tutoring, and career services was mandatory. There were also some voluntary services. Students were provided with:

- Enhanced comprehensive advising on academic, social, and interpersonal issues by ASAP advisors. The student-to-advisor ratio was between 60:1 and 80:1, and ASAP participants met with their advisors very frequently (an average of 21 times in the first semester and 17 times in the second semester; Scrivener et al., 2015). The ASAP advisors worked specifically with ASAP students and had an ASAP supervisor who helped to define their role.

- Required tutoring during their developmental courses and during any semesters they were on academic probation. Each of the participating colleges hired tutors dedicated to ASAP students. Tutors were generally former students at four-year colleges or adjunct faculty.128

- Required meetings with an ASAP-dedicated career and employment specialist. Topics include balancing work and school, finding scholarships, finding part-time work, networking, writing resumes, and aligning college majors with career paths. Participants engaged in an average of five meetings in the first semester and four meetings in the second semester. Students continued to visit with career and employment specialists at “comparatively high rates”129 in the second and third years. Visits began decreasing in the fourth and fifth semesters. The career and employment specialist also hosted career fairs and job talks.

Voluntary services were also provided. For example, ASAP participants could meet with a social work intern one-on-one as needed. Although this service had much potential, it was staffed by rotating interns, and having a new intern each semester was identified as a challenge to the popularity and effectiveness of this offering.130 Additionally, a small number of students were selected to participate in the ASAP Student Leadership Program, which focused on public speaking skills, teamwork, diversity, and advocacy.

5.Provide performance-based monetary incentives and other financial supports. In exchange for program participation, ASAP provided three primary monetary incentives: 1) financial assistance with textbooks, 2) a tuition waiver that covered the difference between a student’s tuition and fees and financial aid, and 3) a monthly transportation pass. The requirement for ASAP participants to receive the program’s financial benefits was enrollment in at least 12 credits or more per term. A 3.0 GPA or higher was required to receive advisors’ discretionary funds for covering students’ winter or summer coursework.

To address course-related financial concerns, ASAP students received free textbooks for all courses. At the end of the semester, students were responsible for returning the books. ASAP staff estimated the textbook cost for a full-time student to be approximately $270 per semester.

ASAP waived the difference between a student’s financial aid package and their tuition and fees. Financial aid offices and college ASAP directors worked together to ensure that the tuition waiver was applied to accounts internally and not paid directly to students. Only 9% of ASAP students needed this extra financial assistance during the first semester of study participation; during the third year of the study, 3–5% of the participants received this waiver. Among students receiving a waiver, the average amount was $719.131

Students in the ASAP program were given free monthly MetroCards for public transportation, contingent on participation in program requirements, such as advising, tutoring, and full-time enrollment. Among the ASAP participants, 93% reported that they had received financial assistance with transportation in the first year of the study, and the vast majority of ASAP students received their MetroCard in most or all of the months when they were taking classes. The cost of the transportation passes increased during the 3-year evaluation. By the end of the follow-up period in 2013, this cost was $112 per month.132

ASAP provided a comprehensive, integrated package of student services, monetary incentives, linked courses, an ASAP seminar, and other supports. Furthermore, CUNY sent consistent, strong messages to ASAP students to enroll full time, take their developmental classes early, and graduate within three years. The available evidence does not show which of these program components, or which combination of components, were responsible for the success of the program. However, based on the relatively large effects achieved by ASAP relative to the effects of an individual program or a combination of supports implemented for a short time, the expert panel recommends adopting a systematically integrated set of interventions over multi-year period over any single intervention in isolation.

For additional guidance, CUNY published a resource guide for college administrators, faculty, and other key stakeholders interested in instituting a program such as ASAP at their own institution. The resource guide shares information about program structure, program components and implementation, program evaluation and management, as well as sample planning templates.133

Potential roadblocks and suggested approaches

Roadblock 6.1. The college is on a limited budget and cannot afford to provide such comprehensive services.

Suggested Approach 6.1. Focus on the long-term benefits of cost per degree from this type of program. Comprehensive and integrated support programs are expensive to implement compared with business-as-usual models. Over 3 years of implementation, Scrivener et al. (2015) estimated the CUNY ASAP program cost (which included the base cost of the program plus indirect costs attributable to the program) per ASAP student per year to be about $5,400 more than the cost for a student who received the usual college services. However, the base operating cost per student for ASAP at CUNY fell about by about 20% after the first few years of implementation, to about $3,900 additional cost, per student per year, compared to students who received typical college services.134

Even though it is more costly at the front end to implement comprehensive and integrated support programs, the benefits can outweigh the costs. For example, the CUNY ASAP program generated more graduates than the business-as-usual comparison (40% versus 22%, respectively). And, although beyond the scope of this guide, a benefit-cost study suggested that the cost per degree was lower.135

Roadblock 6.2. Sustaining student interest and progress through the summer is challenging.

Suggested Approach 6.2. Promote and support students’ year-round enrollment whenever possible. To the extent possible, year-round enrollment should be a priority for underprepared college students. This roadblock may also be alleviated by providing student assistance with employment, providing on-campus employment, addressing students’ financial aid challenges for summer courses, and offering other student supports (e.g., child care).

Roadblock 6.3. Turnover in leadership and objections to change among staff can hinder the implementation of the program.

Suggested Approach 6.3. For a comprehensive, integrated, and long-lasting program to be successful, provide and support leadership, coordination, and collaboration at all levels. Leadership and staff turnover during a long-term implementation process presents a formidable challenge. One strategy for ensuring collective responsibility of long-range implementation of the program is to engage stakeholders throughout the institution in the coordinated design of the program and ongoing learning and modifications. Program developers must provide a structure for collaboration and support the participation of key individuals involved in each facet of the programming. Given the positive outcomes for the ASAP program, available evidence may be effective at convincing staff to participate and make changes to existing practices and policies. Champions should be cultivated from all the important sectors and levels to ensure adaptive implementation, as leadership and staffing changes occur. With structures for collaborative solution-finding, discussion of existing data on current programs’ impact on student outcomes, and broad participation in learning from existing evidence, champions at all levels can be effectively created and nurtured.

Roadblock 6.4. Many students placed in developmental education are part-time, so requiring full-time enrollment would exclude that population.

Suggested Approach 6.4. Depending on the proportion of your institution’s full-time and part-time enrollment, consider designing comprehensive, integrated, and long-lasting programs for both full-time enrolled students and part-time enrolled students, while incentivizing students to enroll full time. About 40% of students enrolled in U.S. community colleges are enrolled full time,136 and, as a sector, community colleges serve the largest share of students underprepared for college. Although some researchers question whether a full-time attendance requirement benefits all students in 2-year programs,137 most research does find a relationship between full-time enrollment and positive outcomes for college students, especially when implemented with sufficient planning and supports for the students—and not in isolation. Some part-time students may be incentivized to enroll full time to have the benefits of an ASAP-like program, while other part-time students may not be able to alter their enrollment and yet benefit from a modified ASAP program geared toward part-time students.

Recommendation 6 implementation planning checklist

Appendix A – Postscript from the Institute of Education Sciences

What is a practice guide?

The Institute of Education Sciences (IES) publishes practice guides to share evidence and expert guidance on addressing education-related challenges not readily solved with a single program, policy, or practice. Each practice guide’s panel of experts develops recommendations for a coherent approach to a multifaceted problem. Each recommendation is explicitly connected to supporting evidence. Using What Works Clearinghouse (WWC) design standards, the supporting evidence is rated to reflect how well the research demonstrates the effectiveness of the recommended practices (see Table A in the Introduction for more details on levels of evidence):

- Strong Evidence means positive findings are demonstrated in multiple well-designed, well-executed studies, leaving little or no doubt that the positive effects are caused by the recommended practice.

- Moderate Evidence means well-designed studies show positive impacts, but there are questions about whether the findings can be generalized beyond the study samples or whether the studies definitively show evidence that the practice is effective.

- Minimal Evidence means that there is not definitive evidence that the recommended practice is effective in improving the outcome of interest, although there may be data to suggest a correlation between the practice and the outcome of interest.

How are practice guides developed?

To produce a practice guide, IES first selects a topic. Topic selection is informed by inquiries and requests to the WWC Help Desk, a limited literature search, and evaluation of the topic’s evidence base. Next, IES recruits a panel chair who has a national reputation and expertise in the topic. The chair, working with IES and WWC staff, then selects panelists to help develop the guide. Panelists are selected based on their expertise in the topic area and the belief that they can work together to develop relevant, evidence-based recommendations. Panels include two practitioners with expertise in the topic. Relevant studies are identified through panel recommendations and a systematic literature search. These studies are then reviewed using the WWC design standards by certified reviewers who rate each effectiveness study. The panel synthesizes the evidence into recommendations.

WWC staff summarize the research and help draft the practice guide. IES practice guides are then subjected to external peer review. This review is done independently of the IES staff who supported the development of the guide. A critical task of the peer reviewers of a practice guide is to determine whether the evidence cited in support of particular recommendations is up-to-date and that studies of similar or better quality that point in a different direction have not been overlooked. Peer reviewers also evaluate whether the level of evidence category assigned to each recommendation is appropriate. After the review, a practice guide is revised to meet any concerns of the reviewers and to gain the approval of the standards and review staff at IES.

Levels of evidence for What Works Clearinghouse practice guides

This section provides information about the role of evidence in IES’s WWC practice guides. It describes how practice guide panels determine the level of evidence for each recommendation and explains the criteria for each of the three levels of evidence (strong, moderate, and minimal evidence). The level of evidence assigned to each recommendation in this practice guide represents the panel’s judgment of the quality of the existing research to support a claim that, when these practices were implemented in past research, positive effects were observed on student outcomes.

After careful review of the studies supporting each recommendation, panelists determine the level of evidence for each recommendation using the criteria in Table 1. The panel first considers the relevance of individual studies to the recommendation and then discusses the entire evidence base, taking the following into consideration:

- Number of studies;

- Study designs;

- Internal validity of the studies;

- Whether the studies represent the range of participants and settings on which the recommendation is focused;

- Whether findings from the studies can be attributed to the recommended practice; and

- Whether findings in the studies are consistently positive.

A rating of strong evidence refers to consistent evidence that the recommended strategies, programs, or practices improve student outcomes for a diverse population of students. In other words, there is strong causal and generalizable evidence.

A rating of moderate evidence refers either to evidence from studies that allow strong causal conclusions, but cannot be generalized with assurance to the population on which a recommendation is focused (perhaps because the findings have not been widely replicated) or to evidence from studies that are generalizable but have some causal ambiguity. It also might be that the studies that exist do not specifically examine the outcomes of interest in the practice guide, although the studies may be related to the recommendation.

A rating of minimal evidence suggests that the panel cannot point to a body of evidence that demonstrates the practice’s positive effect on student achievement. In some cases, this simply means that the recommended practices would be difficult to study in a rigorous, experimental fashion; in other cases, it means that researchers have not yet studied this practice, or that there is weak or conflicting evidence of effectiveness.

A minimal evidence rating does not indicate that the recommendation is any less important than other recommendations with a strong or moderate evidence rating. In developing the levels of evidence, the panel considers each of the criteria in Table A. The level of evidence rating is determined by the lowest rating achieved for any individual criterion. Thus, for a recommendation to get a strong rating, the research must be rated as strong on each criterion. If at least one criterion receives a rating of moderate and none receives a rating of minimal, then the level of evidence is determined to be moderate. If one or more criteria receive a rating of minimal, then the level of evidence is determined to be minimal.

The panel relied on WWC design standards to assess the quality of evidence supporting education programs and practices. The WWC evaluates evidence for the causal validity of instructional programs and practices according to WWC design standards.

Information about these standards is available at http:// whatworks.ed.gov. Eligible studies that meet WWC designs standards without reservations or meet WWC design standards with reservations are indicated by bold text in the endnotes and references pages.

A final note about IES practice guides: In policy and other arenas, expert panels typically try to build a consensus, forging statements that all its members endorse. Practice guides do more than find common ground; they create a list of actionable recommendations.

Where research clearly shows which practices are effective, the panelists use this evidence to guide their recommendations. However, in some cases research does not provide a clear indication of what works. In these cases, the panelists’ interpretation of the existing (but incomplete) research plays an important role in guiding the recommendations.

As a result, it is possible that two teams of recognized experts working independently to produce a practice guide on the same topic would come to very different conclusions. Those who use the guides should recognize that the recommendations represent, in effect, the advice of consultants. However, the advice might be better than what a school or district could obtain on its own. Practice guide authors are nationally recognized experts who collectively endorse the recommendations, justify their choices with supporting evidence, and face rigorous independent peer review of their conclusions. Schools and districts would likely not find such a comprehensive approach when seeking the advice of individual consultants.

—Institute of Education Sciences

Appendix B – About the Authors

Panel

Tom Bailey, Ph.D. Dr. Bailey is the George and Abby O’Neill Professor of Economics and Education at Teachers College, Columbia University. He is also director of the Community College Research Center (CCRC) and two national centers funded by a grant from the Institute of Education Sciences (IES): the Center for Analysis of Postsecondary Education and Employment, established in 2011, and the Center for the Analysis of Postsecondary Readiness, established in 2014. From 2006 to 2012, Dr. Bailey directed another IES-funded center, the National Center for Postsecondary Research.

Dr. Bailey has been the director of the Institute on Education and the Economy at Teachers College since 1992. In 1996, he established the CCRC with support of the Alfred P. Sloan Foundation. In June 2010, U.S. Secretary of Education Arne Duncan appointed Dr. Bailey chair of the Committee on Measures of Student Success, which developed recommendations for community colleges to comply with completion rate disclosure requirements under the Higher Education Opportunity Act. Dr. Bailey and the CCRC won the Terry O’Banion Prize for Teaching and Learning at the annual conference for the League for Innovation in the Community College in 2013, and was also inducted as an American Educational Research Association (AERA) Fellow in the same year. He has been a member of the National Academy of Education since 2012.

Dr. Bailey’s papers have appeared in a variety of education, policy-oriented, and academic journals, and he has authored or co-authored several books on the employment and training of immigrants and the extent and effects of on-the-job training. He authored, with Shanna Smith Jaggars and Davis Jenkins, Redesigning America’s Community Colleges: A Clearer Path to Student Success, which was published by Harvard University Press in April 2015. Other books include Defending the Community College Equity Agenda (Johns Hopkins University Press, 2006), co-edited with Vanessa Morest; Working Knowledge: Work-Based Learning and Education Reform (Routledge, 2004), co-authored with Katherine Hughes and David Moore; Manufacturing Advantage (Cornell University Press, 2000), written with Eileen Appelbaum, Peter Berg, and Arne Kalleberg; and The Double Helix of Education and the Economy (Institute on Education and the Economy, 1992), co-authored with Sue Berryman. Dr. Bailey earned his Ph.D. in labor economics from the Massachusetts Institute of Technology.

Joanne Bashford, Ph.D. Dr. Bashford is president of Miami Dade College‘s (MDC) InterAmerican Campus. She began her community college career at MDC in 1992 and served the college in various roles, including assisting with strategic planning, institutional effectiveness, and institutional research. From 2012 to 2014, she served as associate vice president for developmental education and student success at Broward College. Dr. Bashford has also served as a consultant for the Achieving the Dream initiative since 2005 and consulted with numerous colleges on planning, assessment, and continuous improvement to improve student success. Dr. Bashford earned her Ph.D. in education from Florida International University, a master’s degree in education from the University of Pittsburgh, and a bachelor’s degree in psychology from the University of Florida.

Angela Boatman, Ed.D. Dr. Boatman is assistant professor of public policy and higher education in the Department of Leadership, Policy, and Organizations at Vanderbilt University. Her work focuses on the evaluation of college access and completion policies, particularly in the areas of postsecondary remediation, financial aid, community college student success, and postsecondary instruction and course delivery models. Past projects include an evaluation of the impact of innovations in the delivery of remedial courses, including the use of instructional technology, on collegiate student outcomes, as well as the effects of more traditional remedial and developmental courses for students further down the academic preparation distribution. Currently, she is examining the effects of peers in mainstreamed remedial and college-level math courses, as well as the effects of using learner-adaptive technologies in community college courses. Dr. Boatman earned her Ed.D. in higher education from Harvard University, an M.P.P in public policy and an M.A. in higher education from the University of Michigan, and a B.A. from the University of Minnesota.

John Squires. Mr. Squires is director of high school to college readiness at the Southern Regional Education Board (SREB). He works with states, colleges, and high schools to implement policies and programs to improve student preparedness for postsecondary studies. Prior to joining SREB, Mr. Squires served as chair of the math department at Chattanooga State Community College, where he worked with faculty to improve student learning and outcomes through course redesign. He also worked to expand dual enrollment opportunities with local high schools and he started the Tennessee SAILS initiative, which introduces developmental math in the senior year of high school. While at Chattanooga State, Mr. Squires was the co-principal investigator for the Do the Math grant as part of Next Generation Learning Challenges, and he served as a mentor for colleges in the Changing the Equation grant for the National Center of Academic Transformation. Before joining Chattanooga State, Mr. Squires served as chair of the math department at Cleveland State Community College, where he taught mathematics for 19 years.

Mr. Squires’ work at both Chattanooga State Community College and Cleveland State Community College has been nationally acclaimed. He received the 2014 Bellwether Legacy Award, given by the American Community College Trustees, and the 2009 Bellwether Award, given by the Community College Futures Assembly. Mr. Squires received the 2014 Faculty Excellence Award by the American Association of Community Colleges for his work in departmental redesign and expanding dual enrollment. His math program at Cleveland State has been featured in the Chronicle of Higher Education and was highlighted by President Obama in a 2010 speech on higher education. Mr. Squires was the 2010 Cross Scholar for the League of Innovation in the Community College and he authored the 13th Cross Paper, which focused on course redesign. Mr. Squires serves as a redesign scholar for the National Center for Academic Transformation. He has written articles for the American Association of Community Colleges, the Education Commission for the States, the League for Innovation in the Community College, Inside Higher Education, and the Huffington Post. Upon retiring from the Tennessee Board of Regents after a 25-year teaching career, Mr. Squires was named Professor Emeritus at Chattanooga State Community College. Mr. Squires has an M.S. in mathematics from the University of Tennessee, an M.A.T. from Drake University, and a B.S. in economics from Iowa State University.

Michael Weiss, Ph.D. Dr. Weiss is a Senior Associate at MDRC. Weiss’s work focuses on evaluating programs designed to improve community college students’ chances of achieving academic success. He is also deeply involved in methodological projects intended to improve the quality of random assignment evaluations. Dr. Weiss was recently the principal investigator on an IES-funded study of the long-term effects of two of MDRC’s Opening Doors programs (learning communities and a student success course). He is currently the co-principal investigator on an IES-funded random assignment evaluation of the effectiveness of approaches to accelerate community college students’ progress through developmental (remedial) math courses and principal investigator on an IES-funded random assignment evaluation of the City University of New York (CUNY) Start, a pre-matriculation program for students with significant developmental needs. He was the quantitative lead investigator on MDRC’s random assignment evaluation of CUNY’s Accelerated Study in Associate Programs (ASAP) and is the quantitative lead investigator on the ASAP replication project. In his methodological work, Dr. Weiss has authored peer-reviewed articles on teacher effects in experiments, on measuring school performance, and on a conceptual framework for understanding the sources of variation in program effects. He is currently the primary investigator on an IES research methods grant exploring issues of program-effect variation. He has been invited to give presentations at numerous conferences and organizations, including at IES and the U.S. Department of Labor. Dr. Weiss formerly worked as a data analyst at Educational Testing Services. He earned his Ph.D. from the University of Pennsylvania’s Graduate School of Education.

Staff Leadership Team

William Doyle, Ph.D. Dr. Doyle is associate professor of higher education and public policy coordinator, higher education leadership program in the Department of Leadership, Policy, and Organizations at Vanderbilt University. Dr. Doyle is the senior content expert for the Evidence Review and Support contract with the WWC in postsecondary education. Professor Doyle’s work investigates the antecedents and outcomes of higher education policy at the state and federal level, and uses theoretical and methodological insights from political science to better understand both higher education politics and policy. Dr. Doyle also serves as an associate editor of Research in Higher Education and is on the editorial board of the Journal of Higher Education and Educational Evaluation and Policy Analysis. From 2004 to 2013, Dr. Doyle served as an associate editor of Change magazine, where he wrote the “Playing the Numbers” column, which explained quantitative research to a broader policy and practitioner audience. He earned his Ph.D. in higher education from Stanford University. Prior to Vanderbilt, he held the position of senior policy analyst at the National Center for Public Policy and Higher Education. Dr. Doyle facilitated deliberations by the expert panel and was the lead content expert for the practice guide.

Robin LaSota, Ph.D. Dr. LaSota is a senior research scientist at Development Services Group, Inc. She is a mixed-methods researcher, program evaluator, and technical assistance provider in the field of K–12 and postsecondary education, human services, and youth development. Dr. LaSota coordinates the development of products for the WWC postsecondary education contract and conducts reviews as part of the Reviews, Reporting, Dissemination, and Development (R2D2) contract. Her research on state, college, and student factors associated with community college students’ upward transfer has been published in Research in Higher Education. She conducted post-doctoral research at the University of Illinois Urbana-Champaign at the Office of Community College Research and Leadership. Dr. LaSota has managed professional development programs for urban school leaders and school teams, provided research-based materials to educators, conducted program evaluation and policy analysis, and designed educational programs as part of successfully obtaining grant funding. She received her Ph.D. in education leadership and policy studies from the University of Washington as part of an IES-funded pre-doctoral program called Collaborative Researchers for Education Sciences Training. She led the development of content for the practice guide through collaboration with staff writers, with guidance from the expert panel and leadership team.

Jeffrey C. Valentine, Ph.D. Dr. Valentine is a professor in and coordinator of the Educational Psychology, Measurement, and Evaluation program in the College of Education and Human Development at the University of Louisville. As an internationally known expert, Dr. Valentine has written more than four dozen works that use, explain, or seek to improve the methods of systematic reviewing and meta-analysis. He co-edited (with Harris Cooper and Larry Hedges) the Handbook of Research Synthesis and Meta-Analysis (2nd ed.), is an associate editor for the peer-reviewed journal Research Synthesis Methods, is a statistical editor for the Developmental, Psychosocial, and Learning Problems Group for the Cochrane Collaboration, and is the co-chair for the Training Group for the Campbell Collaboration. Dr. Valentine served as methodological expert for this practice guide, provided input on the practices and recommendations, and oversaw the quality assurance process for the practice guide.

Martha Yeide, Ph.D. Dr. Yeide currently serves as Development Services Group’s project director for the R2D2 contract for IES and the evidence review and support for the WWC postsecondary education. Dr. Yeide also serves as associate project director for the Substance Abuse and Mental Health Services Administration’s (SAMHSA) National Registry of Evidence-based Programs and Practices (NREPP), where she was part of the team that developed new standards and procedures for the review process. She worked with teams refining standards for review and developing scoring instruments for the assessment process for National Institute of Justice’s CrimeSolutions.gov. She is a certified reviewer for the WWC as well as Office of Juvenile Justice and Delinquency Prevention’s (OJJDP) Model Programs Guide, CrimeSolutions.gov, and NREPP. Dr. Yeide previously served as deputy project director for OJJDP’s National Forum on Youth Violence Prevention Training and Technical Assistance Program, which supports the work of 10 cities to effectively address the pressing issues of gang and youth violence. Dr. Yeide was responsible for the oversight and execution of the development of this practice guide.

Staff Writers

Joshua R. Polanin, Ph.D. Dr. Polanin is a senior research scientist at DSG. He is an expert in the application and use of quantitative methodology in education, behavioral health, and social sciences. He currently serves as methodological consultant and research analyst on IES’s WWC postsecondary reviews. Dr. Polanin maintains methodological standards in the review process, trains WWC-certified reviewers, and provides feedback on developing WWC products. In addition, he serves as the methodological expert for SAMHSA’s NREPP. Dr. Polanin received a Campbell Collaboration Systematic Review Grant and led a team of researchers to complete a review on teen dating violence prevention programs. Dr. Polanin received his Ph.D. in education quantitative methodology from Loyola University Chicago and completed a 2-year IES postdoctoral fellowship focused on social policy research and methodology at Vanderbilt University’s Peabody Research Institute.

Elizabeth Spinney. Ms. Spinney is a project director and research analyst at DSG. She serves as a research analyst for the U.S. Department of Education’s WWC. She is also a WWC-certified reviewer for group design studies. Ms. Spinney currently serves as project director OJJDP’s Technical Assistance to End Racial and Ethnic Disparities in the Juvenile Justice System program. As project director, she coordinates training and technical assistance for states on how to reduce disparities at the local level and works with partner organizations to provide recommendations to OJJDP on how to address racial disparities at the national level. Ms. Spinney received a master’s degree in public policy from the John F. Kennedy School of Government at Harvard University.

Wesley Wilson, Ed.D. Dr. Wilson is a research scientist at DSG. He reviews studies and prepares written material for DSG’s evidence review and support for the WWC contract in postsecondary education, as well as the Education Department’s R2D2 project. Dr. Wilson develops content for webinars and other communications for the WWC to connect research to both evaluation and practice. He has more than two decades of experience as a practitioner at community colleges and higher education institutions. Dr. Wilson served as the academic lead faculty for the Community College Leadership Initiative for the state of New Jersey and has been recognized by Austin Community College with the 2012 Faculty Excellence Award. Dr. Wilson received his Ed.D. from the University of Florida, where his work focused on community college leadership.

Sarah H. Young, Ph.D. Dr. Young is a research analyst at DSG. She has worked on a variety of topics, including the use of portfolios in dental education, a meta-analysis of the social interdependence literature, and investigating the mathematical education of future elementary teachers. Her research has employed diverse methods, including qualitative case study, survey research, qualitative observation and interview, meta-analysis, and hierarchical linear modeling. Dr. Young has experience as a university educator at Michigan State University. In addition, as a research analyst and certified reviewer for WWC, Dr. Young helps manage the quality control process so that all products meet WWC standards. She also manages and leads abstract and full-text screening of studies found in systematic searches, retrieves studies and other resources using systematic searches, and constructs and collaborates in constructing search strategies for systematic searches. Dr. Young received her Ph.D. in educational psychology and educational technology from Michigan State University.

Appendix C – Disclosure of Potential Conflicts of Interest

Practice guide panels are composed of nationally recognized experts on the topics about which they are making recommendations. IES expects the experts to be involved professionally in a variety of matters that relate to their work as a panel. Panel members are asked to disclose these professional activities and institute deliberative processes that encourage critical examination of their views as they relate to the content of the practice guide. The potential influence of the panel members’ professional activities is further muted by the requirement that they ground their recommendations in evidence that is documented in the practice guide. In addition, before all practice guides are published, they undergo an independent external peer review focusing on whether the evidence related to the recommendations in the guide has been presented appropriately.

The professional activities reported by each panel member that appear to be most closely associated with the panel recommendations are noted below.

Tom Bailey, Ph.D. He is the director of the Community College Research Center (CCRC) at Teachers College, Columbia University, which produced many of the studies reviewed and used as evidence for this practice guide. For example, CCRC published many of the studies reviewed and used as evidence for Recommendation 4 on compressing or mainstreaming developmental education. CCRC also reported two studies used as evidence for Recommendation 1 on using multiple measures to assess and place students into courses.

Joanne Bashford, Ph.D. She does not have research reviewed for the recommendations presented in this practice guide, or any notable conflict of interest with the panel’s recommendations.

Angela Boatman, Ph.D.Her research on the effectiveness of accelerated instructional approaches in developmental education has been reviewed for this practice guide.

John Squires. He does not have research reviewed for the recommendations presented in this practice guide, or any notable conflict of interest with the panel’s recommendations.

Michael Weiss, Ph.D. He works at MDRC, which produced many of the independent evaluations of interventions reviewed for this practice guide. Two of the practice guide’s recommendations rely in part on evidence from evaluations that Dr. Weiss was integrally involved in, Recommendation 2 on enhanced advising and Recommendation 6 on comprehensive, integrated, and long-lasting supports. Additionally, Dr. Weiss’ wife, Reshma Patel, played a substantial role in much of the evidence reviewed for Recommendation 3 on performance-based monetary incentives.

Appendix D

Appendix D. Appendix Table 1.1

Recommendation 1. Use Multiple Measures to Assess Postsecondary Readiness and Place Students.

1 This study did not meet standards because attrition exceeded WWC thresholds, and the WWC was unable to assess the comparability of the analytic sample at baseline.

2 This means that assignment to treatment condition placed the student in a different course level, compared to the traditional placement measure (i.e., ACCUPLACER score).

3 While this group was not assigned by multiple measures to courses, this intervention group is included as an intervention with an alternate placement method.

4 * indicates a statistically significant result (p < .05). The author reported that students placed into college-level courses by both high school preparation and test scores received higher grades than the students placed into college-level courses by test score alone. However, college-level students placed by high school preparation alone performed equally as well as students placed via the placement test on the measures of academic success in the study. Among students placed at the developmental level, there were no differences in academic success outcomes for the multiple or alternate measures interventions, compared to students placed by test score alone. The author found that single-measure methods may over-select students into developmental education (including students who could have succeeded in college-level courses).

5 This study did not meet WWC standards because the WWC was unable to assess baseline equivalence on socioeconomic status and academic achievement. The authors do use baseline achievement in their estimation models, but do not have a measure of socioeconomic status.

6 This study did not present information in a way that allows standardized effect sizes to be reported. According to the authors, none of these effects were statistically significant, indicating that “boosted” students did as well as students placed traditionally.

Appendix D. Appendix Table 1.2

Recommendation 1. Use Multiple Measures to Assess Postsecondary Readiness and Place Students.

7 No studies for this recommended practice met WWC evidence standards. Due to the rigor of these correlational studies, the expert panel decided to include them as evidence. However, the WWC did not conduct a thorough search for studies that examine relationships between alternative measures of college readiness and college success. The studies discussed here were uncovered during the search for interventions that assessed the effect of using multiple or alternative measures. As such, these results should be considered suggestive.

8 * indicates a statistically significant result (p < .05)

Appendix D. Appendix Table 2

Recommendation 2. Require or Incentivize Students to Attend Enhanced Advising.

9 MDRC defined “academic difficulty” as earning no more than 75% of credits attempted. Other sample criteria included students being aged 18–34 years old; having family income below 250% of poverty level; being beginning or continuing students who completed fewer than 13 credits and had experienced academic difficulties (indicated by not passing courses or withdrawing from courses), having a high school diploma or GED, and did not have an associate’s degree.

10 * indicates a statistically significant result (p < .05)

Appendix D. Appendix Table 3.1

Recommendation 3. Offer Students Performance-Based Monetary Incentives.

11 * indicates a statistically significant result (p < .05)

Recommendation 3. Offer Students Performance-Based Monetary Incentives.

Appendix D. Appendix Table 3.2

12 * indicates a statistically significant result (p < .05)

Appendix D. Appendix Table 4

Recommendation 4. Compress or Mainstream Developmental Education with Course Redesign.

13 * indicates a statistically significant result (p < .05)

14 This study did not meet WWC standards because the authors used imputation for some covariates and outcomes; the WWC currently does not allow imputation for covariates.

15 nr indicates not reported. This study did not present information in a way that allows standardized effect sizes to be reported. See Cho et al. (2012), Table 3, p. 10.

16 This study did not meet WWC standards because an acceptable pre-intervention measure of academic achievement was not available. The authors did control for measures of college achievement and student socioeconomic status.

17 nr indicates not reported. This study did not present information in a way that allows standardized effect sizes to be reported. See Edgecombe et al. (2014), Table A.2, p. 30.

18 These studies did not meet WWC standards because they did not use an acceptable pre-intervention measure of academic achievement in the estimation models.

19 * indicates a statistically significant result (p < .05)

20 This study did not present enough information for the computation of a standardized effect size for this outcome. See Zeidenberg et al. (2010), Table 6, p. 14.

Appendix D. Appendix Table 5

Recommendation 5. Teach Students How to Be Self-Regulated Learners.

21 * indicates a statistically significant result (p < .05)

Appendix D. Appendix Table 6

Recommendation 6. Implement Comprehensive, Integrated, and Long-Lasting Support Programs.

22 * indicates a statistically significant result (p < .05)

References^23

Al-Harthy, I., Was, C. A., & Issacson, R. M. (2010). Goals, efficacy, and metacognitive self-regulation.

A path analysis. International Journal of Education, 2(1), 1–20.

American Association of Community Colleges. (2015). 2015 community college fast facts. Washington,

DC: Author. Retrieved from http://www.aacc.nche.edu/AboutCC/Documents/FactSheet2015.pdf

American Educational Research Association, American Psychological Association, & National

Council on Measurement in Education. (1999). Standards for educational and psychological

testing. Washington, DC: American Educational Research Association Publications.

Attewell, P., Lavin, D., Thurston, D., & Tania, L. (2006, September/October). New evidence on college

remediation. The Journal of Higher Education, 77(5), 886–924.

Bailey, T., Jaggars, S. S., & Scott-Clayton, J. (2013). Characterizing the effectiveness of

developmental education: A response to recent criticism. New York, NY: Community College Research Center, Teachers College, Columbia University. Retrieved from http://files.eric.ed.gov/fulltext/ED542142.pdf

Bailey, T. (2009, Spring). Challenge and opportunity: Rethinking the role and function of developmental

education in community college. New Directions for Community Colleges, 145, 11–31.

Bain, K. (2004). What the best college teachers do. Cambridge, MA: Harvard University Press.

Barnett, E. A., Fay, M. P., Bork, R. H., & Weiss, M. J. (2013, May). Reshaping the college transition:

States that offer early college readiness assessments and transition curricula. New York, NY:

Community College Research Center, Teachers College, Columbia University. Retrieved from http://ccrc.tc.columbia.edu/media/k2/attachments/reshaping-the-college-transition-state-scan.pdf

Barrow, L., Richburg-Hayes, L., Rouse, C. E., & Brock, T. (2012).Paying for performance:

the education impacts of a community college scholarship program for low-income adults (Working Paper Series: WP–09–13, 2009). Chicago, IL: Federal Reserve Bank of Chicago.

Belfield, C., & Crosta, P. (2012).Predicting success in college: The importance of placement

tests and high school transcripts (CCRC Working Paper No. 42). New York, NY: Community College Research Center, Teachers College, Columbia University.

Bettinger, E. P., Boatman, A., & Long, B. T. (2013). Student supports: Developmental education and

other academic programs. The Future of Children, 23(1), 93–115.

Bettinger, E. P., & Long, B. T. (2009). Addressing the needs of underprepared students in higher

education: Does college remediation work? Journal of Human Resources, 44(3), 736–771.

Binder, M., Krause, K., Miller, C., & Cerna, O. (2015, June). Providing incentives for timely

progress toward earning a college progress: Results from a performance-based scholarship experiment. New York, NY: MDRC.

Boatman, A., & Long, B. T. (2010). Does remediation work for all students? How the effects of

postsecondary remedial and developmental courses vary by level of academic preparation (NCPR Working Paper). New York, NY: National Center for Postsecondary Research.

Boylan, H. R. (2002). What works: Research-based best practices in developmental education.

Boone, NC: Continuous Quality Improvement Network with the National Center for Developmental Education, Appalachian State University.

Bracco, K. R., Austin, K., Bugler, D. & Finkelstein, N. (2015). Reforming developmental education to

better support students’ postsecondary success in the Common Core era. Policy Brief. San Francisco, CA: WestEd. Retrieved from https://www.luminafoundation.org/files/resources/reforming-dev-ed.pdf

Bracco, K. R., Dadgar, M., Austin, K., Klarin, B., Broek, M., Finkelstein, N., … Bugler, D. (2014).

Exploring the use of multiple measures for placement into college-level courses: Seeking alternatives or improvements to the use of a single standardized test [Core to College Evaluation]. San Francisco, CA: Retrieved from https://www.wested.org/wp-content/files_mf/1397164696product55812B.pdf

Brock, T., & Richburg-Hayes, L. (2006). Paying for persistence. Early results of a

Louisiana scholarship program for low-income parents attending community college. New York, NY: MDRC. Retrieved from http://www.mdrc.org/sites/default/files/full_472.pdf

Brophy, J. E. (2004). Stimulating students’ motivation to learn. Motivating students to learn (2nd ed.,

pp. 249–306). Mahwah, NJ: Lawrence Erlbaum.

Burdman, P. (2012). Where to begin? The evolving role of placement exams for students starting

college. Cambridge, MA: Jobs for the Future and Achieving the Dream.

Cal-PASS Plus, & RP Group. (n.d.). Multiple measures assessment project – Data match guide to Cal-

PASS Plus. Retrieved from the RP Group web site: http://rpgroup.org/system/files/DataMatch%20Guide_10.26.15.pdf

Cal-PASS Plus, & RP Group. (n.d). Multiple measures assessment project – Pilot college – Developing

a research plan. Retrieved from the RP Group web site: http://rpgroup.org/system/files/MMAP-Sample%20Research%20Plan_10.31.15.pdf

Center for Education Policy Research, Harvard University. (2015). Research project overview: The Study

of pre-college math remediation programs in Tennessee. Cambridge, MA: Author. Retrieved from

http://cepr.harvard.edu/publications/research-project-overview-study-pre-college-math-remediation-programs-tennessee

Cha, P., & Patel, R. (2010). Rewarding progress, reducing debt: Early results from Ohio’s performance-

based scholarship demonstration for low-income parents. New York, NY: MDRC.

Cho, S., Kopko, E., Jenkins, D., & Jaggars, S. S. (2012, December). New evidence of success for

community college remedial English students: Tracking the outcomes of students in the Accelerated Learning Program (ALP). New York, NY: Community College Research Center, Teachers College, Columbia University. Retrieved from http://ccrc.tc.columbia.edu/publications/ccbc-alp-student-outcomes-follow-up.html

City University of New York. (2015). Inside ASAP (Accelerated Study in Associate Programs): A

resource guide on program structure, components, and management. New York, NY: Author. Retrieved from http://www1.cuny.edu/sites/asap/wp-content/uploads/sites/8/2015/05/Resource-Guide.pdf

Coalition on the Academic Workforce. (2012). A portrait of part-time faculty members. Retrieved from

http://www.academicworkforce.org/CAW_portrait_2012.pdf

Complete College America. (2012). Remediation: Higher education’s bridge to nowhere. Retrieved

from http://www.completecollege.org/docs/CCA-Remediation-final.pdf

Cousert, D. (1999). The effects of a mentoring intervention program on retention of students in a

community college (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 304550777)

Dinsmore, D. L., Alexander, P. A., & Loughlin, S. M. (2008). Focusing the conceptual lens on

metacognition, self-regulation, and self-regulated learning. Educational Psychology Review, 20, 391–409.

Duffy, M., Schott, A., Beaver, J. K., & Park, E. (2014). Tracing the development of multiple measures

for college placement across states and systems: Analysis of three state systems: Phase I report.

Philadelphia, PA: Research for Action. Retrieved from http://www.researchforaction.org/wp-content/uploads/2015/02/RFA-Gates-Multiple-Measures-Phase-1-Report-March-2014.pdf

Edgecombe, N. (2011, February). Accelerating the academic achievement of students referred to

developmental education. New York, NY: Community College Research Center, Teachers College,

Columbia University. Retrieved from http://academiccommons.columbia.edu/catalog/ac:146646

Edgecombe, N., Cormier, M. S., Bickerstaff, S., & Barragan, M. (2013a). Strengthening developmental

education reforms: Evidence on implementation efforts from the Scaling Innovation project (CCRC

Working Paper No. 61). New York, NY: Community College Research Center, Teachers College,

Columbia University. Retrieved from http://ccrc.tc.columbia.edu/publications/strengthening-

developmental-education-reforms.html

Edgecombe, N., Jaggars, S. S., Baker, E. D., & Bailey, T. (2013b). Acceleration through a holistic support

model: An implementation and outcomes analysis of FastStart@CCD. New York, NY: Community

College Research Center, Teachers College, Columbia University. Retrieved from http://ccrc.

tc.columbia.edu/publications/acceleration-through-holistic-support-model.html

Edgecombe, N., Jaggars, S. S., Xu, D., & Barragan, M. (2014, May). Accelerating the integrated

instruction of developmental reading and writing at Chabot College. New York, NY: Community College Research Center, Teachers College, Columbia University. Retrieved from http://ccrc.tc.columbia.edu/publications/accelerating-integrated-instruction-at-chabot.html

Education Advisory Board and Complete College America. (2012). Building guided pathways to success:

A survey of innovative approaches for directing student choice and reducing time to graduation.

Washington, DC: Author. Retrieved from http://doingwhatmatters.cccco.edu/portals/6/docs/

Building%20Guided%20Pathways%20to%20Success%20-%20EAB%20Report%20FINAL.pdf

Escobedo, G. (2007). A retention/persistence intervention model: Improving success across cultures.

Journal of Developmental Education, 31(1), 12–14, 16–17, 37.

Fay, M. P., Bickerstaff, S. E., & Hodara, M. (2013, December). Why students do not prepare for math

placement exams: Student perspectives (CCRC Research Brief No. 57). New York, NY: Community

College Research Center, Teachers College, Columbia University. Retrieved from http://ccrc.

tc.columbia.edu/media/k2/attachments/why-students-do-not-prepare.pdf

Finkelman, J., Hudesman, J., Flugman, B., & Crosby, S. (2014). An enhanced formative assessment

and self-regulated learning program: From the classroom to the workplace. Austin Journal of Psychiatry and Behavioral Science, 1(1), 1002–9.

Fong, K., & Visher, M. G. (2013). Fast forward: A case study of two community college programs

designed to accelerate students through developmental math. New York, NY: MDRC. Retrieved from http://www.mdrc.org/sites/default/files/fast_forward_es.pdf

Ford, S. S. (2007). The essential steps for developing the content of an effective advisor training and

development program. Manhattan, KS: The National Academic Advising Association. Retrieved from https://www.nacada.ksu.edu/Resources/Clearinghouse/View-Articles/Advisor-Training-Steps.aspx

Gourgey, A. F. (1998). Metacognition in basic skills instruction. Instructional Science, 26(1), 81–96.

Green, D. W. (2007). Adjunct faculty and the continuing quest for quality. New Directions for

Community Colleges, 140, 29–39.

Grubb, W. N. (2001). “Getting into the world”: Guidance and counseling in community colleges. New

York, NY: Community College Research Center, Teachers College, Columbia University. Retrieved from http://counselors.cccco.edu/Portals/2/assets/Documents/332_23CCRC Grubb Counseling 2001.pdf

Hodara, M., & Jaggars, S. S. (2014). An examination of the impact of accelerating community college

students’ progression through developmental education. The Journal of Higher Education, 85(2),

246–276.

Hodara, M., Jaggars, S. S., & Karp, M. M. (2012). Improving developmental education assessment and

placement: Lessons from community colleges across the country (CCRC Working Paper No. 51). New York, NY: Community College Research Center, Teachers College, Columbia University. Retrieved from http://ccrc.tc.columbia.edu/publications/developmental-education-assessment-placement-scan.html

Howell, J. S., Kurlaender, M., & Grodsky, E. (2010). Postsecondary preparation and remediation:

Examining the effect of the early assessment program at California State University. Journal of Policy Analysis & Management, 29(4), 726–748.

Hudesman, J., Crosby, S., Flugman, B., Issac, S., Everson, H., & Clay, D. B. (2013a). Using formative

assessment and metacognition to improve student achievement. Journal of Developmental Education, 37(1), 3–13.

Hudesman, J., Crosby, S., Ziehmke, N., Everson, H., Issac, S., Flugman, B., … Moylan, A. (2014). Using

formative assessment and self-regulated learning to help developmental math students achieve: A multi-campus program. Journal on Excellence in College Teaching, 25(2), 107–130.

Hudesman, J., Millet, N., Niezgoda, G., Han, H., & Flugman, B. (2013b). The use of self-regulated

learning, formative assessment, and mastery learning to assist students enrolled in developmental

mathematics: A demonstration project. International Journal of Research and Review, 10, 1–17.

Inside Track. (2015). National study of non-first-time students shows full-time enrollment may not be

appropriate for all. Retrieved from http://www.insidetrack.com/2015/01/20/national-study-of-non-first-time-students-shows-full-time-enrollment-may-not-be-appropriate-for-all/

Jackson, C. C. (2014, October 28). Enhanced advising at the University of Akron [Video file]. Retrieved

from https://www.youtube.com/watch?v=a1mZz2mz8ns

Jaggars, S. S., Hodara, M., Cho, S., & Xu, D. (2015). Three accelerated developmental programs:

Features, student outcomes, and implications. Community College Review, 43(1), 3–26.

Jenkins, D., Speroni, C., Belfield, C., Jaggars, S. S., & Edgecombe, N. (2010). A model for accelerating

academic success of community college remedial English students: Is the Accelerated Learning

Program (ALP) effective and affordable? (CCRC Working Paper No. 21). New York, NY: Community

College Research Center, Teachers College, Columbia University.

Jenkins, D., Zeidenberg, M., & Kienzl, G. S. (2009, May). Educational outcomes of I-BEST, Washington

State Community and Technical College System’s Integrated Basic Education and Skills Training

Program: Findings from a multivariate analysis (CCRC Working Paper No. 16). New York, NY:

Community College Research Center, Teachers College, Columbia University. Retrieved from http://ccrc.tc.columbia.edu/publications/i-best-multivariate-analysis.html

Kalamkarian, H. S., & Karp, M. M. (2015, August). Student attitudes toward technology-mediated

advising systems (CCRC Working Paper No. 82). New York, NY: Community College Research Center, Teachers College, Columbia University. Retrieved from http://ccrc.tc.columbia.edu/media/k2/attachments/student-attitudes-toward-technology-mediated-advising-systems.pdf

Kaplan, A. (2008). Clarifying metacognition, self-regulation, and self-regulated learning: What’s the

purpose? Educational Psychology Review, 20, 477-484. doi:10.1007/s10648-008-9087-2

Katiraie, F. (2007). Effectiveness of fast track and advanced fast track programs at Montgomery

College and feasibility of expansion to other colleges and universities (Doctoral dissertation). ProQuest Dissertations and Theses database. (UMI No. 3305792)

Klempin, S. (2014, September). Redefining full-time in college: Evidence on 15-credit strategies. New

York, NY: Community College Research Center, Teachers College, Columbia University. Retrieved

from http://ccrc.tc.columbia.edu/publications/redefining-full-time-in-college.html

Kosiewicz, H., Ngo, F., & Fong, K. (2016). Alternative models to deliver developmental math: Issues

of use and student access. Community College Review, 44(3), 205–231.

Kwon, W. (2014). Essays on education: From Indonesia to Los Angeles, CA (Doctoral dissertation).

Retrieved from ProQuest Dissertations and Theses database. (UMI No. 3644661)

Ley, K., & Young, D. B. (2001). Instructional principles for self regulation. Educational Technology

Research and Development, 49(2), 93–103.

Linder, K. E. (2012). Creating space for adjunct faculty: The multiple roles of centers for teaching and

learning. Journal on Centers for Teaching & Learning, 4, 33–59.

Long, B. T., & Boatman, A. (2013). The role of remedial and developmental courses in access and

persistence. In A. Jones and L. Perna (Eds.), The state of college access and completion: Improving college success for students from underrepresented groups. London, England: Routledge International Studies in Higher Education.

Lynch-Holmes, K. B., Troy, A. B., & Ramos, I. (n.d.). Early alert & intervention: Top practices for

retention. Retrieved from https://issuu.com/connectedu/docs/early_alert_white_paper

Marwick, J. D. (2002a). Charting a path to success: How alternative methods of mathematics placement

impact the academic success of community college students (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 3070381)

Marwick, J. (2002b). Alternative methods of mathematics placement. The Community College

Enterprise, 8(2), 41–50.

Marwick, J. D. (2004). Charting a path to success: The association between institutional placement

policies and the academic success of Latino students. Community College Journal of Research &

Practice, 28(3), 263–280.

Mayer, A. K., Patel, R., & Gutierrez, M. (2015a, April). Four-year effects on degree receipt and

employment outcomes from a Performance-Based Scholarship Program in Ohio. New York, NY: MDRC.

Mayer, A. K., Patel, R., Rudd, T., Ratledge, A., & Blake, S. (2015b, November). Designing scholarships

to improve college success: Final report on the performance-based scholarship demonstration. New York, NY: MDRC.

McClellan, J. L. (2007). Content components for advisor training: Revisited. Manhattan, KS:

National Academic Advising Association. Retrieved from http://www.nacada.ksu.edu/Resources/

Clearinghouse/View-Articles/Advisor-Training-Components.aspx

MDRC. (2015). Frequently asked questions about CUNY’s Accelerated Study in Associate Programs

(ASAP). Retrieved from http://www.mdrc.org/sites/default/files/FAQs_ASAP.pdf

Melguizo, T., Kosiewicz, H., Prather, G., & Bos, J. (2014). How are community college students

assessed and placed in developmental math? Grounding our understanding in reality. Journal of

Higher Education, 85(5), 691–722.

Miller, C., Binder, M., Harris, V., & Krause, K. (2011). Staying on track: Early Findings from a

Performance-Based Scholarship Program at the University of New Mexico. New York, NY: MDRC.

National Center for Education Statistics. (2013). First-year undergraduate remedial coursetaking:

1999-2000, 2003-04, 2007-08 – Statistics in Brief. (NCES 2013-013). Washington, DC: U.S.

Department of Education, National Center for Education Statistics. Retrieved from http://nces.ed.gov/pubs2013/2013013.pdf

Ngo, F., & Kwon, W. W. (2014). Using multiple measures to make math placement decisions:

Implications for access and success in community colleges. Research in Higher Education, 56(5), 442–470.

Ngo, F., Kwon, W., Melguizo, T., Bos, H., & Prather, G. (2013). Course placement in developmental

mathematics: Do multiple measures work? (Policy Brief 4). Los Angeles, CA: University of Southern

California.

Nodine, T., Dadgar, M., Venezia, A., & Bracco, K. R. (2013). Acceleration in developmental education.

San Francisco, CA: WestEd. Retrieved from http://www.wested.org/online_pubs/resource1281.pdf

Patel, R., Richburg-Hayes, L., de la Campa, E., & Rudd, T. (2013). Performance-based scholarships:

What have we learned? Interim findings from the PBS demonstration. Policy brief. New York, NY: MDRC.

Patel, R., & Rudd, T. (2012). Can scholarships alone help students succeed? Lessons from two New

York City community colleges. New York, NY: MDRC.

Patel, R., & Valenzuela, I. (2013). Moving forward: Early findings from the performance-based

scholarship demonstration in Arizona. New York, NY: MDRC.

Radford, A., Pearson, J., Ho, P., Chambers, E., & Ferlazzo, E. (2012). Remedial coursework in

postsecondary education: The students, their outcomes, and strategies for improvement. Washington, DC: MPR Associates, Inc.

Research for Action. (2015). Development and implementation of multiple measures for college

placement across states and systems: Phase I and Phase 2 report summary. Philadelphia, PA: Author.

Richburg-Hayes, L., Brock, T., LeBlanc, A., Paxson, C., Rouse, C. E., & Barrow, L. (2009). Rewarding

persistence: Effects of a performance-based scholarship program for low-income parents. New York, NY: MDRC.

Richburg-Hayes, L., Sommo, C., & Welbeck, R. (2011). Promoting full-time attendance among adults

in community college: Early impacts from the performance-based scholarship demonstration in New York. New York, NY: MDRC.

Robbins, R. (2013). Implications of advising load. In Carlstrom, A., 2011 national survey of academic

advising (Monograph No. 25). Manhattan, KS: National Academic Advising Association. Retrieved from http://www.nacada.ksu.edu/tabid/3318/articleType/ArticleView/articleId/94/article.aspx

RP Group, & Cal-PASS Plus. (n.d.). Multiple measures assessment project pilot college – Getting started

guide. Sacramento, CA: Author. Retrieved from: http://rpgroup.org/projects/multiple-measures-

assessment-project

Schmitt, N., Billington, A., Keeney, J., Reeder, M., Plesac, T. J., Sinha, R., & Zorzie, M. (2011).

Development and validation of measures of noncognitive college student potential. Washington, DC: The College Board.

Scott-Clayton, J. (2012). Do high-stakes placement exams predict college success? (CCRC Working

Paper No. 41). New York, NY: Community College Research Center, Teachers College, Columbia University.

Scrivener, S., & Au, J. (2007) Enhancing student services at Lorain County Community College: Early

results from the Opening Doors demonstration in Ohio. New York, NY: MDRC. Retrieved from http://files.eric.ed.gov/fulltext/ED496600.pdf

Scrivener, S., & Coghlan, E. (2011). Opening doors to student success: A synthesis of findings

from an evaluation at six community colleges. Policy brief. New York, NY: MDRC.

Scrivener, S., & Weiss, M. J. (2013). More graduates: Two-year results from an evaluation of

Accelerated Study in Associate Programs (ASAP) for developmental education students. Policy brief. New York, NY: MDRC.

Scrivener, S., Weiss, M. J., Ratledge, A., Rudd, T., Sommo, C., & Fresques, H. (2015). Doubling

graduation rates: Three-year effects of CUNY’s Accelerated Study in Associate Programs (ASAP) for developmental education students. New York, NY: MDRC.

Scrivener, S., Weiss, M. J., & Sommo, C. (2012). What can a multifaceted program do for community

college students? Early results from an evaluation of Accelerated Study in Associate Programs (ASAP) for developmental education students. New York, NY: MDRC.`

Scrivener, S., & Weiss, M. J., & Teres, J. J. (2009). More guidance, better results? Three-year effects

of an enhanced student services program at two community colleges. New York, NY: MDRC.

Scrivener, S., & Pih, M. (2007). Enhancing student services at Owens Community College: Early results

from the Opening Doors Demonstration in Ohio. New York, NY: MDRC.

Self, C. (2008). Advising delivery: Professional advisors, counselors, and other staff. In V. N. Gordon,

W. R. Habley, T. J. Grites, & Associates (Eds.), Academic advising: A comprehensive handbook (pp.

142–56). San Francisco, CA: Jossey-Bass.

Shults, C. (2000). Institutional policies and practices in developmental education: A national study of

community colleges. Washington, DC: American Association of Community Colleges.

Sommo, C., Boynton, M., Collado, H., Diamond, J., Gardenhire, A, Ratledge, A., … & Weiss, M. J. (2014).

Mapping success: Performance-based scholarships, student services, and developmental math at

Hillsborough Community College. New York, NY: MDRC.

Sperling, R. A., Howard, B. C., Staley, R. & DuBois, N. (2004). Metacognition and self-regulated learning

constructs. Educational Research and Evaluation, 10(2), 117–139.

Tierney, W. G., & Garcia, L. D. (2011). Remediation in higher education: The role of information.

American Behavioral Scientist, 55(2), 102–120. Retrieved from http://abs.sagepub.com/content/55/2/102

Title 5 §55502, California Code of Regulations.

Two-Year College English Association. (2014). TYCA white paper on developmental education reforms.

Urbana, IL: National Council of Teachers of English.

Visher, M. G., Butcher, K. F., & Cerna, O. S. (2010). Guiding developmental math students to campus

services: An impact evaluation of the Beacon Program at South Texas College. New York, NY: MDRC.

Weinstein, C. E., Meyer, D. K., Husman, J., Van Mater Stone, G., & McKeachie, W. J. (2006). Teaching

students how to become more strategic and self-regulated learners. In W. J. McKeachie, & M. Svinicki (Eds.), McKeachie’s Teaching Tips (pp. 300–317). Boston, MA: Houghton Mifflin.

Welbeck, R., Ware, M., Cerna, O., & Valenzuela, I. (2014). Paying it forward: A technical assistance

guide for developing and implementing performance-based scholarships. New York, NY: MDRC.

Westrick, P. A., & Allen, J. (2014). Validity evidence for ACT Compass tests. Iowa City, IA: ACT.

Willett, T. (2013). Student transcript-enhanced placement study (STEPS) technical report. Retrieved from

the RP Group web site: https://rpgroup.org/sites/default/files/STEPSTechnicalReport.pdf

Willett, T., Hayward, C., Nguyen, A., Newell, M., Bahr, P., Hetts, J., … Duran, D. (2015). Multiple measures

assessment project (MMAP) Spring 2015 technical report. Sacramento, CA: Research and Planning Group (RP Group) for California Community Colleges.

Yamada, H. (2014). Community College Pathways’ Program success: Assessing the first two years’

effectiveness of Statway. Stanford, CA: Carnegie Foundation for the Advancement of Teaching.

Young, D. B., & Ley, K. (2003). Self-regulation support offered by developmental educators. Journal

of Developmental Education, 27(2), 2–4, 6, 8, 10.

Zachry Rutschow, E., & Diamond, J. (2015, April). Laying the foundations: Early findings from the New

Mathways project. New York, NY: MDRC.

Zachry Rutschow, E., & Schneider, E. (2011, June 1). Unlocking the gate: What we know about improving

developmental education. New York, NY: MDRC.

Zeidenberg, M., Cho, S., & Jenkins, D. (2010). Washington State’s Integrated Basic Education and

Skills Training Program (I-BEST): New evidence of effectiveness (CCRC Working Paper No. 20). New York, NY: Community College Research Center, Teachers College, Columbia University.

Zimmerman, B. J., Moylan, A., Hudesman, J., White, N., & Flugman, B. (2011). Enhancing self-reflection

and mathematics achievement of at-risk urban technical college students. Psychological Test and

Measurement Modeling, 53(1), 141–160.

Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory into Practice,

41, 64–70.

23 Studies used to support a recommendation are indicated by bold text in the endnotes and references. For more information about these studies, see Appendix D.

Please note that there will still be some footnotes in the guide. Studies used as evidence in support of a recommendation are indicated by bold text in the endnotes and references pages.

Endnotes

1.Among all first-year undergraduates in the United States, about one-third enrolled in at least one developmental course, and among community college students, this proportion is higher (approximately 40 percent) (National Center for Education Statistics, 2013). See also: Bailey, 2009; Bettinger, Boatman, & Long, 2010; Bettinger & Long, 2009; Long & Boatman, 2013; Zachry Rutschow & Schneider, 2011.

2.Attewell, Lavin, Domina, & Levy, 2006; Table 2, p. 904. See also Radford et al., 2012.

3.Zachry Rutschow & Schneider, 2011.

4.Remedial education was a term initially used to describe the function that junior and community colleges performed by offering courses that were not at a college level. Students and faculty who taught pre-college level courses found “remedial education” to be stigmatizing. “Developmental education” became the more favored term to emphasize how faculty work to developmentally and incrementally help “underprepared college students attain their academic goals,” (Boylan, 2002, pg. 3). As the U.S. college completion agenda and college readiness standards have gained prominence, “college ready” is a term used to keep the focus on students succeeding in college and completing their degree. See also: Bailey, Jaggars, & Scott-Clayton, 2013.

5.National Center for Education Statistics, 2013.

6.Howell, Kurlaender, & Grodsky, 2010; Center for Education Policy Research, 2015.

7.Barnett, Fay, Bork, & Weiss, 2013; Tierney & Garcia, 2011.

8.Scott-Clayton, 2012; Belfield & Crosta, 2012.

9.Scrivener et al., 2015.

10. It is typical in WWC reviews of research for only a relatively small proportion of the eligible studies to be rated as meeting standards (overall, about 23% of eligible studies that the WWC has reviewed meet standards at some level). Consistent with previous WWC experiences, the most likely reason a study did not meet standards is that it had issues with group comparability at baseline (e.g., the groups were too far apart on important pretest measures; see the WWC Procedures and Standards Handbook, Version 3.0).

11. This includes randomized control trials (RCT) and quasi-experimental design studies (QED). Studies not contributing to levels of evidence include single-case designs (SCD) evaluated with WWC pilot SCD standards and regression discontinuity designs (RDD) evaluated with pilot RDD standards.

12. The research may include studies generally meeting WWC design standards and supporting the effectiveness of a program, practice, or approach with small sample sizes or other conditions of implementation or analysis that limit generalizability. The research may include studies that support the generality of a relation but do not meet WWC design standards; but, they have no major flaws related to internal validity other than lack of demonstrated equivalence at pretest for QEDs. QEDs without equivalence must include a pretest covariate as a statistical control for selection bias. These studies must be accompanied by at least one relevant study meeting WWC design standards.

13. American Educational Research Association, American Psychological Association, and National Council on Measurement in Education (1999).

14. E.g., Scott-Clayton, 2012.

15. Complete College America, 2012.

16. According to Ngo, Kwon, Melguizo, Bos, and Prather (2013), a lawsuit filed by the Mexican-American Legal Defense and Educational Fund provided the impetus for California’s policy change. In 1991, the Fund challenged the inequity of the Matriculation Act of 1986, which mandated placement testing as a part of enrollment and matriculation services in community colleges. Ngo et al. (2013) reported: “The lawsuit claimed that the California Community College Chancellor’s Office failed to monitor appropriate use of placement tests, resulting in large proportions of Latino students being placed in remediation. The lawsuit was settled outside of court, but Title 5 of the California Code of Regulations was revised in 1992 to mandate the use of multiple measures in placement decisions. The goal was to reduce the “disproportionate impact” of placement tests on different racial and ethnic groups” (p. 2, footnote).

17. E.g., Westrick & Allen, 2014.

18. Marwick, 2002a; Ngo & Kwon, 2014.

19. Marwick, 2002a.

20. Ngo & Kwon, 2014.

21. Belfield & Crosta, 2012; Scott-Clayton, 2012.

22. See “Using multiple measures” on the Bill and Melinda Gates Foundation website: www.gatesfoundation.org; Bracco et al., 2014; Burdman, 2012; Research for Action, 2015; Two-Year College English Association, 2014.

23. See: http://rpgroup.org/projects/multiple-measures-assessment-project. The RP Group is a membership organization of individual California community colleges and districts, focused on providing research, evaluation, professional development, and technical assistance to support evidence-based decision-making. Individuals, colleges, and organizations not in California’s community college system may join as associate members. See also the Education Commission of the States’ (ECS) national scan of states’ implementation of multiple measures for course placement: http://www.ecs.org/moving-from-single-to-multiple-measures-for-college-course-placement. ECS also refers to resources from Research for Action and WestEd, also cited in this guide.

24. Hodara, Jaggars, & Karp, 2012.

25. Duffy, Schott, Beaver, & Park, 2014.

26. “LBCC Promise Pathways,” 2014.

27. Willett, 2013. Willett (2013) investigated the relative effectiveness of various placement indicators on two main outcomes: 1) the students’ placement level of first English course, and first math course attempted; and 2) receiving a passing grade in the students’ first attempted English and math courses (English and math analyzed separately). The predictors that were used in regression models were: a) the most recent California Standards test in English or math; b) count of “A-G” or college preparatory courses completed (a set of seven categories of high school courses to be eligible for admission to the University of California or California State University); c) level of most recent high school course in English or math; d) grade point in most recent high school course in English or math; e) high school grade point average excluding English (or math); and f) level of first college course attempted in English (or math), used only for regressions predicting passing grades in those courses).

28. Willett et al., 2015.

29. Melguizo et al., 2014.

30. Adapted from: Cal-PASS Plus, & RP Group. (n.d). Multiple measures assessment project – Pilot college – Developing a research plan. Retrieved from the RP Group web site: http://rpgroup.org/system/files/MMAP-Sample%20Research%20Plan_10.31.15.pdf and Cal-PASS Plus, & RP Group. (n.d.). Multiple measures assessment project – Data match guide to Cal-PASS Plus. Retrieved from the RP Group web site: http://rpgroup.org/system/files/DataMatch%20Guide_10.26.15.pdf

31. Grubb, 2001.

32.  Scrivener & Weiss, 2009.

33.  Escobedo, 2007.

34. Jackson, 2014.

35. Robbins, 2013; Education Advisory Board, Complete College America, 2012.

36. Kalamkarian & Karp, 2015.

37. Scrivener & Weiss, 2009; Visher et al., 2010.

38.  Cousert, 1999; Visher et al., 2010.

39.  Scrivener & Weiss, 2009.

40.  Cousert, 1999.

41. Ford, 2007; McClellan, 2007.

42. Self, 2008; Kalamkarian & Karp, 2015.

43. Kalamkarian & Karp, 2015.

44. Scrivener & Weiss, 2009.

45.  Visher et al., 2010.

46.   Scrivener & Weiss, 2009.

47.  Scrivener & Weiss, 2009.

48. Lynch-Holmes, Troy, & Ramos, 2012.

49. Sample enhanced advising planning tool from Ivy Tech State College (p. 118) from: Cousert, D. (1999). The effects of a mentoring intervention program on retention of students in a community college (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 304550777). This study meets WWC standards without reservations.

50.  Visher, Butcher, & Cerna, 2010, p. 13.

51.  Scrivener & Weiss, 2009.

52.  Scrivener & Weiss, 2009, p. 27.

53.  Visher et al., 2010.

54.  Patel, Richburg-Hayes, de la Campa, & Rudd, 2013; Mayer, Patel, Rudd, Ratledge, & Blake, 2015.

55.  Brock & Richburg-Hayes, 2006; Cha & Patel, 2010; Patel & Rudd, 2012. The effectiveness of performance-based monetary incentives alone were evaluated in two Ohio colleges in Cha & Patel (2010) and in two New York colleges in Patel & Rudd (2012). Note that in the Louisiana Opening Doors program (Brock & Richburg-Hayes, 2006), counseling was primarily focused on monitoring students’ compliance with program requirements and not on providing academic advising or supports to work through personal challenges to their academic progress. The Louisiana Opening Doors intervention differed from the coupling of enhanced advising, with performance-based monetary incentives, in the VISTA program studied by Miller et al. (2011) and the Adelante program studied by Patel & Valenzuela (2013).

56.  Brock & Richburg-Hayes, 2006; Patel & Rudd, 2012.

57.  Cha & Patel, 2010.

58.  Brock & Richburg-Hayes, 2006; Patel & Rudd, 2012.

59.  Cha & Patel, 2010; Patel & Rudd, 2012.

60.  Miller, Binder, Harris, & Krause, 2011; Patel & Valenzuela, 2013; Sommo et al., 2014.

61.  Miller et al., 2011; Patel & Valenzuela, 2013.

62.  Patel & Valenzuela, 2013.

63.  Welbeck, Ware, Cerna, & Valenzuela, 2014.

64.  Adapted from source: How to Budget a Performance-Based Scholarship, Box 2 (p. 15) from Welbeck, R., Ware, M., Cerna, O., & Valenzuela, I. (2014, February). Paying it forward: A technical assistance guide for developing and implementing performance-based scholarships. New York, NY: MDRC.

65.  Mayer et al., 2015b.

66.  Patel & Rudd, 2012.

67.  Welbeck et al., 2014.

68.  Sommo et al., 2014.

69.  Adapted from: Welbeck, R., Ware, M., Cerna, O., & Valenzuela, I. (2014). Paying it forward: A technical assistance guide for developing and implementing performance-based scholarships. New York, NY: MDRC.

70.  Jaggars, Hodara, Cho, & Xu, 2015; Kosiewicz, Ngo, & Fong, 2016; Bracco, Austin, Bugler, & Finkelstein, 2015.

71.  Edgecombe, 2011, p. 4.

72.  Zachry Rutschow & Schneider, 2011.

73.  Hodara & Jaggars, 2014.

74.  Edgecombe, Jaggars, Xu, & Barragan, 2014.

75.  Edgecombe, Jaggars, Baker, & Bailey, 2013b; This study does not meet WWC evidence standards because the preintervention equivalence of the intervention and comparison groups could not be assessed. Additionally, footnote 12 indicated that 16% of students were both the intervention and comparison groups, which is a confounding factor in the analysis.

76.  Fong & Visher, 2013; This study does not meet WWC evidence standards because the pre-intervention equivalence of the intervention and comparison groups could not be assessed.

77.  Zachry Rutschow & Schneider, 2011.

78.  Cho, Kopko, Jenkins, and Jaggars (2012) evaluated the effectiveness of the Accelerated Learning Program (ALP), however, the study did not meet WWC standards because the authors used imputation for some covariates and outcomes; the WWC currently does not allow imputation for covariates.

79.  Zachry Rutschow & Diamond, 2015; This study does not meet WWC evidence standards because the intervention and comparison groups could not be established as equivalent at baseline on both a continuous measure of prior academic achievement and a measure of students’ socioeconomic status.

80.  Zeidenberg, Cho, & Jenkins, 2010; Jenkins, Zeidenberg, & Kienzl, 2009.

81.  Yamada, 2014; This study does not meet WWC evidence standards because the intervention and comparison groups could not be established as equivalent at baseline, due to the use of a binary measure of prior academic achievement. Additionally, the comparison group was drawn from a prior cohort, rather than comparing two groups from the same time period, which is a confounding factor in the analysis.

82.  Hodara & Jaggars, 2014.

83.  Hodara and Jaggars (2014) also investigated the effects of an accelerated math course compared to the traditional length developmental math course, the comparison group for this analysis had only one college in contrast to five colleges in the intervention group. In this situation, the effects of the intervention cannot solely be attributed to accelerated math instruction, because the college in the comparison group may have unique characteristics affecting the results.

84.  See also Jenkins, Zeidenberg, & Kienzl, 2009.

85.  Adapted from: Nodine, T., Dadgar, M., Venezia, A., & Bracco, K. R. (2013) Acceleration in developmental education. San Francisco, CA: WestEd. Retrieved from http://www.wested.org/online_pubs/resource1281.pdf.

86.  Adapted from: Nodine, T., Dadgar, M., Venezia, A., & Bracco, K. R. (2013) Acceleration in developmental education. San Francisco, CA: WestEd. Retrieved from http://www.wested.org/online_pubs/resource1281.pdf.

87.  Jaggars et al., 2015.

88.  Jaggars et al., 2015.

89.  Fay, Bickerstaff, & Hodara, 2013.

90.  Zachry Rutschow & Diamond, 2015; Katiraie, 2007.

91.  Boylan, 2002; Shults, 2000.

92.  Bain, 2004; Coalition of Academic Workforce, 2012; Green, 2007; Linder, 2012.

93.  Edgecombe, Cormier, Bickerstaff, & Barragan, 2013a; Bracco, Austin, Bugler, & Finkelstein, 2015; Kosiewicz, Ngo, & Fong, 2016.

94.  Adapted from: Nodine, T., Dadgar, M., Venezia, A., & Bracco, K. R. (2013) Acceleration in developmental education. San Francisco, CA: WestEd. Retrieved from http://www.wested.org/online_pubs/resource1281.pdf.

95. This is called self-regulated learning in the research literature; it has been well-researched among K–12 students. Many scholars consider self-regulated learning to be one aspect of metacognition and have discussed the interrelationships of metacognition, self-regulation, and self-regulated learning (e.g., Al-Harthy, Was, & Issacson, 2010; Dinsmore, Alexander, & Loughlin, 2008; Kaplan, 2008; Sperling, Howard, Staley, & DuBois, 2004). See also Gourgey (1998) for a discussion of teaching metacognition and self-regulated learning in basic skills instruction.

96.  See for example, Zimmerman, 2002.

97.  Zimmerman, Moylan, Hudesman, White, & Flugman, 2011.

98.  Zimmerman et al., 2011.

99.  Weinstein, Meyer, Husman, Van Mater Stone, & McKeachie, 2006.

100. Zimmerman et al., 2011.

101.Weinstein et al., 2006, p. 304.

102.Weinstein et al., 2006.

103.Brophy, 2004.

104.Hudesman et al., 2013b, p. 17.

105.Hudesman et al., 2014.

106.In Hudesman et al. (2014), the log form contained the following questions: What was your goal for your students and/or yourself this period? What were the mathematics content and metacognitive strategies your students and/or you were planning to use to achieve this goal? What actually happened? How will the results of this cycle inform your instruction in the future? What worked that you will continue? What adjustments do you make?

107.Adapted with permission from: Hudesman, J. & Crosby, S. (2014). Self-regulated learning (SRL) student workbook. Prepared for Essex County College. Unpublished manuscript. Contact: John.Hudesman@ret.citytech.cuny.edu. See also: Finkelman, J., Hudesman, J., Flugman, B., & Crosby, S. (2014). An enhanced formative assessment and self-regulated learning program: From the classroom to the workplace. Austin Journal of Psychiatry and Behavioral Science, 1(1), 1002–1009.

108.Quiz adapted with permission from: Finkelman, J., Hudesman, J., Flugman, B., & Crosby, S. (2014). An enhanced formative assessment and self-regulated learning program: From the classroom to the workplace. Austin Journal of Psychiatry and Behavioral Science, 1(1), 1002–1009. First cited in: Zimmerman, B. J., Moylan, A., Hudesman, J., White, N., & Flugman, B. (2011). Enhancing self-reflection and mathematics achievement of at-risk urban technical college students. Psychological Test and Measurement Modeling, 53(1), 141–160.

109.Sample Quiz Reflection adapted with permission from: Finkelman, J., Hudesman, J., Flugman, B., & Crosby, S. (2014). An enhanced formative assessment and self-regulated learning program: From the classroom to the workplace. Austin Journal of Psychiatry and Behavioral Science, 1(1), 1002–1009. First cited in: Zimmerman, B. J., Moylan, A., Hudesman, J., White, N., & Flugman, B. (2011). Enhancing self-reflection and mathematics achievement of at-risk urban technical college students. Psychological Test and Measurement Modeling, 53(1), 141–160.

110.Reprinted with permission from: Hudesman, J., & Crosby, S. (2014). Self-regulated learning (SRL) student workbook. Prepared for Essex County College. Unpublished manuscript, p. 23. Contact: John.Hudesman@ret.citytech.cuny.edu.

111.Reprinted with permission from: Hudesman, J., & Crosby, S. (2014). Self-regulated learning (SRL) student workbook. Prepared for Essex County College. Unpublished manuscript, p. 73. Contact: John.Hudesman@ret.citytech.cuny.edu.

112.Reprinted with permission from: Hudesman, J., & Crosby, S. (2014). Self-regulated learning (SRL) student workbook. Prepared for Essex County College. Unpublished manuscript, p. 79. Contact: John.Hudesman@ret.citytech.cuny.edu.

113.Adapted from: Hudesman, J., & Crosby, S. (2014). Self-regulated learning (SRL) student workbook. Prepared for Essex County College. Unpublished manuscript.; Hudesman, J., & Crosby, S. (2014). Self-regulated learning (SRL) supplemental instructor’s lesson plans. Prepared for Essex County College. Unpublished manuscript. Contact: John.Hudesman@ret.citytech.cuny.edu.

114.Scrivener & Weiss, 2013.

115.Scrivener et al., 2015.

116.Source: Scrivener, S., Weiss, M. J., Ratledge, A., Rudd, T., Sommo, C., & Fresques, H. (2015). Doubling graduation rates: Three-year effects of CUNY’s Accelerated Study in Associate Programs (ASAP) for developmental education students. New York, NY: MDRC. Supplemental citations: Scrivener, S., & Weiss, M. J. (2013). More graduates: Two-year results from an evaluation of Accelerated Study in Associate Programs (ASAP) for developmental education students. Policy Brief. New York: MDRC; and Scrivener, S., Weiss, M. J., & Sommo, C. (2012). What can a multifaceted program do for community college students? Early results from an evaluation of Accelerated Study in Associate Programs (ASAP) for developmental education students. New York: MDRC. Disposition: Meets WWC Group Design Standards Without Reservations.

117.Bailey, 2009.

118.Scrivener et al., 2015.

119.Scrivener et al., 2015.

120.A survey of students administered approximately 1 year into the program found that 90% of ASAP participants believed that they had “most or all of the services and supports that they needed to succeed in school” (Scrivener et al., 2015, p. ES–9). Study authors speculate that it would be unlikely that a full-time requirement paired with far more limited supports would be as productive.

121. Scrivener et al., 2015, p. ES–9.

122. Scrivener et al., 2015.

123. Scrivener et al., 2015, p. 23.

124. Scrivener et al., 2015.

125. U.S. Department of Education, 2014.

126. Scrivener et al., 2015.

127. Scrivener et al., 2015.

128. Scrivener et al., 2015.

129. Scrivener et al., 2015, p. 31.

130. Scrivener et al., 2015.

131. Scrivener et al., 2015.

132. Scrivener et al., 2015.

133.CUNY, 2015.

134.MDRC, 2015.

135. Scrivener et al., 2015.

136.American Association of Community Colleges, 2015.

137.Inside Track, 2015.

138.Klempin, 2014.

139.Adapted with permission from: City University of New York. (2015). Inside ASAP, Accelerated Study in Associate Programs: A resource guide on program structure, components, and management. New York, NY: Author. (p. 17) Retrieved from http://www1.cuny.edu/sites/asap/wp-content/uploads/sites/8/2015/05/Resource-Guide.pdf Copyright © 2015 by Office of Academic Affairs, City University of New York.

A practice guide presents recommendations for educators to address challenges in their classrooms and schools. It is based on reviews of research, the experiences of practitioners, and the expert opinions of a panel of nationally recognized experts.

This practice guide was prepared for the WWC by Development Services Group, Inc. under contract ED-IES-12-C-0084.

Table of Contents

Strategies for Postsecondary Students in Developmental Education – A Practice Guide for College and University Administrators, Advisors, and Faculty

Table of Contents

Introduction

Recommendation 1. Use multiple measures to assess postsecondary readiness and place students

Recommendation 2. Require or incentivize regular participation in enhanced advising activities

Recommendation 3. Offer students performance-based monetary incentives

Recommendation 4: Compress or mainstream developmental education with course redesign

Recommendation 5: Teach students how to become self-regulated learners

Recommendation 6: Implement comprehensive, integrated, and long-lasting support programs

Appendix A. Postscript from the Institute of Education Sciences

Appendix B. About the Authors

Appendix C. Disclosure of Potential Conflicts of Interest

Appendix D. Evidence summary of studies in support of ratings

References

Endnotes

List of Tables

Table A. Institute of Education Sciences levels of evidence for WWC practice guides

Table B. Levels of evidence for each practice guide recommendation

List of Figures

Figure A. Prisma flow diagram for study identification, screening, eligibility, and inclusion status/ratings

Figure 1.1. Types of measures to assess college readiness and place students into courses

Figure 2.1. Sample enhanced advising planning tool

Figure 3.1. How to budget for performance-based monetary incentives

Figure 4.1. Student success through accelerated coursework

Figure 4.2. Inventory of strategies currently in place for accelerating students’ progress through developmental education

Figure 4.3. Planning issues and considerations in adopting compressed and mainstreaming developmental education models

Figure 5.1. Overview of the self-regulated learning process

Figure 5.2. Overview of the self-regulated learning process

Figure 5.3. Sample self-regulated learning quiz in mathematics

Figure 5.4. Sample quiz reflection and revision sheet in mathematics

Figure 5.5. Paired discussion among students—evaluation of the quiz format

Figure 5.6. Guided self-reflection about the course

Figure 5.7. Discussion questions for applying self regulated learning skills to other courses