Co-Principal Investigators: Andrade, Heidi; Cleary, Timothy

 Purpose: The purpose of this efficacy study is to determine the impact of the Diagnostic Assessment and Achievement of College Skills (DAACS) on postsecondary students' outcomes and the generalizability of any effects across different types of postsecondary institutions. DAACS is a fully developed suite of free, open-source, online diagnostic assessments of reading, writing, mathematics, and self-regulated learning. Students and advisors use these resources to support newly enrolled students in making successful transitions to college. A previous evaluation of DAACS found that, although there were null effects on on-time progress and credit acquisition overall, students who used DAACS feedback showed significant gains in completing their first 6 months of coursework and were more successful in earning credits than students who only completed the assessment. This previous research led to revisions to DAACS, including student nudges and an advisor dashboard. This current project will examine the impact of the revised DAACS in a variety of postsecondary settings (completely online, brick-and-mortar, blended) with both traditional and non-traditional students.

Project Activities: The researchers will conduct a randomized control trial (RCT) across three institutions. The researchers will compare the outcomes for students who receive DAACS to those who receive business as usual to determine whether DAACS improves postsecondary outcomes. They will also conduct a cost-analysis and cost-effectiveness study.

Products:  The researchers will produce evidence of the efficacy, or lack thereof, of DAACS for improving student outcomes as well as the cost and cost effectiveness of DAACS. They will share their findings through scholarly publications and presentations to practitioners and postsecondary policymakers.

Structured Abstract

Setting: This project will take place  at three postsecondary institutions: one a traditional brick-and-mortar institution in New York one blended (online and in-person), and one fully online.

Sample: Approximately 36,000 newly enrolled undergraduates will participate over the course of the study.

Intervention/Assessment: DAACS is a comprehensive intervention that aims to help students transition into postsecondary education by providing them and their advisors valuable feedback to navigate the early demands of postsecondary education. DAACS has five components:

1. Diagnostic assessments of students' readiness for college in terms of self-regulated learning, reading, writing, and mathematics
2. Feedback, strategies, and links to open educational resources that help students address deficiencies
3. Automated nudges to encourage students to utilize DAACS resources
4. Information and support that enables academic advisors to help students address deficiencies identified by the assessments, including a DAACS dashboard and professional development on DAACS and SRL
5. Predictive models that identify students at risk, as well as the specific risk factors.

Newly starting postsecondary students learn about DAAC during orientation. The students take four online assessments of college readiness, including mathematics, reading, writing, and self-regulated learning. The DAACS gives them  immediate feedback, along with suggestions and links to websites they can use to boost their knowledge and skills. Students' academic advisors use their DAACS results to inform advising conversations and guide students' self-directed learning. Institutional leaders use DAACS data for early identification of students who are at-risk of academic failure and to direct them to individualized supports.

Research Design and Methods: The researchers will conduct a randomized controlled trial at the three institutions, randomly assigning students at each institution to a treatment or control group. The researchers will track students' early credit acquisition, retention, academic achievement, and DAACS use (treatment only) for the duration of the grant. They will also monitor implementation and conduct interviews with advisors, students, and institutional leaders to gather information about their perceptions of DAACS. To determine the impact on students, they will conduct an intention-to-treat (ITT) analyses to whether being assigned to the intervention had an impact on students, and they will also investigate the effect of actually receiving the intervention by estimating the average effect of taking up the intervention among only the students who actually complied with the assignment to condition.

Control Condition: Students in the control condition will receive the business-as-usual student orientation at their institution, without DAACS.

Key Measures: Short-term outcomes include engagement, as measured by students' and advisors' use of DAACS (clicks on assessment results, feedback, resources, or advisor dashboards). Intermediate academic outcomes include time to first credit completion, ratio of credits earned to attempted, and term-to-term retention. Long-term outcomes include one-year retention, grade point average, and graduate rate. Moderating variables include amount and of DAACS use and the components of DAACS used, student characteristics (baseline academic achievement, socio-economic status, demographics, student status), and type of institution (traditional, blended, online).

Data Analytic Strategy: The researchers will use regression analyses to test moderator effects and predictive models to examine the effects of DAACS results on the accuracy of predictions of students' academic success.

Cost Analysis and Cost-Effectiveness Analysis: For the cost analysis, the researchers will use the ingredients method to identify and compute all costs and will use incremental cost-effectiveness ratios to determine the cost effectiveness of implementing DAACS.