Project Activities
In this methods project, researchers will develop a coherent and coordinated package of integrated resources for the design and implementation of rigorous randomized controlled trials (RCTs) of postsecondary education interventions. The team will create, curate, and disseminate resources that will: (a) facilitate conducting more high-quality postsecondary RCTs, (b) reduce the cost of conducting such evaluations, (c) support the implementation of open science practices, and (d) increase consistency across studies in support of cross-study analyses. These freely available resources will be end-user tested, and a network of 23 national experts will help create, curate, and vet these tools and materials, accelerate the formation of field norms, and use and disseminate these products. The research team will conduct analyses and build interfaces to make the resources accurate and accessible to users.
Structured Abstract
Research design and methods
Researchers will assemble a list of key components for carrying out high quality RCTs, verified by the national network of experts. The agreed-upon list will serve as an umbrella for resource curation and development by the research team. Within components requiring tool development, the research team will employ correlational analysis and iterative (continuous improvement) methods to populate the tools with current data and make them accessible to others. Within components where high-quality tools exist, the team will connect researchers with these tools and their supporting materials. The project website will include accessible information about all tools that will make them usable by the broad field of postsecondary researchers, including college administrators, university-based researchers, and research firms.
This project's ongoing theory and empirical evidence-based resource development process will include updating earlier analyses with new data, enhancing analyses with new developments, and integrating resources. Enhancements will include updated empirical benchmarks for interpreting study findings, updated design parameters for power/minimum detectable effects calculations, and updating state funding formulas for improved return-on-investment calculations. Website resources and tools will be integrated by explicitly drawing connections between a tool designed to help researchers assess the likely available sample for a planned study, the minimum detectable effect (MDE) for a planned study, and where MDE falls within the distribution of effects from past evaluations.
User Testing: Resources will be created, curated, reviewed, and vetted by the expert network. They will review plans, test beta versions prior to release, and provide feedback on functionality and usability. The team will use network meetings to obtain additional feedback on products. Each product's development process will start with an initial full committee meeting, including one or more MDRC staff member and network experts who will also serve as end-user testers. Initial committee meetings will clarify the goals of the product, gather information about existing resources that may be useful as a starting point for product development. Next, with the goals clarified and existing document review completed, one or two committee leads will draft the products. This process will be followed by a draft product review meeting with the full committee including network expert end-users. This process will iterate the draft products two times or until the committee is satisfied with the products, whichever comes first. Once completed, the product will be shared electronically and/or at the annual network convening, for product review/vetting with the full expert network. Here, the broader network will have the opportunity to provide additional end-user feedback before the product is finalized.
People and institutions involved
IES program contact(s)
Products and publications
Products: The research team will develop a full set of resources for conducting high-quality RCTs, available to the public on an accessible, free, and user-friendly website. Examples include an R package for processing National Student Clearinghouse data to create core outcomes, a web-based application for conducting power/MDE calculations with built in postsecondary-specific design parameters, and a web-based application with postsecondary-specific empirical benchmarks for improved interpretation of evaluation findings. These resources will also be posted to ERIC where feasible. The expert network will promote materials within their vast networks of graduate students, academics, research firm scientists, and foundation staff. MDRC will disseminate the resource hub through its website and social media accounts, as well as its close relationships with research and practice organizations. The team will convene seminars that explain the resource hub to groups that include Minority Serving Institutions (MSIs), research firms, and pre-doctoral researchers.
ERIC Citations: Find available citations in ERIC for this award here.
Related projects
Supplemental information
Co-Principal Investigator: Kob, Rose
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
