The Higher Education Randomized Controlled Trials (THE-RCT) – Supporting High Quality and Efficient Postsecondary RCTs and Open Science Practices

Researchers will assemble a list of key components for carrying out high quality RCTs, verified by the national network of experts. The agreed-upon list will serve as an umbrella for resource curation and development by the research team. Within components requiring tool development, the research team will employ correlational analysis and iterative (continuous improvement) methods to populate the tools with current data and make them accessible to others. Within components where high-quality tools exist, the team will connect researchers with these tools and their supporting materials. The project website will include accessible information about all tools that will make them usable by the broad field of postsecondary researchers, including college administrators, university-based researchers, and research firms.

This project's ongoing theory and empirical evidence-based resource development process will include updating earlier analyses with new data, enhancing analyses with new developments, and integrating resources. Enhancements will include updated empirical benchmarks for interpreting study findings, updated design parameters for power/minimum detectable effects calculations, and updating state funding formulas for improved return-on-investment calculations. Website resources and tools will be integrated by explicitly drawing connections between a tool designed to help researchers assess the likely available sample for a planned study, the minimum detectable effect (MDE) for a planned study, and where MDE falls within the distribution of effects from past evaluations.

User Testing: Resources will be created, curated, reviewed, and vetted by the expert network. They will review plans, test beta versions prior to release, and provide feedback on functionality and usability. The team will use network meetings to obtain additional feedback on products. Each product's development process will start with an initial full committee meeting, including one or more MDRC staff member and network experts who will also serve as end-user testers. Initial committee meetings will clarify the goals of the product, gather information about existing resources that may be useful as a starting point for product development. Next, with the goals clarified and existing document review completed, one or two committee leads will draft the products. This process will be followed by a draft product review meeting with the full committee including network expert end-users. This process will iterate the draft products two times or until the committee is satisfied with the products, whichever comes first. Once completed, the product will be shared electronically and/or at the annual network convening, for product review/vetting with the full expert network. Here, the broader network will have the opportunity to provide additional end-user feedback before the product is finalized.