IES Blog

Institute of Education Sciences

Guiding Principles for Successful Data Sharing Agreements

Data sharing agreements are critical to conducting research in education. They allow researchers to access data collected by state or local education agencies to examine trends, determine the effectiveness of interventions, and support agencies in their efforts to use research-based evidence in decision-making.

Yet the process for obtaining data sharing agreements with state or local agencies can be challenging and often depends on the type of data involved, state and federal laws and regulations regarding data privacy, and specific agency policies. Some agencies have a research application process and review timeline available on their websites. Others may have a more informal process for establishing such agreements. In all instances, these agreements determine how a researcher can access, use, and analyze education agency data.

What are some guiding principles for successfully obtaining data sharing agreements? 

Over several years of managing projects that require data sharing agreements, I have learned a few key principles for success. While they may seem obvious, I have witnessed data sharing agreements fall apart because one or more of these principles were not met:

  • Conduct research on a topic that is a priority for the state or local education agency. Given the time and effort agencies invest in executing a data sharing agreement and preparing data, researchers should design studies that provide essential information to the agency on a significant topic. It can be helpful to communicate exactly how and when the findings will be shared with the agency and possible actions that may result from the study findings.
  • Identify a champion within the agency. Data sharing agreements are often reviewed by some combination of program staff, legal counsel, Institutional Review Board staff, and research or data office staff. An agency staff member who champions the study can help navigate the system for a timely review and address any internal questions about the study. That champion can also help the researcher work with the agency staff who will prepare the data.
  • Be flexible and responsive. Agencies have different requirements for reviewing data sharing agreements, preparing and transferring data, securely handling data, and destroying data upon study completion. A data sharing agreement often requires some back-and-forth to finalize the terms. Researchers need to be prepared to work with their own offices and staff to meet the needs of the agency.
  • Work closely with the data office to finalize data elements and preparation. Researchers should be able to specify the sample, timeframe, data elements, and whether they require unique identifiers to merge data from multiple files. I have found it beneficial to meet with the office(s) responsible for preparing the data files in order to confirm any assumptions about the format and definitions of data elements. If the study requires data from more than one office, I recommend having a joint call to ensure that the process for pulling the data is clear and feasible to all staff involved. For example, to link student and teacher data, it might be necessary to have a joint call with the office that manages assessment data and the office that manages employment data.
  • Strive to reduce the burden on the agency. Researchers should make the process of sharing data as simple and efficient as possible for agency staff. Strategies include providing a template for the data sharing agreement, determining methods to de-identify data prior to transferring it, and offering to have the agency send separate files that the researchers can link rather than preparing the file themselves.
  • Start early. Data sharing agreements take a lot of time. Start the process as soon as possible because it always takes longer than expected. I have seen agreements executed within a month while others can take up to a year. A clear, jointly developed timeline can help ensure that the work starts on time.

What resources are available on data sharing agreements?

If you are new to data sharing agreements or want to learn more about them, here are some helpful resources:

Written by Jacqueline Zweig, Ph.D., Research Scientist, Education Development Center. Dr. Zweig is the Principal Investigator on an IES-funded research grant, Impact of an Orientation Course on Online Students' Completion Rates, and this project relies on data sharing. 

The Scoop on Replication Research in Special Education

Replication research may not grab the headlines, but reproducing findings from previous studies is critical for advancing scientific knowledge. Some have raised concerns about whether we conduct a sufficient number of replication studies. This concern has drawn increased attention from scholars in a variety of fields, including special education.

Photo array, top left going clockwise: Therrien, Lemons, Cook, and Coyne

Several special education researchers explored this issue in a recent Special Series on Replication Research in Special Education in the journal, Remedial and Special Education. The articles describe replication concepts and issues, systematically review the state of replication research in special education, and provide recommendations for the field. One finding is that there may be more replication studies than it seems—but authors don’t call them replications.

Contributors to the special issue include Bryan Cook from the University of Hawaii, Michael Coyne from the University of Connecticut, and Bill Therrien from the University of Virginia, who served as guest editors, and Chris Lemons, from Peabody College of Vanderbilt University. They shared more about the special issue and their collective insights into replications in special education research.

(In photo array, top left going clockwise: Therrien, Lemons, Coyne, and Cook)

How did you become interested in replication work?

Replication is a core component of the scientific method. Despite this basic fact that we all learned in Research 101, it is pretty apparent that in practice, replication is often ignored. We noticed how much attention the lack of replication was starting to get in other fields and in the press and were particularly alarmed by recent work showing that replications often fail to reproduce original findings. This made us curious about the state and nature of replication in the field of special education.

What is the state of replication research in special education?

It depends on how you define replication and how you search for replication articles. When a narrow definition is used and you require the term “replication” to be in the article, the rate of replication doesn’t look too good. Using this method, Lemons et al. (2016) and Makel et al. (2016) reported that the rate of replication in special education is between 0.4 to 0.5%, meaning that out of all the articles published in our field, less than 1% are replications. We suspected that—for a number of reasons (e.g., perceptions that replications are difficult to publish, are less prestigious than novel studies, and are hostile attempts to disprove a colleague’s work)—researchers might be conducting replication studies but not referring to them as such. And, indeed it’s a different story when you use a broad definition and you do not require the term replication to be in the article. Cook et al. (2016) found that out of 83 intervention studies published in six non-categorical special education journals from 2013-2014, there were 26 (31%) that could be considered replications, though few authors described their studies that way. Therrien et al. (2016) selected eight intervention studies from 1999-2001 and determined whether subsequently published studies that cited the original investigations had replicated them. They found that six of the eight original studies had been replicated by a total of 39 different studies (though few of the replications identified themselves as such).

What were some other key findings across the review articles?

Additional findings indicated that: (a) most replications conducted in special education are conceptual (i.e., some aspects are the same as the original study, but some are different) as opposed to direct (i.e., as similar to the original study as possible), (b) the findings of the majority of replications in special education agreed with the findings of the original studies, and (c) most replications in the field are conducted by one or more authors involved in the original studies. In three of the four reviews, we found it was more likely for a replication to produce the same outcome if there was author overlap between the original and replication studies. This may be due to the challenges of replicating a study with the somewhat limited information provided in a manuscript. It also emphasizes the importance of having more than one research team independently replicate study findings.  

What are your recommendations for the field around replicating special education interventions?

The article by Coyne et al. (2016) describes initial recommendations for how to conceptualize and carry out replication research in a way that contributes to the evidence about effective practices for students with disabilities and the conditions under which they are more or less effective:

  • Many studies evaluate an approach that has previously been studied under different conditions. In this case, researchers should specify which aspects replicate previous research;
  • Conceptualize and report intervention research within a framework of systematic replications, or a continuum of conceptual replications ranging from those that are more closely aligned to the original study to those that are less aligned;
  • Design and conduct closely aligned replications that duplicate, as faithfully as possible, the features of previous studies.
  • Design and conduct less closely aligned replications that intentionally vary essential components of earlier studies (e.g., participants, setting, intervention features, outcome measures, and analyses); and
  • Interpret findings using a variety of methods, including statistical significance, directions of effects, and effect sizes. We also encourage the use of meta-analytic aggregation of effects across studies.

One example of a high-quality replication study is by Doabler et al. The authors conducted a closely aligned replication study of a Tier 2 kindergarten math intervention. In the design of their IES-funded project, the authors planned a priori to conduct a replication study that would vary on several dimensions, including geographical location, participant characteristics, and instructional context. We believe this is a nice model of designing, conducting, and reporting a replication study.

Ultimately, we need to conduct more replication studies, we need to call them replications, we need to better describe how they are alike and different from the original study, and we need to strive for replication by researchers not involved in the original study. It is this type of work that may increase the impact research has on practice, because it strengthens our understanding of whether, when, and where an intervention works.

By Katie Taylor, Program Officer, National Center for Special Education Research

RCT-YES: Supporting a Culture of Research Use in Education

By Ruth Curran Neild, Delegated Director, IES

The mission of the Institute of Education Sciences (IES), at its core, is to create a culture in which independent, rigorous research and statistics are used to improve education. But sometimes research is seen by practitioners and policymakers as something that is done for them or to them, but not by them. And that’s something we’re hoping to change.

IES is always looking for new ways to involve educators in producing and learning about high-quality, useful research. We believe that if state and school district staff see themselves as full participants in scientific investigation, they will be more likely to make research a part of their routine practice. Simply put, we want to make it easier for educators to learn what works in their context and to contribute to the general knowledge of effective practices in education.    

That’s why we’re so pleased to add the RCT-YESTM software to the IES-funded toolkit of free, user-friendly resources for conducting research. Peter Schochet of Mathematica Policy Research, Inc. led the development of the software, as part of a contract with IES held by Decision Information Resources, Inc.

RCT-YES has a straightforward interface that allows the user to specify the analyses for data from a randomized controlled trial (RCT) or a quasi-experiment. Definitions and tips in the software help guide the user and accompanying documentation includes a mini-course on RCTs. When the user enters information about the data set and study design, RCT-YES produces a program to run the specified analyses (in either R or Stata) and provide a set of formatted tables.

The target users are those who have a basic knowledge of statistics and research design but do not have advanced training in conducting or analyzing data from impact studies. But we expect that even experienced researchers will like the simplicity and convenience of RCT-YES and benefit from some of its novel features, such as how it reports results.

When used properly, RCT-YES provides all of the statistics needed by the What Works ClearinghouseTM (WWC) to conduct a study review.  This is an important feature because the WWC often needs to contact authors—even experienced ones—to obtain additional statistics to make a determination of study quality.  RCT-YES could help advance the field by increasing the completeness of study reports.

Another unique feature of the software is that it defaults to practices recommended by IES’ National Center for Education Statistics for the protection of personally identifiable information. For example, the program suppresses reporting on small-size subgroups.

While the user sees only the simplicity of the interface, the underlying estimation methods and code required painstaking and sophisticated work.  RCT-YES relies on design-based estimation methods, and the development, articulation, peer review, and publication of this approach in the context of RCT-YES was the first careful step. Design-based methods make fewer assumptions about the statistical model than methods traditionally used in education (such as hierarchical linear modeling), making this approach especially appropriate for software designed with educators in mind.

The software is available for download from the RCT-YES website, where you can also find support videos, documentation, a user guide, and links to other helpful resources. The videos below, which are also hosted on the RCT-YES website, give a quick overview of the software.

There are many other ways that IES fosters a culture of research use in education. For instance, our 10 Regional Educational Laboratories (RELs) have research alliances that work with states and districts to develop research agendas. The RELs also host events to share best practices for putting research into action, such as the year-long series of webinars and training sessions on building, implementing, and effectively using Early Warning Systems to reduce dropping out.

IES also offers grants to states and districts to do quick evaluations of programs and policies that have been implemented in their schools. The low-cost, short-duration evaluations not only help the grantees discover what is working, but can help others who might use the same program or implement a similar policy. (We’ll announce the first round of grant recipients in the coming weeks).

Visit the IES website to learn more about our work. You can also stay on top of news and information from IES by following us on Facebook and Twitter

 

 

 

The Institute of Education Sciences at AERA

The American Educational Research Association (AERA) will hold its annual meeting April 8 through April 12 in Washington, D.C.—the largest educational research gathering in the nation. This will be a special meeting for AERA, as it is celebrating 100 years of advocating for the development and use of research in education. The program includes hundreds of sessions, including opportunities to learn about cutting edge education research and opportunities to broaden and deepen the field. 

About 30 sessions will feature staff from the Institute of Education Sciences (IES) discussing IES-funded research, evaluation, and statistics, as well as training and funding opportunities.

On Saturday, April 9, at 10:35 a.m., attendees will have a chance to meet the Institute’s leadership and hear about the areas of work that IES will be focusing on in the coming year. Speakers include Ruth Curran Neild, IES’ delegated director, and the leaders of the four centers in IES: Thomas Brock, commissioner of the National Center for Education Research (NCER); Peggy Carr, acting commissioner of the National Center for Educational Statistics (NCES); Joy Lesnick, acting commissioner of the National Center for Education Evaluation and Regional Assistance (NCEE), and Joan McLaughlin, commissioner of the National Center for Special Education Research (NCSER).

On Monday, April 11, at 9:45 a.m., attendees can speak to one of several IES staffers who will be available at the Research Funding Opportunities—Meet Your Program Officers session. Program officers from NCER, NCSER, and NCEE will be on hand to answer questions about programs and grant funding opportunities. Several IES representatives will also be on hand Monday afternoon, at 4:15 p.m. for the Federally Funded Data Resources: Opportunities for Research session to discuss the myriad datasets and resources that are available to researchers.

NCES staff will lead sessions and present on a variety of topics, from The Role of School Finance in the Pursuit of Equity (Saturday, 12:25 p.m.) to Understanding Federal Education Policies and Data about English Learners (Sunday, April 10, 8:15 a.m.) and what we can learn from the results of PIAAC, a survey of adult skills (also Sunday, 8:15 a.m.). Dr. Carr will be a part of several sessions, including one on Sunday morning (10:35 a.m.) about future directions for NCES longitudinal studies and another on Monday morning (10 a.m.) entitled Issues and Challenges in the Fair and Valid Assessment of Diverse Populations in the 21st Century

On Monday, at 11:45 a.m., you can also learn about an IES-supported tool, called RCT-YES, that is designed to reduce barriers to rigorous impact studies by simplifying estimation and reporting of study results (Dr. Lesnick will be among those presenting). And a team from the IES research centers (NCER/NCSER) will present Sunday morning (10:35 a.m.) on communication strategies for disseminating education research (which includes this blog!).

IES staff will also participate in a number of other roundtables and poster sessions. For instance, on Tuesday, April 12, at 8:15 a.m., grab a cup of coffee and attend the structured poster session with the Institute’s 10 Regional Educational Laboratories (RELs). This session will focus on building partnerships to improve data use in education.  REL work will also be featured at several other AERA sessions.  

Did you know that the National Library of Education (NLE) is a component of IES? On Friday and Monday afternoon, attendees will have a unique opportunity to go on a site visit to the library. You’ll learn about the library’s current and historical resources – including its collection of more than 20,000 textbooks dating from the mid-19th century. The Library offers information, statistical, and referral services to the Department of Education and other government agencies and institutions, and to the public.

If you are going to AERA, follow us on Twitter to learn more about our sessions and our work.  And if you are tweeting during one of our sessions, please include @IESResearch in your tweet. 

By Dana Tofig, Communications Director, IES

The PI Meeting in 140 Characters

By Wendy Wei, Program Assistant, National Center for Education Research

How can practitioners and policymakers apply education research to their everyday work if they never hear about it or do not understand it? Communicating and disseminating research findings plays an integral role in promoting the education sciences and advancing the field.

That is why we made communication and dissemination a major theme at the IES Principal Investigators’ Meeting held earlier this month (December 10-11). The two-day meeting in Washington, D.C., featured five sessions that focused on communications – ranging from data visualization techniques to effective dissemination strategies to hearing journalists’ perspectives on how to share scientific results with the general public.

There was a lot of talk about social media during the meeting and plenty of tweeting about the presentations. We used the Twitter hashtag, #IESPIMtg, to foster an ongoing conversation for meeting attendees and to share findings that emerged from sessions.  Any tweet that included #IESPIMtg was automatically pooled together, generating a live Twitter feed that was on display in the lobby throughout the meeting.

 You can see all of the #IESPImtg tweets online, but here are some highlights:

"There is a tremendous sense of urgency to bridge the gap between research and practice..." --John B King #IESPIMtg

— Leah Wisdom (@lifelnglearner) December 10, 2015

.@StanfordEd's Sean Reardon: Good partnership work can lead to new knowledge, change policy+practice, improve data quality #IESPIMtg

— Bill Penuel (@bpenuel) December 11, 2015

#IESPIMtg Practitioner partners play a critical role in making sense of data and analyses in RPPs.

— Jennifer Russell (@Jenn_L_Russell) December 10, 2015

And we can get a little bit meta now…communicating about how to communicate:

Hirsh-Pasek & Golinkoff urges researchers to create "'edible science' that is accessible, digestible and usable." #IESPIMtg

— Tomoko Wakabayashi (@twakabayashi264) December 10, 2015

Awesome presentation on #DataVisualization by @jschwabish: Show the data, reduce the clutter, stop distracting attention. #IESPIMtg

— Rudy Ruiz (@RudyRuiz_BMore) December 10, 2015

.@KavithaCardoza Explaining your research--Don't think of it as "dumbing down." Think of it as simplifying. #IESPIMtg

— Dana Tofig (@dtofig) December 11, 2015

And, of course, what's Twitter without a little fun? When we tweeted this picture...

The poster session is going strong. Principal investigators present findings from #iesfunded research. #IESPIMtg

— IES Research (@IESResearch) December 10, 2015

...Chris Magnuson, Director of Innovation for Live It, Learn It, posted this reply: 

@IESResearch careful...photo looks like it was taken on Death Star! May the force be with all grantees! #SBIR #IES

— Chris Magnuson (@cromagnuson) December 10, 2015

The National Center for Education Research (NCER) and the National Center for Special Education Research (NCSER) have made a commitment to be active contributors in communicating with and engaging the general public in the exciting findings of NCER- and NCSER-funded work. Over the past few years, we have been active on Twitter (you can follow us @IESResearch), and this past year, we launched our blog (the very one you are reading!). These two platforms have provided us with an outlet to share research findings, provide updates about events and deadlines, and connect with audiences we otherwise might not reach.

For those of you who could not make the PI meeting, videos will be posted on the conference website in about a month. So stay tuned!

We hope you’ll continue the conversation started at the PI meeting by following us on Twitter at @IESResearch or sharing your thoughts with us at IESResearch@ed.gov.