IES Blog

Institute of Education Sciences

Recognizing Our Outstanding IES Predoctoral Fellows

Each year, the Institute of Education Sciences (IES) recognizes an outstanding fellow from our Predoctoral Interdisciplinary Research Training Programs in the Education Sciences for academic accomplishments and contributions to education research. This year, IES has selected joint recipients for the 2017 award: Rachel Abenavoli and Callie Little. They will receive their awards and present their research at the annual IES Principal Investigators meeting in Washington, D.C. in January 2019.

Dr. Abenavoli received her doctorate in Human Development and Family Studies from Pennsylvania State University (Penn State). She is currently a postdoctoral research scientist at New York University’s Steinhardt’s Institute of Human Development and Social Change, and is working in the area of early learning and social-emotional development.  Dr. Little received her doctorate in Developmental Psychology from Florida State University (FSU). She is currently a research fellow in the School of Behavioural, Cognitive and Social Sciences at the University of New England in Australia, where she specializes in understanding the development of cognitive and academic skills for individuals from early childhood through college.

We asked our Outstanding Fellows how participating in an IES-funded predoctoral training program helped their development as researchers.

Rachel Abenavoli

The IES-funded predoctoral training program at Penn State provided me with the resources, opportunities, and skills needed to begin building my own program of education research. I began my graduate program with broad interests in young childrenRachel Abenavoli’s social-emotional functioning. The fellowship helped me shape and narrow the focus of my research. By participating in regular seminars, attending talks, and meeting with external invited speakers, I began applying my developmental lens and focus on early social-emotional skills to the study of educational contexts and educationally-relevant outcomes. With generous fellowship funding that gave me the freedom and flexibility to pursue my own research questions, I was able to focus my work in graduate school on the interplay between children’s social-emotional and academic skills as they make the transition to school.

The IES fellowship also provided me with opportunities to learn, practice, and hone my methodological skills. Courses and seminars in program evaluation, causal inference, and multilevel modeling were particularly critical in building my capacity for conducting rigorous school-based research. Invited speakers complemented these core learning experiences by highlighting best practices and innovative approaches in education science. I came away from this training equipped with a range of analytic and methodological tools that are necessary to address the diversity and complexity of education research questions, settings, and designs.

Being an IES fellow has also connected me to a network of established and early career education researchers. Regular meetings with faculty mentors and other IES fellows from different home departments provided a space to discuss new ideas, experiences in the field, and possible collaborations. Conference funding enabled me to attend education research conferences and expand my professional network beyond Penn State. Engaging with this community solidified for me the value of a collaborative and interdisciplinary approach to understanding children’s learning and development in context.

My work is more rigorous and more relevant to education practice and policy because of the IES predoctoral fellowship, and I’m so grateful for the experiences, mentors, and other fellows who are critical to the success of the program at Penn State.

Callie Little

First and foremost, the support and training I received through the IES pre-doctoral fellowship at FSU provided me with an intense and philosophical appreciation of construct validity: whether an investigation can accurately measure what it was designeCallie Littled to measure. This appreciation will continue to guide how I develop and implement research studies.

The rigorous statistical courses included in FSU’s core curriculum provided a solid foundation for building my quantitative skills. Additionally, the combined focus on study-design and methodological training supplied me with a comprehensive knowledge base and the skills to investigate the complex associations among reading skills, and between reading skills and behavioral outcomes. The exposure to multiple advanced statistical methods coupled with the opportunity to directly apply these methods to relevant data so early in my career prepared me for my current projects which use large-scale twin data to conduct high-quality research on individual differences in the development of cognitive and academic skills.

The multidisciplinary environment at FSU, with its rich and diverse range of research programs and faculty, exposed me to a series of analytic techniques and content-area expertise that helped to shape an open-minded and creative approach to formulating research questions. This unique environment was one of the greatest advantages to the IES fellowship, providing the opportunity for strong mentorship, collaboration, and feedback. Most importantly, I used this opportunity to develop ongoing projects with colleagues where we innovatively combine evidence-based methods from several fields.

Fellowship funding enabled me to attend conferences and workshops with other IES fellowship teams, and provided me with access to research resources. I gained new insights into science communication, learned new techniques, and broadened my network of collaborators. I was able to recruit participants and purchase standardized assessment materials, and design and implement several of my own studies during my Ph.D. The resulting rich and unique data sets form the foundation of my current independent research.  From my experiences through the IES fellowship at FSU, I stand well-prepared to continue to conduct innovative and high-quality research into the complex mechanisms underlying achievement.

Katina Stapleton is the program officer for the Predoctoral Interdisciplinary Research Training Programs in the Education Sciences.

The WWC Evidence Standards: A Valuable and Accessible Resource for Teaching Validity Assessment of Causal Inferences to Identify What Works

by Herbert Turner, Ph.D., President and Principal Scientist, ANALYTICA, Inc.

 

The WWC Evidence Standards (hereafter, the Standards) provide a detailed description of the criteria used by the WWC to review studies. The standards were first developed in 2002 by leading methodological researchers using initial concepts from the Study Design and Implementation Assessment Device (DIAD), an instrument for assessing the correspondence between the methodological characteristics and implementation of social science research and using this research to draw inferences about causal relationships (Boruch, 1997; Valentine and Cooper, 2008).  During the past 16 years, the Standards have gone through four iterations of improvement, to keep pace with advances in methodological practice, and have been through rigorous peer review. The most recent of these is now codified in the WWC Standards Handbook 4.0 (hereafter, the Handbook).

 

Across the different versions of the Handbook, the methodological characteristics of an internally valid study, designed to causally infer the effect of an intervention on an outcome, have stood the test of time. These characteristics can be summarized as follows: A strong design starts with how the study groups are formed. It continues with use of reliable and valid measures of outcomes, has low attrition if a randomized controlled trial (RCT), shows baseline equivalence (in the analysis sample) if a quasi-experimental design (QED), and has no confounds.

 

These elements are the critical components of any strong research design – and are the cornerstones of all versions of the WWC’s standards. That fact, along with the transparent description of their logical underpinning, is what motivated me to use Standards 4.0 (for Group Designs) as the organizing framework for understanding study validity in a graduate-level Program Evaluation II course I taught at Boston College’s Lynch School of Education.

 

In spring 2017, nine Master and four Doctoral students participated in this semester-long course. The primary goal was to teach students how to organize their thinking and logically derive internal validity criteria using Standards 4.0—augmented with additional readings from the methodological literature. Students used the Standards (along with the supplemental readings) to design, implement, analyze, and report impact evaluations to determine what interventions work, harm, or have no discernible effect (Mosteller and Boruch, 2002). The Standards Handbook 4.0 along with online course modules were excellent resources to augment the lectures and provide Lynch School students with hands on learning.

 

At the end of the course, students were offered the choice to complete the WWC Certification Exam for Group Design or take the instructor’s developed final exam. All thirteen students chose to complete the WWC Certification Exam. Approximately half of the students became certified. Many emailed me personally to express their appreciation for the (1) opportunity to learn a systematic approach to organizing their thinking about assessing the validity of causal inference using data generated by RCTs and QEDs, and (2) developing design skills that can be used in other graduate courses and beyond. The WWC Evidence Standards and related online resources are a valuable, accessible, and free resource that have been rigorously vetted for close to two decades. The Standards have few equals as a resource to help students think systematically, logically, and clearly about designing (and evaluating) a valid research study to make causal inferences about what interventions work in education and related fields.

 

References

Boruch, R. F. (1997). Randomized experiments for planning and evaluation: A practical guide. Thousand Oaks, CA: Sage Publications.

Valentine, J.C., & Cooper, H. (2008), A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The Study Design and Implementation Assessment Device (Study DIAD). Psychological Methods, 13(2), 130-149.

Mosteller, F., & Boruch, R. F. (2002). Evidence matters: Randomized trials in education research. Washington, D.C.: Brookings Institution Press.

Supporting Military-Connected Students

Through the Systemic Approaches to Educating Highly Mobile Students program, IES supports research to improve the education outcomes of students who face social/behavioral and academic challenges because they frequently move from school to school due to changes in residence and/or unstable living arrangements. This category of students, typically referred to as highly-mobile students, includes students who are homeless, in foster care, from migrant backgrounds, or military-connected. In this guest blog, Timothy Cavell, PhD, University of Arkansas and Renée Spencer, EdD, Boston University discuss their IES-funded research on military-connected students.

Why study military-connected students? Photo of Renée SpencerPhoto of Timothy Cavell

Virtually every school district in the United States educates a child whose parent or guardian is serving in the Armed Forces. Supported by two separate IES awards, our team of researchers is working to understand how schools can better serve military students and their families. Our work focuses specifically on students who have at least one parent/guardian on active (full-time) duty in the U.S. military. We refer to these students as military-connected. Their lives are typified by transition and often entail tremendous sacrifice. For some, the challenges involve a parent deployed into combat or a parent returning from combat. For many others, the challenges are tied to the frequent transitions (e.g., permanent changes of station, temporary duty assignments) required of military families. Our IES-funded research specifically works with military-connected students within the North Thurston Public Schools (NTPS), a school district in Lacey, Washington, about 15 miles southwest of Joint (Army/Air Force) Base Lewis-McChord.

How does your research address the needs of military-connected students?

The Military Student Mentoring (MSM) project which began 4 years ago, is an effort to develop and test the benefits of school-based mentoring for military students. We reasoned that school-based mentoring was a measured response to the needs of students who are often quite resilient but who, at times, might need extra support. A key component of the intervention was developing a mentoring-delivery system anchored by a district-level MSM Coordinator who forged home-school-community (HSC) Action Teams comprised of school staff (e.g., school counselor), military parents, and community leaders. Together, the MSM Coordinator and HSC Action Teams engaged military families, identified military students who might benefit from school-based mentoring, and recruited adult volunteers to serve as mentors. These volunteers were then screened, trained, and supported by a local Big Brothers Big Sisters agency. Preliminary findings from our initial launch and subsequent pilot study support the feasibility and usability of the MSM model and point to expected gains in students’ perceptions of support.  Future steps involve efforts to making MSM more portable and self-sustaining and testing its efficacy more broadly.

Our second project, the Active-Duty Military Families and School Supports (ADMFSS) study, which was funded just this year, explores school supports for highly mobile military students. It is estimated that military students experience 6 to 9 moves during their K-12 years—a mobility rate three times that of non-military children. Most military families and students are resilient and weather these disruptions well, but some are negatively affected by the strain of multiple moves. Growing recognition of these stresses faced by military families has led to calls for schools to offer greater and more targeted support to these students.

We suspect that student mobility is not directly linked to educational outcomes; rather, repeated moves may strain families’ capacity to adjust to new communities and impede students’ ability to connect with yet another learning environment. Therefore, we will be exploring the role that school supports play in fostering military students’ sense of school connectedness. Broadly, the term school connectedness refers to students’ relationship with their school and the extent to which they feel accepted, respected, and supported by others in the school environment. Our basic premise is that high mobility can be harmful to military students’ educational outcomes when it undermines the degree to which they feel connected to school and to students and staff in their school. Supports provided by schools have the potential to buffer military students from the negative effects of high mobility on school connectedness, thereby reducing their risk for poor educational outcomes. Importantly, connectedness is an arena in which schools can take clear and effective action.

Through both of these projects, our ultimate goal is to learn enough to equip and guide other school districts that wish to serve those families who have served our country with courage and distinction.

Katina Stapleton is the program officer for the Systemic Approaches to Educating Highly Mobile Students research program.

Learning about Schools from Teachers and Principals

In the 2015-16 school year, there were approximately 90,400 principals and 3,827,100 teachers in public elementary and secondary schools in the United States. Knowledge about the characteristics and experiences of these key school staff can help inform decisions about education.  The National Teacher and Principal Survey (NTPS) supports these decisions by providing data on a variety of topics from the perspective of teachers, principals and other school staff. Results from these questionnaires provide information such as:

  • Principals’ education. Among public schools, a majority of principals held a master’s degree (61 percent) as their highest degree, compared to an education specialist/professional diploma at least one year beyond the master’s level (27 percent), a doctorate/first professional degree (10 percent), or a bachelor’s degree or less (2 percent).
  • Hours worked by teachers. On average, regular full-time teachers in public schools spent 53 hours per week on all school-related activities. That includes 27 hours that they were paid to deliver instruction to students during a typical full week. Public school teachers were required to work an average of 38 hours per week to receive their base pay.
  • Online courses. Nationwide, about 21 percent of public schools offered at least one course entirely online. This was more common among public charter schools (29 percent) than it was among traditional public schools (20 percent). A greater percentage of high (58 percent) and combined (64 percent) schools offered one or more courses entirely online than all public schools. It was also more common for schools with fewer than 100 students (45 percent) and schools with 1,000 or more students (44 percent). Among schools offering online courses, relatively more public charter schools offered all of their classes online (14 percent) than traditional public schools (5 percent).

More examples of the type of information collected in the 2015-16 NTPS can be seen in the video below:

More information is available in the NTPS online table library. In addition, analysts can access the data using DataLab or obtain a restricted-use license to conduct their own analyses of NTPS restricted-use data files.

 

By Maura Spiegelman

Evaluating Oregon’s Adult Basic Skills Transition Planning Process: An Interview with Judith Alamprese

In her 2017 IES grant, Judith Alamprese (Abt Associates) is collaborating with the state of Oregon’s Office Community College and Workforce Development to evaluate a program that aims to help adults earn a GED® and transition into postsecondary education. This project is funded under the Low-Cost, Short-Duration Evaluation of Education Interventions competition, which supports research that aims to produce meaningful results for local and/or state education agencies quickly. Program Officer, Meredith Larson, interviewed Ms. Alamprese about this current work, how it came into being, and what it might mean for Oregon and adult education more broadly.

Tell us about your area of research and why it’s important to Oregon.

My current research is focused on determining effective interventions for assisting low-skilled adults establish and succeed in a career pathway. Oregon was one of the first states to implement a statewide initiative for transitioning adult basic skills learners (henceforth adult learners) to further education and work, and this project expands Oregon’s activities to support adult learners’ success.  

What is your current project studying?

The Transition Planning Process (TPP) project is a collaboration between Oregon’s Office of Community College and Workforce Development (CCWD) and Abt Associates (Abt). We are using a randomized controlled trial (RCT) to test whether text messaging helps adult learners earn a GED® and transition to postsecondary education and training.

What is TPP, and why is this approach innovative?

TPP is a text messaging intervention in which transition facilitators who work with the adult learners send text messages to the learners to help keep them on track to complete their GED® and enroll in postsecondary courses. The intervention is a supplement to the facilitators’ other transition activities to prepare learners for next steps in education and work.

TPP has a standardized list of text messages to prompt learners to take the GED® tests, set college goals, access information on college planning and other college preparation activities. Facilitators can send texts customized to programs’ specific transition activities.

CCWD chose text messaging because it appeared to be a low-cost approach that could support existing transition activities and provide a boost to ABS learners. The TPP project is an exciting opportunity to determine whether texting can be effective with ABS learners, and may be a promising approach for encouraging specific behaviors in learners preparing to go to college.

How did this project come into being?

The TPP project grew out of Abt’s and CCWD’s work together on an IES Researcher-Practitioner Partnership grant. This grant was a longitudinal study of Oregon’s Pathways for Adult Basic Skills Transition to Postsecondary Education and Work initiative. The findings from Abt’s analyses of adult learners’ GED® attainment and postsecondary participation prompted Oregon to want to try some additional strategies to encourage ABS learners to earn a secondary credential and enroll in postsecondary courses.

What is the current status of the project?

The study is underway, and transition facilitators are providing text messages to encourage adult learners to initiate and complete GED® testing, determine next steps, and begin the postsecondary planning process. The facilitators have found that while many treatment group learners respond to the texts, some learners have chosen to increase their face-to-face interaction with their facilitators. The facilitators report that texting is an efficient way to reinforce learners and check on their progress.

Why is this work important?

This research is particularly important because it is a rigorous test of an intervention that could be beneficial to adult basic skill learners nationwide and could leverage such programs’ existing activities in transitioning learners from basic skills programs to further education and training. We will learn more about the types of information and support that are most persuasive in helping learners succeed.