IES Blog

Institute of Education Sciences

IES Honors Statistician Nathan VanHoudnos as Outstanding Predoctoral Fellow

By Phill Gagne and Katina Stapleton, NCER Program Officers

Each year, IES recognizes an outstanding fellow from its Predoctoral Interdisciplinary Research Training Programs in the Education Sciences for academic accomplishments and contributions to education research. The 2014 winner, Dr. Nathan VanHoudnos completed his Ph.D. at Carnegie Mellon University and wrote his dissertation on the efficacy of the Hedges Correction for unmodeled clustering. Nathan is currently a postdoctoral fellow at Northwestern University. In this blog, Nathan provides insights on becoming an education researcher and on research study design. 

How did you become interested in education research?

I was born into it. Before he retired, my father was the Director of Research for the Illinois Education Association. Additionally, my grandparents on my mother's side were both teachers. 

 

As a statistician, how do you explain the relevance of your research to education practitioners and policy-makers?

I appeal to the crucial role biostatisticians play in the progress of medical research. Doctors and medical researchers are able to devote their entire intellectual capacity towards the development of new treatments, while biostatisticians are able to think deeply about both how to test these treatments empirically and how to combine the results of many such studies into actionable recommendations for practitioners and policy makers.  I aim to be the education sciences analogue of a biostatistician. Specifically, someone whose career success is decided on (i) the technical merits of the new methodology I have developed and (ii) the usefulness of my new methodology to the field. 

Your research on the Hedges correction suggests that many education researchers mis-specify their analyses for clustered designs. What advice would you give researchers on selecting the right analyses for clustered designs? 

My advice is to focus on the design of the study. If the design is wrong, then the analysis that matches the design will fail, and it is likely that no re-analysis of the collected data will be able to recover from the initial mistake. For example, a common design error is randomizing teachers to experimental conditions, but then assuming that how the school registrar assigned students to classes was equivalent to the experimenter randomizing students to classes. This assumption is false. Registrar based student assignment is a kind of group based, or clustered, random assignment. If this error is not caught at the design stage, the study will necessarily be under powered because the sample size calculations will be off. If the error is not caught at the publication stage, the hypothesis test for the treatment effect will be anti-conservative, i.e. even if the treatment effect is truly zero, the test statistic is still likely to be (incorrectly!) statistically significant. The error will, however, be caught if the What Works Clearinghouse decides to review the study. Their application of the Hedges correction, however, will not fix the design problem. The corrected test statistic will, at best, have low power, just like a re-analysis of the data would. At worst, the corrected test statistic can have nearly zero power. There is no escape from a design error. 


To give a bit of further, perhaps self-serving advice, I would also suggest engaging your local statistician as a collaborator. People like me are always looking to get involved in substantively interesting projects, especially if we can get involved at the planning stage of the project. Additionally, this division of labor is often better for everyone: the statistician gets to focus on interesting methodological challenges and the education researcher gets to focus on the substantive portion of the research. 

How has being an IES predoc and now an IES postdoc helped your development as a researcher?

This is a bit like the joke where one fish asks another "How is the water today?" The other fish responds "What's water?" 

I came to Carnegie Mellon for the joint Ph.D. in Statistics and Public Policy, in part, because the IES predoc program there, the Program for Interdisciplinary Education Research (PIER), would both fund me to become and train me to become an education researcher. The PIER program shaped my entire graduate career. David Klahr (PIER Director) gave me grounding in the education sciences. Brian Junker (PIER Steering committee) taught me how to be both methodologically rigorous and yet still accessible to applied researchers. Sharon Carver (PIER co-Director), who runs the CMU lab school, built in a formal reflection process for the "Field Base Experience" portion of our PIER training. That essay, was, perhaps, the most cathartic thing I have ever written in that it helped to set me on my career path as a statistician who aims to focus on education research. Joel Greenhouse (affiliated PIER faculty), who is himself a biostatistician, chaired my thesis committee. It was his example that refined the direction of my career: I wish to be the education sciences analogue of a biostatistician. 

The IES postdoc program at Northwestern University, where I am advised by Larry Hedges, has been very different. Postdoctoral training is necessarily quite different from graduate school. One thread is common, however, the methodology I develop must be useful to applied education researchers. Larry is, as one might suppose, quite good at focusing my attention on where I need to make technical improvements to my work, but also how I might better communicate my technical results and make them accessible to applied researchers. After only a year at Northwestern, I have grown considerably in both my technical and communication skills.

What career advice would you give to young researchers?

Pick good mentors and heed their advice. To the extent that I am successful, I credit the advice and training of my mentors at Carnegie Mellon and Northwestern. 


Comments? Questions? Please write to us at IESResearch@ed.gov.

What is the Price of College?

By Lauren Musu-Gillette

For many students and their families, a college education is seen as an investment in the future. But like all investments, the initial sacrifices can seem burdensome and the future is uncertain. NCES seeks to help students and their families make good decisions by providing access to timely information about the price of college and the availability of financial aid. The NCES College Navigator web site provides an array of search tools to help students locate institutions that meet their financial needs and academic interests. Additionally, the Higher Education Opportunity Act of 2008 requires the U.S. Department of Education to report  current information and recent changes in net prices for college attendance, and tuition and fees charges at different types of institutions. Data on high and low cost institutions are published on the College Affordability and Transparency Center website. These sites can help students and their families make informed decisions about their most affordable options.

NCES recently released a report that examines the total, net, and out-of-pocket prices by type of institution in 2011-12. Although it is presented for institutions at an aggregate level, rather than for individual institutions, this report provides valuable information on the average prices for different types of degree-granting colleges, including breakdowns for students from families with different income levels. Overall, students at public 2-year colleges had the lowest average total price of attendance in 2011-12 at $15,000. Public 4-year institutions had the lowest average total price ($23,200) among 4-year colleges, and the average price of attendance at 4-year for-profit institutions was $29,300.  The average total price was highest at private non-profit 4-year institutions ($43,500); however, 4-year private nonprofit schools also awarded the most grant money and had the greatest percentage of students who received grants.

Factoring in financial aid, such as grants, loans, and work study, reduces the out-of-pocket costs that students and their families pay at various institutions. As a result, many full-time students pay less than the advertised total price of attendance. The out-of-pocket net price of attendance is based on the total price of attendance, but also accounts for the amount of grant and loan aid that students typically receive. Additionally, because many sources of aid are based on students’ financial need, there are differences in the out-of-pocket net price of attendance for dependent students from lower and higher income families. 

Overall, students at public 2-year colleges had the lowest out-of-pocket net price (after grants and loans) at $9,900 in 2011-12. The average out-of-pocket net price was $11,800 at public 4-year institutions and $15,000 at for-profit institutions. The largest difference between total price of attendance and out-of-pocket net price was at private nonprofit 4-year institutions, with the out-of-pocket net price ($18,100) being about $25,400 less than the average total price.  

Across all types of institutions, dependent students from lower income families (the lowest quarter of family incomes) had the lowest out-of-pocket net price. The average out-of-pocket net price for students from lower income families was $7,500 at public 2-year institutions, $7,100 at public 4-year institutions, $11,000 at private nonprofit institutions, and $15,000 at private for-profit institutions. In contrast, students from families in the top quarter of incomes had average out-of-pocket costs of $13,100 at public 2-year institutions, $16,800 at public 4-year institutions, $26,600 at private nonprofit institutions, and $22,300 at private for-profit institutions.

For more information on the total, net, and out-of-pocket prices by type of institution, please download the entire report: What is the Price of College: Total, Net, and Out-of-Pocket Prices by Type of Institution in 2011-12, or watch the video below.

Where Are They Now? A Q&A With the Creators of EcoMUVE – A Virtual Environment for Middle School Science

Where Are They Now? showcases completed IES research projects. The feature describes the IES project and research findings, and updates the progress since IES project completion.

By Ed Metz, NCER Program Officer

In this inaugural Where Are They Now? feature, we take a look back at a 2008 grant to researchers at Harvard University for the development of EcoMUVE.

EcoMUVE uses Multi-User Virtual Environments (MUVEs), which have the look and feel of video games, to help middle school students gain a deeper understanding of ecosystems, scientific inquiry, and causal patterns. The MUVEs recreate authentic ecological settings within which students explore and collect information. Students work individually at their computers and collaborate in teams within the virtual world. EcoMUVE includes two modules, Pond and Forest; each module is a two-week inquiry-based ecosystems curriculum. EcoMUVE received the First Place award in the Interactive and Immersive Learning Category at the 2011 Association for Educational Communications and Technology conference, and has received follow-on support from the National Science Foundation and Qualcomm Wireless. 

In this blog, we catch up with two of the researchers who led the development of EcoMUVE, Chris Dede and Shari Metcalf, to look back at their IES project and to learn about recent developments.

How and when did the idea to develop a virtual environment for science learning come about?

Chris Dede’s prior research with the River City project looked at supporting student inquiry using immersive exploration in a virtual world. Meanwhile, Harvard Professor Tina Grotzer was developing ways to support students in understanding complex causality in ecosystems. They worked together on a grant proposal to IES to combine their interests.

How does a virtual environment provide meaningful learning opportunities that otherwise might not be possible?

Ecosystems are complex systems shaped by relationships that often happen at microscopic levels, at a distance, and over long periods of time. Immersion in virtual environments can transform the learning experience by situating the learner in a rich simulated context in which new visualization opportunities are possible – e.g., zooming in to the microscopic level, or traveling to different points in time.

Students start to get a feel for the ecosystem and its relationships through tacit sensory clues. It is an uphill walk from the pond to the housing development, and students can walk down along a drainage ditch and through the pipe where runoff flows into the pond. The pond becomes noticeably greenish during the algae bloom. 

Students can experience turbidity directly by walking under the water of the pond and seeing how murky it looks on different days.

 

What was an unexpected outcome of the development process?

The types of “big data” about motivation and learning for each student that EcoMUVE can generate include: time-stamped logfiles of movements and interactions in the virtual world, chat-logs of utterances, and tables of data collected and shared. Other digital tools can provide data from concept maps that chart the flow of energy through the ecosystem and that document each student team’s assertions about its systemic causal relationships, with adduced supporting evidence. Using Go-Pro cameras, students’ collaborative behaviors outside of digital media can be documented. We would like to use this data to provide near-real time feedback to students and teacher, through various forms of visualization.

What were your main research findings from the IES development project?

After using EcoMUVE, students showed gains in learning of science content, and also improvements in their attitudes towards science, particularly in the beliefs they were capable and interested in being scientists. Teachers felt that the curriculum was feasible, well-aligned with standards, and supported student engagement and learning of science content, complex causality, and inquiry, and had multiple advantages over a similar non-MUVE curriculum. A study that looked at student motivation found that, while at first students were most enthusiastic about the 3D virtual world and game-like environment, over time their engagement centered on the inquiry-based pedagogy and the collaborative problem-solving.  Gains were also found in students’ complex causal reasoning about non-obvious causes; distant drivers of ecosystems dynamics and the system parameters; and processes, steady states and change over time.

How has the EcoMUVE project proceeded in recent years since the IES research project ended?  

Beginning in May, 2012, we’ve been pleased to be able to offer a standalone version of the EcoMUVE software for download through a free license from Harvard University. As of January, 2015, over 1,200 users have registered with the website. The EcoMUVE project receives e-mail inquiries almost every week from educators who are interested in the curriculum. In some cases, whole districts have adopted the EcoMUVE curriculum, including Cambridge, MA, and Peoria, AZ.

Internationally, researchers at the University of Hong Kong have been working with Harvard University to use EcoMUVE for professional development, to help teachers understand how to use scientific investigations as learning activities for students. Other collaborators include Sydney University, and Aalborg University in Copenhagen.

Looking ahead, what does the future hold for EcoMUVE?

We continue to make EcoMUVE available for download from our new website, http://ecolearn.gse.harvard.edu. We have been extending our research to develop EcoMOBILE, an extension of the EcoMUVE curriculum that blends immersive virtual environments with the use of mobile technologies during field trips to real ecosystems for observations and data collection. EcoMOBILE is funded by the National Science Foundation (NSF) and Qualcomm’s Wireless Reach Initiative. We have also just started a new research project, EcoXPT, also funded through NSF, designed to work alongside EcoMUVE to support experiment-based inquiry in immersive virtual environments.

Questions? Comments? Please send them to us at IESResearch@ed.gov.

About the Interviewees

Shari J. Metcalf is Project Director of the EcoMUVE project at the Harvard Graduate School of Education. She holds SB and SM degrees from MIT, and a PhD from the University of Michigan, where she designed and developed Model-It, a software tool for students building models of dynamic systems. Her professional focus is the design of educational software tools, and in particular on using modeling, simulation, and virtual immersive environments to support inquiry-based science learning.

Chris Dede is the Timothy E. Wirth Professor in Learning Technologies at Harvard’s Graduate School of Education.  Chris was the Principal Investigator of the EcoMUVE project. His fields of scholarship include emerging technologies, policy, and leadership.  His research includes grants from NSF, IES, and the Gates Foundation to design and study immersive simulations, transformed social interactions, and online professional development.  

What is the Forum on Child and Family Statistics?

By Grace Kena

The Federal Interagency Forum on Child and Family Statistics, is a working group of Federal agencies that collect, analyze, and report data on issues related to the well-being of children and their families. The Forum on Child and Family Statistics’ mission is to promote coordination and collaboration among member agencies and to improve efforts to collect and report Federal data on children and families. This forum is unique in that it compiles key findings across many domains of children’s lives. 

The National Center for Education Statistics (NCES) has been involved with the Forum on Child and Family Statistics since the early stages of its development. Founded in 1994, the Forum on Child and Family Statistics was formally established by Executive Order No. 13045 in 1997. The Forum’s main activity is to produce the report, America's Children: Key National Indicators of Well-Being, which is a collection of national indicators of child well-being. Through the report, the Forum aims to improve the reporting of Federal data on children and families; make these data available in an easy-to-use, nontechnical format; and stimulate discussions among policymakers and the public, and between the statistical and policy communities.

Using Federal data, the America’s Children series presents a set of key indicators on aspects of children’s lives that measure their well-being and influence the likelihood that a child will become a well-educated, economically secure, productive, and healthy adult. While there are many, interrelated aspects of children’s well-being, America’s Children reports on seven major domains:  family and social environment, economic circumstances, health care, physical environment and safety, behavior, education, and health. Currently, 23 agencies contribute to the report, including NCES, the Environmental Protection Agency, the Bureau of Justice Statistics, the Bureau of Labor Statistics, the Economic Research Service, the U. S. Census Bureau, and the National Center for Health Statistics and the Substance Abuse and Mental Health Services Administration. 

The Forum on Child and Family Statistics has published the America’s Children report since 1997. Beginning in 2004, the Forum started producing a brief report, America’s Children in Brief: Key National Indicators of Well-Being in even-numbered years; the full report is still published in odd years. Although this shortened version of the report focuses on selected indicators, data for all indicators are updated on the website each year. In 2014, the Forum published a one-time, special issue report titled America’s Young Adults. In addition to producing reports, the Forum collaborates with partner and other organizations on a number of research projects and in supporting conferences, workshops, and policy seminars. Most recently, NCES experts participated in a day-long workshop on Measuring and Reporting Social-Emotional Development in Early Childhood. NCES experts also authored the 2013 special feature on the academic knowledge and skills of kindergarten students using data from the Early Childhood Longitudinal Study, Kindergarten Class of 2010–11 (ECLS-K: 2011).
 
The 2015 America’s Children report shows several improvements in children’s well-being. The number of babies born prematurely has continued to decline, and recently, the percentage of children with asthma has decreased. High school completion rates have increased, particularly for Hispanic students. On the other hand, some aspects have not improved. The percentage of children experiencing a major depressive episode has continued to increase over the past several years. 

This year’s report also contains a special feature on health care quality, which provides information on well-child and well-adolescent visits, preschool vision screenings, asthma management plans, and access to care.

Learn more about the Forum on Child and Family Statistics and its activities, and the 2015 America’s Children report at the website. Also, tune in to a recent podcast describing findings from the latest report.

Beyond Wikipedia: Reading and Researching Online

By Becky McGill-Wilkinson, NCER Program Officer

Gone are the days of library card catalogs and having to consult the 26-volume hardbound encyclopedia gathering dust on your parents’ bookshelf. Students these days have seemingly infinite information at the tips of their fingers. Most households in the U.S. have a computer, and most teachers report at least one computer in their classrooms. Research shows that the majority of high school students use the Internet to complete school assignments, and 71 percent of students use their laptop computers for school. In this changing world, it becomes more and more important to understand how reading and researching on the Internet are different from performing those tasks with books and other paper texts.

Don Leu and his team at the University of Connecticut have been examining this topic for several years. First on their agenda was studying whether reading online is the same as reading on paper. They discovered that students who are poor readers on paper may be good readers online, and students who are good readers on paper are not necessarily good readers online, suggesting that reading online requires some unique skills. Leu and his collaborators argue that reading online requires that students be able to: (1) use search engines; (2) choose appropriate search result; (3) judge whether the source can be trusted to be accurate and unbiased; and (4) consolidate information across multiple websites or online texts.

Of course, it’s not enough to understand the process of reading and researching online. As with any skill, some students are better at it than others, and as computers, tablets, and smart phones become more common, it becomes more and more necessary for students to hone their online reading and research skills if they are to succeed in college and career. Teachers need to be able to teach these skills, and teachers need to be able to identify when their students need extra help or practice. In 2005, Leu received a grant from NCER to study Internet use in adolescents at risk for dropping out of school, and developed an intervention to help teach seventh-grade students specific strategies to locate, evaluate, synthesize, and communicate information on the Internet.

Building on this earlier work, in a 2009 grant from NCER, Leu and his team set out to develop measures of online reading comprehension. The end result of this project is a set of Online Research and Comprehension Assessments (ORCAs) for use with seventh grade students. The team developed both a multiple choice version and a version that allows students to work in a simulated internet environment. In both versions, the student is tasked with answering a research question posed by a simulated peer, and must use a search engine, choose the appropriate search result, determine whether a source is trustworthy, and then tell their simulated peer about what they found. The ORCAs were tested with 2,700 students in two different states, and the researchers surveyed teachers and other practitioners to determine whether the ORCAs were usable.

Leu has been especially interested in thinking about how changing ideas about literacy may impact low-income students differently from middle- and high-income students. In a recently published paper, Leu shows that students who came from families earning approximately $100,000 per year were more than a year ahead of students whose families earn approximately $60,000 per year on online reading abilities as measured by the ORCAs. This study highlights the importance of considering the achievement gaps between high- and low-income students on a variety of domains, including those not typically measured by standardized tests, such as online reading comprehension.

The ORCAs are available online for free, as is a professional development module to help teachers learn to use it. 

Questions? Comments? Please email us at IESResearch@ed.gov.