IES Blog

Institute of Education Sciences

See How IES is Supporting Technology-Delivered Assessments

For decades, student assessments have looked the same: multiple-choice or short-answer questions administered with pencil and paper, with all students receiving a common group of questions. Today, innovations in assessment design, greater understanding in the learning sciences, and new technology have all contributed to the way that assessments are administered and taken, and how the resulting information is shared with teachers, students, and families.

Since 2002, the Institute of Education Sciences (IES) has made more than 100 awards for the development of new assessments that are driven and delivered through technology.  The awards were made to a mix of academic researchers, entrepreneurial firms, and larger education research organizations. All of the projects included a rigorous research and development process, with studies to validate that assessments are measuring what is intended and pilots to test the promise of the technologies for improving student learning outcomes.

To highlight some of the technology-delivered assessments, IES has created YouTube Playlists that feature 57 videos in seven areas:

The assessments highlighted are delivered via mobile apps or through web-based computers and administered for different purposes. Some are diagnostic assessments used to screen students at the start of a new unit or year to identify areas where students struggle or areas to target with intervention. Many serve as performance assessments to determine how well students analyze information and draw conclusions when engaging in complex scenario-based activities. Others are summative assessments used to measure student performance during and at the end of the school year. Many also include a formative assessment component that adjusts based on the level of performance, and are designed to provide feedback and cues to students to inform the learning process.

A good number of the assessments are administered as simulations, games, scenarios, and puzzles, allowing for complex challenges where students can demonstrate mastery of knowledge and skills. Several enable new opportunities for assessment through the application of technological advances, such as natural language processing and machine-learning, read-aloud stories, fast-paced tasks that require students to respond, and speech recognition programs. Many save classroom time because the assessments are self-administered, and teachers benefit from the automatic grading as students go.  Most of the assessments also provide teachers information to guide practice through data dashboards or generated reports. Several of the assessments are already being used in school around the country.

Below are highlights from each of the playlists. It is important to note that none of assessments highlighted are wide enough in scope or configured to measure the full depth and breadth of State learning standards. Therefore, they are not sufficient to replace statewide summative assessments used for accountability and reporting purposes.  Collectively, however, the examples highlight the promise of technology-delivered assessments to improve and expand on existing approaches for measuring student learning and social and emotional skills, and for informing teacher instruction.  

Mathematics and Science

ASSISTments is web-based mathematics platform that assesses and then provides immediate feedback to students in grades 3-12, and generates teacher reports use to inform instruction. 

SimScientists is a simulation platform that formatively and summatively assesses science inquiry skills and knowledge aligned to middle school Next Generation Science Standards. 

Reading and Writing

RAPID is an adaptive literacy diagnostic and summative assessment system for students in Kindergarten through grade 12. 

Revision Assistant provides automated sentence-level in-line feedback to students during writing tasks aligned to Common Core State Standards. 

Social and Emotional Development

VESIP is a web-based simulated environment that measures the ability of students in grade 3-7 to interpret social cues which research demonstrates are needed to resolve conflicts. 

Early Learning

The School Readiness Curriculum Based Measurement System provides universal screening, benchmarking, and progress monitoring in language, literacy, mathematics, and science, for students in Pre-K and Kindergarten students. 

English Learning

ONPAR assesses the science and mathematics content knowledge and skills of English- and Spanish- speaking students using hyperlinks and animations to make questions accessible to all students. 

Tools for Teacher Practice

CLASS 5.0 automatically analyzes classroom discourse (student and teacher talking during class) and provides reliable profiles to guide and optimize how teachers lead instruction. 

Students With Disabilities or At Risk for Disabilities

NumberShire is a game-based mathematics intervention for students with, or at risk for, disabilities in Kindergarten through Grade 2. The game embeds instructional supports such as providing explicit, systematic, and frequent instruction, goal setting, and allowing students to work at their own pace. 

AnimalWatchVi Suite is an iPad app covering pre-algebra mathematics for middle and high school students with visual impairments. The app includes accommodation tools such as problem narration, audio hints, braille, and tactile graphics to provide accessible assessment. 

Written by Edward Metz, ED/IES SBIR program manager and IES Education Technology topic program officer.

A Look at Private Schools and Homeschooling

By Dana Tofig, Communications Director, Institute of Education Sciences

Much of the data you will find on the NCES website is related to public schools. It makes sense because a majority of students do attend public schools and those schools are required to gather and report a lot of information. Still, NCES does collect a significant amount of information about non-public elementary and secondary schools and a more limited amount of information about homeschooling.

Two recently released NCES reports provide information about other types of educational programs that serve millions of students—private schools and homeschooling. 

Private Schools

Characteristics of Private Schools in the United States provides a first look at data from the 2013-14 Private School Universe Survey, which is conducted every two years to gather information about the schools that approximately 10 percent of elementary and secondary students attend. This report, released on Nov 1, provides a tremendous amount of information, such as the number, type, and religious affiliation of private schools, as well as data about enrollment and programs offered.

The report shows that there were 33,619 private schools in 2013-14, serving 4.6 million students. The majority of these schools—about 69 percent—had a religious affiliation and 68 percent were located either in cities or suburbs, rather than towns or rural areas.


Source: Characteristics of Private Schools in the United States: Results From the 2013-14 Private School Universe Survey: National Center for Education Statistics, November 2016


The new report also provides a look at the percent of seniors who graduate and the subsequent postsecondary enrollment of students in private schools and breaks that information down by a number of categories. In 2012-13, slightly more than a quarter (26 percent) of private schools had students in 12th grade, and the graduation rate at those schools was 97 percent. The graduation rate was highest (99 percent) in schools with 750 or more students and lowest (83 percent) in schools with fewer than 50 students.  

Of 2012-13 private school graduates, 65 percent attended a four-year college by fall 2013, but there was wide variance in that rate by school type and location. For instance, 85 percent of graduates who attended Catholic schools enrolled in college by fall 2013, while the percentage was lower for students who attended other religious private schools (63 percent) and nonsectarian schools (56 percent). The four-year college enrollment rate was higher in schools that were located in the city (69 percent) and suburbs (66 percent) and lower in schools in towns (61 percent) and rural areas (58 percent).

Homeschooling

Homeschooling in the United States: 2012 estimates the number and percentage of homeschooled students in the U.S. in 2012 and compares that with estimates from previous years (1999, 2003, and 2007). It also provides demographic characteristics of homeschoolers and information about the reasons parents chose to homeschool their children and where they get curricular materials. The data come from responses to the Parent and Family Involvement in Education Survey that is part of the National Household Education Survey Program.

The report shows that, in 2012, there were approximately 1.8 million students who were homeschooled, representing about 3.4 percent of all students, ages 5-17, enrolled in elementary or secondary grades. Since 1999, the percentage of students who are homeschooled has doubled, with significant increases seen between 1999 and 2003 and 2003 and 2007. 


* - Statistically adjusted

SOURCE: U.S. Department of Education, National Center for Education Statistics, Parent Survey of the National Household Education Surveys Program (NHES), 1999; Parent and Family Involvement in Education Survey of the NHES, 2003, 2007, and 2012


When asked why they chose to homeschool their children, 25 percent parents said the most important reason was concern about the environment at other schools, such as safety, drugs, or negative peer pressure. Other parents said the most important reasons were dissatisfaction with the academic instruction at other schools (19 percent) and a desire to provide religious instruction (17 percent). About 21 percent of parents said there were other reasons, such as family time, finances, travel, and distance.

The report also provides information about how parents accessed the curriculum and books they used for homeschooling. Non-retail website and homeschooling catalogs, providers, or specialists were the most reported sources at 77 percent each, followed by the public library (70 percent), and retail bookstores or other stores (69 percent). Other significant sources were education materials were publishers not affiliated with homeschooling (53 percent), homeschooling organization (45 percent), and church, synagogue, or other religious organization (38 percent). 

Changes in America’s Public School Facilities From 1998-99 to 2012-13

By Lauren Musu-Gillette

The physical condition of school facilities is an important element of the school experience in the United States. To ensure that school facilities provide optimal conditions for learning, districts may need to build new facilities or upgrade existing facilities. During the first decade of the 21st century, public school systems in the United States spent, on average, over $20 billion annually on school construction (Baker and Bernstein 2012).[i] A recently released NCES report examined changes in school facilities in the U.S. from 1998-99 to 2012-13.

How did the average age of schools’ main instructional buildings change from 1998-99 to 2012-13?

In school year 2012-13, schools’ main instructional buildings were 19 years old, which was older than the average age of 16 years in the 1998-99 school year.[ii] In addition, on average, large schools were newer than small schools (by 8 years) and medium-sized schools (by 5 years) in 2012-13.

How did dissatisfaction with schools’ environmental factors change from 1998-99 to 2012-13?

In the 1998-99 school year, ventilation was the factor rated as unsatisfactory for the highest percentage of public schools while lighting was the lowest. However, by the 2012-13 school year, the percentage of public schools for which ventilation was rated as unsatisfactory had dropped, and the percentage for lighting increased. In fact, lighting was the only environmental factor that was rated as unsatisfactory for a higher percentage of public schools in 2012-13 than in 1998-99.


SOURCE: National Center for Education Statistics, Changes in America’s Public School Facilities: From School Year 1998-99 to School Year 2012-13


Was there a difference in the percentage of schools that needed money for repairs, renovations, and modernizations from 1998-99 to 2012-13? What was the estimated cost of these projects?

The percentage of public schools that needed money for repairs, renovations, and modernizations to put onsite buildings in good overall condition was lower in 2012-13 than 1998-99 by 23 percentage points (53 vs. 76 percent). However, the average cost of these projects was estimated to be $1.4 million more per school in 2012–13 than in 1998–99, adjusting for inflation.

Was there a difference in the percentage of schools with plans for building improvements in the next 2 years from 1998-99 to 2012-13?

A lower percentage of public schools in the 2012–13 school year had plans for building improvements in the next 2 years, compared to 1998-99 (39 vs. 48 percent). However, approximately 39 percent of public schools in the 2012–13 school year had plans for major repairs, renovations, or replacements to at least one building feature in the next 2 years.


[i] Baker, L., and Bernstein, H. (2012). The Impact of School Buildings on Student Health and Performance: A Call for Research. New York: McGraw Hill Research Foundation. Retrieved March 10, 2015, from http://www.centerforgreenschools.org/sites/default/files/resource-files/McGrawHill_ImpactOnHealth.pdf.

[ii] The age of the school was defined based on the year of the most recent major renovation or the year of construction of the main instructional building if no renovation had occurred.

NCSER Investigators Receive Awards from the CEC’s Division of Early Childhood

In October, the Council for Exceptional Children’s Division for Early Childhood (DEC) honored recipients of the DEC Awards at their Annual International Conference on Young Children with Special Needs and Their Families. These awards are conferred upon individuals who are making a difference in the lives of young children with disabilities and their families. A number of NCSER-funded Principal Investigators (PIs) were among those honored by the DEC.

Kathleen Hebbeler (left) was one of two recipients of the Mary McEvoy Service to the Field Award, which recognizes an individual who has made significant national or international contributions to the field of early childhood special education. Dr. Hebbeler, of SRI International, has served as the PI on two NCSER-funded awards. She explored participation in and characteristics of early intervention services that predict child outcomes in kindergarten using data from the National Early Intervention Longitudinal Study. She also examined the reliability and validity of the Child Outcomes Summary Form, a tool used by many states in reporting annual child progress for the Individuals with Disabilities Education Act (IDEA) preschool programs.

Karin Lifter (center) was the recipient of the Merle B. Karnes Award for Service to the Division for Early Childhood. This award recognizes an individual who has made a significant contribution to DEC in areas of leadership, service, research, advocacy, or publications. With NCSER funding, Dr. Lifter, of Northeastern University, has been validating the Developmental Play Assessment, an instrument designed to generate a profile of a child’s skills in play for progress monitoring and instructional planning. 

Michaelene M. Ostrosky (right) was awarded the DEC Award for Mentoring, an honor that recognizes an individual who has provided significant guidance to the development of students and/or new practitioners in the field. This award highlights the importance of training and guiding the next generation of leaders in the field. Dr. Ostrosky, of the University of Illinois at Urbana-Champaign, served as PI on a project to evaluate the efficacy of Special Friends, a class-wide kindergarten program designed to improve the social outcomes of children with disabilities. She is currently serving as co-PI on a project that is developing a class-wide motor skills intervention for preschool children with developmental disabilities, called CHildren in Action: Motor Program for PreschoolerS (CHAMPPS).

Written by Amy Sussman, program officer, NCSER and Wendy Wei, program assistant, NCSER/NCER

The Scoop on Replication Research in Special Education

Replication research may not grab the headlines, but reproducing findings from previous studies is critical for advancing scientific knowledge. Some have raised concerns about whether we conduct a sufficient number of replication studies. This concern has drawn increased attention from scholars in a variety of fields, including special education.

Photo array, top left going clockwise: Therrien, Lemons, Cook, and Coyne

Several special education researchers explored this issue in a recent Special Series on Replication Research in Special Education in the journal, Remedial and Special Education. The articles describe replication concepts and issues, systematically review the state of replication research in special education, and provide recommendations for the field. One finding is that there may be more replication studies than it seems—but authors don’t call them replications.

Contributors to the special issue include Bryan Cook from the University of Hawaii, Michael Coyne from the University of Connecticut, and Bill Therrien from the University of Virginia, who served as guest editors, and Chris Lemons, from Peabody College of Vanderbilt University. They shared more about the special issue and their collective insights into replications in special education research.

(In photo array, top left going clockwise: Therrien, Lemons, Coyne, and Cook)

How did you become interested in replication work?

Replication is a core component of the scientific method. Despite this basic fact that we all learned in Research 101, it is pretty apparent that in practice, replication is often ignored. We noticed how much attention the lack of replication was starting to get in other fields and in the press and were particularly alarmed by recent work showing that replications often fail to reproduce original findings. This made us curious about the state and nature of replication in the field of special education.

What is the state of replication research in special education?

It depends on how you define replication and how you search for replication articles. When a narrow definition is used and you require the term “replication” to be in the article, the rate of replication doesn’t look too good. Using this method, Lemons et al. (2016) and Makel et al. (2016) reported that the rate of replication in special education is between 0.4 to 0.5%, meaning that out of all the articles published in our field, less than 1% are replications. We suspected that—for a number of reasons (e.g., perceptions that replications are difficult to publish, are less prestigious than novel studies, and are hostile attempts to disprove a colleague’s work)—researchers might be conducting replication studies but not referring to them as such. And, indeed it’s a different story when you use a broad definition and you do not require the term replication to be in the article. Cook et al. (2016) found that out of 83 intervention studies published in six non-categorical special education journals from 2013-2014, there were 26 (31%) that could be considered replications, though few authors described their studies that way. Therrien et al. (2016) selected eight intervention studies from 1999-2001 and determined whether subsequently published studies that cited the original investigations had replicated them. They found that six of the eight original studies had been replicated by a total of 39 different studies (though few of the replications identified themselves as such).

What were some other key findings across the review articles?

Additional findings indicated that: (a) most replications conducted in special education are conceptual (i.e., some aspects are the same as the original study, but some are different) as opposed to direct (i.e., as similar to the original study as possible), (b) the findings of the majority of replications in special education agreed with the findings of the original studies, and (c) most replications in the field are conducted by one or more authors involved in the original studies. In three of the four reviews, we found it was more likely for a replication to produce the same outcome if there was author overlap between the original and replication studies. This may be due to the challenges of replicating a study with the somewhat limited information provided in a manuscript. It also emphasizes the importance of having more than one research team independently replicate study findings.  

What are your recommendations for the field around replicating special education interventions?

The article by Coyne et al. (2016) describes initial recommendations for how to conceptualize and carry out replication research in a way that contributes to the evidence about effective practices for students with disabilities and the conditions under which they are more or less effective:

  • Many studies evaluate an approach that has previously been studied under different conditions. In this case, researchers should specify which aspects replicate previous research;
  • Conceptualize and report intervention research within a framework of systematic replications, or a continuum of conceptual replications ranging from those that are more closely aligned to the original study to those that are less aligned;
  • Design and conduct closely aligned replications that duplicate, as faithfully as possible, the features of previous studies.
  • Design and conduct less closely aligned replications that intentionally vary essential components of earlier studies (e.g., participants, setting, intervention features, outcome measures, and analyses); and
  • Interpret findings using a variety of methods, including statistical significance, directions of effects, and effect sizes. We also encourage the use of meta-analytic aggregation of effects across studies.

One example of a high-quality replication study is by Doabler et al. The authors conducted a closely aligned replication study of a Tier 2 kindergarten math intervention. In the design of their IES-funded project, the authors planned a priori to conduct a replication study that would vary on several dimensions, including geographical location, participant characteristics, and instructional context. We believe this is a nice model of designing, conducting, and reporting a replication study.

Ultimately, we need to conduct more replication studies, we need to call them replications, we need to better describe how they are alike and different from the original study, and we need to strive for replication by researchers not involved in the original study. It is this type of work that may increase the impact research has on practice, because it strengthens our understanding of whether, when, and where an intervention works.

By Katie Taylor, Program Officer, National Center for Special Education Research