Inside IES Research

Notes from NCER & NCSER

Fieldnotes: Reflections from an Adult Education Instructor on Research and Practice

Approximately 18 percent of US adults are at the lowest levels of literacy and nearly 30 percent are at the lowest levels of numeracy. The adult education system serves adults with low skills, but many education researchers know little about the students or the setting.  Recently, NCER convened a working group of adult education instructors, administrators, and researchers to discuss adult education’s research and dissemination needs.

Mr. Marcus Hall, an adult education instructor at the Community Learning Center and JEVS Human Services in Philadelphia, participated in this working group. He spoke with Meredith Larson, NCER program officer for adult education, about his experiences and interests in research. A copy of the working group meeting summary is available here.

Please describe your adult education classroom.

I once taught a 7-week course with students ranging from 18 to over 60 years old who had low literacy or math scores. I tried to contextualize instruction around their career interests and differentiate it to their learning needs. For example, some students were proficient readers but needed comprehension and math practice while others struggled with one or more of the basic components of reading. Somehow, I needed to help those learning phonics and those struggling with fluency while also challenging those ready for comprehension work. It’s hard to meet student needs in such a short time without teacher aides or adaptive technologies.

Why is research particularly important for adult education?

The challenges we face are monumental. Despite the large number of adults in need, adult education feels under-funded, under-staffed, and under-appreciated. Our students need complex, comprehensive, and well-rounded intervention, but we often have to make the most out of slightly targeted, inexpensive, and difficult-to-implement solutions. We need researchers to provide practical information and recommendations that we can use today to help adults learn and retain information.

Have you used research into your teaching?

Specifically, for reading instruction, I use techniques and activities built on evidence-based reading interventions. I start with tested diagnostic assessments to determine the needs of my students followed by strategies such as Collaborative Oral Reading or Repeated Reading exercises to support my students.

What topic during the meeting stood out to you?

The discussion about the workforce and professional development resonated with me. Many of our educators are part-time, come out of K-12, close to retirement, and may not have specific training for working with adults. They are asked to teach subjects they may not have any certification in, and their programs may not be able to provide the professional development they need. Just as we need supports for our learners, we need research to develop supports for us educators.

What additional research would you like to see?

Many of my students have had traumatic experiences that, when relived in the classroom, can cause them to disengage or struggle. I feel that understanding triggers and signs of discomfort has greatly enhanced my ability to help my students. Many educators want to leverage mental health approaches, like trauma-informed care, but we could use help learning how to integrate these strategies into instruction.

What do you hope researchers and educators keep in mind regarding one another?

It seems that researchers publish and promote their work to other researchers and then move to the next topic. This may be due to time constraints, publishing demands, or institutional requirements. I hope researchers take the time to come into our settings and observe us in action. I want researchers to work with us to help us understand and accept what is and isn’t working.

As for educators, we need to not try things and then stop using them when something unexpected occurs. At times, we revert back to what we know and are most comfortable with in the classroom. We educators can and must think critically about our norms and be ready and willing to enhance our practice with new information. 

Guiding Principles for Successful Data Sharing Agreements

Data sharing agreements are critical to conducting research in education. They allow researchers to access data collected by state or local education agencies to examine trends, determine the effectiveness of interventions, and support agencies in their efforts to use research-based evidence in decision-making.

Yet the process for obtaining data sharing agreements with state or local agencies can be challenging and often depends on the type of data involved, state and federal laws and regulations regarding data privacy, and specific agency policies. Some agencies have a research application process and review timeline available on their websites. Others may have a more informal process for establishing such agreements. In all instances, these agreements determine how a researcher can access, use, and analyze education agency data.

What are some guiding principles for successfully obtaining data sharing agreements? 

Over several years of managing projects that require data sharing agreements, I have learned a few key principles for success. While they may seem obvious, I have witnessed data sharing agreements fall apart because one or more of these principles were not met:

  • Conduct research on a topic that is a priority for the state or local education agency. Given the time and effort agencies invest in executing a data sharing agreement and preparing data, researchers should design studies that provide essential information to the agency on a significant topic. It can be helpful to communicate exactly how and when the findings will be shared with the agency and possible actions that may result from the study findings.
  • Identify a champion within the agency. Data sharing agreements are often reviewed by some combination of program staff, legal counsel, Institutional Review Board staff, and research or data office staff. An agency staff member who champions the study can help navigate the system for a timely review and address any internal questions about the study. That champion can also help the researcher work with the agency staff who will prepare the data.
  • Be flexible and responsive. Agencies have different requirements for reviewing data sharing agreements, preparing and transferring data, securely handling data, and destroying data upon study completion. A data sharing agreement often requires some back-and-forth to finalize the terms. Researchers need to be prepared to work with their own offices and staff to meet the needs of the agency.
  • Work closely with the data office to finalize data elements and preparation. Researchers should be able to specify the sample, timeframe, data elements, and whether they require unique identifiers to merge data from multiple files. I have found it beneficial to meet with the office(s) responsible for preparing the data files in order to confirm any assumptions about the format and definitions of data elements. If the study requires data from more than one office, I recommend having a joint call to ensure that the process for pulling the data is clear and feasible to all staff involved. For example, to link student and teacher data, it might be necessary to have a joint call with the office that manages assessment data and the office that manages employment data.
  • Strive to reduce the burden on the agency. Researchers should make the process of sharing data as simple and efficient as possible for agency staff. Strategies include providing a template for the data sharing agreement, determining methods to de-identify data prior to transferring it, and offering to have the agency send separate files that the researchers can link rather than preparing the file themselves.
  • Start early. Data sharing agreements take a lot of time. Start the process as soon as possible because it always takes longer than expected. I have seen agreements executed within a month while others can take up to a year. A clear, jointly developed timeline can help ensure that the work starts on time.

What resources are available on data sharing agreements?

If you are new to data sharing agreements or want to learn more about them, here are some helpful resources:

Written by Jacqueline Zweig, Ph.D., Research Scientist, Education Development Center. Dr. Zweig is the Principal Investigator on an IES-funded research grant, Impact of an Orientation Course on Online Students' Completion Rates, and this project relies on data sharing. 

Equity Through Innovation: New Models, Methods, and Instruments to Measure What Matters for Diverse Learners

In today’s diverse classrooms, it is both challenging and critical to gather accurate and meaningful information about student knowledge and skills. Certain populations present unique challenges in this regard – for example, English learners (ELs) often struggle on assessments delivered in English. On “typical” classroom and state assessments, it can be difficult to parse how much of an EL student’s performance stems from content knowledge, and how much from language learner status. This lack of clarity makes it harder to make informed decisions about what students need instructionally, and often results in ELs being excluded from challenging (or even typical) coursework.

Over the past several years, NCER has invested in several grants to design innovative assessments that will collect and deliver better information about what ELs know and can do across the PK-12 spectrum. This work is producing some exciting results and products.

  • Jason Anthony and his colleagues at the University of South Florida have developed the School Readiness Curriculum Based Measurement System (SR-CBMS), a collection of measures for English- and Spanish-speaking 3- to 5-year-old children. Over the course of two back-to-back Measurement projects, Dr. Anthony’s team co-developed and co-normed item banks in English and Spanish in 13 different domains covering language, math, and science. The assessments are intended for a variety of uses, including screening, benchmarking, progress monitoring, and evaluation. The team used item development and evaluation procedures designed to assure that both the English and Spanish tests are sociolinguistically appropriate for both monolingual and bilingual speakers.

 

  • Daryl Greenfield and his team at the University of Miami created Enfoque en Ciencia, a computerized-adaptive test (CAT) designed to assess Latino preschoolers’ science knowledge and skills. Enfoque en Ciencia is built on 400 Spanish-language items that cover three science content domains and eight science practices. The items were independently translated into four major Spanish dialects and reviewed by a team of bilingual experts and early childhood researchers to create a consensus translation that would be appropriate for 3 to 5 year olds. The assessment is delivered via touch screen and is equated with an English-language version of the same test, Lens on Science.

  • A University of Houston team led by David Francis is engaged in a project to study the factors that affect assessment of vocabulary knowledge among ELs in unintended ways. Using a variety of psychometric methods, this team explores data from the Word Generation Academic Vocabulary Test to identify features that affect item difficulty and explore whether these features operate similarly for current, former, as well as students who have never been classified as ELs. The team will also preview a set of test recommendations for improving the accuracy and reliability of extant vocabulary assessments.

 

  • Researchers led by Rebecca Kopriva at the University of Wisconsin recently completed work on a set of technology-based, classroom-embedded formative assessments intended to support and encourage teachers to teach more complex math and science to ELs. The assessments use multiple methods to reduce the overall language load typically associated with challenging content in middle school math and science. The tools use auto-scoring techniques and are capable of providing immediate feedback to students and teachers in the form of specific, individualized, data-driven guidance to improve instruction for ELs.

 

By leveraging technology, developing new item formats and scoring models, and expanding the linguistic repertoire students may access, these teams have found ways to allow ELs – and all students – to show what really matters: their academic content knowledge and skills.

 

Written by Molly Faulkner-Bond (former NCER program officer).

 

Weighted Student Funding Is On The Rise. Here’s What We Are Learning.

Weighted student funding (WSF) is a funding method that aims to allocate funding based on individual student needs. While large districts are increasingly using WSF systems, little research exists to assess their effectiveness. In this guest blog, Dr. Marguerite Roza, Georgetown University, discusses her team’s ongoing IES-funded research study that seeks to document and understand WSF designs and features as implemented in the field, and to gauge the extent to which WSF designs are associated with reducing achievement gaps. The study’s initial findings chart the WSF landscape across 19 U.S. school districts that used WSF in 2017-18.

Over the last two decades, dozens of big districts (including those in New York City, Boston, Denver, Houston, and Chicago) have shifted to using a weighted student formula to distribute some portion of their total budget. Instead of distributing resources via uniform staffing formulas, these districts use a student-based formula to allocate some fixed sum of dollars to schools for each student based on need (for example, allocations are typically higher for students with disabilities and students with limited English proficiency). The 2015 Every Student Succeeds Act (ESSA) authorized a WSF pilot, allowing up to 50 districts to incorporate key federal program dollars into a district’s formula.

As WSF systems now serve millions of K–12 students—and the number of WSF districts continues to grow—our research begins to document the range of these WSF formulas and gather details around how they are being implemented in school systems around the nation.

Why do districts adopt WSF?

Our study of school board and budget documentation indicates that nearly all districts identify equity (89%) and flexibility for school principals (79%) as a key rationale, with nearly half also citing a goal of transparency (49%). Interestingly, not one of the 19 districts cite “choice” (whereby families choose their school) as a driving factor in the rationale for using WSF even though much of the literature links choice and WSF. Despite the goal of transparency, only a third of the districts actually post their formulas online (like this posting from Houston ISD)—a finding that surprised us and them.  In fact, after we shared the finding with our study districts, several updated their online budget materials to include their formulas. Whether districts are meeting their goals of equity and flexibility will be more fully investigated in Phase 2 of the project.

Is there a typical WSF model that districts are using?

No. We find that there is no standard WSF: Each district has developed a home-grown formula and differences are substantial. On one end of the spectrum, Prince George’s County deploys only 20% of its total budget via its WSF, while Orleans Parish deploys 89%. Most districts deploy some 30-50% of their annual funds via their WSF formula, indicating that they are adopting a hybrid approach. They deploy the rest of their funds via staff allocations, program allocations, or in whatever ways they did before moving to WSF.

 

 

Districts define their “base” allocations differently, and no two districts use the same student weights. Most commonly, districts use grade level as a student weight category, but they do not agree on which level of schooling warranted the highest weight. Seven districts give their highest grade-level weight to elementary grades, four give it to middle school grades, and four give the highest weight to high schoolers.

Two thirds of districts use weights for students identified as English Language Learners (ELL) and as having disabilities, while half use weights for poverty. Even the size of the weights differs, with ELL weights ranging from 10% to 70%. Several districts use tiered weights.

We also found a range of unique weights designed within the districts for categories of locally identified need (for example, Boston uses a weight for students with interrupted formal learning, and Houston uses a weight for students who are refugees).

What other trends exist in districts implementing WSF?

We found that non-formula features and exemptions reflect local context. Small school subsidies, magnet allocations, and foundation amounts are common examples of non-formula features that several districts use. Some districts exempt some schools from the formula, grant weights for school types (vs student types), or fund selected staffing positions outside the formula. Districts seem to be layering their WSF formulas on top of long-standing allocations, like subsidies for small schools. Clearly, it is difficult for most districts to deploy a strict formula, and these exemptions or adjustments have the effect of mitigating the formula’s effects on some schools.

We also found that nearly all districts continue to use average salaries in their budgeting, likely limiting their goals for equity. In this practice, schools are charged for their teaching staff based on district-wide average salaries, not the actual salaries of teachers in the building. Districts in Boston and Denver have experimented with the use of real salaries for a subset of their schools (allowing for roughly one-third of their schools to budget and account for spending based on actual salaries).  Both the formula exceptions and this continued reliance on average salaries may be limiting the extent to which WSF is making progress on equity. Analysis in Phase 2 of the project will quantify the effects of these formula adjustments on spending.

What kinds of budget flexibilities do principals have?

With WSF, districts give principals flexibility in staffing, stipends, and contracts, but not base compensation. In virtually all WSF districts, principals had at least some flexibility in choosing the number and type of staff in their buildings and in awarding stipends. Interestingly, most principals had power to issue contracts with their funds, and half could carry over funds from one year to the next.  Despite these flexibilities, base teacher compensation is generally off limits for principals and continues to be controlled centrally.

How difficult is it for districts to design and implement their own versions of WSF?

Changing district allocations is hard work. At each point in our study, we find districts building “homegrown” approaches to WSF that reflect their own spending history and local context. We could see this as a practical transition of sorts between old and new allocation strategies, where district leaders straddle both the desires to change allocations and the pressures to keep allocations the way they are.

What are the next steps in this research?

Future analysis in this project will explore the degree to which WSF is delivering on the goal of increasing equity and outcomes for poor and at-risk students. However, the homegrown nature of WSF makes it tough to generalize about the WSF model or its effects. Undoubtedly, the variation poses problems for research. Clearly there’s no way to analyze WSF as a single model. Also challenging is that districts use different definitions (even on formula items such as the “base” and what constitutes a student weight). Perhaps this is unsurprising as there is no common training on the WSF model, and no prevailing terminology or budgeting procedures for district leaders to use in their work.

We see our study as a first step in a broader research agenda that will explore the scope and range of implementation of WSF in U.S. school districts and offer deeper analysis of the extent to which WSF is helping systems meet commonly cited goals of greater equity, flexibility and transparency. Meantime, we hope WSF systems and those considering shifting to WSF will be able to learn from this work and what peer systems are doing, perhaps with the ultimate effect of creating a common vocabulary for this financial model. 

 

 

Rethinking Connections Between Research and Practice in Education

IES-funded researchers from the Center for Research Use in Education (CRUE) at the University of Delaware recently learned that their article, “Rethinking Connections Between Research and Practice in Education: A Conceptual Framework,” was the 8th most-read article in any AERA journal in 2018!

In the article, the authors argue that “Recent efforts to improve the quality and availability of scientific research in education, coupled with increased expectations for the use of research in practice, demand new ways of thinking about connections between research and practice.”

Elizabeth Farley-Ripple, University of Delaware, CRUE
Elizabeth Farley-Ripple, University of Delaware, CRUE
 

Under the Every Student Succeeds Act (ESSA), education leaders must use evidence to inform their practice. However, the CRUE researchers argue that this simple framing “risks reducing evidence use to an administrative task rather than multiple activities constituting a political and social practice within a complex organizational process.” In fact, “the field lacks a comprehensive understanding of what evidence-based decision-making looks like in practice—for example, when is evidence brought into the decision-making process? Who engages with it? How is it understood in the local context? How often is it reviewed?”

CRUE’s research on this issue reveals gaps in the assumptions and perspectives of the research and practice communities, including the usefulness of research products; the nature and quality of research; the problems that research addresses; the structures, processes, and incentives for research production and use; and the specific relationships between the communities. They present a conceptual framework that highlights how these differences in understanding affect both the depth of research use and the depth of research production. Their article in Educational Researcher explains each of these and how they work together.

The framework shows that increasing education research use in practice is a complex, bidirectional issue, in which characteristics of both communities play a part: researchers need to produce work that is “decision-relevant,” and practitioners need to make decisions that are “research-attuned.” 

Written by Corinne Alfeld, NCER Program Officer