Inside IES Research

Notes from NCER & NCSER

Working to Understand the Policy Process in the Development of Michigan’s Read by Grade Three Law

In recent decades, there has been an emphasis on quantitative, causal research in education policy. These methods are best suited for answering questions about the effects of a policy and whether it achieved its intended outcomes. While the question of, “Did it work?” remains critical, there is a need for research that also asks, “Why did it work? For whom? In what contexts?” To answer these types of questions, researchers must incorporate rigorous qualitative methods into their quantitative studies. Education research organizations like the Association for Education Finance and Policy have explicitly invited proposals using qualitative and mixed methodologies in an effort to elevate research addressing a range of critical education policy questions. Funding organizations, including IES, encourage applicants to incorporate qualitative methods into their research process. In this guest blog, Amy Cummings, Craig De Voto, and Katharine Strunk discuss how they are using qualitative methods in their evaluation of a state education policy.

 

In our IES-funded study, we use qualitative, survey, and administrative data to understand the implementation and impact of Michigan’s early literacy law—the Read by Grade Three Law. Like policies passed in 19 other states, the Read by Grade Three Law aims to improve K-3 student literacy skills and mandates retention for those who do not meet a predetermined benchmark on the state’s third-grade English language arts assessment. Although the retention component of these policies remain controversial, similar laws are under consideration in several other states, including Alaska, Kentucky, and New Mexico. Below are some of the ways that we have integrated qualitative methods in our evaluation study to better understand the policy process in the development of the Read by Grade Three Law.  

Collecting qualitative sources helped us understand how the policy came to be, thereby assisting in the structure of our data collection for examining the law’s implementation and subsequent effects. In our first working paper stemming from this study, we interviewed 24 state-level stakeholders (policymakers, state department of education officials, early literacy leaders) involved in the development of the law and coded state policy documents related to early literacy to assess the similarity between Michigan’s policy and those of other states. Understanding the various components of the Law and how they ended up in the policy led us to ensure that we asked educators about their perceptions and implementation of these components in surveys that are also part of our evaluation. For example, because our interviews made clear the extent to which the inclusion of the retention component of the Law was controversial during its development, we included questions in the survey to assess educators’ perceptions and intended implementation of this component of the Law. In addition, it confirmed the importance of our plan to use state administrative retention and assessment data to evaluate the effect of retention on student literacy outcomes.

To trace the Read by Grade Three Law’s conception, development, and passage, we analyzed these qualitative data using two theories of the policy process: Multiple Streams Framework (MSF) and policy transfer. MSF says that policy issues emerge on government agendas through three streams: problem, policy, and political. When these streams join, a policy window is opened during which there is a greater opportunity for passing legislation. Meanwhile, policy transfer highlights how policies enacted in one place are often used in the development of policies in another.

We found that events in the problem and political streams created conditions ripe for the passage of an early literacy policy in Michigan:

  • A national sentiment around improving early literacy, including a retention-based third-grade literacy policy model that had been deemed successful in Florida
  • A pressing problem took shape, as evidenced by the state’s consistently below average fourth-grade reading scores on the National Assessment of Educational Progress
  • A court case addressing persistently low-test scores in a Detroit-area district
  • Previous attempts by the state to improve early literacy

As a result of these events, policy entrepreneurs—those willing to invest resources to get their preferred policy passed—took advantage of political conditions in the state and worked with policymakers to advance a retention-based third-grade literacy policy model. The figure below illustrates interviewee accounts of the Read by Grade Three Law’s development. Our policy document analysis further reveals that Michigan’s and Florida’s policies are very similar, only diverging on nine of the 50 elements on which we coded.

 

 

Although this study focuses on the development and passage of Michigan’s early literacy law, our findings highlight both practical and theoretical elements of the policy process that can be useful to researchers and policymakers. To this end, we show how particular conditions, coupled by policy entrepreneurs, spurred Michigan’s consideration of such a policy. It is conceivable that many state education policies beyond early literacy have taken shape under similar circumstances: a national sentiment combined with influential brokers outside government. In this way, our mixed-methods study provides a practical model of what elements might manifest to enact policy change more broadly.

From a theoretical standpoint, this research also extends our understanding of the policy process by showing that MSF and the theory of policy transfer can work together. We learned that policy entrepreneurs can play a vital role in transferring policy from one place to another by capitalizing on conditions in a target location and coming with a specific policy proposal at the ready.

There is, of course, more to be learned about the intersection between different theories of the policy process, as well as how external organizations as opposed to individuals operate as policy entrepreneurs. As the number of education advocacy organizations continues to grow and these groups become increasingly active in shaping policy, this will be an exciting avenue for researchers to continue to explore.

This study is just one example of how qualitative research can be used in education policy research and shows how engaging in such work can be both practically and theoretically valuable. The most comprehensive evaluations will use different methodologies in concert with one another to understand education policies, because ultimately, how policies are conceptualized and developed has important implications for their effectiveness.


Amy Cummings is an education policy PhD student and graduate research assistant at the Education Policy Innovation Collaborative (EPIC) at Michigan State University (MSU).

Craig De Voto is a visiting research assistant professor in the Learning Sciences Research Institute at the University of Illinois at Chicago and an EPIC affiliated researcher.

Katharine O. Strunk is the faculty director of EPIC, the Clifford E. Erickson Distinguished Chair in Education, and a professor of education policy and by courtesy economics at MSU.

Effective Postsecondary Interventions: Early Colleges Combine High School and College to Benefit Students

Across a set of research grant programs, IES generates knowledge of how to increase students’ access to, progress through, and completion of postsecondary credentials and degrees. Funded projects develop and test a range of interventions from state-level policies to classroom practices, with an emphasis on strategies that promote college attainment for students historically underrepresented in postsecondary education. This new blog series called Effective Postsecondary Interventions highlights interventions with evidence of effectiveness generated through IES-funded research.

 

The Early College High School Model

The Early College High School (ECHS) model addresses barriers to college attainment commonly experienced by students historically underrepresented in higher education. Students from low-income families and minoritized racial and ethnic groups often attend high schools that lack rigorous pre-college courses, strong support for college enrollment, and established connections to colleges and universities. For those students, cost is an additional barrier to enrollment and persistence in college. The ECHS model addresses these barriers by combining secondary and postsecondary instruction within the same school, prioritizing college-preparatory high school courses, offering opportunities to enroll in college courses while in high school, and providing comprehensive supports for academic and social-emotional development—all at little or no cost to the students. Although concurrent enrollment in high school and college courses (dual enrollment) is a core component of the model, the ECHS model is more expansive than dual enrollment because it includes a broader set of intervention components and has a clear equity objective.

All early colleges reflect these four design principles:

  • Enrollment of students historically underrepresented in higher education
  • Partnerships including a local education agency, a higher education institution, and the surrounding community
  • An integrated program of secondary and postsecondary education with the goal of all students earning 1 to 2 years of college credit prior to high school graduation
  • A comprehensive support system for students to develop academic skills as well as social and behavioral skills.

 

Attending Early Colleges Increases Postsecondary Attainment

Four IES-funded projects have evaluated impacts of the ECHS model. Prior studies within these projects found that significantly larger percentages of early college students completed a college preparatory course of study during high school and enrolled in postsecondary education within six years of entering high school. The two most recent projects assess postsecondary attainment:

In addition, early college students earned associate degrees at rates that exceeded their counterparts in traditional high schools by 22% and 18%, respectively, while earning bachelor’s degrees at equal or higher rates. These impacts are substantial, and especially noteworthy because both evaluations studied early colleges across a range of settings. Moreover, the impacts are similar for different student subgroups, regardless of gender, race/ethnicity, or family income.

 

Why have early colleges been so effective?

Early colleges set high expectations, provide high-quality interactions between staff and students, and encourage college access and success for all students. Several studies confirm that early colleges substantially improve high school experiences and outcomes. Students in North Carolina early colleges reported higher expectations, more rigorous and relevant instruction, stronger academic and social supports, and better relationships with teachers than their counterparts in other high schools. Early college students in AIR’s five-state sample reported significantly higher levels of college-going culture and instructor support than their counterparts in traditional high schools. The combination of high expectations and supportive relationships promotes better outcomes for early college students beginning in ninth grade (compared with students in other high schools). For instance, early college students are more likely to persist in college-preparatory math courses, attain a significant number of college credits during high school, and graduate from high school. Importantly, these positive results hold for students from all racial and ethnic groups, including students who enter high school at low levels of math proficiency.


For more information about the studies, the SERVE Center at the University of North Carolina at Greensboro has posted a brief of impact findings from their evaluation of North Carolina early colleges. The American Institutes of Research has posted a brief of impact findings from their five-state evaluation of early colleges.  

Written by James Benson (James.Benson@ed.gov), a Program Officer for Postsecondary Education within NCER’s Policy and Systems Division.

Representation Matters: Exploring the Role of Gender and Race on Educational Outcomes

This year, Inside IES Research is publishing a series of interviews showcasing a diverse group of IES-funded education researchers that are making significant contributions to education research, policy, and practice.

 

The process of education transmits sociocultural values to learners in addition to information and knowledge. How individuals are represented in curricula and instructional materials can teach students about their place in the world. This can either perpetuate existing systemic inequalities or, conversely, provide a crucial counternarrative to them. With an exploration grant from IES, Anjali Adukia (University of Chicago) and Alex Eble (Teachers College, Columbia University) are exploring how  representation and messages about gender and race in elementary school books may influence student’s education outcomes over time. The researchers will develop and use machine-learning tools that leverage text and image analysis techniques to identify gender- and race-based messages in commonly used elementary-school books.

 

Interview with Anjali Adukia, University of Chicago

Tell us how your research contributes to a better understanding of the importance of diversity, equity, and/or inclusion in education.

In my work, I seek to understand how to reduce inequalities such that children from historically (or contemporaneously) marginalized backgrounds have equal opportunities to fully develop their potential. I examine factors that motivate and shape behavior, preferences, and educational decision-making, with a particular focus on early-life influences. Proceeding from the notion that children are less likely to be able to focus on learning until their basic needs are met, my research uses both econometric methods and qualitative approaches to understand the specific roles different basic needs play in making these decisions. My research, for example, has explored the role of safety and health (sanitation, violence), economic security (road construction, workfare), justice (restorative practices), and representation (children’s books), particularly for marginalized groups.

 

As a woman and a minority, how has your background and experiences shaped your career?

My research is informed and influenced by my own experiences. When I was a child, I never understood why there weren’t more characters that looked like me or when there were, why they had such limited storylines. For me personally, the motivation underlying our IES-funded project was borne out of my lived experience of always searching for content that reflected who I was. I think of representation as a fundamental need: if you don’t see yourself represented in the world around you, it can limit what you see as your potential; and similarly, if you don’t see others represented, it can limit what you see as their potential; and if you only see certain people represented, then this shapes your subconscious defaults.

It was a real watershed moment when I realized that academia allowed me to pursue many of my larger goals in life, in which I hope to meaningfully improve access to opportunities and outcomes for children – especially those from disadvantaged backgrounds. I hope to accomplish this in various ways, paying forward the many kindnesses generously given to me by: (1) producing rigorous policy-relevant evidence that expands our understanding of big questions and opens new avenues for inquiry; (2) translating my research such that it helps inform policymakers and practitioners in the design of school policies and practices; (3) understanding issues with a depth and sophistication that comes from “on-the-ground” insights, knowledge cultivated in multiple disciplines using different methodologies, introspection, humility, and courage; (4) directly working with government agencies, non-profit organizations, and community groups to positively inform policies; (5) contributing to the larger public discourse; and (6) by training and advising students to have the fortitude to ask hard questions, to be able to defend different perspectives on issues, to learn that knowledge brings more questions than answers, and to be willing to take risks and fail (and in the process, I will certainly learn more from them than they will ever be able to learn from me).

 

What has been the biggest challenge you encountered and how did you overcome the challenge?

Life is always filled with challenges, but one challenge starting from when I was young was to feel comfortable in my own skin and to find legitimacy in my own voice. I grew up as an Indian-American daughter of Hindu immigrants in a rural, predominantly white and Christian setting. I was different from the other kids and did not always feel like I fit in. I remember literally trying to erase my skin hoping that it would make it lighter. I found the helpers, as my parents (and Mr. Rogers) would suggest, and tried to focus on the voices that lifted me up – my family, teachers, other mentors, those friends who loved me no matter my differences. My mother always told me to find the kindness, the good, the love in people; to find the common ground and to embrace and learn from the differences. I surrounded myself with love, focusing on what I had and on what I could do rather than what society was telling me I couldn’t do. I turned to concentrating on things that mattered to me, that drove me. I don’t think there is a single challenge in life that I overcame alone. I have been very lucky, and I am deeply grateful for the many gifts in my life, the many loved ones – family, friends, colleagues, mentors, healthcare workers – who have lifted me up, and the opportunities that came my way.

 

How can the broader education research community better support the needs of underrepresented, minority scholars?

The notion of what is considered to be an important question is often driven by the senior scholars in a field, for example, the people considered to be “giants.” Demographically, this small set of leading scholars has historically consisted of people from the most highly represented groups (particularly in economics). And because the field is thus shaped mainly by researchers from a “dominant” group background, the key questions being pursued may not always reflect the experiences or concerns of people from underrepresented backgrounds. Education research has pockets where these different perspectives are being considered, but it can continue to evolve by becoming more open to approaches thought to be less traditional or to questions not typically asked (or asked from a different point of view). Expanding the notion of what is considered important, rigorous research can be difficult and cause growing pains, but it will help expand our knowledge to incorporate more voices.

 

What advice would you give to emerging scholars from underrepresented, minority backgrounds that are pursuing a career in education research?

Keep a journal of questions that arise and topics that pique your curiosity and interest. Soon, you will find questions in the fabric of everyday life, and you will start to articulate the wonder you see in the world around you and what inspires you to action, to understand the universe further. I find that when I return to past writings and journal entries, I am reminded of questions that have ignited my fires and see some of the common themes that emerge over time. Find your voice and know that your voice and views will grow and evolve over time. There are so many interesting and important questions one can pursue. Most importantly, you have to be true to yourself, your own truth. Find circles of trust in which you can be vulnerable. Draw strength from your struggle. There is deep truth and knowledge within you.

 


Dr. Anjali Adukia is an Assistant Professor at the University of Chicago Harris School of Public Policy and the College.

This interview was produced by Christina Chhin (Christina.Chhin@ed.gov), Program Officer, National Center for Education Research.

 

Building Knowledge About Adult Education

Adult education programs aim to support the millions of American adults who wish to strengthen their basic skills, earn a high school degree or equivalent, or become U.S. citizens. During the National Adult Education Family and Literacy Week (September 20-26, 2020), IES is highlighting the ongoing research it supports to help expand our knowledge of and innovations for the adult education system and the communities they support.

Roughly 30 million U.S. adults may lack a high school degree or equivalent. And according to an assessment conducted in 2017, approximately 30 percent of adults may have very low numeracy skills, and approximately 20 percent may have very low literacy skills. Adults with low levels of academic attainment or low basic skills may face barriers to full economic or civic engagement.

Adult education programs aim to help such adults. These programs often run on lean budgets and need to meet the needs of a highly diverse population with a multitude of learning goals.

In order to help understand and improve the ability of the adult education system to provide services, IES researchers have been conducting various research projects in partnership with adult education providers.

For example, Career Pathways Programming for Lower-Skilled Adults and Immigrants conducted mixed-methods research in partnership with Chicago, Houston, and Miami. This work helped each city understand what types of career pathways adult education programs were offering and who was participating in such programing. This type of information helps programs, cities, and the system more broadly understand and adjust to the needs of learners and communities.

The New York State Literacy Zone Researcher-Practitioner Partnership focused on improving the ability of case managers to help adult learners leverage wrap-around services and access and succeed in adult education and training programs. This project helped develop tools and training for case managers and conduct an exploratory pilot study of these tools to see if they predicted learner outcomes, such as persistence in a program.

The Georgia Partnership for Adult Education and Research (GPAER) is a collaboration among researchers at Georgia State University and leadership at the Georgia Office of Adult Education: Technical College System of Georgia to help understand adult literacy programs across the state. This ongoing work is conducting mixed-methods studies to understand program features, learner characteristics, and indicators of beneficial learner outcomes.

There is still much to learn to help improve the ability of programs to find and support adult learners. IES encourages additional research to further help us understand the landscape of adult education, the needs and interests of adult learners and their instructors, and the outcomes and impacts of improving adult basic skills.


For more information about adult education research at the National Center of Education Research, contact Dr. Meredith Larson.