Inside IES Research

Notes from NCER & NCSER

Learning from CTE Research Partnerships: How Michigan Built Trust with Researchers to Better Understand State Data

As part of our ongoing blog series aimed at increasing state research on career and technical education (CTE), Austin Estes, Senior Policy Associate at Advance CTE, and Corinne Alfeld, Research Analyst at the Institute of Education Sciences (IES), are conducting interviews with individuals who are part of successful CTE State Director research partnerships. The first interview was with Jill Kroll of the Michigan Department of Education and Dan Kreisman of Georgia State University (and Director of CTEx). [Note: this interview has been edited for length; you can find the full interview transcript here].

 

Jill Kroll Dan Kreisman
Michigan Department of Education Georgia State University

 

The first question we have is about the projects that you work on together: what were some of the research questions you came up with, and how did you come to settle on those research questions?

Jill – I first connected with Dan and with Brian Jacob at University of Michigan when I saw Brian present to our P-20 council about some research that he was doing connecting the wage record data for five community colleges. I was like “Gee, is there any way you can do something similar with the statewide secondary student data?” And he said it was possible. So I worked within our department procedures to find out how we could go about establishing a relationship that would allow this opportunity.

Dan – That led to a whole bunch of other discussions of things that we thought were interesting. So, to say that there is a set of research questions is not the way I view our relationship. We talk with folks in Jill’s office regularly to hear what questions are pressing for them, and then we try to help facilitate answering those and then see where those lead us. I think one of the important things is we try to think about where there are policy levers, so we want to say “If we answer this question, how can the state or the districts use that information to further their mission of providing CTE programming to students in Michigan?”

Jill – I’ve been really happy with the extent to which Dan and the research team have consistently focused on the “so what?” Rather than focusing on vague research questions of interest only to other researchers, they have emphasized their interest in doing research that has practical application, that can be used by educators in the field.

Could you share an example of how you’ve been able to use some of this evidence and research to change policy, or at least to shape your understanding on some decisions that you’re making at the state level?

Jill - When we were starting to work on our Perkins V [the Strengthening Career and Technical Education for the 21st Century Act] state plan, we had a short time to determine what we wanted to consider for our secondary indicator of program quality. Because Brian, Dan, and their students had been working with this data for so many years, they had the capacity to very quickly do the matching and

 come up with an approximation for us about what postsecondary credit attainment would look like, and what strengths and weaknesses they saw in the data. It would have been really difficult for our office, or even multiple state agencies, to have been able to work that quickly and give it the critical analysis that they did.

The other thing they did when we were making the decision for that indicator is look at the data that we had for work-based learning and tell us what could be done with it. What came out of that was that the data was not in any form that could be analyzed (text and PDFs). This was really revealing to our State Director Brian Pyles, and it led him to set a policy that we are going to build a consistent way of collecting data on work-based learning. So that is another piece where it influenced practice and policy. One of the most exciting and valuable things that I find about the partnership is that Dan and the other researchers have a lot more capacity to analyze the data in a way that we just don’t have the time to do. Sometimes we don’t have the expertise, and sometimes we just don’t look at the data in the same way.

Dan –And there’s a flip side that without their input, we often are looking at data and can’t make heads or tails of something. And we can get on the phone or write an email to someone over there and say “Hey we’re seeing this thing. Can you tell me what that means?” And they will come back with “Oh, the system changed” or “There was this one policy,” and “Here’s what you have to do to make it fit everything else.” And this happens all the time. We would be completely lost without this open channel that we have to their office.

I think it’s important not to dismiss the power of good descriptive work. Lots of times, the questions that states are grappling with can often be illuminated with some really careful and good descriptive work. You can say, “This is what we’re seeing, this is the big picture,” if you step back for a minute, and that information lots of times has been as valuable as the stuff we try to do that is more causally oriented in our research.

Jill – I agree, and I want to follow up on the whole issue of how important trust is. I cannot emphasize enough how important it is to me that Dan and the other researchers come to us with those questions, that they check in with us. That’s absolutely critical. Anyone who works with any kind of data knows that it’s just so complex. If you link tables wrong, or misunderstand a data field, you can come to a completely wrong decision. So that communication and that interaction and trust are key to accurate outcomes.

As you’re both looking ahead, what’s next on the agenda? What are some of the research questions and priorities you have for this partnership?

Dan – Number one is tracking students into the labor market. That’s our biggest and most outstanding question. And the degree to which CTE programs are preparing students for college and the labor market and careers. In terms of other projects, one of the things we’re interested in is technical assessments. We’re also part of a consortium of several states – that’s the CTEx group. We meet annually together, and that allows us to harmonize things across states to see how trends are similar, how enrollment rates work, all sorts of different questions across multiple states.

Jill – One of the things we’re talking about right now is that we don’t have, in an accessible form, data on access to a particular program. We know that career centers serve certain districts, but if someone asked, “If student A is going to Central High School, what programs do they have access to? we don’t have a good way of answering that at the moment. We’ve had a couple of discussions about how we can work together to build basically a dataset that clarifies that. That would be mutually beneficial and would take resources from both in order to do something like that.

Thinking back on this partnership, is there any advice you would give to other State Directors or CTE researchers?

Dan – Building a strong relationship is the first thing you have to do. And part of that is spending time face to face talking about questions, moving around ideas, looking at data together. We had the benefit of a long windup period. We spent at least a year just talking about questions and putting together data before we even started doing any analyses. We also had buy-in from Jill’s office up and down the line from folks who were doing the research to people who were in policymaking roles. And without all of that, none of this would even have been possible.

And the second part is to not downplay the value of just providing good information. A lot of us on the research side don’t realize how little time folks in the state offices have to take a step back and say, “What’s going on with our data? Let’s look at the big picture.” And one of the things we can provide them is just giving them that big picture and handing it to them in a digestible way. And doing that is the first step, is a really good way to start building that trust. They really see the value of what you can do early on. And then you can start to get into more difficult or longer-term questions.

Jill – The first advice I would give is: Do it! Partner with researchers. I can’t say enough positive about it. The second is: Follow department procedures and be transparent with department leadership. You know that windup might be really, really slow while you jog through the channels that you need to in your department to do things by the book, but I think it pays off in the long run.

My third one is: Be transparent and open with school districts. Share what you’re doing and invite their input. Anybody who works with state data would probably know, you’re always a little hesitant about what the public would think about this use of data. The way that Dan and the postdocs and graduate students have openly shared the work that they’ve done with our CTE administrators has really helped, in that I have not gotten any doubt from districts.

The full transcript can be accessed in Advance CTE’s Learning that Works Resource Center. Other blog posts in this series can be viewed here.

An Example of the Unquantifiable Effect of Research on Practice

At IES, we continue to think about ways to positively impact education practice through research. It is relatively straightforward to count and share the publications and research outputs produced by our grants. A bigger challenge is measuring the impact IES-funded research has on implementing evidence-based practice after the research project is complete. So we were thrilled when we received the following letter from Patrice Bain—a middle school teacher, author, education specialist, speaker, and consultant—who has worked closely with IES for many years.

I used to think of government agencies as impersonal bureaucracies often hidden from the public eye. One agency, IES, not only proved me wrong, it positively changed my life.

In 2006, Drs. Henry Roediger III and Mark McDaniel from Washington University in St. Louis obtained a grant from IES to research how students learn in an authentic classroom. The classroom where this research began was mine. And this is where the life-changing impact began.

The IES grant paid for technology to be used in my school’s classrooms and research assistants to aid our teachers. Heading up the research at my school was Pooja Agarwal, and this began a collaboration lasting over a decade.

In 2007, IES invited me to be the sole K-12 educator to co-author a practice guide. The large organization, to me, now had a face: Elizabeth Albro, who warmly welcomed me. I clearly recall sitting at a large table in Washington, DC, surrounded by my cognitive science superheroes: Drs. Hal Pashler, Mark McDaniel, Brian Bottge, Art Graesser, Janet Metcalfe, and Ken Koedinger. Each talked about important research that would impact learning in classrooms, and I knew my newly-expanded teaching repertoire now would be based in the science of learning. The final result of our meetings became the highly cited practice guide Organizing Instruction and Study to Improve Student Learning. In addition, information from this guide was featured on the website Doing What Works.

As Pooja and I delved into how retrieval, spacing, and metacognition played a role in student learning at my school, I was contacted by REL Mid-Atlantic, a part of IES that offers research-based professional development in Delaware, Washington, DC, New Jersey, Pennsylvania, and Maryland. Touting the benefits of teaching using the seven recommendations in Organizing Instruction and Study, I gave professional development presentations in the Mid-Atlantic regions with Drs. Hal Pashler, Ken Koedinger, and Nate Kornell.

Pooja and I also gave several presentations that included the research happening in my classroom. With IES funding, that research became a multi-year project involving over 1500 middle and high school students. With the passing of each year and research on learning becoming more defined, I was able to develop strategies utilizing retrieval, spacing, and metacognition. Pooja and I continued our collaboration. I was seeing success in the eyes of my students: I wasn’t just teaching content, I was teaching them how to learn.

A wealth of information on the science of learning seemed to be making a mark. Yet learning myths—those based on anecdotes and fads—were still circulating. To combat this, IES and NCER invited me to be on a working task group to tackle Neuromyths vs. Neurotruths. Once again, as I sat around a table in Washington, DC with learning superheroes, we explored how to begin to dispel prevalent myths of learning.

Because of IES and the opportunities I was given, I wanted to shout from a mountaintop that we can transform teaching. I’ve seen it. I’ve done it in my classroom. I realized a book started to brew within me. I’m not sure how the decision occurred, but I knew my collaboration with Pooja Agarwal was worthy of documenting. And so it began. We wanted to write a practical, evidence- and research-based book. Books had been written by cognitive scientists; books had been written by teachers. However, our book would be the first written by a cognitive scientist and an educator.

Powerful Teaching: Unleash the Science of Learning was released in June 2019. The ideas have resonated with educators across the globe. We are transforming education.

And it all started with IES approving a grant.

 

IES Celebrates Computer Science Education Week and Prepares for the 2020 ED Games Expo at the Kennedy Center

This week is Computer Science Education Week! The annual event encourages students from Kindergarten to Grade 12 to explore coding, with a focus on increasing representation among girls, women, and minorities. The event honors the life of computer scientist Grace Hopper, who broke the mold in the 1940s as a programming pioneer. Coding and computer science events are occurring in schools and communities around the country to celebrate the week.

This week is also a great time to highlight the computer science and engineering projects that are coming to Washington, DC for the 2020 ED Games Expo at the Kennedy Center on the evening of January 9, 2020. Developed with the support of the Institute of Education Sciences and other federal government offices, the projects provide different types of opportunities for students to learn and practice computer science and engineering skills with an eye toward examining complex real-world problems.

At the Expo, expect to explore the projects listed below.

  1. In CodeSpark Academy’s Story Mode, children learn the ABCs of computer science with a word-free approach by programming characters called The Foos to create their own interactive stories. In development with a 2019 ED/IES SBIR award.
  2. In VidCode, students manipulate digital media assets such as photos, audio, and graphics to create special effects in videos to learn about the coding. A teacher dashboard is being developed through a 2019 ED/IES SBIR award.
  3. Future Engineers uses its platform to conduct STEM challenges for Kindergarten to Grade 12 students. Developed with a 2017 ED/IES SBIR award.
  4. Fab@School Maker Studio allows students to design and build geometric constructions, pop-ups, and working machines using low-cost materials and tools from scissors to inexpensive 3-D printers and laser cutters. Developed with initial funding in 2010 by ED/IES SBIR.
  5. In DESCARTES, students use engineering design and then create 3-D print prototypes of boats, gliders, and other machines. Developed through a 2017 ED/IES SBIR award.
  6. In Ghost School, students learn programming and software development skills by creating games. In development with a 2018 Education Innovation and Research grant at ED.
  7. In Tami’s Tower, children practice basic engineering to help Tami, a golden lion tamarin, reach fruit on an overhanging branch by building a tower with blocks of geometric shapes. Developed by the Smithsonian Institution.
  8. In the Wright’s First Flight, students learn the basics of engineering a plane through hands-on and online activities, then get a firsthand look at what it looked (and felt) like to fly it through a virtual reality simulation. Developed by the Smithsonian Institution.
  9. In EDISON, students solve  engineering problems with gamified design software and simulate designs in virtual and augmented reality. In development with support from the National Science Foundation. 
  10. May’s Journey is a narrative puzzle game world where players use beginning programming skills to solve puzzles and help May find her friend and discover what is happening to her world. Developed with support from the National Science Foundation. 
  11. In FLEET, students engineer ships for a variety of naval missions, test their designs, gather data, and compete in nationwide naval engineering challenges. Developed with support from the U.S. Navy’s Office of Naval Research.
  12. Muzzy Lane Author is a platform for authoring learning games and simulations without requiring any programming skills. Developed in part with a Department of Defense award.

About the ED Games Expo: The ED Games Expo is the Institute’s and the Department of Education's annual public showcase and celebration of educational learning games as well as innovative forms of learning technologies for children and students in education and special education. At the Expo, attendees walk around the Terrace Level Galleries at the Kennedy Center to discover and demo more than 150 learning games and technologies, while meeting face-to-face with the developers. The Expo is free and open to the public. Attendees must RSVP online to gain entry. For more information, please email Edward.Metz@ed.gov.

Edward Metz is the program manager for the ED/IES Small Business Innovation Research program.

Christina Chhin is the program officer for the Science, Technology, Engineering, and Mathematics (STEM) Education research program.

Measuring Social and Emotional Learning in Schools

Social and emotional learning (SEL) has been embraced by many schools and districts around the country. Yet in the rush to adopt SEL practices and support student SEL competencies, educators often lack assessment tools that are valid, reliable, and easy to use.

 

Washoe County School District in Nevada has moved the needle on SEL assessment with support from an IES Researcher-Practitioner Partnership grant. The district partnered with the Collaborative for Academic, Social, and Emotional Learning (CASEL) to develop the Social and Emotional Competency Assessments (WCSD-SECAs)—free, open-source instruments that schools can use to measure SEL competencies of students in 5th through 12th grade.

Long and short versions of the SECA are available to download from the school district’s website, along with a bank of 138 items across 8 SEL domains that schools around the country can use to modify SECA assessments for their local context. The long-form version has been validated and aligned to the CASEL 5 SEL competency clusters and WCSD SEL standards (self-awareness, self-management, social awareness, relationship skills, and responsible decision making). The assessment is also available in Spanish, and the Metro Nashville Public schools offer the assessment in 8 additional languages.  

Students complete the long-form SECA as part of Washoe’s Annual Student Climate Survey by rating how easy or difficult SEL skills are for them. Under the Social Awareness domain, students respond to items such as “Knowing what people may be feeling by the look on their face” or “Learning from people with different opinions than me.” Under the Responsible Decision Making domain, students rate themselves on skills such as “Saying ‘no’ to a friend who wants to break the rules” and “Thinking of different ways to solve a problem.”

The SECA is one component of Washoe County’s larger School Climate Survey Project that is marking its 10th anniversary this year. Washoe provides district-level and school-level reports on school climate to support the district’s commitment to providing safe, caring, and engaging school environments for all of Washoe’s students and families.  

Written by Emily Doolittle, NCER’s Team Lead for Social Behavioral Research

IES Honors Dominic Gibson as Outstanding Predoctoral Fellow

Each year, IES recognizes an outstanding fellow from its Predoctoral Interdisciplinary Research Training Programs in the Education Sciences for academic accomplishments and contributions to education research. The 2018 winner, Dr. Dominic Gibson completed his Ph.D. in Developmental Psychology at the University of Chicago. He is currently a Postdoctoral Researcher at the University of Washington where he specializes in understanding how children learn words and mathematical concepts. In this blog, Dominic discusses his research and his experience as an IES fellow.  

What inspired you to focus your research on early mathematics?

So many everyday activities as well as many of humanity’s greatest achievements rely on math. Simple math becomes so second nature to us that it is often difficult for older students to conceptualize what it would be like to not have a basic understanding of numbers. But children take months and often years to learn the meanings of just the first few number words (one, two, three) and to learn how the counting procedure really works. Children’s acquisition of other math terms (angle, proportion, unit of measurement) is similarly marked by misconceptions and slow, difficult learning.  

Overcoming these learning challenges relies on an interesting mixture of uniquely human abilities (like language) and skills we share with other animals. Moreover, children’s ability to master early math concepts predicts their future academic success. Therefore, by studying how children learn about math, we can better understand the sources of humanity’s unique achievements and apply this knowledge to reducing early achievement gaps and maximizing our potential.

Based on your research, what advice would you give parents of pre-kindergartners on how to help their children develop math skills?

My biggest piece of advice is to talk to children about numbers and other basic math concepts. Children benefit from abundant language input in general, and “math talk” is no different. Even simply talking about different numbers of things seems to be particularly important for acquiring early math concepts. Numbers can be easily incorporated into a variety of activities, like taking a walk (“let’s count the birds we see”) or going to the grocery store (“how many oranges should we buy?”). Likewise, good jumping off points for using other types of early math talk such as relational language are activities like puzzles (“this one is too curvy to fit here—we need to find a piece with a flat edge”) and block building (“can you put this small block on top of the bigger one?”).

It also may be useful to note that even when a child can say a word, they may not fully understand what it means. For instance, two- to four-year-old children can often recite a portion of the count list (for example, the numbers one through ten) but if you ask them to find a certain number of items (“can you give me three blocks?”) they may struggle when asked for sets greater than two or three. Therefore, in addition to counting, it is important to connect number words to specific quantities (“look there are three ducks”). It may be especially helpful to connect counting to the value of a set (“let’s count the ducks—one, two, three—there are three!”).

My last piece of advice is to be careful about the types of messages we send our children about math. Many people experience “math anxiety,” and if we are not careful, children can pick up on these signals and become anxious about math themselves or internalize negative stereotypes about the types of people who are and are not good at math. Ensuring that children feel empowered to excel in math is an important ingredient for their success.

How has being an IES predoctoral fellow helped your development as a researcher?

The diverse group of people and perspectives I encountered as an IES predoctoral fellow made a huge impact on my development as a researcher. As an IES predoctoral fellow pursuing a degree in psychology, I met many students and faculty members who were interested in the same questions that interest me but who approached these questions from a variety of other disciplines, such as economics, public policy, and sociology. I also connected with networks of educators and policymakers outside of academia who alerted me to important issues that I may have missed if I had only worked within my own discipline. Through these experiences, I gained new tools for conducting my research and learned to avoid the types of blind spots that often develop when approaching a problem from a single perspective. In particular, I gained an appreciation for the challenges of translating basic science to educational practice and the number of interesting research questions that emerge when attempting to do this work.

Compiled by Katina Rae Stapleton, Education Research Analyst and Program Officer for the Predoctoral Interdisciplinary Research Training Programs in the Education Sciences, National Center for Education Research