Minutes of Meeting: December 4-5, 2023

Date

December 4th, 2023

Location

Suite 206
Heritage Reporting Corp.
1220 L Street, N.W.
Washington, D.C. 20005-4018
(202) 628-4888
contracts@hrccourtreporters.com

The parties met remotely, pursuant to notice, at 10:00 a.m.

Proceedings

(10:00 a.m.)

DR. LEE: Good morning everyone. A special good morning to our colleagues on the East Coast. Not the East Coast, the West Coast. Sorry about that. I think we weren't thinking West Coast friendly when we set that time. So this will be really sort of our first full meeting where I'm hoping we'll be able to sort of scope out a sense of our vision as a Board, and how we want to organize ourselves as a Board. And I want to thank everyone, the various subcommittees that we established just in terms of prep for this meeting for the work you've done. I'm going to suggest, and if we can finalize this on our meeting tomorrow, that we establish formal subcommittees that will be able to meet in the interim outside of the Board meeting, so we knew we needed to do that for the committee working on the process for our hiring an Executive Director to support the Board, but the other committees need to be able to also meet outside of Board meetings. So I'm going to recommend that we do that. So with that, Ellie, can you call the roll?

MS. PELAEZ: Yes. Will do. Dr. Anaya?
DR. LEE: He will not be able to attend today.
MS. PELAEZ: That's right. That's right. Thank you. Ms. Hernandez-Legorreta?
MS. LEGORRETA: Present.
MS. PELAEZ: Dr. Darling-Hammond?
DR. DARLING-HAMMOND: Present.
MS. PELAEZ: Dr. Fuchs?
DR. FUCHS: Present.
MS. PELAEZ: Dr. Gandara?
DR. GANDARA: Present.
MS. PELAEZ: Mr. Guy?
(No response.)
MS. PELAEZ: Dr. Harper?
DR. HARPER: Present.
MS. PELAEZ: Mr. Hilliard?
(No response.)
MS. PELAEZ: Dr. Klasko?
(No response.)
MS. PELAEZ: Dr. Lee?
DR. LEE: Present.
MS. PELAEZ: Dr. Scott?
DR. SCOTT: Present.
MS. PELAEZ: Ms. Sullivan?
MS. SULLIVAN: Present.
MS. PELAEZ: Dr. Yoshikawa?
DR. YOSHIKAWA: Present.
DR. KLASKO: This is Dr. Klasko. My microphone was off, so I was —
MS. PELAEZ: Okay. Thank you. Got you.
DR. KLASKO: I'm here.
MS. PELAEZ: Dr. Schneider?
DR. SCHNEIDER: Here.
MS. PELAEZ: Dr. Carr?
DR. CARR: Here.
MS. PELAEZ: Dr. Albro?
DR. ALBRO: Present.
MS. PELAEZ: Dr. Soldner?
DR. SOLDNER: Present.
MS. PELAEZ: Dr. Jones?
DR. JONES: Present.
MS. PELAEZ: Mr. Robert Santos from the Census Bureau, or a designee?
(No response.)
MS. PELAEZ: Dr. Brett Miller?
DR. MILLER: Present.
MS. PELAEZ: Mr. Wiatrowski?
MR. WIATROWSKI: Here.
MS. PELAEZ: Thank you. Dr. Moore?
DR. MOORE: Here.

MS. PELAEZ: Thank you. Okay. We have a quorum.
DR. TURLEY: Ellie, you did not call out my name.
MS. PELAEZ: Oh, I apologize.
DR. TURLEY: This is Ruth Lopez Turley.
MS. PELAEZ: Oh, I'm sorry. I'm sorry, Ruth. I had you on my list. Sorry. Dr. Lopez Turley?
DR. TURLEY: Thank you.
MS. PELAEZ: Thank you.
DR. LEE: All right. Great. And so now we are going to call for approval of the Board minutes. I assume that you all received, and had an opportunity to review, so unless there's any discussion, I'd just call for the vote.
DR. HARPER: I move to approve.
DR. LEE: And a second?
MS. PELAEZ: Second.
DR. LEE: All approve aye.
(Chorus of ayes.)
DR. LEE: All opposed nay?
(No response.)
DR. LEE: All right. That's approved. So our first report is from the committee who will discuss a process for us responding to the National Academy, and other reports on IES. And that is Hiro, Ruth and Denisa. So I'm not sure who wants to take the lead; who is chairing.

DR. TURLEY: I will start. Thank you, Carol. So yes, this committee is made up of Hiro, Denisa, and myself, and I'm going to share — I don't know if you all already have this, but I'm going to share in the chat a link to — there we go — a link to a document, just a Google document, and you all should have access to it, but, of course, if you don't, please let me know. Did you all receive the link just now? Okay. Good. So what you'll see in this document obviously it's a draft document. We're very open to your input on what this process should look like, but what you'll see here is that we — as Carol just mentioned this committee met to try to develop a process that would ensure that our next Director of IES will develop a list of priorities that will be shared with the Board, and the Board will have an opportunity to review, and eventually approve that list of priorities. And there was a discussion among us about the use of that language, so we are open to hearing your input about that as well, based on the IES website that describes the duties of this Board. It does describe one of our duties as approving — it actually uses that language of approving. So that's why you see that language here as well. But we're open to discussion regarding the extent of the authority that we have to review and approve priorities, but right now this document is written with that language. So what you'll see here is we divided it into a couple of sections, or three sections. The first one is focusing on the process for evaluating the IES Director's priorities. The second and third are focused on the process for evaluating recommendations on any NASEM reports. The third one is for processing IES responses to those recommendations. So second and third are related in that way. So going to the first section, the process for evaluating the IES Director's priorities, you'll see that there is a required — the IES Director is required to post those priorities on the federal register for a mandatory 60-day comment period, but we would like for the IES Director to submit to us, the Board, prior to that period to submit to us those proposed priorities so that we have an opportunity to review them, and discuss them. And you'll notice that I just put a blank — blank weeks for now, because we are open to getting your input on what would be a reasonable amount of time. The goal here, of course, is we don't want to be a burden to the IES Director, but we definitely want to be a part of this process, and we would like for it to be a reasonable amount of time, so that we have an opportunity to review and discuss those priorities. And you'll see here in these steps that there is an opportunity for there to be a little bit of back and forth, that we can actually, you know, discuss them, but then if we recommend any changes to those priorities, or anything along those lines, that we can convey that to the IES Director, and then that person will have an opportunity to respond, and make revisions as-needed, and that we can do this a few times. The main thing I want to point out here is that in terms of the criteria that's recommending that are used to assess these priorities, there are mainly two that we have a starting point. First, alignment with the Education Sciences Reform Act, or ESRA, as well as alignment with recent NASEM report recommendations, which then brings us to the second and third pieces. But I'd be happy to pause here for any comments before I turn to the second and third sections, or would you like for me to continue? Carol, what do you think?

DR. LEE: I'd like to pause for a moment because I'd like to get some feedback from Mark on this. So there is, and, I believe, Mark had mentioned at some point at one of our meetings about some specific priorities that he had that I think he wanted the Board to comment on, and then there's the issue of the timing for the new Director, which I think starts in April. And so I'm a little bit unclear at this point in terms of any relationship at this point to existing priorities that Mark may have, as it relates to the timing of the appointment of the new Director.

DR. TURLEY: That's a great question, Carol, and I have to admit that this subcommittee assumed that we were referring to the next IES Director. So all of — yeah, these details, these timelines, I assumed that it was probably too late for the current Director, but I'm open to suggestions about that regarding the current Director.

DR. LEE: Mark, do you have any comments on that?
(Technical issues.)
DR. LEE: Mark, are you there?
DR. SCHNEIDER: Yeah. (Audio interference).
(Technical issues.)

DR. SCHNEIDER: So first of all, yes, I agree that most of these comments apply to the next Director, and that person should be in place hopefully the last week in April, or early May. The White House has asked several people to fill out screen aires, but as you know, that's all being driven by the White House Office of Personnel. So, where that stands, and who is actually being actively considered, is pretty closely held, but I know that the intention is to have someone in place so that there's not the kind of hold that existed in leadership in the — several times in the past. That's number one. Number two, with regard to existing priorities, as you know, there were only three people on the Board during — into the last week of the Trump Administration. There were a bunch of people, and none of them were ever finalized. So there exists on the website priorities that were established by John Easton (phonetic) in 2011, I think, and that went through the formal process of review and approval by the Board. When I came to the Director, I proposed different proprieties, but the Board was pretty much — I kept waiting for the Board to be constituted, and to turn into more than just three people, so I never submitted it to review by the three-person Board, which, of course, I could have done because the legislation just says that quorum is — would have been two of the three people. We could have had that meeting, but I didn't like the optics of it. So, if you look at the — if you look at what is there on the Federal Register, I mean, I think it's the right level of priorities for a statement like that. And I think you have the choice of either approving those, right, to take the place of the ones from 2011 or not, and just waiting until April when you start working with the — with whoever the new Director is who will hopefully be seated, as I said, at the end of April, or the beginning of May. I mean, I think it's clearer for you to approve these, but if you don't like them, then you have the opportunity to just wait until five months from now.

DR. LEE: So, thoughts from the Board the question of whether we want to visit the priorities that have been — that are already online, or simply wait until the appointment of the new Director.

DR. KLASKO: This is Steve. I mean, I just think that given the timing and proximity, having a new Director see what these were would have — then be able to look at what they now view as what they want to spend the next few years doing, I think is important as opposed to unnecessarily saddling them with what was already decided.

DR. TURLEY: Yeah. I'm not sure that it's worth our time to go through the effort of approving the current priorities under Mark Schneider given that he's got very little time left in this role. So I would prefer to just — and this is assuming, when I was talking about this proposed process that this would apply to the next Director.

DR. LEE: Okay. I have a sense, unless someone wants to speak otherwise, that the sense of the Board would be to wait until the new appointment to the new Director is appointed and begin that process at that time. Okay. We'll go onto the reports. So let me just say that part of my thinking is that the work of each one of these committees are ones that at this stage we're going to be sort of discussing, and giving feedback on process, but not that we're making a decision because there's so much information embedded just in the mere list of recommendations from the future of education report, but to get a general sense of the direction of those recommendations of that, and the other reports, and then to talk about a process and timeline through which we would respond.

DR. TURLEY: Correct. So, the idea for today is not to actually make any decisions about any specific recommendations, but just to go over the process. I do want to mention that I only talked about the process for evaluating the Director's priorities, but I also want to mention briefly the process for evaluating NASEM report recommendations that you can see listed here as well, that the idea would be to — for the Board to hopefully be alerted of any releases of new and relevant NASEM reports as soon as they become available, and then to have the Board review and discuss those reports at one of our Board meetings. One possibility that you see here is it could be the whole Board, or we could assign a reading committee depending on how we'd want to proceed here. But the idea would be that then the Board would have an opportunity to share reactions, advice, or anything — or timeline if a timeline isn't already mentioned in the actual report, to discuss these things with the IES director. And then the last part that I'll mention is just that there's an opportunity for the Board to evaluate the responses to any recommendations listed on NASEM reports. So that's also listed here. The main goal is, of course, just to have — to do this in a timely manner, to actually come up with a timeline so that — both a timeline for responding to any recommendations, but also a timeline for acting on those recommendations. But of course, there's an opportunity for the IES director to say, well, some of those recommendations are not feasible, or not — or maybe not necessary. So, there's an opportunity to say that. But either way, if the Director's response is this is a good recommendation, then we'd like to see a plan of action for those recommendations, and if the response is, no, this is not feasible, then we'd like to see an explanation for that. So yeah, so that's just the — just to wrap up the process of — you know, for the Board to be involved in both the Director's priorities, and NASEM report recommendations. So, with that, I do — as you all know, there are currently several reports that, I'm sure, many of us have already read, and I just want to highlight all the recommendations. I'm going to go over briefly the Future of Education Research at IES report that came out last year, and then Denisa and Hiro will go over briefly the other two reports that are listed here. But you can see here just for our purposes all the recommendations of this first report are listed in this document. I'm not going to read through all of them, but I will say that at a high level a few things that are worth highlighting are that the general aim here is to expand research topics, to expand research design, to expand research methods, expand even units of analysis, not just focusing on the student level. So those are the kinds of recommendations that you'll see listed here. The general idea here is they mention a need for expanding the type of research that IES funds. Another overarching goal here that stood out with these recommendations is, of course, an emphasis on equity, and really focusing on more transparency as well. Transparency in several ways, and transparency in terms of, you know, how decisions are made in terms of training, who is going through training, but also, of course, who is getting funding. So there is currently some level of transparency there, but the recommendations here are to — are requesting for more transparency. And two last things I'll mention with these recommendations is just more — a better process for both adding and removing topics for funding IES projects, and so that you just — the need for a process for adding and removing topics was highlighted. And the last thing that was mentioned that stood out with these recommendations is to just — a better process for partnering with — you know, between researchers and practitioners, and folks that are actually putting that research to use. Currently the process, the application process, requests letters of support from practitioners, but the recommendation here to move beyond that to actually have a partnership plan in place, perhaps a working group in place, things like that. So those are the overarching things that stood out in terms of these recommendations, and as Carol said, we don't necessarily have to make decisions right now about the specific recommendations, but we should definitely be aware of them, and have them in mind for perhaps the next time we meet, and especially once a new IES director is in place, to talk about how the new director will respond to these specific recommendations. And then if there are any comments about this first report, I'd be happy to hear them, but then I'll turn it over to Denisa to talk about the second one.

DR. LEE: Denisa, you go ahead.

DR. GANDARA: Okay. Thanks, Ruth, for that overview. So, the second report was a vision and roadmap for education statistics. That one was also published in 2022, and that one was specifically addressing NCES, the National Centers for Education Statistics. And here the recommendations — there's actually some alignment with the recommendations from the previous report that was just described, so some common themes across were strengthening the focus on equity in different ways. So, for the NCES report focusing more on data collection, and expanding research methods and design, examining — engaging in strategic planning. So that was really central to this other report for NCES and having regular evaluations of responsiveness of research to contemporary issues. And also similar to that previous report, engaging with stakeholders, improving transparency, and leveraging educational technology. And one of the other key recommendations was to strengthen state longitudinal data systems. So that's one that was specific to NCES in strengthening the capacity to link data across state systems. So again, the specific recommendations are listed on the document, but those are sort of the key themes, and the crosscutting themes across the first two reports. And I think Hiro is going to talk about the third report.

DR. YOSHIKAWA: Yeah. And I can jump in. Of course, there was one other 2022 report, which is on NAEP, and because that's under the authority of NAGB, we memorize it here. It's not under our Board technically speaking. So I think we did want to highlight one prior report because there has been some follow-up from NCES, and it seems like some of it is ongoing, which was the report in 2019 on monitoring educational equity, which basically had as its heart a proposal for indicators for educational equity that IES could collect at the national level, though it also had recommendations for both state and district level collection of these elements. So, we pasted the relevant table here, which is in the summary of the report, which is a list of educational equity indicators, and as you see, it is not just about equity of educational outcomes, but equity of educational inputs. And so that, I think, was the point of that report. And NCES in its summer — this last summer, its update, indicated that they were in the process of creating a composite indicator based on this report. So, they have already been in the process of implementing based on this 2019 report. But I just wanted to let our committee members know about this report, and that there were active activities to kind of follow-up on its recommendations. So that's why we included it here.

DR. LEE: All right. So, we're going to have next on the agenda an opportunity for the various Commissioners to respond to report on how they have responded to the recommendations in these various reports. I think that one of the recommendations that I'm going to make that we can resolve probably tomorrow because I think it will apply to the overall committee structure of the Board. And that is — so we were made aware that one of the subcommittees that we had identified in planning for this meeting was one around the process for the Board identifying, and hiring an executive director to work on behalf of the Board, and we were told because that involved personnel issues we needed to create a formal subcommittee that could meet outside of the public meeting of the Board. And I think we need to do the same for — we would create almost like standing committees coming out of this meeting. And I think this group would be one of those because resolving how we're going to work through responding to this range of recommendations is not something we can do within the context of a Board meeting. I also am going to recommend that this body be constituted into a formal subcommittee of the Board with the authority to meet outside of the Board meeting. At the same time, I think that because of the scope of this work, and the technical specificity of it, I'm imagining there's a lot of work that's going to need to be done in reconciling these recommendations with the existing priorities for not only IES, but the various centers. I also am imagining that there's some timing issues in terms of how any of these recommendations might relate to issues that are under consideration for the ESRA reauthorization. And in that case we probably need to be making some evaluations and recommendations relatively soon in terms of the timing for the process as that's unfolding, and that the committee that we would formally constitute as a formal committee of the Board with the capacity to meet outside of Board meetings, would also be able to recruit experts in various fields who might advise that committee in working through the details of how the Board should respond to his recommendations. So, any feedback around that structure? And I'm assuming that the Board has the authority in constituting standing committees to also in those internal meetings invite stakeholders who are able to give us technical advice. The other thing that I think is relevant here, and again, I'm not quite sure how to handle that, but, I think, it would come under the purview of this committee, is we've gotten two statements of recommendations for the Board to consider; one from the Institute for Higher Education Policy, which we sent out, and shared with everyone, and the other from the Alliance for Learning Innovation. And all of their recommendations, I think, again, are relevant to this question of priorities for IES that they want the Board to consider. And again, I think the details are such that I don't think we can respond to them today, but that it would be most efficient to assign them to this committee to look at in relationship to the NASEM reports, et cetera. So, any thoughts on that as a process for moving forward? I can't hear you, Ruth.

DR. TURLEY: Yeah. Thanks. Thank you for that recommendation. I really like the idea of sitting on subcommittees to keep the process moving bit faster than just depending on these big meetings with the entire Board. But just to be clear, when you said this committee were you talking about this submitted, or this —

DR. LEE: No. This subcommittee.

DR. TURLEY: This subcommittee. Okay.

DR. LEE: Right. My general sense is that to work through the details of the areas that we're going to need to address, we need working groups who are going to delve into them so that when we come to the Board you have recommendations that you've been able to vet internally through a process that would involve, again, other people who can advise you and the Board through the process.

DR. TURLEY: All right. That sounds good to me.

DR. LEE: Yeah. I think that's a wise way to proceed. Just to publicly acknowledge, I'm assuming they may be in the public presence that's here, but to assure the Institute of Higher Education policy, and the Alliance for Learning Innovation, that the Board has received, and will plan to formally take up their recommendations.

I think before we move to the response of the various Commissioners to the recommendations from the various reports, we probably need to think about a timeline for the work of this committee, and perhaps if anybody here has any sense of — and I know this is nothing definitive, but any sense of the timeline for the process of ESRA moving through the Congress, because this would inform, I think, how soon we need to have the committee able to do the work.

DR. SCHNEIDER: Carol, this is Mark. So could I just add a couple of things on that?

DR. LEE: Sure.

DR. SCHNEIDER: So we know that the FACA committee, so FACA as it governs your activities, and I would suggest — and it's very complicated, and, actually, when we talk to the General Counsel's Office they refer everything to a separate FACA group within the OGC. So, I would suggest that everything you do you would need to make sure that it's kosher with the FACA. That's number one. Number two, in terms of timeline for ESRA, well, I mean, it's both very tight, and very loose. So very tight in the sense that the Health Committee is actively working on its reauthorization proposal and depending on the day they're talking about moving to the next step within a week or so. So, they are actually moving very quickly. So, you need to understand that. But there will be multiple bites at that apple, right? And the ultimate bite is if the House acts on the ESRA reauthorization, then it has to go to committee, but you know this. I mean, so it's both a short timeline for help, and a much longer timeline that's much more problematic because ultimately the House has to pass a version of ESRA reauthorization, and then it would have to be a committee, to a conference committee, to iron out the final stages. There are a couple of opportunities still for input into the Health Committee's discussions that they're intending to get a bill out pretty soon.

DR. LEE: So a couple of responses. We certainly will follow through in terms of any legal constraints. This is one of the reasons that we moved to formulate the committee working on the process for this Board hiring its executive director to a standing committee that would be able to meet outside of the public meeting because it dealt with personnel issues. I am in communication with a staffer from the White House about the scope of our authority as a Board. We will certainly follow through, but I am very confident that we have the ability to constitute our working committees as we so choose. The other is this is related to the — which we'll get to toward the end of the meeting the process for our identifying an executive director; that the executive director, as I understand it, is a staff person that is hired directly by the Board to be our support person, and liaison for our activities both with IES and with the Congress. And I think we are going to try to have some more conversation tomorrow about the whole reauthorization process with ESRA, but, I think, we need to be able to create some line of communication with the Health Committee about our interest as a Board in weighing in on this process, so that we have some feedback relationship around the timing of our response. Any thoughts on what I've just shared?

DR. HARPER: That sounds good to me, Carol.

DR. LEE: In other words, there's no point in the White House, it seems to me, appointing a Board if we have absolutely no relationship to the process just in terms of getting some sort of feedback, but we will pursue that, and figure that process out. So now we're going to move on, which will be very helpful in getting reports from Nathan, from Liz, and from Peggy, and Matt, to how internally you all have responded to the recommendations in these various reports, and this will — since I think everything we — there's a direct transcript for this meeting, so that will be a helpful resource for this policy committee, to be able to review how internally you all have responded to the recommendations in the reports. So, we'll start with Nathan.

DR. JONES: Okay. And I'm actually going to hand it off immediately to Liz, who is going to get us started, and, I believe, Liz, you are going to be in control of the PowerPoint slides; is that correct?

DR. ALBRO: I am. Here, give me just a second here. Let me pull up the slides, and share, and we are going to view this in a shared fashion because the reality is that as we have been reading through this report it's clear that it will take more than a village. So actually, Nate, I just — so you guys will hear me calling on my colleagues as we switch without calling; we'll switch. If you all can't hear anything, please just let me know. I think the way we would like to do this is we would like to get through our deck, and then we will be happy to engage in conversation. So Nate, I actually think you are the first person.

DR. JONES: Oh, that is right. So just to get us started, our team thought it would be useful to just take a step back, and just at the onset provide just an overview of the structure of IES. This may or may not be helpful for folks. This may or may not be too much, or too little information, so to the extent that before we jump in, and start talking about our responses to these recommendations, if folks do feel like they need a bit more, we're happy to pause, and hear folks out. But one of the things that we wanted to just make clear, and lay out, is that IES does have four centers, so Liz and I, I think, were formerly tasked with making a response to the NASEM report, but to the extent that we thought it would be useful, we're also going to be calling in Matt Soldner from NCEE, and Anne Ricciuti from the Office of Science. And I think the reason for doing so is just to give a brief overview of the charge of the NASEM committee. So they were charged with four tasks. The first was to identify critical problems where new research is needed. The second was how to organize the request for applications. This is, I think, why we've brought in the Office of Science in our response. The third is to focus on new methods and approaches. And the fourth is new and different kinds of research training investments. So, to the extent that it is useful, we will walk-through our responses kind of around each of those, but we did want to just say at the onset both structurally, and because of this set of recommendations, we thought it would be important to at times turn it over to Anne, or to Matt, for responses. Okay. Liz, you can move on.

DR. ALBRO: Great. Thanks, Nate. Sorry. I've got too many things going on here. All right. So, we've decided that we will — we thought it was really important to actually start with one of the last recommendations in the report because as Ruth really made it clear when she was summarizing at its very highest level what the report is asking us to do, it involved expanding, right? Expanding our investments, expanding what we're doing. And we are, of course, constrained by the fiscal and personnel resources that we actually have present. And so just to highlight what we're actually talking about in terms of the amount of money that Congress has appropriated to us, we only have numbers available for fiscal '22 and fiscal '23, as I think most of you are aware. Appropriations for fiscal '24 have not yet been made. So, we are operating currently with the assumption that we have flat funding, but we, in fact, don't know what our actual dollar amounts will be available for fiscal '24. As you will note, there are two budget lines that support research here at IES. The first is our research, development, and dissemination line, which you all are going to hear me talk about as the R&D line. And importantly, that line supports not only the research activity at NCER, but it also supports activities at NCEE, as well as across IES functions specifically focused on dissemination. I also just wanted to highlight that you'll notice that there was a nice bump-up in '23, but I do want to underscore that that bump-up was actually associated with directive language in our appropriations connected up to coming up with high reward, solutions for seemingly intractable problems, which we haven't been acting on over the course of both '23, and activities so far this year. You will also notice that the research in special education line, the mixer, the rise line, it's substantially less, unfortunately, than the available funds that we have in R&D, but it also did have a bump-up in '23. So, I just wanted to set that there, and again, one of the very last recommendations of the NASEM report was to increase the funding that is available for us to support education research, and unsurprisingly, in the role that I currently inhabit I agree with that recommendation wholeheartedly. All right. So as we were thinking about, and as we have been thinking deeply about the report, we have been thinking about more than just the education research grants program, which is — and the research training grants program, which is where the report focused in part because the breadth of recommendations really pointed out to us opportunities to leverage existing mechanisms that we have in order to really meet the spirit in the full, we believe, of the NASEM report. So, we are also over the course of this morning going to share with your investments that we have been focused on in terms of the research and development centers, and NCER is required to support. The research network is a portfolio of networks that we have leveraged, we think, to really answer critical problems of policies and practice. Our investments in collaborations, and just to be clear, sometimes these things cross, as well as our required small business innovation research program. This is required by all federal agencies that have more than $100 million in investment and research, and NCER holds the responsibility for that program for the entire department. Back to you, Nate.

DR. JONES: And so, I think I will just echo that within NCSER we have a set of levers that we can use as well to respond to recommendations like those in the NASEM report. And I think we do want to prioritize these two main ones in our research grants program, and also our research training. I'll note there on the research training side that it's going to be useful later. There is a long history of how the training came to be within NCSER, but one of the things that has been kind of a demarcation historically is that the Office of Special Ed programs, OSA, has run pre-doctoral programs, and so what that has meant for us within NCSER is historically where we've placed our emphasis on the training side is with regard both to early career training, so mentoring for early career faculty, and researchers, as well as post-doctoral training. But, I think, what we wanted to emphasize is as we're thinking about these recommendations we do have this variety of levers, some of which are better suited to responding to the recommendations than others. Okay, Liz?

DR. ALBRO: Thanks, Nate. All right. So, for the next few minutes what we are going to do for you is share with you these crosscutting co-PIs that we have identified for the recommendations. As you all know, there's a list of recommendations which is very, very long, and so what we can provide is an itemized list, but we really thought it was more helpful to sort of bump up a couple levels to share with you how we have been thinking about the co-PIs in shaping the work that we are doing. So, the first is responsiveness in community engagement. The second is expanding competitions, which Ruth has already showcased for us. The third is the broadening participation lens, and then, of course, equity, which crosscuts everything that we do. So, let's start with responsiveness in community engagement. We're going to share with you all some of the activities that we have already undertaken so far. So, the first we've used so far is three different approaches in terms of identifying and responding to community priorities within the two research centers. So, the first is that we have been leveraging our ability to put out requests for information, or RFIs, through the Federal Register. So, this goes out to all the entire nation, and anyone is available to provide feedback. So the full science of the fiscal '24 RDD centers we put that out in February where we knew we were required to do this, and as you all know, in law we have very, very broad topics, and we asked for input, and we still didn't receive any two responses out of 84 responses, including, just so you all know, someone's entire dissertation. So, there was lots of content to review, which we enjoyed, and have really pulled on, and as I hope some of you noticed, we will be posting our new competition notice for our R&D centers later this week. We have also put out an RFI, which recently closed. Our new investment responding to the directive in appropriations that we're pulling from seedlings to scale, we've not had a chance to look carefully yet at those responses, but we did receive 62 comments, and we're very excited to provide a synthesis, and pull that together as we think about how we provide responses, both to the directive that the Hill has provided us with, but also with the broad needs that our communities have identified, and their hopes for this new line of investment. Nate, do you want to turn it back to the working group?

DR. JONES: Yeah, because I think the first of these is a nice example of us trying to be directly responsive to recommendations in the report, and then I think the second of these, I think, reflects kind of how we are thinking more broadly about responding. So, in spring 2023 NCSER hosted a technical working group focused on the special education teacher workforce. The educator workforce was something that was clearly articulated as a highlighted topic within the NASEM report, and we wanted to be responsive to that. So, as we'll talk about a little later, one of the outputs of that type of a working group was an RFA for a new center focused on the special educator workforce that we announced this fall. I think the second of these two that at least we're doing within NCSER is both because I'm stepping in as a new Commissioner, but also because I think we want to make sure that we are being responsive to the needs of the various communities that we serve. We will be hosting in some form or another a technical working group in spring 2024 that is going to be a part of an internal strategic planning process on our part. So, the goal here is to say let's bring together stakeholders through a range of communities and have them nominate pressing issues and challenges facing the field of special education, and then we will work internally to try to figure out how we can best respond to those nominations.

DR. ALBRO: Thank you, Nate. And then the last point that I wanted to highlight here is that we do have current opportunities, or current investments, that are happening right now that engage state and local partners in identifying the co-PIs for post-research, and where state and local partners are, in fact, partners in the work that is supported. The first is a program that uses, leverages, the state longitudinal data system called using longitudinal data to support state education policymaking. If you all are not familiar with that, I do encourage you to look at the range of projects that we've supported, again, in partnership with our state folk who are in charge SLDS, and this is inclusive — just so everyone knows, it's inclusive not only at the K–12 space, but also thinking about ways to connect SLDS all the way up to post-secondary education, to labor market outcomes, as well as the early childhood sector. So, it is actually broader than the primary focus of the SLDS. So, it's pretty exciting. We've got some really, really interesting projects that have come through that. As was said, during '22 we received funding through AARC, and we established two networks focused on pandemic recovery efforts in education agencies, as well as a highlight for folks that both of those networks require partnerships for folks who are the research teams who are participating in both of those networks. So the work that we're supporting through that is in response to what this (audio interference) on the ground using our funds in response to pandemic recovery efforts, and we're beginning to see some funding from that, again, both in the pre-K–12 sector, as well as in the community college sector. So just some thoughts about where we are, some ideas, or discussions of what we've done. And I'm going to now turn this over to Matt, who is going to provide a different lens on some of the work that we've been doing. I believe I can push the right button. There we go.

DR. SOLDNER: There we go. Okay. Thanks, Liz, and good morning, everyone. So, the work you just heard Liz and Nate talk about is part of the larger enterprise, a really shared enterprise across the work of all the four centers here at IES. So, we'll take you through this diagram, and help you have a sense of how we think about this work, and the various parts of the Center's kind of interactions which give you the broad goals and mission that I know we all share. So, you just heard on the far left in this notion around showcasing community priorities and needs. You heard Liz and Nate talk about RFIs, and TWGs, the technical working groups. Here at NCEE we use many of the same tools. We have RFIs, we host TWGs. Our latest is a TWG we call Sweet, which I will talk a bit later about this morning, but a small working group of education practitioners brought in to advise us on how to work around topics that focus for NCEE, usability of our products, and strategic dissemination. NCES also contributes to this notion of servicing priorities and needs. As you can see, all of their work whether it's NAEP, whether it's the work on the conditions of education, all designed to service and information about what's going on on the ground that IES might respond to. And then finally, the fourth element to that kind of surfacing community needs, we all do direct need sensing, right, beyond using surveys, beyond assessments, beyond RFIs, beyond panels. We go on the ground talking with folks about what it is they most wish IES was doing. So, our regional education labs are done on a routine basis. There are education agencies. Shout out to NAPE helping us work with CEC to do some direct needs sensing around needs of folks who work with students with special education needs, all designed, as you can see, as we move from the far left to the second dock, all designed to inform our evidence-building work. And inside there you can see how those needs inform not just the innovation activities at NCER and NCSER are undertaking, not just our shared applied RDD agendas at NCER, NCSER, and the co-PIs take on. Of course, our program evaluation work. But it also involves, and it might not be immediately apparent in kind of surfacing needs as the role of evidence synthesis activities, accumulating what's already known, and making sense of it, and making sense of it, and importantly, servicing needs and gaps from a kind of different constituency, if you will, the extant literature. And so, most of the evidence synthesis activity happened inside, and NCEE is part of our overall house. You probably are familiar with our practice guides. One of our most important publications, which is focused on educators, but an important new initiative inside NCEE is designed to complement what we're hearing from the field, is this notion of servicing in a more rapid manner what we are seeing in the following literature, and in the gray literature, to inform where there may be gaps in high quality evidence that could be brought to bear to improve policy and access. And this activity for what we're thinking about as evidence will be designed not to just surface gap the literature that — or not distribute literature to the practice guide, or surface gaps in literature, which might compliment what we hear from the community to help NCER, and NCSER program officers, to think about where they might shape their portfolio. So, we're moving in the direction to making sure we're always responding to gaps whether they're field-initiated, or literature-initiated. And then finally, all of that is kind of an input into the final box on the slide are knowledge use activities. You can see how the evidence-building activities, and our evidence-based activities, are all designed to inform knowledge use. Again, the whole goal of this is to see the high-quality research that IES and others do, inform policy and practice. We do that through technical assistance of our RELS. Our grantees certainly do that as part of their dissemination activities. IES shares things wildly, of course. But importantly, a bit of nuance, notice that there then is the cycle. Once we have knowledge use, we inform that back to into our community needs and practice as we share information entering the world, and really have a question as to whether it's changing conditions on the ground. And so it's a continuous cycle. Needs sensing, evidence building, synthesis, sharing knowledge, and then repeating, right, always moving hopefully in an upward trajectory to improve practice. Finally, I should just say — well, not say. I was excited to talk about expanding competition. I know. I know. You want to get to it. Just in knowing that this cycle at the bottom, this knowledge use and needs sensing, really is bidirectional. I think as a whole you would hear from the IES today, and over the two days we're together, about how it is we really want the community to be helping us improve our knowledge use activities at every turn, so it is not a one-way push. It's truly a communication between the community needs, and the activities we are kind of putting out there to the world based upon our high-quality research. So hopefully keep this virtuous cycle in mind as you think about our work together over the next couple of days, and I'm glad as we move through it to take questions on your thoughts on any part of it. And now onto Liz to talk about expanding competition.

DR. ALBRO: All right. I was just trying to make sure that my arrow was in the right place.

DR. SOLDNER: There we go.

DR. ALBRO: All right. So Nate, I think this is yours, although I'm not sure here. I want to see if this image shows up if I hit this button. Ah-ha. Go for it, Nate.

DR. JONES: It does, and I'll get to the analogy in a moment. But I'm really excited about that last figure that Matt shared in part because I think it really aligns nicely with the report's recommendation. So, Chapter 5 of the report focuses on topics, right, And I think one of the charges of a committee was to identify critical problems where research was needed. And I think one of the things the report does nicely is rather than simply say this is a laundry list of things that are important to that set of 15 people who are in the room writing that report, it instead says there is a real need to develop this kind of process of learning information from communities, generate evidence, synthesize that evidence, and kind of build this cycle. So, in doing that stuff the hope is that new topics will emerge over time, and that we will be able to pivot as an organization accordingly. The analogy that we wanted to put out here because I do think it's a useful one is to think about the core of what we have historically funded, and at least on the NCSER side. If I look back in the last 20 years, NCSER has tremendously expanded our knowledge around promising effective interventions for students are both academic and behavioral, and we've really made tremendous progress. And I think what is reflected in the report, and, I think, what is reflected in our perspectives as an organization, is really let's take that core of what we've been developing, and strengthen it by adding new strands, adding new strands to kind of strengthen the contribution we're making to the field, and the knowledge we're able to produce on behalf of the communities we serve. So that analogy aside, which, I don't know, we may like, or may not like, but, I think, what we did want to do is much like the report nominates a handful of topics that would be useful for the field to consider moving forward on, we wanted to just take time to walk through a couple of topics that were highlighted in the report, and describe how we've been responding to them.

DR. ALBRO: Thanks, Nate. All right. And this is explained with many words, and I apologize for that; there are fewer words than there were to begin, but Nate, as he just said, we're going to — it should be sort of two examples in the research centers, and then Matt is going to have an opportunity to share some of the work he's been thinking about looking at some of these topics and scenes that have been going through. And given that, this report came out in the midst of the global pandemic, and technology was pervasive, and all of the work that was happening with our students and learners, we wanted to showcase how we have been thinking about this request, this call to invest more in education technology and learning analytics at the two research centers. And the reason I showed that very first slide with the six boxes is really to sort of showcase how we thought about that not only thinking about our verified A competitions for technology is a crosscutting theme, and technology is just in all of the topics that exist there, but how we've tried to embed education technology and learning analytics as deeply in our investments across all of the work that we're doing. So, I think I'm going to just quickly highlight a couple of things here. The first is that we know that there is a deep need in the field to build capacity and broaden participation in the youth of data science and learning analytic techniques in the context of education. So, recognizing that, and in our limited funding year last year, we actually ran a competition, and are now thrilled to say that we have three training institutes across the nation focused on data science and learning analytics, and you also, I hope, are signed up to our news flash, and as we spread out, please share this as a community. There are opportunities for individuals to come and participate in this training without cost to themselves. The second is that we've launched a variety of network type activities. AI Institute's broad investment looking at how can we think and learn together about the best ways to use ed technology, and lean into the learning analytics that can help people use the data that they collect every day in the context of a learning management system, in the context of how they're teaching their kids and learners in terms of that, all of the electronic stuff that exists right now, right? The SEERNet is our digital learning platform network where we've been excited to invite our first research team to join that network, and we're excited to have the opportunity to join more folk to help us learn what can we do making use of an electronic data that already exists in school systems whether they're in the K–12 sector, or in the Pre-K–12 sector. We've launched, I'm sure you guys have heard about this, two new AI Institutes working in partnership with the National Science Foundation, which we are thrilled to be able to be a part of. The Learn Network, right, there's a call for adaptation and scaling in the context of the work that we're supporting. The Learn Network is intentionally designed to support teams of high-quality evidence, products that have high quality evidence, to think about what we need to do to be adaptive to the needs of communities who want to implement this work, and to prepare it so that these interventions can be used more broadly. We have been under our 305-T are transformative application. If you didn't notice that that whole partnership brand, although it includes a different set of partners perhaps than one that we're already thinking about, trying to make sure that industry partners are part of the conversation because the reality is is that they are the ones who are pushing ed tech; they are the ones pushing learning analytics, and if we're not part of that conversation, we are worried that we will not as a research community be able to shape the products that come out of that. We, as I hope you all notice, one of the topics included in our new R&D center competition that will be coming out later this week is focusing on the use of generative AI in classroom instruction, again, something that we know is happening, and yet the research community knows remarkably little about. And so we are really going to hopefully provide national leadership in that focus. The ceilings to scale competition I've already mentioned quickly, and throughout all of this the Small Business Innovation Research Program, which we have been supporting, and have been charged supporting since 2002, sits at this interesting locus where we are really working with the tech developers who are small businesses, and trying to ensure that the products that they are developing and sharing using technology builds high quality evidence to the degree that we're able to, and support innovation so that we can take all the good work that the research community is doing, and make that it is invested, immersed, infused, whatever the right word is, in terms of the products that come out. So that's just one example of the way that we have been trying to think about these broad themes across the body of work that we are (technical interference.) And Nate, I think this is for you.

DR. JONES: It is for me, and it will always be for me. I am deeply committed to focusing on teacher education, and educator workforce. This is something that clearly was identified in the report as a priority. I think what it is is an acknowledgment that if I go back to that rope analogy, especially I use the case of NCSER, that the interventions that we have been successful in developing do not get carried out in a vacuum, right? Those are implemented by educators who themselves are sense makers, who are kind of making sense of these new interventions vis-á-vis their existing set of priorities, and those things are existing within systems. And so, the report really pushes us to make a more concerted investment in teacher education, and the education workforce. Before we talk about some of our responses, I think this is an interesting case that lets you all into, I think, some of the challenges, or the tensions within a space like this, or in the complexities, because to an extent we can prioritize investigators to apply for certain projects. But what we have seen historically at times is that both because of the conditions that are in place within IES, but also because of the conditions of fields, a field's ability to be responsive to new and emerging areas is not always in place. Why is that? Part of that is measurement challenges, and part of that is simply the priorities within a field. So, I think when we think about teacher education, for example, I know that there is a group, Heather Hill (phonetic), Suzanne Alobe (phonetic), Pam Grossman (phonetic), David Chard (phonetic), who for the last several years have been trying to increase the amount of causal research being conducted in teacher education. Liz and I both have met with that group at various times. And I think the fact of the matter is we see very see very few of those folks' kind of doing this kind of work in the teacher ed space. So, we can do certain things to incentivize work, but I also think it is a deeper set of challenges, and one that I think we are all thinking about how we respond. Okay. That is kind of the high-level diatribe, but, I think, in a more specific sense. So, I want to talk a little bit about our center. This one we announced this September. This is an RFA for research and development center on the Special Education Workforce. I think it is probably apparent to folks in this room that we have had longstanding shortages in special education that only appear to be getting worse over time. So, we are establishing the center with the hope of providing national leadership on ongoing staffing challenges. The way that this RFA was written it requires that investigators are working in partnership with state Departments of Education. The goal here is so that researchers can be responsive to local state needs and provide recommendations that hopefully will influence both policy and practice. So that is us on the NCSER, and then, Liz, I will hand it over to you to talk about some of your efforts at NCER.

DR. ALBRO: Yes. And my two are very quick here. The first is that while we have provided investments, and invited applications focused on teachers for our entire history, it became clear to us in reading the report that sometimes there are challenges in folks picking that up, right? We had changed the name of our teacher-focused topic in the RFA, and we've renamed it again, sort of really making it clear that, yes, we want to know about teachers teaching in the education workforce, and those are the applications that we're hoping to receive. As you all know, A applications have already come in for '24. I don't have any data yet to share with you on that, but we're hopeful that that shift, right? You guys know that a new name can sometimes really change things, and we're really hopeful that that has changed the field. And in parallel to NCSER's RFA, one of the other topics for R&D centers is we got lots of feedback from the community that there was a real need to understand teacher shortages in the K–12 sector, and really understand what kinds of policies and practices are states and districts putting in place to really try to address recruitment and retention in the K–12 genre education workforce. Stay tuned for more details on that. Again, that will be coming out soon. And now, Dr. Soldner, this is you.

DR. SOLDNER: Thanks, Liz. Lastly, but not least in this section we want to talk a bit about our investments in research on the use of research evidence, or URE as we call it. Also sometimes called knowledge mobilization by folks. You'll see on this slide that IES' founders must have been unbridled optimists when it came to the idea of the evidence us, right? That if we wrote the responses to this wonderful research, and we published academic journals, or posted this to our website, things would magically get from their hands into those of policymakers and practitioners to let all of us know, you know, Ruth, especially you know, not so. Kevin Costner was not entirely correct. I mean, if you do build it, they do not always come, right? And so there are many reasons why that is, and it is no party's fault at all, right? But it is something to manage if you hope to see your fellowship views and condition changed on the ground, and we very much do. And so, I want to — briefly about our investments. To be clear, IES has been investing in improving use of research evidence for more than a decade, so in one way with our emphasis on the use of RPPs is an investment in seeing research evidence used, and that dates back to the 2010s, right, both through work in the RELS, and through NCER. NCER has also made grants to RDD centers specifically focused on this activity both in 2014 and 2015 at our National Center for Research and Policy and Practice, and our National Center for Research Used in Education. So today we continue that thread, just in some different manners. So, we're further extending this work in a new investment by being a bit more thoughtful about how we can use the REL program, something in our own backyard, not only as a place where we're doing the work of use of research evidence, but as a place where we can study it, right? We have ten literal labs in which we can do this work. I know RELS are the nation's single largest investment in education RTTs, more than 70 of them, and so we are currently setting aside funds, investing $2 million in the first year to think about how we are, and maybe how we're not yet, improving the use for research evidence. And this afternoon, or whenever we come to it, when I talk about the work that's at NCEE, I can talk a bit more about what that work looks like besides they were investing in this phase yet again after some real foundational work done by our colleagues at NCER.

DR. ALBRO: All right. I know this is a lot. Thanks, guys, for giving us the opportunity to share. There is a lot of stuff that we've been doing, and thinking about, and we're happy to have a chance to share it with you all. Broadening participation, and transparency about broadening participation, we're definitely seeing a theme that came through the report. And so, I wanted to share with you where we are, the information that we have so far, and where we hope to go. So, let's just share with you that we have done today in terms of trying to understand where we sit. So, we started with trying to understand what institute — which institutions apply for IES funding, right? I know this is a question, and a concern of the report, and so we will share with you some information about that in a second. Describing the specific key personnel named on the project, we will share with you the information that we have, and I'm happy to talk about this. We've listened to communities, as I think many of you are aware; that we have a whole variety of listening sessions. This was, I'm trying to think about a year and a half ago, where (audio interference) about who completed (audio interference). Someone is unmuted. Can someone please mute?

MALE SPEAKER: 245746.

DR. ALBRO: 245, yeah. The — okay. If we mute us, then no one can hear me, which perhaps you guys are ready for. I don't know. All right. It sounds like that has been accomplished. Thank you. We shared who completed our IES-funded pre-imposed sectoral training programs on our project abstract, if folks are interested. That information is now available. And we have been systematically gathering information about best practices for broadening participation in federal grant programs, looking at what they have done at the National Institute of Health, and the National Science Foundation have done, as well as others, and including what folks have been doing around peer review. So, the next couple of slides are going to be some graphs, and some charts rather, and I will just share these with you. And I think what you will see is that the story is pretty much the same across each of these, regardless of which characteristic we're talking about. So, let's first start with the named gender of PIs in applications and awards. These are for applications that were received in fiscal '21 and fiscal '22 and is inclusive of some of the funding now that's in place that we did in fiscal '23. The data that you're seeing here is percentage of the total number of applications received, reviewed, and then the awards that we made. Just for context, we received 1,900 applications. Of those, 1,763 were forwarded for peer review, and we made 261 awards. As you will see, this should not surprise anyone on this call, we have many females who are applying to receive funding for us, but I want to note, and again, this tan bar that is here, the tan bar is individuals who chose not to provide us that information, right? As a federal office we ask, or we required, folks to complete the OMB form that includes information about gender and demographic features, but there is always an option to not wish to provide, and as you will see, both here, and throughout the other slides, anywhere from about 20 to 32 percent of folks who are filling in this form are not providing us with information. So that given that data, I just want to put out there is a certain amount of caution that we try to try to think about in terms of drawing inferences from the data that we have received that has been shared with us.

DR. LEE: Liz, can I ask you a quick question?

DR. ALBRO: Yes, ma'am.

DR. LEE: Have you all considered — because I know we've been kind of wrestling with this at the National Academy, in terms of more than this dichotomous gender identification?

DR. ALBRO: Yeah, Carol. That's a really good question. Unfortunately, the OMB form that we are using, right, that that is a form that has been approved that we must use in order to collect data, currently only has these three options. I do know at the federal level there is a conversation about expanding that description, and once we receive an OMB-approved form to use, that will replace this form, and they will have whatever the agreed upon classifications are. So,, thanks, Carol. That's definitely an important question for just to take into consideration. All right. Similarly for the race of PIs, and co-CIs, again, one of the things I think that — one of the things that's interesting to me is noticing how the patterns that we are seeing across proceed, to reviewed, and awards made, it's actually not that different across these different categories. Because of the concern about protecting privacy, we had to collapse individuals who identified as Native American, Hawaiian American, and I realize I didn't write this down, but you will see that the vast majority of applications that we receive have PIs and co-co-principal investigators who identify as white. We still have this large number of individuals who are choosing not to report their race to us. I do think that there seems to be some indication, again, I don't like to draw conclusions from this data, particularly with so much of it missing, but that the co-principle investigators are actually somewhat more diverse than the PIs that we're currently seeing, or that we saw in our applications that came in in '21 and '22. But again, that's perhaps the statement I shouldn't have made out loud, but that's what the data looked like. In terms of ethnicity, I think we can — there's no argument with this. We receive very few applications from principle investigators who identify as Hispanic. Again, I do not wish to provide, and one could think about reasons why we don't have that information in terms of ethnicity. But again, we have no reasons, we have no explanation, for why. These are simply people checking boxes on a form that they submit to us at the time of application. Institution type. So, for those who ever applied for federal funding you probably know well the Standard Form 424, and part of the Standard Form 424 there's an opportunity to that your sponsored project office has to select what is their institution type. I went back and looked at the list. I think there's something like 16 different institution types, probably more, but that was what I was able to count. And so, all I have here are the four greatest types of institutions who have applied and received for funding. As you all can see here, public institutes of higher education are our predominant constituent, our predominant customer in terms of folks who are coming in and applying for money, followed by our nonprofit customers, followed by private institutes of higher education, and then we have state government, and MSI, and those categories are small. Again, trying to draw conclusions from this data is complicated. I just ask that you're only allowed to select one of these categories, and we know that all of these MSIs, or many MSIs, could also be classified as either a public institution of higher education, or a private AAG. So in some ways it's — we just don't know to what degree there is additional overlap amongst those categories. So the lessons that we've taken from this is that broadening our applicant pool has got to be something that we focus on, and as you all know, this is something that we've been concerned about even before we had availability of this data, which is in some ways was why we started to try to collect this data. And so we have been trying to think about how we'd let folks with — who come with different backgrounds know that you can actually be an education researcher when you grow up if you wanted to. And so just under the leadership of Dr. Katina Stapleton (phonetic), one of the things we did was establish the pathways to the Education Appliances Program, which requires a partnership between minority-serving institutions, and a Research 1 institution, the idea being try to — to try to pull in individuals with bachelor's and master's degrees into sort of understanding and learning more about research. We've so far run two cohorts in 2016 and — 2016 and 2017 is like one cohort, and 2021. As you might imagine, the 2021 cohort was deeply impacted by the pandemic, but the data that we have so far indicates that we have 264 pathways fellows who have completed their training program at these six institutions that were awarded in '16 and '17. Early career is a newer opportunity for NCER, and we were building on the success of the NCSER program. As you all know, we established a program focused on early career faculty support for faculty who were at — who were on faculty at minority-serving institutions. That we have been working hard to build, again, the number of applications that we received, and we have made our first award, and we're excited about that. So we also heard in our listening sessions that there are many individuals who bring different demographics characteristics who are working at predominantly white institutions, or other institutions, and so we thought it was really, really important to broaden the opportunities for early career scholars to put in applications to IES, and to have the — and to NCER, and to have the opportunity to work with my dedicated team to really help them build and improve their applications. And, finally for institutions, again, this is really redundant, right? We've been trying to continue to find ways to invest in supporting work at minority-serving institutions. Part of this is through embedding these kinds of partnership requirements. I will say that using longitudinal data programs we have been trying to right size the partnership on that, so initially the requirement was that the state come in as the lead, but it turns out that state SLDS systems are not always in the best position to quicken and support a research project in the context of a federal program, so now we've moved this to a partnership diagnostic about who comes in as the lead institution. All right. Now, I'm going to turn this over to Anne because the other question that we got lots of input on was understanding and broadening who reviewed.

MS. RICCIUTI: Great. Thank you, Liz. Good morning, everybody. I'm Anne Ricciuti, Deputy Director for Science, and I have the responsibility for scientific peer review research grants in my office. So I'm going to take just a couple of minutes because I know this is a lot that we're going through, but I wanted to address a couple of the issues around broadening who reviews, and then I am also on the agenda later today, and, actually, will try to provide a little bit more background around the review to conceptualize some of this information. So just very briefly, there are two parts that we have been thinking about around broadening who reviews our research grants. One is around understanding who our reviewers are and reporting out that information publicly while also preserving privacy, individual privacy, and the confidentiality of the peer review process. Currently you can see on our website, if you are able to dig down deep enough on the website to find it, you can find lists by fiscal year of our reviewers by panel. We do combine some panels, and sometimes we combine across years, again, to try to preserve some of the confidentiality of the review process, but we have, and we have had for a number of years, online the names of the reviewers, and their institutional affiliation, and we also indicate who has served as chairs of panels. We will soon be posting the list for the most recently completed fiscal year, fiscal '23, so soon that will be up on the website. We are working on figuring out best ways to collect demographic information about our peer reviewers in a way that preserves privacy, and that we can report out publicly in aggregate. We have been looking at how and what NIH and NSF collect, and working on exactly what we should be collecting, how, and personing appropriate OMB approval. So, we are looking at wanting to collect information not just about race and ethnicity, but also about some of the kinds of factors that Liz was just reporting about around applicants, including gender, disability status, and others. We are also working on expanding diversity and being able to report about diversity of our reviewers, and a number of other aspects as well, including seniority, so you know, early career versus mid versus later career, disability status, et cetera. And as Liz mentioned, all of this, of course, would be collected voluntarily from reviewers. We can work on ways to try to make sure that folks are answering in some way, but we can't require that anybody actually provides us the information. We are also working on the systems and tools for collecting and maintaining that information, and I'll get to that a little bit later. At the same time where also we're working on expanding the diversity of our reviewer pool in terms of all of these areas. I have a dedicated member of my team currently, my very small team, which you'll hear about later, and this past year that individual traveled to a number of different conferences to try to get information out, and recruit reviewers from a number of different areas that are new to us. We also have participated in the listening sessions that NCER and NCSER did with a number of groups and communities and were part of the contract that NCER and NCSER led looking at what other federal agencies are doing around broadening participation. For a number of years, we've been working to get reviewers with experience in different kinds of settings, so from state and districts. Also, other nonacademic settings both industry and other kinds of settings. It is challenging for a number of reasons; however, we have been continuing to work along those lines anecdotally because I don't have the status report, but anecdotally I think we've been able to improve in a number of those areas. One example is the recent panel we held on to the transformational research competition that Liz mentioned earlier. We did have on that panel reviewers representing industry, education, and educators and administrators, education administrators, as well as researchers. And I think it was a really, really good mix of folks and perspectives. Our goal is to be able to provide lots of information publicly about our review process, these are the procedures, our people, our results, quickly and efficiently. We've been working on — my sort of dream would be to be able to have the kind of information that NIH, Center for Scientific Review, has been able to put out. I believe very strongly in transparency while also preserving privacy. And I will stop there. I am on the docket for later, so you'll hear more from me later, and I'm happy to talk more, and answer questions. And now I'm going to turn it back to Matt, I believe.

DR. SOLDNER: All right. Thank you, Anne. So all the issues we've discussed this morning whether it's engage in the communities with whom we work and serve, whether it is considering how our competition can help us innovate, and build evidence and serve supporting all learners on a wide range of topics, or whether it's participation, so that we are bringing more voices to the table in all aspects of our processes. All of those are examples of how we are responding to issues of equity that are raised by the Academy support. There are a couple others that we'd like to highlight for you this morning before we wrap up. I promise the end of our conversation is in sight thanks to the next slide. Great. So first, in June of 2022 with the help of the ISY DNI committee, we established the SEER equity standard. You'll see it on your slide. If you aren't familiar with SEER, it began in 2018 as a series of initially recommendations, now principles and standards, that we believe that researchers could adopt beyond having studies which have strong internal validity that if they were adopted could result in their research being more transformational. Equity was not one of the original standards, but in June of 2022, we added it. You can see the standard itself on the slides. It's supported by four additional recommendations. There's things that we think are nice to have, or things that should be on researchers' minds, including recommendations that researchers be more transparent about how their conceptualizing of equity in their work, that they were to better understand the heterogeneity of impacts in their studies, so we can achieve the mission was often talk about, which is knowing what works for who, and under what conditions. We advise researchers to design interventions that are responsive to the context in communities in which they're meant to be implemented. And then finally, engaging those communities in all aspects to the RDD process. So not now, but later, if we go to the SEER website, and check those out, since publishing that standard in 2022 we have convened one technical working group to help us initially identify tools and resources that are already in the community that might help researchers make forward progress on that goal, and we will continue to work to find ways we can support researchers achieving a trajectory standard in the years ahead. And, I believe, I'm finally turning it over — back to —

DR. ALBRO: Nate.

DR. SOLDNER: — back to Nate.

DR. ALBRO: And then onto questions.

DR. JONES: And then, yes, we will turn it over to questions momentarily. I just wanted to highlight a couple of things related to NCSER on equity. I think the one that's not here is we're giving a lot of thought to the ways in which individuals with disabilities are often multiply marginalized. So I've been excited that over the last couple of years we have funded a handful of studies focused on English learners with disabilities. I'd like to see more of that work moving forward. But, I think, also one of the things that I think that we are seeing within the disability community, and especially the research community, is this kind of continued push around nothing about us without us, right, which is a slogan used by the disability community to ensure that folks with disabilities are included in policies, or practices, or programs that impact them. So in response to that, in our most recent RFA we've tried out asking investigators to document how they're collecting the perspectives of students with disabilities within the research process. We had specific grantees who have worked really hard to include researchers with disabilities in our projects, and we are trying to figure out how we can elevate those examples, and make them more commonplace in our field. And I think kind of going back to Matt's original point where we're still exploring options to have routine ongoing feedback from the disability community about how our research maps onto their needs and perspectives. So I will close there, and Liz, I don't know if you had any final thoughts, or if we were — if you just had —

DR. ALBRO: Well, first of all, thank you. I know that the Board wanted to have a chance to ask questions, and respond, and react. As you guys can see, our desire to try to sort of slim down how we responded given the breadth of responses in the report, I don't know how well that worked, but it was our attempt to really give you all a sense of how we've been engaging broadly with the report across all of the work that we're supporting across the three centers, as well as through our peer review. And with that, I'm going to open this up. Carol, do you want to be traffic cop, and direct, ask questions? That would be great.

DR. LEE: Sure. Just one quick question, and I gradually see hands coming up here.

Nathan, and I know that at this point the way the field is established nothing on here is going to change, but I do know that in the special education community, if you will, there is discussions and debates around even the whole construct of disability versus diversity. And to the extent to which you — the extent to which the focus of the work that you're doing in the center being able to try to shift the categorization. I'm just curious.

DR. JONES: Yeah. I mean, I think it's an important question. I think those — some of those thoughts, I think, reflect two things, right. One is the perspective that some within specialized research have historically approached things from a deficit-oriented perspective. I'm not sure that that is always true, but I do think that we are working hard to think about how we can prioritize externally facing research that is acknowledging the assets that individuals with disabilities bring. So that's thing one. I mean, I think thing two is because of concerns around over-representation historically there have been kind of concerns about special education as an enterprise, and the extent to which we name something as a disability. I will also say that within the disability community there are many who acknowledge that we have disabilities. We don't want to shy away from that, and we just want schools to be anti-ablest, and to ensure that schools are supportive of those individuals. I don't know if there are other perspectives of members of the Board on this issue. I saw Doug's hand pop up, but —

DR. LEE: So we're going to —

DR. FUCHS: Carol, just — I know you want to move on. I just want to just briefly add to what Nate said. I think the disability community is very aware of a potential tension between diversity and disability. And I think that the disability community in part strongly supported by IES has pursued policies and practices like response to intervention that try hard to distinguish in practice between kids with disability versus kids who may seem like they have a disability, but really are not disabled, and are performing poorly because they've not had access to appropriate education. So there is this sensitivity to the issue that you're raising.

DR. LEE: Thanks. Hiro?

DR. YOSHIKAWA: Yeah. First of all, thanks for this really comprehensive overview of all the activities in these four big bucket areas. So yeah, very helpful, and exciting to see all the activities that are related to the Board recommendations. Just a couple areas I wanted to ask a little bit more about, which have to do with the definition of outcomes in education research, and then also the report's recommendations around methods and research designs. Recommendations in the report around use of outcomes beyond the student level to get to indicators that are more structural and contextual, that are about measures related to educational equity, and how those — there's a research agenda around developing and validating those kinds of measures. So under that bucket, which is kind of more — yeah, whereas the traditional outcomes have been really at the kind of student, and maybe teacher levels, or that kind of thing. Then around the choice of methods and research designs, I think the report made some recommendations for some methods that were certainly mentioned in what you presented like AI, but there is also one on supporting qualitative and mixed methods approaches to research design, and methods. There were a variety of other kind of design things in Recommendation 6.12. So those two areas I'd love to hear what the activity has been related to those.

DR. ALBRO: (Audio interference).

DR. JONES: You unmuted yourself, Liz, so just mute yourself.

DR. ALBRO: Thank you, Nate. I'm glad you saw that clearly. I was doing really well until you asked me a question, Hiro. Two things. Yeah. Thanks. We were trying to slim down the slides, so definitely these are all the things that we are continuing to think about. In terms of the definition of outcomes this is actually a really, really interesting and important question because for us one of the things that IES, and the way that we read the legislation, and as derived, that we are obligated to fund work that directly links to the student outcomes, right; that we need to have measures of student outcomes in addition to all of the other measures that we care about. And so, from my perspective nothing precludes anyone coming in and proposing to have additional structural — you know, measures of structural outcomes, or measures in educational equity, but I do know that it is challenging to include those types of measures, and also include measures connected up to student outcomes. We would be happy to continue conversation around this and help us think creatively about how we need our legislative mandate, but also broaden the kinds of measuring and outcomes that folks are including. I will say that if you look at many of the research projects that we fund very few of them only include one type of outcome, right, and very few only include only education and (audio interference). So we have all kinds of measures of attainment, of progression through the education system. We have measures observational measures, say, for things like school climate, of conditions of schools in the school systems that exist. But, I think, we as a field need to work together to think about ways to improve and increase the measurement capacity and skills that we have to measures systems-level change, for example. And Nate, I don't know if you want to jump in on that one before we move to the next one.

DR. JONES: No. The only way I'll jump in is I think clearly there is a need to invest on measurement kind of across the research enterprise. I think IES has invested a lot in measurement of student outcomes. I think we have seen less proposals focused on educator outcomes, or systems-level outcomes, but, I think, our ability to make progress in those areas that kind of get further away from a focus on students is always going to be dependent on our ability to have high quality measurement systems surrounding that work.

DR. ALBRO: Yeah. And I do — and just to jump in. We have certainly been talking about the measurement project type, and about measures, and we're super proud of that work, and we're really glad that the committee is proud of that work as well. But, I think, as I — I mean, my default is not always to go back to training, but I do wonder, and would love your all's thoughts, on how do we continue to build the community of scholars who have high quality psychometric expertise, and reliability, and validity who can help us tackle these really critical problems because we feel like — I mean, in terms of what we see there are fewer individuals who are doing that work than we think are needed. So, the second is around methods and research design. The first thing that I want to say is that we invite mixed methods work, and that has been part of our RFA for a very long time, and the reality is we actually see a bunch of mixed message work, right, as this is coming in. And I want to say any time anyone puts an application in, we encourage them to do both kinds of work, right, to provide us with quantitative data that (audio interference). But we also are extremely interested in (technical issues.)

DR. LEE: Liz, we can't hear you now. We can't hear you, Liz.

DR. JONES: I think the phone is now muted, Liz, the 202 number.
(Pause.)

DR. ALBRO: The phone decided that we should know why you guys can't hear my voice anymore. They said nope, no more. Down with Liz. Anyway, so when I look at the work that we support, we actually have lots of work that is in both qual and quan. It's really trying to understand the context within which this work is happening. Anytime we support development in innovation work, which is actually in terms of numbers, the largest pop of applications that we support there's always qualitative work that's helping us understand how communities are responding to that work. So we appreciate its knowledge that is important for that kind of mixed methods work in the work that we support. So with that, did I answer all of your questions, or is there more in there? Nate, did you want to jump in?

DR. JONES: Just when your phone cut out you said we are also extremely interested in?

DR. ALBRO: We're extremely interested in, good question, and you think I'm going to remember what I was going to say? I mean, I think one of the things that I'm extremely interested in is trying to figure out how we, again, as a community can report the information that's collected on sort of both sides of this, if you will, qual and quan work together, so that the field is aware of both of these projects, right? I sat on a mixed message committee that Spencer pulled together probably about a decade ago, and one of the real challenges that we talked about is how do we share this information together, right? The way our current publishing system is set up we actually don't have opportunities where people can sort of both share, if you will, an impact study outcome, the quantitative outcome and all of the contextual information that becomes incredibly important for interpreting and understanding those findings. So that's, I think — may I say, again, I can talk for hours on this, but I know I'm not the only person on the agenda. Nate, did you want to say anything?

DR. JONES: No. They have lots of questions. I'll hold off.

DR. LEE: Denisa?

DR. GANDARA: Thank you. I'll just start by echoing Hiro's appreciation for the presentation. I think you all did a really impressive job of distilling your responses to a 285-page report to accurate and concise, but very thorough, presentation. So thank you for that. I do have a couple of follow-up questions related to transparency. Would you please comment on any efforts to make the data on applicants publicly available, and I'm especially interested in seeing these data in a way that allows us to capture this over time. And my other question is about whether you have any data on study samples to share? That is also one of the recommendations of the NASEM report that we just covered, and I just think it's really important to understand who is being questioned, and I'm wondering were there any efforts around collecting data on that.

DR. ALBRO: Yeah. Thanks, Denisa. Super great questions. So yes, we are working hard to try to figure out ways to be transparent in terms of sharing this information. We will be sharing out tables, so just tables of that information, so as long as it conforms with sort of aggregate — the requirements of aggregate data. And we've had to pool stuff together in part because the fiscal '22 funding years, and the fiscal '23 funding years were odd, right, in the sense that we didn't have most of our primary competition. So, our numbers are really small. Like Anne talked about, like NIH has their wonderful data that they shared on this DSR, we would love to be able to have something like the NIH data book, and we're actually doing some back-end work to try to make sure that we can have that. I will say, however, that our numbers are small enough that the ability to track over time is going to be complicated by sort of the general number of applications, and information that we get. But to the degree that we're able to share that data, we will. And we've just been delayed because we did this thing where we funded down this way where we funded grants from '23 — funded grants in '23 from the 22 applications. We had to come up with a shared agreement of how we were going to report that data. So that is entirely the goal, and we will be sure to let the Board know when that information is available publicly so you all can see it. In terms of some examples. So yeah, so we started work on — we did an initial survey, a small survey, I don't know. I don't want to mess up and get in trouble here. We reached out to our grantees, and asked them what kinds of information do you have available in the data that you're collecting about your study samples because what we are trying to figure out is what information is already available that folks could then report to us, and how much information would we really like adding additional burden to our grantees if we're asking them to do that because there are requirements in terms of OMB, and the Paperwork Reduction Act that we have to be responsive to. So, we have that data. Unfortunately, we've not had a chance to really dig into it, and come up with solutions, but it's absolutely something that we care a lot about, and we know, again, that reporting standards. And I am focused on the publishing side of this, but reporting standards are all over the place in our journal, so it's very, very difficult to try to understand the demographic characteristics and features of individuals who participated in research funded not only by us, but by the National Science Foundation, and the National Institute of Health. So, we really appreciate that recommendation and are trying to come up with a good solution that doesn't over-burden grantees, but also make sure that we have information that we think is really important for the community to know about. Nate, anything to add?

DR. JONES: Nope. Now would be a good time.

DR. LEE: Doug?

DR. FUCHS: Yeah. Just briefly. First, thanks very much to Liz, Nate, and Matt. Wonderful presentation. I'd like to — and I don't have a question. I'd like to just add a brief historical note to perhaps better understand the 2022/2023 appropriations for NCSER, 60.3 million, and then 64 million. It should be understood, I think, in an historical context in roughly 2002 the Office of Special Education Programs was required to give up its research portfolio. It switched from the Office of Special Ed Programs to IES. And in that year 2002 the investment — the research portfolio which began NCSER's, was over $70 million, and that was in 2002 monies. So, we can look at 60 million going up to 64, and that's nice, but the historical fact is that way back when it was 70. That's all.

DR. ALBRO: Thanks, Doug.

MALE SPEAKER: Could I add to that? So we've been trying, and Doug knows this, and we've been trying really hard for years now trying to get to restoration of a more reasonable amount of money for NCSER. So I think my understanding is, the history is, in 2011 about a third of NCSER's budget disappeared for payback for other work that the department was doing, and they had to find — a pay for, not a pay back, and they had to find money to do some other programs. And I think it's arguable about how good the research community was for special ed in 2010 and '11. And I think when people look towards it because it's off the field wasn't, shall we say, as strong as it should have been in ten years later, 12 years later. I don't think there's any question that the work on special education that I have supported since in the field, has just gotten, I know, an inordinate magnitude better, and the problem, of course though, is that we are in a hole from 2011 and 2012, and no matter — I mean, I've tried in — and Nate, I mean, all — many other people have tried to get even a modest restoration of where we should be, which would probably be the vicinity of post-90 or $100 million. And I think Carol said right at the beginning about the question of what the Board's role should be, so helping us make a case to the Congress, the appropriators for more money is one of the things that we really would like from the Board, and for me getting more money for the Special Education Research Program is incredibly important because I think what happened in, I think, it was 2011 where sort of the budget disappeared was not an appropriate action, and certainly the field has gotten so much better, and so much stronger that it deserves much more support than the money that it currently has.

DR. LEE: So, Stephen, and other staff, and members of the Board, do you have any ideas about processes for the Board to engage in to support both this particular recommendation that (audio interference) increasing the SPED allocation even more broadly as we try to have some way of communicating with regard to ESRA reauthorization?

MS. LEGORRETA: This is Conchita. Sorry. I raised my hand. I'm not sure how to unraise it. So, Stephen and I's report actually go into that. We do have ideas, so whenever that would be a good time. I don't know if now a good time is, or if we should wait, but we do have some recommendations with the Policy Subcommittee.

DR. LEE: Okay. Great. We'll hold it until that time.

DR. KLASKO: And we're really sensitive to, you know, what we're willing to do versus what we're allowed to do from the Board's point of view. Because, I mean, I keep getting excited, and then getting regulations of what we can't do. So I think that would be — once we do our report it would be very interesting to hear from that perspective.

DR. JONES: And I will just add — oh, sorry, Carol.

DR. LEE: Go ahead.

DR. JONES: Just one thing really quickly, and to the extent that we can provide data, or information to support your efforts, we'd be happy to do so. You know, we have shared publicly that this year in particular there were more than a couple of projects that were deemed meritorious. We simply did not have funding for it. I think it's evidence of where we're at as a field that our reviewers are saying we should fund this work, but we are in a position where our funding is just too limited for us to be able to be faithful on that issue.

DR. LEE: Hiro, your hand is up again. Conchita, did you finish what you wanted to say? We're going to come back when we come to your report. Hiro, is your hand up or down?

DR. YOSHIKAWA: It's down. Sorry.

DR. LEE: Okay. So James?

DR. MOORE: For what I would like to say is that I think it's very important that the Board consider some kind of action that it can take. IES, and EDU, and NSO, we're the largest entities in the federal government that funds STEM education research. And it's clear that IES has its challenges even within my directory, we have our challenges primarily because the STEM education aspect of my portfolio is not congressionally mandated. So, if I have to take cuts, I'm going to have to hit the core, and our core is education or research. So, we're the only game in town at the federal government. When you think about STEM research broadly you can go to almost every federal agency, I won't say every, but many other federal agencies, but in this educational research realm, particularly in STEM, you don't have a lot of entities you can tap into. So typically, when IES is hit, you know, we're hit as well. So, in turn, the whole educational research enterprise is hit in a major way. I don't know if Mark has anything to add to that or not, but I just thought it would be important to provide that context.

DR. SCHNEIDER: So, there's a major opportunity with regard to STEM, and we — you know, James and I have been — we've had a road show going. We have another one on Tuesday, I think, is our next road show. But STEM is an area that is very, very contemporary; it's very much needed to increase the STEM workforce, the diversity in the STEM workforce. And I've written about this many times. James has also. I mean, 75 percent of students with disabilities are below basic in science and math according to NAEP. Of the subgroups that NAEP measures, that's the largest percentage of students that are below based, and we need to address that. STEM is an area that mostly is in NSF's domain. We support a lot of work. We collaborate with NSF on this, but it's a window that I think the Board needs to think about about how to make the argument for more funding for IES because the nation is falling behind on its STEM workforce. In terms of students with disabilities we have to solve that problem.

DR. LEE: So just a general comment on time. I think if we can go one hour more — I mean if you need to just take a break, do it, and then take like a half-an-hour break for lunch, and then come back to try to complete. And I can only see the hands of the faces, but I know there are more people, so that if you have something you want to say, and I don't see your hand go up, just speak. But I wanted to make a couple of comments, and I think this discussion, again, has been very helpful, and again, to the staff I appreciate the breadth and synthesis of your sharing with us. So on the question of like outcomes at the student level, and this is something — it's — there is this emerging — I won't call it a feel yet because it's not that, but sort of an emerging conception of something that people are calling the science of human learning and development that focuses on, and integrate a way of understanding human learning and development with some very interesting, I think, new work that's coming from work in the neurosciences. And so, for example, in terms of thinking about even student level outcomes, I think that we tend to focus on what we think of as cognitive indicators in terms of achievement in some fashion or another, but less attention to the role of also — what I think about is psycho social indicators that are contributors to cognitive outcome and achievement issues having to do with identity, you know, self-perceptions, and the like. And so how that might be another kind of area of thinking about a sort of impact. And equally around the methods question that in addition to thinking, as we sometimes want to think about dichotomies between quantitative, qualitative, and what we're thinking about is mixed methods. Again, I think there's some really emerging work around data mix systems, and methods attached to understanding dynamic systems which would encompass both the level of the individual bias of the level of context and participation within all spaces, and the extent to which this is a set of issues that are emerging, the possibilities of IES sort of taking this up to support this. And I was going to call on you, James, because I also had thought that these could be another interesting possible opportunity of relationships, some sort of joint kind of activity between the National Science Foundation and IES, and thinking about the National Science Foundation not only in terms of what we think about as sort of traditional STEM activities, but the science of learning, which seemed to me they are much situated in the work that NSF does, but the possibilities of really moving a whole new field forward. And related to that is the question of — so the efforts that you all have done in terms of studying impacts, and some of the documentation that I've seen around that. And I think this is addressed in that report, the Future of Education, is what kinds of initiatives might be developed that the Board could potentially give some feedback on it, thinking about it, of how do you measure the impact of the work that's done. So, we have the What Works Clearinghouse, and Eric and the practitioner reports, but the whole question of who takes them up, and what happens with them. And I think Peggy will be next. I see she's back. In terms of our sort of reporting, the field of education we still have these persistent — and if we could just take NAEP, NAEP — you know, I'm more on the reading side of things. NAEP, you have a little jump here and there on the reading. It's still relatively flat. And the vast majority of children even beyond the students with disabilities that you talked about, James, or I think Mark, at your measures of science achievement, on the whole we have such significant challenges that despite all the efforts and money we put in, we still haven't been able to fundamentally shift fields. And so what it might mean to try to figure out, and what I'm thinking of is like a theory of change, of what are the variety of levers from the policy community at state levels, at federal levels, of all of the varieties of institutions we have, what is the theory of change that can sort of push the things that we research, and find afterward to be taken up in practice. And then the last comment on the diversity, so it was great seeing the data that you all have collected, and the data shows — I mean, there are no surprises in the data. So, the question underneath that, I think, becomes what sort of practices, efforts, relationships, need to be built into — if you think about a theory of change, of expanding, the diversity of the pool of both applicants and persons through institutions have received awards, they're beyond the collection of the data. What would be the theory of change to be able to expand that? So, I don't have answers to any of these questions. I'm just saying I think it's part of thinking about IES priorities that these would be important issues to sort of take into consideration. Any thoughts relative to that?

DR. ALBRO: Thanks, Carol.

(Simultaneous discussion.)

DR. ALBRO: I just wanted to say thank you, Carol. I am mindful of time, and so we have lots of things that we could think about. I do think that the measurement challenge is interesting. I did want to call out for folks if you all had not been to edinstruments.org, this is the download. So initiated into this at Brown University, and it is really an attempt to crowd source, and pull together extant measures that exist in the field. And so, as we're thinking about like communities, and trying to understand what's available, Ed Instruments is a place to start, and there's lots of measurement on that — in that particular database, focus on like the social, the social side of all of this work. So, if you're not familiar with it, I just wanted to make sure that you had a chance to see what is there, and if you have better, different, new, other measures, to please submit it to that group so that we can continue to collect. We're always looking for opportunities to continue our collaboration with the National Science Foundation. We appreciate the suggestion around thinking about dynamic systems, and different ways of approaching human learning and development. I mean, as a cognitive scientist I appreciate, of course, thinking about human stuff, human learning and development as a group together. And theory of change is hard. We've been working on it. We've got some theory of changes, but you know, coming up with a single theory of change is really challenging, and so we appreciate thoughts and insights that you all have as we're trying to think about how we as a federal bureaucracy can make changes, and shift and change the lives and outcomes for individual learners, right, that we're all deeply committed to.

DR. LEE: So one of my takeaways from this discussion, which I have found to be very rich is, one, part of what I had hoped to come out of this meeting, was for us as a Board to sort of develop our own theory of action in terms of the scope of what our commitments are, and how we want to operate. And so, one, I think, is thinking about the role that the Board can potentially play in supporting some of these big-ticket issues that we've raised. Certainly, the reauthorization of the funding around us, SPED, but equally important some way of engaging in some fundamental self-reflections about how we achieve impact, and what are sort of new frontiers that perhaps are not currently captured in existing priorities. That would be helpful. So maybe now if we can move onto Peggy. I see you're back now.

DR. CARR: Yes. Thank you. And thank you for waiting. I had to drop off for the PETA release that are scheduled a year ahead of time, so the date I could not control. So, thank you for waiting. If someone could put up my presentation. Is it part of the — no one has it? Okay.

MALE SPEAKER: Peggy, I could share it if you would like.

DR. CARR: Yes. That would be great, so I have that.

MALE SPEAKER: One second.

DR. CARR: Let me say while he's bringing up the presentation the NASEM report for NCES was bold, it was daring, it was forward-thinking, and we really were excited to receive the report. I likened this report, this NASEM report for NCES addition for the future, NCES to be very similar to what Emerson Elliott received in the '90s from the Academy, the National Academy of Science, and they were very blunt. They said that if NCES did not improve what it was doing to take seriously its role as a statistical agency, they might as well close the doors, shut the doors. Emerson took what that report said to heart. And I think what we see today, as an NCES well-respected organization, had a lot to do with what Emerson did, and how he responded to that report. So, I'm glad to see that we have good, bold, and forward-thinking as part of our frontier, and so I'm going to take time today to tell you what we've done to respond to it. I should point out that in our conversation — we can go to the next slide, Stephen. In our conversation with the panel, they made it clear that they did not want to be confined by the lack of resources, the fiscal resources, the Human Resources. They wanted to explore what should be possible, not be confined by any limitations of resources. So, with that as the context we have done as much as we can to forward the good thinking of this panel. Next slide. What I thought I would do is just basically go through the themes. There are 15 recommendations here. We have addressed, I think at some level, almost all of them, but to go through the themes, and to identify how these things relate to the recommendations, and then take the bulk of the conversation to tell you what we have done in response to these things. The first thing was to develop a bold, and there's that word again, bold, strategic plan to make tough decisions, and we've done that, and that will be the basis for my discussion with you today. In April and May, we actually released a plan, a strategic plan. It's on our website. And then following not long after that we actually released an action plan, an implementation plan relevant to this strategic plan, what we have been able to accomplish, what we're hoping to accomplish in the future. So I invite you to go online and look at the details with regard to this plan. Another thing that was in bold, NCES to set its own priorities, and there are a lot of thoughts there about independence, and the Evidence Act. Actually, I think these next two things are related, maximize NCES' ability to fulfill the Evidence Act. As you know, ESRA is being reauthorized, and there are a lot of components about the independence and autonomy of NCES embedded in ESRA, particularly in Title 1. And the version that we have seen, I think, struggles, in my opinion, with what independence should look like. It is not commensurate, and my opinion with what the trust regs, and the Evidence Act reports that independence and autonomy should look like for an agency such as ours. So, I think that that ship is still out. Next slide, please. Diversity, and awareness of equity issues, is a major part of this report. Chapter 2 would ask you to focus on that. If you really want to understand what they're asking us to do is, I think, a major theme throughout this report, and most notably in this chapter. And we have taken it very seriously as we implement it, and, well, developed, and now implementing our strategic plan Another theme was to expand data acquisition strategies for a new insight. And I think what they're asking us to do there is to look at data science, and newfound sources of data, administrative data, and we've taken that challenge very seriously as well. Next slide, please. Prioritize data to increase relevance is something that we're going to have to work on operationalizing because we want to be cost-efficient, we want to examine what it is that we've been doing for decades, and perhaps figure out ways to do it differently, and that's going to take — that's going to take considerable effort. And I think we're going to need our experts, our stakeholders, to help us get through that, which really brings me to that next thing there, create engagement feedback loops. The report asked us to put together a nimble consulting body, and I don't think I get to say much more about that later in this presentation, but I want you to know that we have put together some experts to develop maybe a charter for our consideration of what that nimble consulting body might look like. And they will help us in our vision to help us prioritize our data collections, maybe even content of what we're collecting, but this is an important part of how we want to engage consistent with what the report is saying with our stakeholders moving forward. They are also asking us to enable data access in a way that we have not done before. So, they're asking us to think more creatively about our state longitudinal data systems, and how we cannot just provide technical assistance for their development of longitudinal data sets, but how we can ask them to, or hopefully get them to, allow us as a federal agency to be part of that community, not just providing technical assistance. That is what they want. It's going to be difficult to achieve, but we are exploring possibilities there. Next slide, please. Improved dissemination and focus on accessibility and usefulness. I will say more about this later in the talk, but we are looking at ways to track our products, and monitor them to see what stakeholders are finding useful about them, or not finding useful about them, so we can use that information to improve our products, and to improve accessability to them. What kinds of products are we producing, not just reams and reams of tables, for example, that people will not always find very useful, or accessible. We also are going to examine consistent with the recommendations of internal structure, and operationalization of NCES' responsibilities. There's a lot of silos, and you know we have four centers, but within NCES we have divisions, and these divisions, these three divisions that we have, and we have two units as well, often work more in silos than one might imagine. So we're looking internally how we can recentralize some of our operations, for greater efficiency consistent with the recommendations of the panel. So on page 94 of the report is this — don't worry if you can't see it, but it is a roadmap, and the report actually gave us a roadmap. They didn't just talk about it conceptually, and in terms of a framework, but they actually gave us a roadmap as to when we should be getting these major activities done, such as getting the strategic plan together, getting the action plan associated with implementing the strategic plan. Well, I can just tell you we are not on the timeline that they've laid out here, but we are on track. Next slide, please. So this is our strategic plan that I think is the major part of what the panel wanted us to do. We developed this plan not just from the top down, but from the bottom up, so all of NCES employees, and managers, had an input in what is in this plan. And we also shared it with our colleagues across IES. There are four goals, and here are multiple objectives beneath these goals. I won't go into them, but the first goal is to align products to data on needs. And I'm going to take some time today to go through each one of these. The second goal is to improve and innovate NCES operations, and I think we have some really good examples with what we've done, and what we're planning on doing with regard to this particular goal. Foster and leverage mutually beneficial partnerships. I mentioned the nimble consulting group that we are excited about, but we've also started to develop stronger relationships with stakeholders that we really haven't had that kind of ongoing, mutually beneficial relationship before, such as HCBUs. And to then go forward to embed and integrate these principles, diversity, inclusion, accessibility, into all of what NCES does. And I don't mean just sampling. I don't mean just reporting. I mean everything, and so we're going to go through that, and tell you what we're doing there. Next slide, please. I thought it would be interesting to lay out why I think — going through the strategic plan today is the best way to sort of respond to your inquiry about how we are addressing the NASEM report. There is a huge overlap between what's in our plan, and what NASEM asked us to do. Next clip, please. These are the things that I just went through, and you can see the recommendations behind them, and we're going to go through the strategic plan, but what I want to demonstrate is that all of these things are embedded in all of these themes from the NASEM report, are embedded in the NCES strategic plan. But there are some parts that are not that we still need some collaboration, some work with stakeholders both in terms of IES, and the Department. So, their recommendation in the plan — I'm sorry, recommendation in the report that encourage the Department and IES to collaborate for ensuring the independence of NCES, and to support the Evidence Act, and the NCES role in the Evidence Act, and not just NCES, but also the statistical official also identified in the Evidence Act. So, I think this is still a work in progress. The trust regs, if you have not seen them, delineate — the draft trust regs because they're not final yet, delineate how the IES, the parent organization, is to operationalize, or support, and enable, the independence and autonomy of NCES as a statistical agency. So I think we need to have some finalization of that trust reg. And also what Congress wants to do with regard the independence of the NCES commissioner whether it remains a presidential appointee, for example. And some of the other very clear independence signals in the reauthorization of ESRA. The new contracting arrangements that's still something that we need to think more about, but you will not see much of reference to those activities in our strategic plan. Then there are those things within our strategic plan that are not in the overlap between the report, the NASEM report, and our plan. I want to point those out to you. We are revisiting as I speak, our statistical standards. One of the things that Emerson Elliott was asked to do back in the '90s when NASEM gave him his report for the future was to develop some standards, some real strong defensible standards. And we have revised these standards a couple of times since I've been here at NCES, but it's time to do it again. And we are in the process of doing that from the top down, and from the bottom up. Everyone is involved in the revision of these standards, and they're going to include DEIA, not just as a chapter, or a portion of the standards, but everywhere throughout the standards is our goal. We also have a goal to have psychometric standards, which were not sufficiently dealt with in the statistical standards, so we now have a chief of psychometrics, and that chief will work with the chief of — the chief statisticians to develop standards. And we have some outside experts that are helping us to articulate what the scope and framework might be for psychometric standards. We want to continuously address issues of data quality. There are data quality issues as all statistical agencies are struggling with, for example, around response rates. So that is part of our strategic plan, and not really dealt with that much in the NASEM report. And we also think that we need to work on internal and externals, and internal, but we need to work on internal communications both within NCES, and within the Department about what our role is, particularly now as it has been defined in the Evidence Act, and will be defined more, and in more detail, in the trust regs about what it is that we should be doing as a statistical agency. I have found that as these documents have developed, have been circulating in the federal statistical system, and is parent organizations and partners, that we were not all on the same page. So, I think there has to be some improved communication in that regard. Next slide, please. So, these are the goals. And go to the next slide. So, what I'm going to do now is sort of go through each of the things that we have done and give you some exemplars with regard to each one of these goals. Just as a reminder, this particular goal addresses the NASEM themes of data acquisition, prioritizing data for relevance and feedback loop that I've described several times now as I've gone through my talk. We are most excited, I think, when it comes to a new and innovative product about our School Pulse Panel, our SPP. As you know, IES was asked — was given funding, and asked to collect information about the impact of COVID, and to monitor it. Some of those funds were allocated to NCES to collect data using this new instrument, the School Pulse Panel survey, that goes out every 30 days, with a set of modules some of which are constant, but many of which rotate in and out every 30 days. Here we're getting realtime information from the K–12 schools about the impact of the pandemic, what the recovery looks like, not just for students, but for teachers, for the schools, the entire context. What you see here on the right are the latest results. We just released some results to stakeholders from the August Pulse Panel, and you can see a very interesting finding; 45 percent felt they would understand; 45 percent of the schools, K–12 schools, felt they were understaffed entering into the '23-'24 school year. I happen to know the results. We just released them. It's better than it was when we did this last year, but there are still challenges with schools being understaffed. On the right here, the furthest to the right here, you'll see what we're planning for December. That will be school facilities, something that many have expressed interest in, and also learning recovery. This is one of the modules that we continue to cycle in and out of. So, we are very, very proud of this new product, and aligning these products with the needs of our stakeholders. I'm reminded by teachers, for example, when they see these data, they say they for the first time see themselves in data that we are releasing. So, we're very happy to hear about that. We have also undertaken a listening tour of districts. I mentioned earlier that we have challenges with response rates. There are certain states, and certain large urban areas within states that always say no to our requests, or rarely say yes to our requests, to participate in our studies. So, what we have done is to develop a listening tour asking for nothing, but just to talk to the chief superintendents of these districts, sometimes within the state at a higher level, and just tell them what we have to offer, what kind of data that we produce, and what we could do to make their participation more meaningful for them. And we've held a series of these listening tours over the course of about three months, and we've gotten districts who normally would just say no. Baltimore, for example, is good for saying no. We really can't do it. Montgomery County where I live almost always say no, but we've gotten some good response from this listening tour. The other thing we've done to align our products to data needs is to launch a different way of developing our portfolio of products that we release in terms of our publications. What we used to do years ago is to sort of sit around the table very much like you would do in a dissertation defense, and people would defend — each unit would defend why they think a particular publication is worthy of being part of our portfolio. When we have put that process in this writ system, and it has an opportunity for everyone to comment on why a particular area should be the focus, and we can also see what everybody is proposing to plan for their next publication, so we can decide if there needs to be an alignment, there needs to be some editing, or trimming of what our portfolio, or releases of publications, might look like for the coming up year. It's a wonderful system. Next slide, please. If we had more resources, I know the panels that don't worry about resources, but we have to worry about resources. We would increase granularity of our data of products by doing things like more small area estimations. We have done small area estimations for adult literacy, data collection, a PEAC (phonetic), for example. Just smashing success. But we think we could probably use this methodology more with more funding, and more FTE to do it. Another example would be concatenating samples across low response rates for private schools. We are having a heck of a time getting private schools to understand why it's important for us to have their responses to our national surveys as well. They're part of our education system. And so often we're not able to report on them, so we're looking into some methodology that will allow us to concatenate across years, so that we can increase our samples, and decrease the bias around those samples, and report on Lutheran schools, or more than Catholic schools, I should say. We want to build state and local capacity by leveraging geospatial data. This is the pot of gold that no one seems to, at least internally, realize that we have, that this geospatial data is such a valuable asset. And we get millions of hits on geospatial websites a month, not a year, a month. People are touching these data, and we could do a lot more if we could figure out how to link across states to use state of the art methodologies to link to our data, our state data, and to data from other agencies using this kind of leveraging with something like geospatial data. It's very exciting work. We would also, if we had additional funds, we are now working in a pilot, for example, with three other statistical agencies on ways to use AI to crawl around in peer review journals to gather information about how our data sets are being cited in these journals. Researchers are using them, what they're saying about these data sets, how they're using them, which variables they're using, and then to put all of this information in displays, so that we can share it internally, and share it with our stakeholders. GOIs are something that we're working with our colleagues across all of IES to mark every publication that we have so we can track it and see how it is being used. And, of course, AI is everywhere, and we want to use AI with additional funding not just in NAEP, but other places, in other surveys, in NCES to do things like item generation, or doing item scoring, but item generation as well. Next slide. So,, Goal 2 is to improve innovation — is to improve and innovate operations. I mentioned this earlier. We want to address the silos that we have, and these are the things that this particular goal, and these objectives, address. Next slide, please. So, I mentioned earlier that we hired a chief psychometrician, and this is important because we often think about NAEP when we think about assessments, but we have assessments elsewhere in NCES. We have longitudinal ECLSK, the high school longitudinal assessments; there are assessments there. But the advances, and the innovations, and assessment activities that we see in NAEP are not necessarily being applied to these other large scale data collections that we have, for the national data collections that we have. And this psychometrician is also going to look to other areas where we can develop indices, such as the equity indicator that was mentioned earlier. The edge data, that's just this geospatial data that I mentioned earlier, has so much potential, and stakeholders know that it has a lot of potential, but we haven't done, we believe, as much as we could do moving forward. The other thing I wanted to mention I've already talked about the private schools in the automated scoring, but I wanted to mentioned that our longitudinal data collections are exploring using remote administration, remote testing, not just going to the schools, and this is going to be very efficient if we can actually fully operationalize this because some of these longitudinal study participant students by the time they get to, say, middle school, or in the higher grades, they have dispersed so much so you might only have a handful of students in a school. So, it's very expensive to go there to just test a few students. So, testing them in settings outside of the brick and mortar is going to be very efficient, and very cost-efficient if we can do it. Next slide, please. We would love to have an R&D unit, and the NASEM report says a little bit about this, but with additional funding, and resources, and for staff, we think that this would go a long way to modernizing our methodology, ways of collecting data, ways of using data not just survey data, but other data that we perhaps don't collect that might be useful. We want to do more of that. And I've mentioned the geospatial work, but I want to point out that this Geospatial work that we want to do with additional funding could address these issues of better poverty measures than the ones that we've had. Free and reduce indicators, free and reduce price lunch has been used since the proxy for SES, and, of course, that indicator was not developed for that purpose. But these Geospatial studies that we are exploring we believe will help us to develop not just school level poverty measures, but student level poverty measures that can be generated in such a way that we can have more accurate poverty measures for a school. We also want to look into centralizing our sampling methodology. A lot of the contractors who do our sampling across NCES are the same contractors working for different data collection activities. We think we might be able to centralize some of the contracts, or some of those sampling methodologies, so that we can have more efficient sampling, and more cost-efficient activities across the center. Next slide, please. Goal 3, fostering and leveraging mutually beneficial partnerships. Let's just go to the next slide. To address this particular goal, we hired a director of partnerships. We have a lot of outreaches through a form, which is actually mentioned in our legislation, and the National Post-Secondary Education Cooperative, so we have a lot of outreaches, but it's not as well-coordinated as we think it should be. Our director of partnership is going to work to improve our coordinated outreach. She is also going to work at the level of operation to make sure we know why our partners are having challenges with our data collections, who is saying no, why they are saying no, how often they say no, what did they say yes to, those sort of nitty-gritty pieces of information that we kind of known in silos, but we don't know as a collective. And similarly, we need to take a stronger look at the state longitudinal data system to see if there are ways that the federal government can work with these grant recipients to the benefit of us all. Next slide, please. We would love to have a state coordinator group very much like NAEP has a state coordinator for every state. And I think this is actually mentioned in the NASEM report. If we had funding to do that, we would perhaps not have at the state level, we might have at the district level, but we would like to have a coordination group of that type to further our mutually beneficial partnerships. We have some of the most renowned psychometricians and statisticians in our field who came through internships with NAEP — I'm sorry, NCES, many, many years ago; the Larry Hatchers of the world, or the David Kaplans of the world. They came through NCES years ago as young scholars, and through an internship, or fellowships as it will, and now they're some of the best in the world, and we would like to, if we have funding, to rekindle that activity. I think I'm going to move on. I think I have actually already said enough about this latter two points. So, let's move onto the next slide. And more on the embedding DEIA principles across NCES. I think maybe since the last time we met perhaps aligned with that time period. We released an equity in education dashboard. We actually have a dashboard now that pulls all of the publications, or products that we have, tools as well, into one dashboard where it's like a one-stop shop for equity with regard to our product, or products that we partner with our other statistical agencies in collecting. And this particular dashboard is based on that equity report that someone mentioned earlier published by the National Academy back in 2019, has 16 or 17 indicators, and a really solid framework for not just outcomes, but also variables that mediate equity, and that's how we pull together that dashboard. That NASEM report is also the framework for our equity indicator that hopefully will be not just national, but state level as well, using as a guideline the indicators identified in that report. Our IPEDS data collection is also exploring some equity indicators through piloting of some particular areas of concern by our stakeholders, like how we operationalize sex or gender in our report, so there is a question that's being piloted with IPEDS now. Legacy admission is another one that has been fairly — has been at the top of mine, by many in the post-secondary world. So we are piloting a Legacy question, and also race ethnicity admissions. We have information about who applies, but we don't have information about who is actually admitted. And you know the Supreme Court in their decision, this is kind of related to this, but not because of that. We have included a pilot question here. We were actually thinking about it long before the pilot — sorry, long before the Supreme Court made its decision. And last, but not least, I wanted to point out that we've taken leadership roles in various interagency equity working groups, as many have across IES, to provide a presence in the statistical practitioners' community. NCES is leading, for example, the ED SOGI plan, which is required by House, the OMB. We also are co-chairs on a couple of the interagency subcommittees on defining, redefining race ethnicity SPD, Statistical Policy Directive 15 that the chief statistician has been working on and will finalize in the summer of 2024. And I am one of the four cochairs of the White House Equity — the EEO, the Equity Initiative for 13-985 that the President signed on his first day of office back in 2021. And we have a lot of different projects there. So, we have been a presence in the community, and taking leadership roles in the community, and this, we believe, is relevant to the NASEM recommendations, and to our strategic plan. And, finally, I think there's one more slide, we want to do more. We think the resources that are needed here are probably more in terms of number of FTEs as opposed to funding. We just are short on people as it is. Our FTE is smaller than it has been in ten years by 20 head counts. And we want to do more in this area in reaching out with MSIs, and HBCUs. For example, the first time I've seen in a decade attended the HBCU conference that they have here annually and made a big presentation. It was standing room only. We need to do more of those sorts of things, but we need to do it with concentrated FTE that can be assigned and can focus on this kind of work. And we also need to be able to publish more in this area. Actually, analyzing our own data. Resiliency analysis, for example. We can't just continue to report on gaps, how big the gaps are, where the gaps are. We need to do more diagnostic work. That is what we would do if we had more people, more FTE. I believe that is the conclusion of my presentation. I said a lot, and I'd be happy, Chair, if there's time for questions. I'd be happy to come back if your break — if you are targeting your break for right now, but —

DR. LEE: Well, if people are willing to stay on for a few minutes more, I think it would probably be most efficient to address questions right now, and then once we do, we'll go onto our break. So do you want to close that screen down, so I can see who's —

DR. CARR: Stephen, if you could close that. Thank you.

DR. LEE: So, Conchita?

MS. LEGORRETA: Yes. So, thank you so much for that information. Stephen and I, when we give our report, the information you gave that's going to be kind of the model that we're going to look for others. So, thank you. I have two questions. One is how many schools, or systems, about are taking part in the SPP? Just to know like is there a large part? How big kind of the sample is. And then the second question I have is around the DEIA focus, and I love that it includes the DEIA, as the executive order mentioned. Is there a requirement, and, I guess, it would — my question is kind of broader than just the specific center. Is there a requirement that all applicants be required to include their results, products, research, in an accessible manner, and is this included as part of the RFPs?

DR. CARR: The Pulse, I believe, has something in the range of 1,200 respondents, and it is — I think the most important thing about the Pulse is that it is nationally representative. It is a strong sample. We drill down to the local level, locale level, I should say, as well as the regional level, to make sure that parts of — all parts of the United States are represented. So even though it's not a huge sample, it is a very strong and reliable sample. Your question about equity issues, inclusion, and diversity in our RFPs is a really good one. We have — there's room for improvement there. I think NAEP has a really strong model with regard to requiring as vendors to talk about how they're going to ensure equity in their administrators, for example, or how they're going to encourage professional opportunities for up and coming psychometricians. There are like some internship programs that are in some of the NAEP contracts, and they were collaboratively across the alliance contracts. But there needs to be more about who is sitting around the table, who is involved in item development, who is involved in the analysis of the data because that is the perspective they're bringing to those particular activities. If that's your question, there's lots more to do in that area.

MS. LEGORRETA: And my question is more specific on like accessibility. So do we require inner RFP —

DR. CARR: Oh.

MS. LEGORRETA: — for all of the products, and everything be accessible to both the participants who made the — in the research, but then also their final products.

DR. CARR: Oh, yeah. They have to be 501 — 508 compliant. Yes. Everything has to be 508 compliant. But in addition, for accessibility regarding data collection, we also require accommodations. We require universal designs in our data collections, the devices, the instruments. Yes. Those are part of the RFPs.

MS. LEGORRETA: Thank you.

DR. CARR: But 508 compliance is required across the Board.

DR. LEE: Okay. Shaun?

DR. HARPER: Thank you. Peggy, thanks for such a spectacular, very, very, very thorough presentation. It was very impressive. Two questions. One, could you give us a sense of the integrated R&D unit? What level of resources do you imagine would be required to pull something like that off?

DR. CARR: Well, we need expertise that we probably don't have now. We want to be able to crawl around, you know, do web scraping. We have researchers that do web scraping, or vendors that do web scraping for us, but we need someone internal to NCES to manage that kind of work. Data scientists. I know that, you know, everyone is talking about data scientists, they're unicorns. We need the kind of unicorns that really understand education data, and that can take their knowledge of data science, and apply it to our work. Figure out creative ways to use found data, whether it's transaction data, whether it's administrative data that's at another agency. HUD, for example, has reached out to us, and they want us to figure out how we can work together to help them understand equity issues with their data. So, we need dedicated staff more than anything else, to be quite honest, to implement this. It's not — funding, of course, would be helpful to have contractors to help you operationalize those activities, but I think it needs to start with highly specialized, focused FTE.

DR. HARPER: That's very helpful. My second question is a quick one. The equity and education dashboard, how do we get to it?

DR. CARR: We'll send you the link. Stephen, you can put the link — Carol, Dr. Lee, if it's okay, we'll just put the link right in the email. I mean, in the chat.

DR. HARPER: Okay. Fantastic. Thank you.

DR. CARR: I just want to say I love this dashboard. Everyone loves this dashboard, but it is so much more we could do. You talk about what else needs — what else we could do. We need to push it down to the district, and state level. We need to add more tools. We have a lot of tools across the center that we think could be tweaked, so that they are focused on equity and inclusion. There's item mapping. There are all sorts of things that we have that just with a little bit of a tweak they could have an equity focus. So that dashboard is just the beginning. As happy as we are about it, it could do a lot more.

DR. HARPER: Fantastic. Thank you so much, Peggy.

DR. LEE: In addition to putting in the chat, Ellie, could you maybe after the meeting just send it to us via email since everybody isn't necessarily downloading from chat right now? Denisa?

MS. PELAEZ: Yes. Thank you.

DR. GANDARA: Thanks, Peggy. It's really exciting to see all the excellent work that you're doing at NCES. Thank you for sharing with us. I'm thinking about the public comments we received from the Institute for Higher Education Policy regarding post-secondary sample studies, and in full disclosure, I'm on the Board for IHEP, and I'm also a higher ed researcher who uses a lot of these data products. So, I'm curious to learn about the process for evaluating the relevance and usefulness of data products, and to what extent you engaged stakeholders in the decision-making process. And just for further context in case others haven't seen the letter from IHEP, IES has decided to — well, my understanding is IES has decided to discontinue baccalaureate and beyond, and potentially scale back NPSAS, which is the post-secondary student aid survey, and PBS. And again, as someone who uses these data, and as a member of the community who uses these data, I was just really surprised to learn about these decisions. I'm curious to know more about the decision-making process. So if you could please share it, that would be great. Thank you.

DR. CARR: Thank you for your question. I, fortunately, did read the letter, and thank you for your interest. NPSAS is one of the few mandated data collections that we have. IPEDS is mandated. NAEP is mandated. NPSAS is mandated. And so, we have to do it. I mean, it is the only source of information about how low-income families, families of different — at the student level, I should say. Students from low-income families of different racial ethnic backgrounds, where we can get that information, and understand how they are paying for college whether it's Pell Grants, or loans, or whatever. So, this is valuable, but it is expensive. And I should point out that the decisions about B&B were made before I took on this role. So, I am not totally privy to exactly how that decision was made, but baccalaureate and beyond is one of those — it's a spinoff, as you know, of others on our call who may not know, but it's a longitudinal spinoff of NPSAS. But as I understand it, prior to my coming we have to figure out how to stay within our budget. Our budget is not improving, it's not increasing, and so we have to cut back, I think is the bigger point. With regard to what we're doing with NPSAS itself, what we're planning to do is to have an administrative data collection year to be alternated with a sampling year in which we also collected administrative data. So, we would go in that cycle every three years, and that's going to save us a lot of money because collecting information from students is very expensive, but we believe that we are still going to have meaty information about how students pay for college. Our biggest problem with NPSAS, I don't want to belabor this point because it wasn't in your letter, is that it looks like we will not have access to federal tax information, which we used to get from our partners over in federal student financial aid when the data got here, but because of the way FAFSA now collects information from families, it just gets put right into the system, it's protected, and we can't get it. So, the biggest concern I have is, quite honestly, and Matt, my colleague over in evaluation, has expressed a similar concern for his activities, is that we don't have access to these data for NPSAS. B&B, yes, we need B&B. I would love to have funds to implement it, but the biggest problem I see coming in the next couple of years is that at some point someone is going to realize we don't have that data. We are in contact with our stakeholders, so stakeholders, let us know when we make hard decisions. Remember that first recommendation I showed you that NPSAS said we had to do? It said you have to make hard decisions because we can't do everything. So I don't think we're in a vacuum this nimble supporting, advising group that I mentioned earlier. That's one of the things we're going to ask them to help us do. How do we make these decisions? What will be the criteria for making these kinds of decisions? Who should we talk to? What are the tradeoffs? So it's not an easy answer unfortunately.

DR. LEE: Denisa, your response to Peggy's response? Our response to this request from — how do you pronounce it? IHEP?

DR. GANDARA: IHEP. Yes.

DR. LEE: Our response to IHEP is going to be a purview for this subcommittee to work on, but I'm just curious as to whether you have any thoughts or response?

DR. GANDARA: Thank you, Peggy, for your response, and for sharing the additional context about the fact that this decision about B&B was made long ago it sounds like, and in the additional context regarding FAFSA, and tax information, it's not something that I was familiar with, and it does seem like another cause for concern. I will be really curious to know more about the advising group, and to learn more about the process that will be put in place for making these kinds of decisions about which products to discontinue, and what areas to expand in. So I'd like to get an update on that once that advising group is in place.

DR. CARR: Well, we can certainly provide — Madam Chair, we can certainly provide more information at your request regarding the FAFSA group because we are working on a charter, and perhaps this group might have some input, or some ideas about what should be in the charter before it's finalized, because this is one of the goals that we're hoping this group help us to address prioritizing of our data.

DR. LEE: So two issues that have emerged, not to discuss at this point, but, I think, should come up perhaps even tomorrow when we get to sort of bigger issues, is, one, is that I'm hearing that, one, there are ESRA questions, and one of which I understand has to do with some potential recommendations about the appointment of the director of the center that I think is emerging in the ESRA discussions on the Hill. And the other is — again, I think this would be useful as a topic tomorrow after we've gone through all of these reports, are the funding questions relative to certainly the ESRA reauthorization. There seemed to be a lot of complicated kind of issues about what rationales for increased funding for whom, you know, under what circumstances. So, I've heard this generally on the IES side, and now specifically relative to NCES that would be helpful. I have one quick question, Peggy, and that is how do you all — and I guess this is a Peggy, Mark, other directors' kind of question, about relationships across the centers? So there seems to be some common issues like around equity, or the issues of the kinds of data that NCES collects, and its relationship potentially to priorities that other IES centers develop. So, I'm just curious as to what kind of structures do you all have in place for communication, and feeding on one another's data? Like how does some of the kinds of reporting coming out from NCES inform some of the kinds of targets that the other centers seek to address? 

DR. CARR: Mark, you can start.

DR. SCHNEIDER: So, a couple of things. So first of all, NCES — I'm sorry, IES is a relatively small organization. We were all in close physical proximity, and that really matters. And so, you walk down the hall and talk to people, right? And even after the pandemic people are here an awful lot of the time, and that makes coordination a lot easier. I mean, we all know this, right? I mean, these meetings — we have to have meetings like this, but the fact of the matter face-to-face contact and discussion is actually quite important. And just an aside, which is a very human side, we're a five-minute walk from the Wharf, which I don't know if any of you have been to the Wharf, which is a relatively new area in Washington, D.C. with lots of bars, and lots of restaurants, and like a 5:30 trip to a bar is actually, again, incredibly important for human interaction. So, I mean, this is really important. The second thing that I'll note is that I've been in various circumstances, education, universities, et cetera, and I would say that the senior leadership team is one of the most cohesive and friendly groups that I've ever encountered. So again, I mean, the close proximity, the bars, all this matters, and it's actually an amazingly collaborative and friendly senior leadership team. So, you have to keep that in mind also. The third thing, I think the third is that when I came on Board IES had a terrible history with regard to communications, and communication really matters because we're producing reports, and agreeing on reports, and things like that actually matter. So I hired a director of communication about four years ago, and just recently promoted her to be the deputy director, which is a pretty serious step forward for Comms, and that actually becomes another mechanism for ensuring discussion. So, I think that the — I'm sorry, the last point is Matt laid out his, I'm not sure, theory of action. Is that what you want to call it, Matt? Where there's like this virtuous cycle with data that comes out of NCES, and there's studies from NCER and NCSER get fed into each other, and that really matters.

DR. LEE: Well, let me just say I think it's wonderful that you have the personnel infrastructure. That's really very, very important. I was more concerned a question about a strategic set of strategies or policies that you all have in place for formal ways of thinking about the data that's collected, for example, and its relationship to priorities for funding, or the joint interest across all the centers around issues of equity where your efforts to address them. Like Peggy, you were talking about going to the HBCU conference for not necessarily only on behalf of NCES, but the broader across centers. So, it's more I question about strategically how these relationships inform one another. 

DR. CARR: Well, from my perspective we have lots of room for improvement in terms of having a systematic, strategic process for ensuring that that kind of shared purpose and activity is actually happening. There are some good examples, however, notwithstanding of when it has worked. NCES, for example, collects process data on not just how students answer a question yes or no, correct or not correct, we also have information about the process they undertake to arrive at that arrive, and we put that data set together, and to Matt's group — Nate's group, sorry, Nate, you're still new, give it to Nate's group, and they put out an RFA using that data. Or whether students with disabilities are using accommodations, or universal designs that we have built into it, and is it really helping them. That's the kind of coordination that we should be doing more of that we're not doing more of, I think, and we would do if it were put together in a very strategic way. I think another good example would be the IES summit that was just done, and we all participated in the summit. Over 2,000 participants, over 120 representatives, who made fantastic, very impressive presentations, including the Dr. Bernard. I hope I got his name right, the black astronaut who kicked it off. But many of those participants recommended by our colleagues down the hall that they need to be involved. So, I think that there are good examples, but we haven't codified it, and stratified it into goals, and objectives, and procedures in a way that it happens, and it just shouldn't happen because somebody came up with a good idea, and we go down the hall and bring everyone in.

DR. LEE: Well, again, that's helpful, and for me the reason that I'm kind of raising the question is that I think a fundamental issue for the Board, and IES together to wrestle with, is the question of uptake and impact of all the work that's being done. And so, it seems to me the relationship between the breadth of kinds of data that NCES collects should be an important sort of indicator, if you will, on level of the impact of the work that IES is supporting. Denisa?

DR. GANDARA: Yes. I'm sorry for keeping us from our lunch break here. Just a quick clarifying question. Peggy, I think I heard you say that there might be an opportunity for us as a Board to provide input on the advisory group's procedures for prioritizing data products, and I just wanted to ask, it sounds like you're nodding yes. Carol, is that something we could also discuss either later today, or tomorrow, the possibility of having NBES offer input on NCES' advisory group's procedures for prioritizing data products?

DR. LEE: So, one of the big issues that we're going to discuss, and probably that will be tomorrow, is the relation — as we're talking about forming sort of permanent committees is to whether or not we want to establish some formal committees that have relationships with specific centers. So, we can — we'll kind of, I think, come back to that tomorrow. So why don't we — 

DR. SCHNEIDER: Could I just add a point of information —

DR. LEE: Sure.

DR. SCHNEIDER: — or whatever order, something, some point to that? So, remember under the existing legislation you're an advisory board, and almost all of your formal responsibilities as specified in ESRA are — is to advise the director. So just keep that in mind as you start thinking about these committees, and who is talking to whom. But under the existing legislation the — I think there are probably nine or ten activities that are specified in ESRA, and almost all of them are designed to say the Board shall provide advice to the Director.

DR. LEE: So these again are some issues. I think one of the challenges, I'm just going to speak personally, is that I think that the charter, and I'd mentioned this to Jacob I think is the contact that we've had with the White House, that I think that the charter for the Board is very vague and conflicting, and it moves back and forth between sort of advising, and more direct kinds of roles. But, I think, that — and part of this may come back to the conceptualization, Mark, that you have about the pathways through which you as director get advice, and I'm thinking that the ability of the Board to interact with centers can still be viewed as ultimately, you know, going up the chain of command, if you will, to you as opposed to everything that we do is only at the level of interacting with you, which I think — I mean, the reason we had all these reports is because getting data from the various centers, and understanding the challenges that they're working with informs, any kind of recommendations that we make. But we can, you know, pursue that further because I think this is going to be a learning journey for all of us in terms of the work, and the focus of this Board to be effective as an advisory board. So why don't we take a break. I have 12:24 Central Time. So why don't we come back at 1 o'clock, 1 o'clock Central, 2 o'clock Eastern, and you all can go down the line. So basically a little over 30 minutes. Does that work?

MALE SPEAKER: That works.

DR. LEE: Ellie, we'll all just sort of stay here online, and just leave, so that you're not having to keep readmitting people. Will that work?

MS. PELAEZ: Yeah. That's fine.

DR. LEE: Okay.

MS. PELAEZ: All right.

DR. LEE: Thanks everyone. Thanks, staff, for all the wonderful reports, and we'll see you in a half-an-hour.

DR. SCHNEIDER: Thanks.

(Whereupon, at 1:25 p.m., the meeting in the above-entitled matter was recessed for lunch, to reconvene at 2:00 p.m., later that same day.)

Afternoon Session
(2:00 p.m.)

DR. LEE: How are we looking, Ellie, in terms of we're back?

MS. PELAEZ: I mean, it appears that we have a quorum, folks are back.

DR. LEE: Okay. Great.

MS. PELAEZ: Mm-hmm.

DR. LEE: So, I'd like to move on to Conchita and Stephen's report around the Board and upcoming policy legislation.

MS. LEGORRETA: I don't see Stephen. I'm not sure if he's back. Okay. I will go ahead and get started. If Stephen joins, we can get that. So, we had different conversations looking at the lens of legislation in different areas. So, one of the things we talked about was internal review of systems at our centers, and kind of looking at it in that way, and then also given the Administration's concerns about learning loss, and educational equity, and the pandemic, what does that look like for us to be able to improve our kind of internal policies and procedures. And then we also wanted to touch on the NIH designation. They recently designated people with disabilities as a population with health and educational disparities. So, we wanted to kind of focus on how we can make sure that our information is kind of aligned with that. And then also what does it look like to require demographic indicators in research for the projects that we support, specifically around disability. So that was kind of the broad implications. Mark and Steve had a call, and as we all know, there's a lot of things that we can and cannot do in terms of Board members. So, one idea that we had specifically around the funding, and how as different people have mentioned before, over time, the funding has gone down, and what does that look like. So, we're proposing, and we definitely want to get the feedback of Mark, and others who know kind of what we can and cannot do, but one of the ideas was a day on the Hill, and this would be specific with the Help Committee, and the Appropriations Committee, in keeping it very bipartisan, very — not bipartisan. Sorry. And as a way to not advocate, but rather, educate the people on the Appropriations and Help Committee as to this is our need. And the example that Peggy gave was really perfect in her presentation. That's the kind of information we would need if this is something we would want to do. The best time to do Hill visits are in March and April, so we could start small this year where it's just the Board who goes, and then the following years we open it up to our stakeholders to support. But what this would look like is they — I've been a bunch of times, but they can provide us with space on the Hill to have like a pre-meeting where we could have people who are kind of supporters of our work come give a couple words, and the main focus would be creating a fact sheet. So, this fact sheet would include like, for example, the information Peggy shared, but coming from kind of all of the different centers, and why it's important that these initiatives be funded, and it would be very simple, easy to understand. They would be pretty much training everybody during that visit on these are the things to talk about, and then breaking up and visiting the different people on the Help and Appropriations Committee. And it does take time for this tour, but, I think, beginning with the visibility of this is who we are, this is why it's so, so important, and getting people in the legislation on our side in order to support us with this. Again, we're not sure if this is something that we can do, but what a lot of organizations do to kind of get around this is as a way to educate legislators, not as an advocate, and so the fact sheet would reflect that in making sure we have kind of that information. So that's our brief and mighty report. So, if we were to decide if this is something we, a) can do based on the regulations, and we would want to do, what we would need from the different centers is kind of similar information to what Peggy shared, and then we would work on putting together a fact sheet. And I have contacts with the Appropriations Committee, so we can get the meeting and rooms reserved, and get that set up with the idea that this year is kind of smaller in the Board, and the following year we would have a bigger, inviting educators and constituents, and then not only hitting the Appropriations and Help Committee, but also representatives so that people can go to their offices. So, Stephen, are you on by any chance? Okay. So that's kind of our report. We'd love feedback from a) is this possible with our parameters of the Board, and if people are interested that this is a way to do that to kind of uplift funding that we really need.

DR. LEE: So I'm assuming this might fit in with broader discussions that I think we ought to hold tomorrow about any relationship that we may have relative to ESRA reauthorization. So, you might include what is the scope of specific recommendations we would make, some of which we will vet in a broader discussion tomorrow. But I am curious, Mark, or Ellie, or any staff who can advise us on this question that Conchita has raised about our role — I would assume that I would be interested in hearing back from them that as an NDS Board advising IES that we could, in fact, advocate. It's not like we are personally advocating, taking the political positions, but in some sense, I would assume that's part of our role as a Board. Mark? Is Mark back? If not, maybe Liz, you may know. 

DR. JONES: They may be having trouble with their microphone.

DR. ALBRO: We got it. I got it, Stephen. Thank you. Yeah. Hello, Carol. We were just thinking that this is probably a question for the FACA attorney, and we were all just kind of trying to look to see if the FAFSA attorney is on in terms of making sure that we follow the guidance. So I don't know who that person would be, but Ellie can connect us, connect you all, up with the appropriate attorney, who can help you provide guidance, and sort of a framework by which you should be interacting with our colleagues on the Hill.

DR. LEE: So part of the discussion that I had with David, I think is his name, the last Board chair, around the functions and the process that they had from the last Board of hiring an executive director, and part of what he said was that that person can also serve as a kind of liaison among other responsibilities between this Board, and the Congress, and the Secretary of Education's office. So again, we will pursue that, but I would assume that this should be within our purview. Doug?

DR. FUCHS: Yeah. Conchita, thank you for that report. As someone who has participated in education efforts on the Hill in the past both for IES generally, and NCSER particularly, I have a sense that it can be helpful, but as I'm sure you know, it's not the necessary and sufficient condition for moving people in the direction of greater funding. So, I mean, it's got to be part of a larger strategy, and I'm certainly not asking for the specifics of a larger strategy right now but would be very interested to hear about that at some point.

DR. LEE: Do you all as staff have any — I mean, what are the activities in which you engage certainly as you're, you know, arguing for increased budget funding, for example, or other kinds of priorities that are going to be impacted by legislation at the congressional level? How do you all engage in communications or seek to inform them? 

MALE VOICE: Yeah, Carol, that's a great question. And so, we do often have occasions to provide feedback to both our appropriators and authorizers. We tell them what we — you know, what we are doing, and what we might be able to do is increase resources, but it is in the spirit of giving information only to inform it's their deliberation; it's not for the sake of lobbying. So, it really is kind of a descriptive for lack of a better word kind of account, you know, what we're doing, where we'd like to head, and what the kind of delta between those two things are, just so we're sure we don't run afoul of any actual kind of legal requirements.

DR. LEE: So, I would assume once we get a sense of the sort of scope of authority, we have relative to the relationship of this Board to the activities of the Congress coordination of information would certainly be very important between internally what you generate, and what we would attempt to try to support, you know, externally.

DR. FUCHS: Carol, if could just quickly add. My limited experience in work being — you know, working the Hill to try to get more money for IES, a really great group of potential supporters, collaborators, are the representatives — what do they call them? The Washington, D.C. representatives of respective institutions of higher education. These people — the people that I've worked with are extremely sophisticated. They know exactly who to speak to, and it's rarely — you know, it's rarely a Congress person. It's staff, it's key staff, who oftentimes seem to run the show. And they know these people and were able to get us to speak with them directly, and for long periods of time, and it was — I think in the long run it was very helpful. 

DR. LEE: Does anyone know about this group that's reached out to me that's — they're connected with A.E.R.A., but the friends of IES? And presumably they would be a kind of entity I think — some of these stakeholders you're talking about I think are part of that — part of that group. Caroline?

MS. SULLIVAN: Yeah. Some of this probably, and I guess if we're having a discussion tomorrow about sort of larger communications, because I think a piece of it is, you know — in this hearing you have your external communications to the public, and to parents, and whatnot, and then your internal communications. This goes into that. Some of this is sort of what is the story of what we're doing. I think especially it's quite difficult when you're a very data centric organization because that is your job, but we have to tell the story because that's how human beings, and really politicians, they think in story, right? And so, I think maybe the Board can be helpful with some of that comms strategy, but then also who are your people because whenever you're going to advocate for something it is — I think you need to have a strategy in that too, right? Who are all of the people who are your families, and think about family I'm meaning like who can help you with this, and it's not just — it's the associations, and it's also the association of say school superintendents and the associations of all of the higher ed people and the ed prep schools and all of these groups, you know, and we can even drill down in the special ed and to the Autism Society and the ARC and all of these groups that can help amplify what we're doing as well.

MS. LEGORRETA: Yeah. This is Conchita. What you're saying is exactly what we discussed. So, the idea was to bring these groups together and do it at — because March and April are so close for this year, if we could do this, it would be something smaller. But in the following years bringing together all of those supporters and constituents who can tell those stories because what staffers like to hear is they like to hear both, so they want to hear the data, and they want to hear the data, and they want to hear the personal stories on how this is impacting a student in special education, or whatever. And so, the more we can provide the stories, and the data with all of these people who are allies for lack of a better word, who are providing this information would be super, super helpful. So, thank you. Yeah, that's absolutely one of the things we discussed.

DR. SCHNEIDER: So, could I just add to that? So, one of the problems that IES has had, And I think why it's not as visible or successful as it could be, is that we were never in the storytelling business. And we have many really, really truly compelling and successful stories. Liz has been working on them. We had Ogilvie as a contractor because contracting law, and processes, and administration in the Department is not so good shall we say. We have no contract comms in place right now. We are struggling to get it back in place, but one of the reasons we need a good strong comms shop is exactly what you're talking about. There's nothing as compelling as stories, especially when we talk about successful stories of students, and there are so many — there are so many stories that we have, so many contributions that are both at a very high level, and ultimately it gets down to the lowest level of a student, you know, who all of a sudden can, in fact, have support services that they would never have before. So, Liz has been working on this. Nate actually have this too. I mean, so the question is whether the stories — and this has to do with the research side, right, what are the stories that our research has — to show that our research has actually done something. So, the other part of our problem is that — and I think problem is the right word, is that we — most of our money in research goes academic researchers, and their incentive structure is not the same as what we're talking about, right? And we are — I mean, it's been a struggle, but we are reorienting IES to be more — I mean, we do basic research obviously, but we need to do the transnational work. We need to do the communications work. We need to do all that kind of work that's essential for turning all the work we've done, all the good work we've done, all the accomplishments, into stories that people could grasp. And that is — I mean, it's not — it's a cultural issue as much as anything else, but we're changing the culture, so people understand the importance of getting those stories into the world because that is how we build support. So, what — just one last point. In earlier discussions there seems to be some confusion between authorization and appropriations, right? So, we've got to be careful about that, right? I mean, we call them the four corners, right, the two appropriations committees, and the two substantive committees, authorizing committees. So, the ESRA reauthorization is in only one of those corners. The Senate Help Committee, the House Education Committee, may never take up this bill, right, and this bill is going to happen — I mean, it's happening fast, right, at least in the Senate side, whether or not they could get a floor vote, and whether or not it passes the floor vote in the Senate is a different thing. If it goes to the Senate floor and gets a lot of bipartisan support, the chances of it going through the House are greater, but right now the House has not touched ESRA reauthorization and don't seem much interested in doing it. So that's one thing. But you've heard several times that part of our problem is our budget, so, I mean, we have our budget around $800 million, which in the real word sounds like a lot of money, but in the world of science, federal science agencies — I mean, someone said, oh, you must be a rounding error in NIH's budget, you know, one of our fellow science agencies. We're not even a rounding error in their budget, right? And I don't know if James is back on, but, I mean, you know, they've had so much more success in the last year or two in terms of getting money, new monies. And, I mean, the Chips Act gave tons of money to Commerce and NSF. We got nothing. We weren't even mentioned in the Chips Act, which is crazy. So, part of our problem — and part of that is the sales. I mean, its literally sales, right? We haven't sold ourselves correctly. So, then it's easy to ignore us when the money machine starts cranking out money for Commerce and NSF, and like — I mean, we lost money actually. In the recission that happened, we lost like $44 million. So, this is — I mean, so this is part of a problem on our side, right, that we haven't done our job in terms of telling stories. So we need your help on that. So, Conchita, I mean, you're absolutely — or I'm sorry, Caroline, both of you are totally correct. We need a — it's a P.R. campaign as much as anything else.

MS. SULLIVAN: And, I mean, I think while it would be great to have a whole bunch of money and hire an LV, I mean, it is — you don't need to — I mean, a lot of times this does need to happen organically anyway, so I think just saying, oh, because we don't have a giant budget for a big old comms shop doesn't mean we can't do some of this work, right? And also, let's face it, a lot of your research partners, all those universities have comms shops too, and it behooves them to tell these stories as well, especially if it's coming out of their institutions. So there are ways to do it, but it is, I think — it's just problematic, especially in this time where a lot of people aren't paying attention to data to be able to knit that through, look, we found out this that's a best practice for helping students with special needs be able to do better in school, and this is where we do it right, and this is how it can help you.

DR. FUCHS: You're absolutely right. All you guys are absolutely right. We need to tell stories, but we also need to be able to speak the language of the people we're talking to. And Vanderbilt's got a very clever congressional liaison person, and she said to us before we went to the Hill she said, look, one of the people you're going to be talking to is a congressman from Tennessee. He's a small businessman. Speak to him as a small businessman. And so, one of us in conversation with him talked about research as an engine for innovation. Research is an innovation engine, and he — that metaphor really connected to him. He didn't — we were talking about research, you know, research for like 10-15 minutes, and he didn't know what the hell we were talking about. When we talked about innovation, you know, an engine for innovation, that connected with him. So that's also — I mean, part of this is really art, you know. I mean, you really — you really need to know how to be a good salesperson.

MS. LEGORRETA: How visits work is knowing — so let's say we're targeting the Help Committee. Knowing who all of those people are, and knowing their values, and their backgrounds, and being able to train. That's why the training would also happen before the sessions, is being able to train everybody on this is who you're going to go see; this is how you frame it differently. It's the same issue, but you need to frame it differently based on even if they're Republican or Democrat. The values are going to be very, very different. So that's part of the research that goes into it is figuring out who is — who you're going to be speaking to, and then as well — because ideally the way this is done is you have a couple of people who are kind of people that support this work who are on the Hill who would come and do a little intro. So maybe like, you know, someone from Senator Casey's office. They do a lot with students with disabilities. But then you also need a Republican, so that it shows that we — when we go and talk to these individuals that like it was we had people from both sides of the aisle, and this is people that — So yeah, there's a lot of work, and a lot of kind of research that goes into making it in a way that makes sense, and then also trying to pair if there is somebody who is going that's from — that's a constituent of one of these people that's the ideal because that's who they're going to listen to more than somebody who is not a constituent. So, all of these things are kind of part of those visits. The question is how much we can do, or not do.

DR. SCHNEIDER: I'm sorry. Just one last thing. So, I think in terms of authorization the timing is not going to work for you, quite frankly. I mean, there's going to be some discussion, but almost everything is baked already. But it's the appropriations side that will have some more runway, right? And I think — I mean, Carol is obviously very forward-thinking, and whatever you said — I mean, look, it's an annual process, right. I mean, the authorization is going to happen or not, right, and most people don't think it's going to get through, but what we're trying to do is get as strong a bill as possible to be the floor, the guidepost for next year, right. So, I don't want to — I don't want to discourage you from working on the ESRA Reauthorization, but the timing for this cycle is probably a little compressed. I mean, the House hasn't taken it up yet, so there's entry points in the House. And then — so Stephen wanted me later to talk about the NEED Act, but there's another act in the House which may or may not get through, and the NEED Act — I don't remember what — National Education Evaluation whatever, it actually does two things. The first one is — it sets up NCADE, the ARPA Ed, and some day we can talk about that, or not someday, maybe later. And the second one is SLDS. So, the NEED Act has two titles to it. Both of them are really good, and really important, right? So, we could talk about ARPA Ed, NCADE, and, you know, whenever you want. But the SLDS is really worth discussing because that is — Peggy, you alluded to this before. So that is an incredibly important data source. The federal government has spent $1 billion building it. Almost all that money was done by 2011, or 2012, for tech process where the biggest investments were 12 years ago. I mean, I don't need to finish the dot, dot, dot in that. And we need SLDSV 2, and that would require whole bunches of money. Probably not another billion, but a lot of money. But SLDS, if we properly conceive of it, can solve so many of our problems, right, because the vision of SLDSV 2 is actually to integrate data into the state longitudinal data system. So rather than — so we spent 20 years — 17 years actually, figuring out how to build SLDS from a K–12 system pushing it down to early childhood, pushing it up into post-secondary. That's what we've done for the last 10-12 years, right? And we haven't talked about the importance of labor market outcome data, but that's in ESRA reauthorization. So, the importance of labor market outcome data, but that's — in ESRA reauthorization the labor market outcome data is all over the place, right? We internally, and using our own authorities have expanded, but to have the Congress say we want these data, we want labor market outcome data, is a huge expansion of our ability to find out actually what happens with education. So that's number one. So, we're expanding this, right. It's gone down, it's gone up, but for many issues that we care about in education, chronic absenteeism, you need data coming in from everyplace, right? So, Peggy mentioned the Evidence Act, which is on the federal level where agencies are supposed to share their data. My reaction is good luck getting the Census, or IRS data, to — you know, giving us these data. Peggy mentioned that also. But at the state level, I mean, what we want to do is we want authorization in SLDS so that the education department that runs these systems can, in fact, bring in data from other — I mean, it can already, some states are doing it, but we want to encourage states to bring in data about housing, for example, about food security, or insecurity. Criminal justice, whatever it is, so that we get a 360 view about what the kids are experiencing. So, to me, SLDS is like something that we all should go to bat for that is in the NEED Act, and there's a longer runway for that. And then again, I think most of you know about our plans, or attempts to get NCADE and ARPA Ed into the legislation, but it didn't get into the Help Committee. We're still trying to get them to revisit that. But the NEED Act has it in it.

DR. KLASKO: This is Steve Klasko, and when you and I talked, you know, And I think this gets down to what I'd really like to hear from Carol, and Conchita, and I, have talked about is, you know, we talked about, I think four things that could really matter. You mentioned the absenteeism and getting the funds to help the states get the resources to obtain actual data, you know, the whole issue of the seedlings to scale piece about research is an area for students with neurodiversity, you know, almost like an A.I. institute for neurodiversity like you talked about sort of looking at how we can scale academic progress through partnerships with — between new technologies and academic research. I'm in the VC world now that would love to do that. The government ends up, you know, very much creating a barrier to some of the companies in the — I guess I think you called it a $10 million opportunity to create the actual efforts from school Pulse data to be analyzed in months as opposed to years. You'll have a more rapid turnaround from the point of modern technology. So, I guess my naive question is let's just — I mean, forgetting the whole health authorization, and centers committees, and stuff like that, you know, how can we help you even if we just take to those things that would seem obvious as being needed, you know, all over the lay press about absenteeism, nor diversity. And other countries, frankly, I was just in Brazil, on the nor diversity side, and they're doing in some respects a better job than we are. So how do we get that word out, and not have it just be another thing, and what's the role of this advisory Board to you in our relationships with representatives. Like I don't even know can I call my representative as a member of the National Board, and say, hey, you know, you should really — you should really look at this. So, to me, I think, that's where Conchita and I were really trying to get some direction.

DR. SCHNEIDER: So, two things. So, the first thing is about what your authority — of what your freedom is as both a member of the Board, and as a U.S. citizen. They may not align perfectly, but you need to have advice about that, right? But you need to have advice about that, right? And again, I think the FACA attorneys, and OGC, could give you advice about that. The other thing — I'm sorry. Just to follow that up. So, my daughter is a pretty good attorney, and her job is to get to yes. Government attorneys, often their job is to get to no. So, I don't know if any of my friends from OGC are on there, but — okay. So the fact of the matter is like you need to be pretty assertive about this is what we need to do. Tell me if this is illegal, right? And look, I don't want to visit you in jail, quite frankly, right? So let's get good advice about what you can do as a citizen and a member of NBES. That's number one. Number two, I think — so I don't want to — I don't want you, I don't want any of us to be in the position of going to the Hill to the appropriators and say, oh, give IES more money, right, because they're good guys, they need more money, right? So what I've been trying to do, and only with modest success, is to pitch, you know, bright, shiny objects, right? So NCADE, ARPA Ed, we got $40 million for ARPA Ed, okay? And that was like — we were never going to get $40 million for statistics or NCER, right? We made a case, which is unfortunately a true case that we needed a modern education R&D infrastructure. We needed to have more timely releases of our data. We needed to focus on more applied issues. We needed to scale, right? So we haven't even talked about scaling. The scaling is so incredibly important, right, and part of the reason that people like DARPA, and all the ARPAs that are beginning to show up, is because they admitted to taking science and turning it into products, and things that change facts on the ground. So in part because ARPA is a thing across the government we were able to pitch ARPA Ed, which by the way when I was in IES, and when I was at Peggy's job as Commissioner, we started in 2003 or '04 we started talking to DARPA about ARPA Ed, and now 20 years later we are getting close almost — we're almost there, right? It's not in the — it's not the Help Committee write-up, but we haven't given up on that yet. And then the NEED Act is yet another possibility to get ARPA Ed done. But you need — so for me SLDS if we can get like NGA, or CCSSO, to endorse the importance of these data sets for the Governor, or for the chief of the school system, I could talk forever, I do talk forever about SLDS. It doesn't matter, right? We need NGA to stand up and say the Governors need this. We need CCSSO to stand up and say, you know, the school superintendents need this. So that to me is a marketing strategy, but we have to have something to sell. If we just say, oh, we're good guys, look at all the wonderful things we've done, which we should be doing anyway, right? This is why telling stories is so important. But people are going to be much, much more interested in something innovative, right? We haven't been able to crack — we haven't been able to crack this nut, but give us some money in the ARPA situation, and we'll push. We'll push, we'll push, we'll push in a totally different way than our standard approach. That to me is the — is what we — we're doing that, and that's what we need to do more of. 

DR. KLASKO: That makes a lot of sense. Thanks.

DR. LEE: So, what I'm hearing is that this can be a useful project of support from the Board to IES, and that one of the things — and also that we need a targeted group in terms of some targeted persons from this Board, certainly form this committee, staff from IES, and possibly may even want to pull in some other advisors. This is why I'm suggesting that at the end of the meeting I want to propose a vote to create standing committees that are able to meet outside of the Board meeting to work through details that will be recommended to the Board, and that such committees can include pulling in stakeholders who can give advice to how they reason through whatever the project may be. So, to that, Mark, what I'm hearing in particular would be the need of multiple stories. One is a general branding set of stories, but the other around particular initiatives that may be related to particular policy initiatives, or that may be related to part of the argument for increasing overall funding to the institute. I think it would be helpful tomorrow when I wanted to have a broader discussion around the ESRA issue, and thanks, Mark, for the distinction between reauthorization and funding to be able to perhaps create a representation of a number of initiatives that are on the horizon, and timelines associated with them, so that as we begin to try to create a plan and infrastructure we have a sense of the scope of what we'd want to do, and what might be different timelines for that. James?

DR. MOORE: I'll try to be succinct. You know, I can easily give you the names of our Board members for NSB and NSF. I know our Board have ongoing meetings on the Hill. We got the biggest budget increase that we've had since we were founded. But I can tell you some of the talking points because I sat on a lot of committees beyond just the educational research space, the workforce development space. It's one of those spaces that Congress is really interested in. And so part of the vernacular that people use when — regardless of what side of the aisle is around the Chips and Science legislation, and if you haven't reviewed that, I encourage you to review that because that is the space in which people tend to operate is — some would say is the spending moment of our time, the Chips and Science. So that might be a good frame to kind of frame some of your discussions, but like I said, our budget — our Board have continuous meetings with people on the Hill to advocate for increasing budgets. But I will say I agree with Mark, it does seem that things are going downstream; however, you know, we had the omnibus bill that we got to supplement, and that's in my directorate is the biggest beneficiary of that omnibus. But when you talk about, yeah, NSF did get funds, but it's like taking a five-gallon bucket out of the ocean and say you stole the ocean. It was $200 million over five years. We're talking about $25 million per year the first two years, and $50 million for the last three years. Now, I won't ever say $200 million is chump change. It's more money than I ever, ever dealt with in my lifetime, and probably ever will, but nevertheless, when you think about the grand challenges around these issues, it's really not that much money. So I would — I'm willing to share the names of our chairs, and I know they have ongoing meetings with individuals on the Hill, and because it's clear that the way my budget comes in is very different from how everybody else's budget comes in in NSF I always know what my budget is. It's a blessing, and this occurs at the same time, and you'll probably say what's the blessing. The blessing is I always know what my budget is because it says EDU; however, the curse is that some people still don't think educational researchers do research. We're marginalized sometimes in an Agency like mine. And so, I just think there is a path forward, and I do know the sense of the emergency, but how I kind of navigate is when I talk to people on the Hill, outside of the Hill, they really know the significance of the educational space, particularly as it relates to STEM, although their world view is STEM. But like you said, Carol, the science of learning is a big part of what we do within our portfolio. And so, yeah, my budget is 1.3 billion, but it's driven, or it's a lot of congressional mandates, right, that the money is already carved out in many ways. And so, I really think in these times, particularly around there's this emphasis on rural STEM education, which is underscored throughout Chips and Science. And I think that is one of the many reasons why we have bipartisan support in Chips and Science. So framing is going to be really important, but rural in Ohio means something different, and Shaun could probably bear witness, than being in Georgia and South Carolina. It's just totally different. And when you think about some of the demographic shifts, and when you think about the diversity, when you talk about what we call an ed scores states, the correlations with poverty, and the states that get, again, the least amount of funding in the federal government. So, these are priorities of the Biden/Harris Administration, and so I won't be long-winded, but that's just something to think about as a Board. 

DR. LEE: So, one of the things, and then Caroline, I'll call on you, that I think — and we obviously would have to pull together a group, and relatively soon, to sort of try and map out the plan for this. But part of the storytelling seems to me, obviously, is around the impact of work that has been done. But also, it seems to me as one of the other story lines around the possibilities of additional kinds of work that meet needs that have been persistently not addressed, some of which I think do include — I'm going to keep coming back to this science of learning and development. That's the new horse I've been riding for the last few years, and it goes back for me to this notion of theory of change that the extent to which human activity takes place within and across ecological systems, and not in any single side of activity. And the fact that all human actors involved in trying to design and support learning are bringing to bear their perceptions of themselves of others, the relevance of whatever the activities that are engaged in, and wrestling with implicit biases, and conceptual change. That there's basic scientific work to be done in that area that may fundamentally even shift how we think about this notion in moving to scale because I think even the understandings we have around reaching scale, one, don't take into account the complexity of the education system in this country from the city or town to the district to the state to the feds, the multiple parties, the different assumptions about what are the outcomes of interests. A very complicated, you know, system to manage. And I think that on the one hand there's this powerful story of the work that IES, and all of its centers, have accomplished over the years, but there's also standing this sort of bear in the back of saying whatever kinds of data we're looking at, we're nowhere near where we should be, and what does it mean to wrestle with that question, and convince other stakeholders that they need to be investing in. And also the issue of developing partnerships, so it's not only NGA, the what is it, Council for Inner City Schools, practitioner organizations, the whole variety of stakeholders we may need to just outline, you know, a variety of kinds of stakeholders in terms of relationships that we want to form, some of which obviously — much of which you would be doing internally, but also ways in which the Board could potentially help facilitate some of those relationships. Caroline?

MS. SULLIVAN: Yes. So like, James, the workforce thing, I've never seen such a focus on workforce, and I've been doing workforce for a while. So, I think however as we talk — talking about how this impacts workforce, it is compelling to elected officials, as well as to pretty much everybody else. So, I think that's critical. And on these large acts like the Chips Act, it is critical that we're growing that workforce. I mean, this is something that I think is just sort of dawning on people is that if you're going to bring everything back to the states, and need a technically skilled workforce, you got to start that way deeper in K–12 than just trying to do solutions at community colleges, or get people to do short-term credentials, because we need to grow more. And also, we've got demographics that really aren't working in our favor just because of the birth rate, and things like that. So, I think some of this caging around workforce, and things like that, I think, and knowing the data, and understanding what works, and understanding how you help students reach their full potential so they can participate in these emerging sectors is critical. And Matt, I think maybe it might be helpful to not think about this as just some data people in education. It's sort of laying a foundation for all of these other things that are in different departments, and different acts. And the other thing, the NGA I'm happy to help with that. We do a lot of stuff with the NGA. But also, what Governors, and elected officials at all levels desperately want is some of this longitudinal data; what are the outcomes, and it's very difficult. And I think another thing that I think the Board might be able to help is talking about that, and how with additional funding we can support states to help their common follow-ups in longitudinal data because it's rough out there, and people do need to know what happens to people.

DR. LEE: So I think too that the earlier comments that you were just making, Caroline, to me exemplified this point I'm trying to make about the sort of theory of change having to address the multiple levers in ecological systems that impact whatever outcomes are of interest, and the extent to which we can articulate that kind of vision in a way that's accessible, and human-focused, but also is the case for the funding that's going to be required to take that shift, or expansion of the efforts of all of the centers to address these issues in more complex ways. So my kind of takeaway from this is that it sounds like this is a target in terms of as we're trying in this meeting to come up with kind of a vision of priorities for this Board that one of them would be how we can establish collaboration, support, partnership with IES, and all its entities around developing a strategic plan for this in a timely manner. So as Conchita was saying about even whatever limited things we might be able to do as of March or April, that, one, for me this would become a rational for constituting this group as a standing committee of a Board that would allow the committee to meet outside of the public before a public Board meeting to work through the details of such a plan, including in working through that close work with whomever IES determines, I would think, some representative from all of the centers would need to be involved, and our ability to invite some other stakeholders not as members in any way of such a committee, but to advise the committee on how to do the drafting. And then when we determine, which I think will probably come up at the end of the meeting tomorrow in terms of setting a calendar for our meetings, we can set a standard calendar, but we also may want to set a special date for another meeting that would allow this committee to report back on the specifics of a plan to begin to actually start this work in March. Does any of that seem reasonable as an approach?

MS. LEGORRETA: Yes. Sounds great. Thank you.

DR. LEE: So, I'm going to — I think I'd like to hold off on the vote for establishing committees of the Board until we've gotten the next two reports done because I think that those discussions will inform the breadth of what standing committees, we think we should have as a Board. So, anything else on this upcoming policy issues in the world of the Board? And in the meantime,, before the Board — the committee meets to start as planning Mark will give whatever feedback we need from the lawyers about the scope and restrictions of whatever it is we can do as a Board. I do suspect, however, that when we're speaking as individuals we should speak as individuals, and that the only time we should speak as Board members is when we are speaking on behalf of some decisions that the Board has made. I think that's going to be a very fine line that we're going to have to navigate. Okay. So, the next group, the notion of organization of NBES subcommittees. Caroline and Elmer.

MS. SULLIVAN: Well, we weren't able to spend — weren't able to like to have official meetings, but, I think, just from the discussion that we've been having today it sort of — this is coming pretty clear on some of the buckets that we might need to think about committee work. One of them, I think, is communications both external and internal, And I think that — at least it seems to be it doesn't really involve any of the advocacy, or the lobbying, and so I think that is something that is certainly doable for the group. The policy work that was shared earlier, I think that is another area where we can have a subcommittee that also seems like it would not run counter to whatever we end up understanding is our place going into the Hill. Yet when we were going through the minutes from the last meeting there was an awful lot of interest in supporting research for exceptional children, and So I think that crosses different — I think different buckets, but, I think, that is one that a lot of us have expressed to, and interest in. And I think it might make sense to have it be a separate committee. Another one is the equity focus, how are we supporting staff to have that equity and accessibility focus both on the studies that are coming out, and how we can — how can we support some of the institutions that are receiving funding with those goals that we have. Let me pull up my document. I want to see something. Because I was also thinking that part of it is I think some of how we want to work, I think, is going to be gleaned then from our discussions today and tomorrow, but I really also think some of this we keep talking about alignment, and crosscutting, and the different systems, and how people talk to each other. Carol, your whole system of change thought, I think, is an interesting one as well on how we can do this work. I spent a lot of time silo busting in my job, and it is sort of amazing how nobody knows what we're doing, and I think that is helpful as well.

DR. LEE: Other thoughts? So part of what I was thinking about this question, And I think maybe we could come back to this again after we have some discussion about the executive director search in part because it's my understanding that this is a position of someone who works directly for the Board, is paid out of the Board's budget, and the scope of what kinds of support we want that person to fill I think should be related to the scope of the kind of broader commitments and vision that we have as a Board if we want, you know, support for. But I think the discussion — so I think this idea of a communications committee is really important. I think the policy committee, again, is very important because we've had a lot of discussion, and I can imagine that both communication and policy would want to have a lot of crosscutting conversations because they're going to be interrelated in the scope of work. I think the issue, and I'm just speaking for myself at this point, so the focus of discussions that have come up relative to exceptional children, the issues of equity, a number of these, I think, are embedded in some ways in the recommendations from the various reports, evaluation studies of the different centers' work. And also, just lack of kind of confirmation. My sense from Peggy is that NCS pretty much agreed with the recommendations of that report, and statistics on education. The other centers when you all did the sort of summarization, which makes sense because there are so many recommendations, is how do you kind of get a handle on it, but I didn't necessarily get a sense, but it would be really helpful to know if there were any of the specific recommendations that were made that you all said no, we just really don't think these are valuable. If that's not the case, then I think a committee, and this is complicated, and kind of gets back to one of the questions Mark had raised earlier, but I had been thinking that if we were able to form maybe a committee, and it might have — this is sounding complicated, but maybe sort of two subgroups within it in terms of what — the issue of what kinds of supports, and understandings of the unfolding of the work of the various centers to address the recommendations, and the visions that they have of which themes around equity, around special need populations, all the variety of kind of buckets if you will in the report could be a valuable group, but the understanding that's going to take time because there's a lot of stuff, right, to do. Could be buckets of standing committees. And, again, my understanding of the logic of standing committees is that they have the authority to meet outside of this public meeting where we'll never be able to work through the details of complex stuff in the context of a Board meeting like this, but we can have specialized groups that prepare all the necessary research, collaboration, whatever it is, to come and be able to make recommendations, including how the two — the feedback we've gotten from IHEP, and I'll ask for a learning innovation, a cite for their recommendations to be taken up so we're not getting feedback from the public that somehow get lost in the mix could be a way of moving forward. And again, with the understanding that these committees would have the authorization to invite not as members of these committees, but as advisors to come in who bring specialized knowledge that could help inform whatever the work that we're doing. So I'm just — I'm going to turn this into a formal recommendation after we discuss this search for the executive director, but I'm just curious for your feedback about this sense of a structure for the Board moving forward. 

DR. HARPER: I'm in favor of what you proposed, Carol. It makes perfect sense to me.

DR. LEE: You know, we're going to be the most active NBES Board in years, so we're going to maybe pat ourselves on the back or not. I don't know. Ruth?

DR. TURLEY: I was just giving the thumbs up to say, yes, this is a good plan. 

DR. LEE: Okay.

DR. FUCHS: I do too, Carol. I think it's — having subcommittees, and subcommittees able to draw upon others with expertise as consultants is really the only way to go about this to get as much done as I think we would like to get done. 

DR. LEE: Great. All right. So why don't we just transition, and we'll do a formal vote on this after the discussion of the search for the new executive director. So, I'm going to just express what I think I understand the position to be, and then Mark, or staff, anybody else, can correct me if I'm wrong. So, it's my understanding that we are authorized — that we have a budget of something like $345,000, something in that range, to support the work of this Board, and that part of those funds can be used to hire an executive director. It's a very weird-sounding position to say that you got an advisory Board, and you're hiring somebody to help you, and you call them an executive director. But anyway, that's what they seem to have in the charter, or someplace, that they're hiring someone to basically fulfill the role that Ellie and Andrea had done and take this time to publicly thank Andrea. And she's moved on professionally, but how supportive she had been from the very beginning of this process with all of us. And Ellie has taken her place and been very wonderful in the transition. But it's my understanding that the executive director is a position that the Board conducts the search, and hires this person, and identifies the duties that we want this person to fulfill, which is basically to support carrying out the work that the Board has identified. When I talked to David, the prior chair, and that's why I'm recommending that that committee once we form it meet with David. And he also indicated that he could put us in contact with the former executive director that he hired, or that last Board hired in terms of the scope of the work. But it seemed to have included things like the work we're talking about in terms of connecting with the stakeholders on the Hill for establishing relationships with the other potential organizations that could be helpful. Our contacts internally with IES that this person would handle all of those kinds of work for carrying out the wishes of the Board. I think there's a question also in terms of our giving this committee some direction about the scope of the work we want, the qualities, and professional experiences we think that person should have, and also whether this should be a full-time or part-time position. I'm sort of leaning to thinking that it could be a part-time position except I can't imagine who would take it as a part-time position for the scope of experience that we want.

DR. FUCHS: Carol, if could just — if I could just add a —

DR. LEE: Go ahead.

DR. FUCHS: If I could just — if I could just add to that. Jim, and Dana Hilliard, and I, as part of this subcommittee thinking about the executive director communicated by email, and we're in agreement that we will go forward. I contacted — I have an email into David Chard, whom I know, and we're going to set up a contact with him, and then after him the — and David Chard was the former chair of the Board, and then after speaking with him the executive director under him. And just speaking for myself, I think the commonsense way to go here is to use our conversations with David Chard and the former executive directors as a kind of data gathering to get a sense of, as you just said, what seem to be the desirable professional and personal characteristics of an executive director. And I think to me it kind of makes sense for us to come back to all of you with a very preliminary set of — criteria are too strong a word, but a set of characteristics that we think would make sense as we move forward in trying to find somebody. And I'm not suggesting that we first conceptualize what it is, or who it is that we want, and then to start the process of trying to find people. I mean, they don't have to be in lock step, but it seems we should all be in agreement, at least in principle, about the kind of person we want as executive director. So, I think it would be helpful to get some feedback from Board members at this stage about what — how you're thinking about the sort of skill set that you think we should be recruiting for. I assume we would have to write a job description. And then a question — Mark, this may be a question for you, or one of the other staff, and that is technically how does this process work, the process of sending out the call, promoting the position? I assume the interview process should take place within the committee. I don't see that we would be doing interviewing potential employees at a public meeting, and then that the committee would then make a recommendation for hiring that would be voted on in the public meeting of the Board.

DR. SCHNEIDER: Okay. So, Carol, the one thing that you have to understand is that you're hiring a government employee, which means that the hoops that you have to jump through to make this happen are rather consequential, and rather difficult. You can see Nate is nodding his head because he's two months into this, and he's run his query into all the H.R. problems. So, you have one major advantage, and that is Ellie is the designated official, and she is unbelievably good, incredibly well-organized, and does a lot of the H.R. work us, you know, for the institute. Right now, I believe, and she could add what's needed, right now we're working with historical records, and practices, to get a job description, which you need, and has to be classified. I mean, it then has to go through H.R., and has to be publicly advertised. I mean, there's just a lot, a lot, a lot of stuff. I mean, we all in IES bear the scars of these processes. And I keep looking at Nate. He's gotten himself under control. I think that's earlier when we started talking about hiring you could just see like, oh, my God. So, you just have to be — you just have to know that — two things. One is that the hiring process is rather difficult, and you want to be careful about this is designated — we're expecting this to be a half-time job, not a full-time job, and I'm not sure — I mean, you have a lot of ambitions, but a full-time job is expensive, and you have a fixed budget like all of us. So, you need to think this out. But hiring is not simple, and Ellie, fortunately, has a lot of experience, and a lot of scars negotiating this.

DR. LEE: So let me say that one of the things that I have found absolutely fascinating about this whole process of joining this Board, and understanding the expectations of it, is that it's my understanding that we are required to hire this person, right? I had assumed coming in before I even learned that we were required that we would just have a staff liaison from IES who would support the work of the Board only to find out we're supposed to hire somebody to help us. And the time — and so all of these hoops I can quite imagine on the one hand — and, you know, Mark, I've raised this with you, and I did with Jacob, and with Andrea, I don't think any of us even know the term of our appointments, or when our appointments end, and when and if we're going to be reappointed considering that in theory we were appointed two years ago, and we're just now, you know, getting to meet, that I can easily imagine the hoops that we have to go through to hire someone we're required to hire. Our terms could be over by the time you go through the hoops. So, I just find the whole process to be quite fascinating. 

DR. SCHNEIDER: Well, fascinating is a nice word. I would use words like horrifying, you know, words like that.

DR. LEE: And one of the issues relative to the finances of this too, Mark, that you had raised is my understanding that we have a fixed budget of something like 345, or whatever, and that is to pay both the staff position along with the meetings, face-to-face meetings, that we would have. So again, this will be part of the equation, and we need in budgeting out — like this budget of this 345, whatever it is amount, is for what period starting when through what period of time, so as we are anticipating face-to-face meetings, and what the anticipated costs would be. I mean, all of these sort of budgetary issues have to be worked out as well, and its clearly part of the equation as to whether this is full or part-time. But David said he had a full-time. I don't know. 

DR. FUCHS: Carol, I got two questions for Mark.  Mark, I was listening carefully to what you were saying before. It sounds like when you said the position was part-time that was more of a recommendation than anything else, and the reason it was a recommendation was because to hire a full-time person is very costly, and the budget is a given. So, my question is am I correct in that assumption? That's my first question.

DR. SCHNEIDER: Yes, right. Absolutely. 

DR. FUCHS: And my second question is Carol has mentioned several times that the budget is several hundred thousand. I seem to recall, and I may be wrong, that it was closer to 600,000. Can you tell me whether you know exactly how much our budget is? And also, can we actually see the budget?

DR. SCHNEIDER: I'm not sure what you mean see the budget.  Ellie, are you here?

MS. PELAEZ: Yes. I'm here.

DR. SCHNEIDER: So the number, where is — that number in the budget document, right?

MS. PELAEZ: Yes. It's in the FY-24 President's budget. Uh-hmm.

DR. SCHNEIDER: And what is —

MS. PELAEZ: It's 339,000.

DR. SCHNEIDER: 339,000. That's public — that's a public document?

MS. PELAEZ: Uh-hmm. 

DR. SCHNEIDER: So we could obviously — obviously we could send it to you?

MS. PELAEZ: Yep.

DR. FUCHS: Thank you. 

MS. SULLIVAN: Is that the President's budget, or is that the omnibus that was passed?

DR. SCHNEIDER: Right, right. No, Caroline, that's the right question. So we'd have to go back and look at what was in the controlling budget, and that's the fiscal — and to answer Carol's, that's a fiscal year.

DR. FUCHS: That's '24. That's not been approp'ed yet. 

MS. PELAEZ: Right. Just to be clear — this is Ellie again — it hasn't been appropriated yet. That's what has been recommended I guess you could say, right?

MS. SULLIVAN: And that's the levels of funding, correct, right? So the deal that was reached with '23 funding that's where this is, correct? No?

DR. SCHNEIDER: No. So the '23 level was zero, right? Because there was no executive director.

MS. SULLIVAN: Right. Right.

DR. SCHNEIDER: And right now you have zero dollars. So until you have a budget you have zero dollars for R&D, right?

MS. SULLIVAN: Yeah. That's what I was wondering. 

DR. SCHNEIDER: Yeah. I mean, that's something to float you through until they pass the budget in February, but that's — I don't know about that though. 

DR. FUCHS: And Mark, not to put you on the spot, but, I mean, Carol did raise in her characteristically diplomatic way the fact that none of us knows how long we're going to serve on the Board, whether we're going to be reappointed. I mean, it seems to me this is a fundamental issue that needs to be addressed almost before anything else.

DR. SCHNEIDER: So, to be fair, right, I mean, this is — you're appointed by the White House, and not by the Secretary, and not by me. So, all these issues that you're talking about I sympathize, but the fact of the matter is I have no control over them, and no input into that. So you all — but my understanding at one time didn't you get a memo that said that — So I think Linda is not on the phone anymore, on the call, but she was appointed, and then reappointed, even like months ago because the way the legislation worked was that there are staggered terms, and people get appointed into those existing terms. So, when she got appointed, it was — I'm going to — she got appointed in the fall, and her — the seat that she was taking expired, the term of office expired at the end of October. I'm making these dates up, but, I mean, you get the point. So, the White House reappointed her already. So, I'm not sure in your letter of appointment it must say when — didn't get a letter that said you're done in 2029, 2028, 2027? You got nothing that looked like that?

DR. FUCHS: I believe I did get the letter some time ago saying when my term would be — would finish, and then I got an email from somebody saying disregard that, don't take that seriously. 

DR. SCHNEIDER: Okay.

DR. FUCHS: So, I think that's where I'm at.

DR. SCHNEIDER: Okay. So, I would take the letter with dates on it as a good starting point, but the fact of the matter is that — and quite frankly, this is all driven by the White House, your appointments, as you know, your Presidential appointees. At one time we were also Senate confirmed, which meant that we wouldn't even be having this meeting probably for four more years if we were waiting for you all to be confirmed by the Senate. But the fact of the matter is — and Carol obviously has good ties in the White House, So I think we need — you need to get that straightened out with the White House. 

DR. FUCHS: Thank you.

DR. LEE: I'm sorry. Just a quick response, and then Shaun. Jacob more or less said the same thing to me, this kind of warning. It could be helpful if each of you could maybe just send me the dates that are on the letters that you have received, and then as we start this process then maybe we can talk together with this liaison from the White House. Shaun? Sorry.

DR. HARPER: Yeah. So I'm going to be a bit less diplomatic than Carol, unfortunately. Despite Carol's extraordinary leadership really, and a heroic effort, I'm just going to name it, this doesn't feel like a real thing to me. It feels like we are a low authority, no authority group that's going nowhere fast. It doesn't feel like a real thing, at least to me. That is enormously frustrating I must say.

DR. SCHNEIDER: So, I understand your frustration. Welcome to federal government. Got it. So that's number one. Number two, I mean you are governed, like I am, like Liz, like Matt, by existing legislation. So your activities, and your authorities, are clearly specified. There is some confusion in the charter, but the charter doesn't supercede the law. The law is the governing document. So that is part of the deal. And I understand your frustration, and I totally understand it. Look, we have spent now five, six, seven hours. I mean, you've met the staff. They're incredibly responsive. They're incredibly professional. They will do what they can to support you, but, I mean, quite frankly, again, they too are limited by the law, right? They also have full-time day jobs that they have to do. So, there's always going to be this tension. I mean, look, that's the reality. And Shaun, I mean, I understand why you'd be frustrated, but that's the reality. And I'm sorry, you know, when I start talking like this it drives me crazy because like I know in all of our personal lives we're like get this done, get this done, like I need this done, I need this done, but the fact of the matter is that we have governing authorities and legislation, and we have to abide by them. 

DR. LEE: So let me just say, Shaun, that one of the things that's been very helpful for me is the fact that there is a staffer from the White House who has reached out to me, and we've had several conversations, and I have found when issues come up that are kind of conundrums from me, he's been very helpful. So, I think as we move through this process then I suspect one will be the sort of legal questions around the whatever boundaries around advocacy work that we do on behalf of IES, you know, relative to policy, so we can get clarity on whatever the bounds of that may be. And also, clarity around this position, this executive director position. Again, I think talking to David is going to be very helpful. They did, as I understand it, hire this person that's a full-time employee, and he said that he would be willing to connect us with that person so we could talk with him directly about what they're experiences in the scope would be, And I think that will be meaningful to do as we come up with a possible job description for this work. I will also say that it may be because Andrea knew that she was planning to make a transition out of IES that we had some discussions about trying to move forward on the EDD search, so that she would not have to continue to carry that — you know, that work out. This may be a different situation with Ellie. I don't know. But as Mark said, welcome to the federal government. This is the playing field that we're on right now. So, what I'm going to do — 

DR. HARPER: You know, Carol —

DR. LEE: Go ahead.

DR. HARPER: Carol, I want to ease up on this. I will say that there is something about this that is particularly and unusually off. I've done several things with the federal government over the past 20 years, several, several things in collaboration with the White House, and I don't know, like forward movement was not as stalled in those previous engagements. So, I'm just not sure that we can attribute it entirely to, well, this is just a bureaucracy of the federal government because it's unlike other federal government experiences that I've had. And again, Carol, my problem for sure is not with you. You are doing the very best that you can to lead us, and you're leading us great, but, I think, we all said yes to this because we saw it as an opportunity to contribute, and we're people who are accustomed to accomplishment, and it just feels like it's a low authority, low accomplishment group at this point.

DR. LEE: Well, again, I totally understand, and sympathize with what you're saying. I think we just have to be forward-looking, and create our own vision, and our own action plans, and then if they want to step in and tell us we can't do it, then we respond at that point, but I don't think we should restrict the vision of the kind of support that we want to create. And at the end of the day, and Shaun, this is precisely, you know, for all of us, and this is why I'm saying I want us to try to figure out a way of how to continue to support the really productive, positive work that IES has done. You know, I've been in several grants, and I'm very thankful, Liz, for both of them, and to be able to continue to support those efforts on the one hand, and on the other hand for all of us to continue to wrestle, which is not on any person, on any division, et cetera, about the inability that we have as a nation to figure out how to restructure opportunities to support children and adults. And Peggy was saying she had this meeting around PISA. It absolutely fascinates me that the PISA data comes out, the PEARLS data comes out. We're supposed to be the city on the hill at the top of everything, and no matter what those education statistics outcomes are, we are never in the top. And yet all the things that they do in other countries, we're talking about the teacher workforce, you know, conditions of the healthcare system, we pay absolutely no attention to the fact empirically that other people who are able to get strong educational outcomes don't do anything like what we do, and it doesn't seem to matter. And so, Caroline, one of the questions, you know, of the stories we tell somehow has to wrestle with there's some kind of underlying kind of fixed deep-seated beliefs. I don't know if it's the pull yourselves by your bootstraps, or whatever the case, or education doesn't matter, you know, educators are like factory workers, that is so deep-seated that it's hard to shift people's notion that why in Finland do teachers make the same thing that doctors and engineers make. Doctors and engineers don't make in Finland what they do in the United States. But the fact that there's parity says something, right? Linda's work in that world in education, these places — teachers are not in classrooms with teachers all day. They have time for professional development, the kind of PESA data that's captured around supports in a system that's — we're not doing that stuff. And that's why I want to keep planting this little seed around what I think — and James, I'm glad you're here because I still think that NSF and IES need to be collaborating in some way around this. And I keep coming back again to the work in the neurosciences in supporting this that participation and culture for practices matter. Genes don't determine. Genes follow experience. Genes are inherently malleable, malleability in their expression. Diversity is what allows humans as a species to survive. So why is it that we have people who are in special education — because you've got to have somebody who is in and out, when these are people you're talking about neuro diversity that have different sets of resources that they use to navigate the world, and because all of this is evolving in dynamic systems with dynamic relations, and we're still doing, you know, causal studies, and quasi experimental designs. So, whatever the error is we don't pay any attention to it because we have to find that there's some fundamental kind of shifts in reorganizations that as we continue to support the work that all of the centers do, you're also able simultaneously to interrogate some questions about some fundamental reconceptualizations as some streamline, you know what I mean, of the work. So that the NCES data shows — you know, gave them NAEP, and they're jumping off the chart. You know what I'm saying. My speech, sorry for it, but this is part of what drove me to even being interested in trying. It's been an interesting journey.

DR. JONES: Carol, you know, you mentioned PESA, and you're right. When you look at the PESA data, the international comparisons, the United States at best is middling. It's not — it's hardly in the top 50 percent, or top half.

DR. LEE: Right.

DR. JONES: But if you look at the PESA data by states, Massachusetts the last time I looked was sixth in comparison to all countries in the world, and why was it sixth? Well, probably for lots of reasons, but one of which was it tends to be more research, or evidence based. And so, if someone is trying to make a case to a skeptical staffer on the Hill that research — you know, when we say that research matters, there's a story right there.

DR. LEE: And so again, I think that idea of having a standing committee that may have some subgroups in terms of relationships with the various — the work of the various centers can just become — these kinds of cross conversations, I think, are really important for new ideas to emerge, and this become an important part of the story that that — that in our complex, you know, decentralized system we do have good stories to tell, and can identify the research, including research that I'm sure has come out of the work of IES over the years that inform how Massachusetts thinks about organizing their system not only of education, but healthcare if my memory serves me correct. So, I want to ask for a — I'm probably — I think probably older than anybody here. I just turned 78. I have all this gray hair, and I can be starting to say something, and then completely forget what I was going to say. Hopefully the gray hairs are an excuse for it. But a resolution, whatever you call it, to form four standing committees of the Board. One that focuses on supporting a communication policy of a collaboration, again, between the Board and IES. Two, a policy committee that will focus on identifying policy targets to which IES and this Board should respond and make the recommendations for how we engage in that work. Three, a standing committee focusing on the hiring of an executive director. And fourth, a standing committee that will focus on working closely with the various IES centers around responding to both the recommendations of the National Academy of Reports, and transformation targets that the centers themselves have established. And as standing committees they then would have the authority to meet outside of the regular public Board meeting. It's my understanding, Mark, you can correct me, that either Ellie, or the executive director we hire, would always need to be present at those meetings with the expectations that those committees do not make decisions; they do the research to make recommendations to the Board.

DR. SCHNEIDER: That is my understanding too. 

DR. LEE: Okay. So, with that, can I get — I don't know if I'm making the whatever you call it, I need a second.

DR. ALBRO: A motion? Do you need a motion?

DR. LEE: Yes. That's what I need is a motion. 

DR. JONES: I move — go ahead. Go ahead. 

MS. SULLIVAN: I was going to move that we establish four committees, or standing committees, to advise — and we might want to put — if this is going to be a formal motion, what Mark was sharing before, like our primary job is to advise the Director, so maybe do we want to put that in the language as well? I don't know. Or do you all want to make this — do you all want to make this broader, or do you want — do you feel that it needs to be a little more specific?

DR. LEE: I think that's a question for Mark. At the end of the day we advise the Director. From that point in terms of these committees, and particularly the one to support the various centers, is a mechanism through which to at the end of the day advise whomever the representations from the various centers. That's a Mark decision, whatever feedback that the directors, or whomever he appoints to be the sort of liaison to those committees, that was all his decisions.

DR. SCHNEIDER: So, I'm not sure if I'm supposed to comment on that. I mean, that's factually correct. 

DR. LEE: All right. 

MS. SULLIVAN: So, Mark, would that be an appropriate motion, or do you need more specifics, or do you think that would work?

DR. SCHNEIDER: Well, I'm just going to — I'm going to harp back to the comparison of earlier when I said like all our good decisions come from hanging out in a bar together as compared to Peggy's much more formal view of how things get done. Look, I mean, I think — so first of all, I just want to tell you that the Board is a welcome addition because of all the voices, and hopefully all the political energy that you could help mobilize to get us the things that we need, right? I think that — I some of you, not all of you, I think that there's going to be a lot of give and take, and a lot of, shall we say, evolution as we go forward. I myself think that — and again, you're going to need advice from people that live in the legal world what is possible within the terms of FACA, et cetera. My own preference would be because we really — I mean, look, we're embarked on a journey, and I have no idea where it's going to end up, right? I would prefer — And I think if I were you, I would prefer something that's not as specific, and much more narrowing since we really don't know — we really don't know what we're — where we're going.

MS. SULLIVAN: Okay. Well, then why don't we say I move that the Board establish four subcommittees to guide the Board's work, and that's pretty open. And if you want Carol — if we wanted the motion, I can put the names of the four committees, or we can just move to create the committees, which might give us a little more flexibility if we needed that.

DR. LEE: I think we have to identify the committees in order for them to be constituted, not just that we'll — we already have the capacity to create committees under ESRA. 

DR. SCHNEIDER: Okay. So we know where the subcommittees — we know what the target of the subcommittees are, so that's fine. And I'm not sure if we want more sentences then to advise, or to — you know, to gather data, whatever the right words are, in order to help the Board come up with recommendations and advice to the Director. That to me is — anything more than that is, I think, we're looking for trouble. 

DR. LEE: Well, that is the essence — I think the essence of this is that committees do not make decisions. Committees are the forms through which data is examined, points of view are brought to bear, and that then come to the Board to make recommendations in the areas that we have identified. And again, just the complexity of the discussions today just around the future of education report, and all of those recommendations. There's no way we can wrestle through this in a Board meeting. 

DR. FUCHS: Yeah. Caroline, I agree. I agree. I would — and I agree with everyone, the general sentiment that it should be more general than specific. And I'm just wondering whether we should limit ourselves by saying four. I mean, are we sure that it's going to be four, and it's not going to be five or six? I mean, I just don't see the imperative right now to specify a number. I mean, we could say — we could say, you know, that the motion could be about forming subcommittees, four of which are, if you want to name them, and that leaves open the possibility that we add to them later.

DR. LEE: We can certainly do that. I just don't want to lose the four areas of focus, but to say that we are creating a structure of standing committees of the Board of which the original selection will be focused on these four topics. To me, the importance of creating the standing committees is it allows that group of people to meet and to vet through ideas before we come to Board meetings. So can I get a second?

MS. SULLIVAN: Whoever is taking the minutes, do you want to read back the formal motion? Do we have a Secretary?

DR. LEE: Ellie, are you taking notes that you could —

MS. PELAEZ: No. We have a court reporter on who is taking the minutes. 

MS. SULLIVAN: Okay. So, it's a transcript, not necessarily minutes. So, all right. I will move that the Board establishes subcommittees to support and advise the work — it's very hard to dance between specific and vague. To advise the work of the Board. Why don't we just leave it wide, wide open? Including standing committees to address communications, policy, research for the executive director, and support of IES centers. Carol, do we need to go to recommendations, or just — that's pretty vague, but, I think, that would — that hits it. 

DR. LEE: I think that's fine. 

MS. SULLIVAN: All right. 

DR. LEE: Can I get a second? Because this is a big question, Ellie, why don't you call the roll.

MS. PELAEZ: Okay. I can do that.

DR. HARPER: And we're doing yea and nay?

DR. LEE: What did you say?

DR. HARPER: No. I was just going to say the options are yea and nay, or abstain?

DR. LEE: Correct.

DR. HARPER: Okay.

MS. PELAEZ: Dr. Anaya? We know he's not on.

Ms. Hernandez-Legorretta?

MS. LEGORRETA: Yea.

MS. PELAEZ: Okay. Dr. Fuchs?

DR. SCHNEIDER: That would be yea. He gave a thumbs up.

MS. PELAEZ: Oh, he did. Okay. Dr. Gandara?

DR. GANDARA: Yea.

MS. PELAEZ: Mr. Guy?

MR. GUY: Yes. Yea.

MS. PELAEZ: Dr. Harper?

DR. HARPER: Yea.

MS. PELAEZ: Mr. Hilliard?

MR. HILLIARD: Yea.

MS. PELAEZ: Dr. Klasko?

DR. KLASKO: Yea.

MS. PELAEZ: Dr. Lee?

DR. LEE: Yes.

MS. PELAEZ: Dr. Lopez Turley?

DR. TURLEY: Yes.

MS. PELAEZ: Dr. Scott?

DR. SCOTT: Yea.

MS. PELAEZ: Ms. Sullivan?

MS. SULLIVAN: Yea.

MS. PELAEZ: And Dr. Yoshikawa?

(No response.) 

DR. LEE: So that's it, right? 

MS. PELAEZ: Yes. That's it.

DR. LEE: Okay. Great. Thank you. 

MS. PELAEZ: Uh-hmm.

DR. LEE: So, we're going to be able to finish early today because I'd rather wait and take tomorrow to talk about both the ESRA, and other policy initiatives that have a timely basis to them in terms of how we're thinking about them. And also at that meeting to discuss a calendar for Board meetings. And I'm thinking that it might be a particularly formal policy committee. So, one, I would say that we had sent out early a request for people to decide on what committee they wanted to work. I would say that we can send perhaps Ellie, you know, a communication out to the Board members identifying the four subcommittees so that people can decide if we want to stay with the same committee, or another committee. You can be on more than one committee, you know, if you like. But the two committees, I think maybe three really, that are going to need to meet very quickly will be one, the executive director search, and the other is the group — some combination between the policy group and the communication if we're going to take up Conchita's recommendation about possibly being able to prepare something, small though it may be, by March or April for issues that are on the table coming up. So, you can kind of think about that, you know, tonight, and we'll meet tomorrow morning. And we probably won't need to take the full time for that tomorrow. I'm praying that Linda's husband is okay, so she'll be able to be on Board of the meeting tomorrow, but she's been gathering a lot of information relative to the ESRA authorization. That will be kind of helpful information, as well as some communications had with a number of stakeholders who are kind of weighing in on that upcoming legislation.

DR. FUCHS: Carol, what time do we meet tomorrow Eastern Standard Time?

DR. LEE: 10:00.

DR. FUCHS: Okay.

DR. LEE: Same time. 10:00 East Coast, 9 o'clock my time. 9:00, 8:00, 7:00 Eastern. Shaun, 7:00 again in the morning. Get a good night's sleep, you all, and Shaun.

DR. HARPER: Maybe that's why I was so cranky today. Maybe.

DR. LEE: We'll make sure we have later times so you'll be in a good mood when you come, right?
All right. Thanks, everyone. I really appreciate everybody's time and effort.

MS. PELAEZ: Thank you, guys. Take care. 

(Chorus of goodbyes.)

DR. LEE: Take care, everyone.

(Whereupon, at 3:50 p.m., the meeting in the above-entitled matter was adjourned, to reconvene at 10:00 a.m. the following day, Tuesday, December 5, 2023.)