Ask a REL Response
Evaluating Digital, Virtual, Online, and Blended Teaching — June 2020
Question
What does research say about evaluating digital, virtual, online, and blended teaching? Is there any research on educator evaluation in response to COVID-19?
Response
Following an established REL West research protocol, we conducted a search for research reports and resources on evaluating digital, virtual, online, and blended teaching, and on educator evaluation in the era of COVID-19.
The sources included ERIC, Google Scholar, and PsychInfo. (For details, please see the methods section at the end of this memo.)
We have not evaluated the quality of references and the resources provided in this response. We offer them only for your reference. Also, we searched for references through the most commonly used sources of research, but the list is not comprehensive and other relevant references and resources may exist. References are listed in alphabetical order, not necessarily in order of relevance. Access to the full articles is free unless indicated otherwise.
Research References
Black, E., DiPietro, M., Ferdig, R., & Polling, N. (2009). Developing a survey to measure best practices of K–12 online instructors. Online Journal of Distance Learning Administration, 12(1). Abstract available from https://eric.ed.gov/?id=EJ869272 and full text available from https://www.westga.edu/~distance/ojdla/spring121/black121.html
From the abstract: “Limited data exists related to teaching and learning in K–12 virtual schools. This paper builds upon a recent study related to successful practices of K–12 online instructors. The paper describes the utilization of a survey built upon qualitatively derived best practices of K–12 online instructors and provides the opportunity to relate these practices to teacher’s perceived professional development needs. Outcomes indicate that virtual school instructors identify online presence, diligent student monitoring and an enjoyment of technology among factors that contribute to virtual school instructor success. Instructors also identified face-to-face student mentors as a key component for success. Respondents felt that they would benefit from professional development focused on technological skills, content-based technological integration and evaluative resources for online learners. The paper concludes with a call for additional research to refine and implement the assessment.”
Natale, C. F. (2012). Teaching in the world of virtual K–12 learning: Challenges to ensure educator quality. Educational Testing Service (ETS). Full text available from https://www.ets.org/s/educator_licensure/ets_online_teaching_policy_final_report.pdf
From the Introduction: “This report presents the results of a six-month research project examining virtual K–12 teaching and learning. It was first necessary to understand the current state of K–12 virtual education across the country: various contexts and dimensions of online learning, various categories and models of online delivery systems and recent and projected trends in online learning. From there, one can begin to examine how teaching in an online environment differs from ‘traditional’ or ‘brick-and-mortar’ classrooms, what key skills or knowledge are needed for effective online teaching and if and how higher education is preparing teachers for the new world of virtual K–12 learning. Next came an in-depth examination of what was going on in the states: what states have significant virtual K–12 learning initiatives? What specific requirements do states have to ensure the quality of their online teaching force? How many full- and part-time online teachers are there? What interest do states have in pursuing endorsements specific to online teachers and/or an assessment of online teaching competencies? The results of this study do not produce simple answers, as the world of virtual K–12 education has been described by several individuals as the ‘wild, wild west’; that is, a largely unregulated, fluid and rapidly changing environment influenced by factors beyond the current jurisdiction of many state departments of education. Nonetheless, as this report will show, there is a growing recognition on the part of many state agency personnel that, while policies often lag behind practice, the issue of teacher quality remains critically important, and attention is gradually shifting to focus on such a critical matter.”
Nittler, K., & Saenz-Armstrong, P. (2020). Teacher evaluations and support during COVID-19. National Council on Teacher Quality. Full text available from https://www.nctq.org/blog/Teacher-evaluations-and-support-during-COVID--19-closures
From the introduction: “Now more than ever, teacher evaluations, albeit retooled, could provide the support teachers require and the oversight students need. Providing feedback and support to teachers can both equip them presently as they adjust their practice to distance learning, as well as guide focus areas for future growth once students and teachers return to their physical classrooms. While we have a good sense of how states are responding on teacher evaluation policy, so far there’s been less focus on what districts are doing, which is where the rubber actually meets the road. In the last few weeks, some districts have begun to define their evaluation policies to fit the current times, although it appears that most of the large school districts NCTQ tracks have yet to do so. We hope that our review and analysis of this first set of districts will serve as guidance to other districts as they undertake this important work.”
Powell, A., Rabbitt, B., & Kennedy, K. (2014). iNACOL Blended Learning Teacher Competency Framework. International Association for K-12 Online Learning. Full text available from https://eric.ed.gov/?id=ED561318
From the abstract: “In recent years there has been a dramatic rise in interest and early adoption of blended learning to improve the educational experiences of students. A great amount of work has been done to codify approaches, with tools and resources emphasizing the structural components of new models, such as the configuration of physical learning space, use of time, distribution of staff, and applications of technology. While there is widespread recognition that great in person teaching remains essential within these structures, there has been less exploration of the human factors and effective practices that make them successful. Schools and districts are asking for more support for understanding teachers’ new roles and effectively supporting them in transitioning to new models of teaching and learning. To respond to this need, iNACOL and The Learning Accelerator (TLA), two organizations committed to helping educators succeed at adopting and implementing blended learning at scale, assembled a national committee of blended learning practitioners, thought-leaders, and experts to explore one critical question: What are the key characteristics of teachers in successful blended learning environments? Over the last year, this committee worked together to review existing practices and research (including an earlier framework developed by TLA), to develop emerging hypotheses with each other and then field-test them with a broader set of external stakeholders. This process culminated in the development of the work presented here, the ‘iNACOL Blended Learning Teacher Competency Framework.’”
Schulte, M. (2009). Efficient evaluation of online course facilitation: The “quick check” policy measure. Journal of Continuing Higher Education, 57(2), 110–116. Abstract available from https://eric.ed.gov/?id=EJ866286 and full text available for a fee from https://doi.org/10.1080/07377360902995685
From the abstract: “In distance education and online learning, the exponential growth of programs and the need for instructors has forced proper analysis of instructor teaching and learning to the background. To meet the immediate needs of students and technical operations, distance learning institutions often fail to evaluate how well instructors follow needed online policies and online best practices, and do not provide general mentoring or remediation for instructors. One task confronting many distance learning programs is to properly evaluate a large number of instructors in a short time. Another task is to then use the completed evaluations to promote professional development. Park University’s College for Distance Learning (CDL) has implemented a Quick Check evaluation mechanism to gauge online instructor facilitation. The Quick Check is an elemental online course facilitation measure that efficiently and objectively evaluates online instructors within their online classroom environment. In this article, the author reviews relevant literature, describes the Quick Check process, and presents research on its effectiveness in increasing facilitation of online courses.”
Ternus, M. P., & Faulk, D. R. (2007). Benchmarking quality in online teaching and learning: A rubric for course construction and evaluation. The Journal of Effective Teaching, 7(2), 51–67. Full text available from https://eric.ed.gov/?id=EJ1055643
From the abstract: “Online courses have many components and dimensions. Both the form (structure) and the content (expression) are situated in an overall environment. The sum of these elements results in student outcomes and learning. In order to facilitate construction and evaluate the quality of an online course, a four-part rubric was designed to reflect: Structure (Context, Organization, and Environment); Content (Presentation of Information); Processes (Relationships and Interactions); and Outcomes (Mastery of Content and Course Evaluation). This rubric was designed to provide quantitative and qualitative standardized evaluation for faculty self-evaluation, peer evaluation, and administrator evaluation. The rubric was piloted at two universities and shown to be highly effective in eliciting effective and usable feedback for course instructors and program directors. It was concluded that a uniform rubric that can be applied to any discipline could facilitate evaluation of all online courses within a program to a set standard that can then be used for course enhancement and improvement with structured comprehensive evaluation from instructors, peers, or program directors. It was found that a well-designed course (structure), with relevant and credible information (content), as well as mechanisms for interaction and collaboration (processes), could result in enhanced student learning (outcomes).”
Tobin, J. (2004). Best practices for administrative evaluation of online faculty. Online Journal of Distance Learning Administration, 7(2). Abstract available from https://eric.ed.gov/?id=EJ1114201 and full text available from https://www.westga.edu/~distance/ojdla/summer72/tobin72.html
From the abstract: “This presentation demonstrates how to evaluate the materials and teaching in online courses. Topics covered include similarities with evaluation of on-ground teaching, factors unique to online courses, technological considerations, helping administrators unfamiliar with online courses, and national standards, rubrics, and benchmarks.”
Additional Organizations to Consult
The Aurora Institute – https://aurora-institute.org
From the website: “The Aurora Institute, formerly known as the North American Council for Online Learning (NACOL) (2002–2010) or International Association for K–12 Online Learning (iNACOL) (2010–2019), is an international K–12 nonprofit organization focused on enhancing K–12 online learning quality.”
REL West note: The Aurora Institute has one resource that is relevant to this request:
National standards for quality online teaching. Full text available from https://eric.ed.gov/?id=ED509639
Education Commission of the States (ECS) – www.ecs.org
From the website: “Education Commission of the States partners with education policy leaders to address issues by sharing resources and expertise. We are proud to serve both the people who develop and implement education policy and the students who directly benefit from effective policy change.”
REL West note: ECS has two resources that are relevant to this request:
State information request: Teacher evaluations during COVID-19 closures Full text available from https://www.ecs.org/state-information-request-teacher-evaluations-during-covid-19-closures/ and https://www.ecs.org/covid-19-update/
State information request: What states are doing about teacher evaluations during COVID-19. Full text available from https://www.ecs.org/wp-content/uploads/State-Info-Request-Teacher-Evaluations-During-COVID-19-Closures.pdf
Education Week – www.edweek.org
From the website: “Education Week is an independent news organization that provides comprehensive coverage on K–12 education news, analysis, and opinion. We believe that an excellent education is possible for all students and our mission is to inspire and empower our readers to continue advancing the field every day.”
REL West note: Education Week has one resource that is relevant to this request:
Should teachers be evaluated during coronavirus school shutdowns? Full text available from https://www.edweek.org/ew/articles/2020/04/20/should-teachers-be-evaluated-during-coronavirus-school.html
Florida Virtual School (FLVS) – https://www.flvs.net
From the website: “Florida Virtual School (FLVS) is an established leader in developing and providing virtual K–12 education solutions to students all over Florida, the U.S. and the world. A nationally recognized e-Learning model and recipient of numerous awards, FLVS was founded in 1997 and was the country’s first, state-wide Internet-based public high school. Today, FLVS serves students in grades K–12 and provides a variety of custom solutions for schools and districts to meet student needs.”
REL West note: FLVS has one resource that is relevant to this request:
Florida Virtual School. (2018). Florida Virtual School instructional personnel evaluation system. Orlando, FL: Author. Full text available from https://www.flvs.net/docs/default-source/district/flvs-instructor-evaluation-plan.pdf?sfvrsn=19337a2a_16
Southern Regional Education Board (SREB) – https://www.sreb.org
From the website: “The Southern Regional Education Board works with states to improve public education at every level, from early childhood through doctoral education. We help policymakers make informed decisions by providing independent, accurate data and recommendations. We help educators strengthen student learning with professional development, proven practices and curricula. And we help policymakers, institutions and educators share scarce resources to accomplish more together than they could alone.”
REL West note: The SREB has two resources that are relevant to this request:
Southern Regional Education Board. (2006a). Online teaching evaluation for state virtual schools. Full text available from https://www.sreb.org/publication/online-teaching-evaluation-state-virtual-schools
Southern Regional Education Board. (2006b). Standards for quality online teaching. Full text available from https://www.sreb.org/publication/standards-quality-online-teaching
Method
Keywords and Search Strings
The following keywords and search strings were used:
[(Evaluation OR evaluating OR evaluate) AND (teacher OR educator) AND (digital OR virtual OR online OR blended) AND (teaching OR instruction)]
<[(Educator OR teacher) AND evaluation AND “COVID-19”]
Databases and Resources
We searched Google Scholar and ERIC for relevant resources. ERIC is a free online library of over 1.6 million citations of education research sponsored by the Institute of Education Sciences.
Reference Search and Selection Criteria
When searching and selecting resources to include, we consider the criteria listed below.
- Date of the Publication: References and resources published within the last 15 years, from 2005 to present, were included in the search and review.
- Search Priorities of Reference Sources: Search priority is given to study reports, briefs, and other documents that are published and/or reviewed by IES and other federal or federally funded organizations and academic databases. Priority is also given to sources that provide free access to the full article.
- Methodology: Priority is given to the most rigorous study designs, such as randomized controlled trials and quasi-experimental designs, and we may also include descriptive data analyses, survey results, mixed-methods studies, literature reviews, or meta-analyses. Other considerations include the target population and sample, including their relevance to the question, generalizability, and general quality. Priority is given to publications that are peer-reviewed journal articles or reports reviewed by IES and other federal or federally funded organizations. If there are many research reports available, we select those with the strongest methodology, or the most recent of similar reports. When there are fewer resources available, we may include a broader range of information. References are listed in alphabetical order, not necessarily in order of relevance.
This memorandum is one in a series of quick-turnaround responses to specific questions posed by educational stakeholders in the West Region (Arizona, California, Nevada, Utah), which is served by the Regional Educational Laboratory West at WestEd. This memorandum was prepared by REL West under a contract with the U.S. Department of Education’s Institute of Education Sciences (IES), Contract ED-IES-17-C-0012, administered by WestEd. Its content does not necessarily reflect the views or policies of IES or the U.S. Department of Education nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.