Description:

Previous Award Number: R305N160013
Previous Awardee: Arizona State University

Related Network Teams: Early Learning Network Lead (PI: Susan Sheridan, R305N160015); Early Learning Contexts in Rural and Urban Nebraska (PI: Susan Sheridan, R305N160016); Boston P-3: Identifying Malleable Factors for Promoting Student Success (PI: JoAnn Hsueh, R305N160018); Building an Effective PK–3 Education System: Actionable Aspects of Policies, Programs, Schools, and Classroom Processes that Promote Children's Learning in the Nation's 11th Largest School District (PI: Robert Pianta, R305N160021); Early Education in Rural North Carolina (PI: Margaret Burchinal, R305N160022); Early Learning Network: Critical Contributions of Classroom Ecology to Children's Learning (PI: Laura Justice, R305N160024)

Description: A Research Network involves several teams of researchers who are working together to address a critical education problem or issue. The objective is to build new knowledge, encourage information sharing, and assist policymakers and practitioners to strengthen education policies and programs and improve student education outcomes. Members of the Early Learning Network will identify malleable factors that support early learning from preschool through early elementary school grades. They will also develop a classroom observation tool that practitioners can use to assess aspects of classroom quality that are associated with positive academic and social-behavioral outcomes in preschool and early elementary school. The Network includes five Research Teams, an Assessment team, and a Network Lead.

Purpose: The Assessment Team will build on three existing, integrated, well-validated, and reliable measures and develop a technology-based observational tool that can be used by practitioners in prekindergarten through third grade (P-3) classrooms to assess instruction, teacher-child interactions, and the classroom learning environment. The tool, to be called Optimizing Learning Opportunities for Students' (OLOS) Early Learning Observation System, will address the need for a comprehensive, multidimensional assessment of structural features, classroom processes, and the quality of instruction in different subject matter areas that are associated with child outcomes across the transition from preschool to early elementary grades.

Project Activities: The Assessment Team will pursue an iterative design, development, and testing process based upon well-established principles of design-based implementation research. In years 1 and 2, the Assessment Team will develop a prototype of the OLOS observation system. In year 3, the Assessment Team will work with other Network members to conduct a field test at preschool programs and elementary schools. In year 4, the Assessment Team will refine the observation system. In year 5, the Assessment Team will work with other Network members to conduct another field test and finalize the observation system for use in prekindergarten and elementary school classrooms.

Products: This project will result in the Optimizing Learning Opportunities for Students' (OLOS) Early Learning Observation System, an innovative, multidimensional assessment system that can be used by practitioners to conduct observations of preschool and early elementary general education classrooms. The technology-based observation system will generate reports to match effective recommended instructional practices with observed learning opportunities by creating charts and reports for practitioners that will provide clear guidance for improving instructional practices and enhancing learning opportunities for students.

Structured Abstract

Setting: This Assessment Team will begin developing the observation tool in prekindergarten and elementary schools in southern California that serve ethnic/racial minorities, English Language Learners (ELL), and children living in poverty. It will then work with other members of the Early Learning Network to pilot-test and validate the observation system in Boston, Massachusetts; Fairfax County, Virginia; rural and urban school districts in Nebraska; rural communities in North Carolina; and selected school districts in Ohio.

Sample: For Years 1 and 2 (prototype development of OLOS), the researchers will use an existing database of observations of preschool through third grade classrooms as part of other research projects. Video-taped observations of classroom instruction and assessments of students' literacy, math, and social-behavioral outcomes were conducted in pre-k and elementary school classrooms in Arizona (144 students), Florida (2351 students and 291 teachers), and Michigan (215 pre-k children and 43 pre-k classrooms; 285 kindergarten children and 62 classrooms; and 206 first graders and 89 classrooms). Additional Year 1 data will come from a sample of pre-k classrooms and elementary schools in Southern California. The sample for the Year 3 field test will include participants from the assessment team's research site in California and participants from other Early Learning Network research sites. In Year 5, the sample will include 300 classrooms and 20 schools.

Assessment: The Optimizing Learning Opportunities for Students' (OLOS) Early Learning Observation System will draw upon three existing, valid and reliable observational tools: Individualizing Student Instruction (ISI)/Pathways, Creating Opportunities to Learn from Text (COLT), and the Quality of the Classroom Learning Environment Rubric (Q-CLE). OLOS will be a web-based system that can be used across different computer platforms (PC, Mac, Chromebook, and tablets). The OLOS observational system will capture both structural and process aspects of classrooms, and the amount and type of learning opportunity in preschool classrooms and the early elementary grades. OLOS allows practitioners to observe individual students. OLOS will be used to observe both individual children and global aspects of teaching quality. Specific constructs that will be assessed include content of the interaction, context, role of the teacher, type of child participation in the learning opportunity (e.g., off task, listening, interactive), and teacher quality (e.g., teacher affect, warmth, and responsiveness in teacher-child interactions and classroom organization). OLOS will capture multiple language settings, such as Spanish instruction and interactions. OLOS will also capture literacy and emergent literacy, numeracy, math and science learning, teacher warmth and responsiveness to students, teacher-student interactions, classroom management and organization, discipline, teacher knowledge, and social and emotional climate. In the OLOS observation system, the ISI/Pathways measure will capture the match between recommended and observed instruction in specific content areas; the COLT-Student (COLT-S) will capture students' participation in learning opportunities at the individual student level; the COLT-Teacher (COLT-T) will capture teacher-child interactions; and the Q-CLE will capture teachers' practice and the quality of the classroom environment.

The OLOS observation system will capture three key dimensions of students' opportunity to learn and the quality of the classroom environment from preschool through early elementary school including: (1) the match between recommended amounts (in minutes) and types of instruction and observed amounts (ISI/Pathways measure); (2) students' participation in learning opportunities (COLT-S); teacher-student interactions (COLT-T); and (4) global characteristics of the classroom environment that are associated with student academic and social-behavioral outcomes (Q-CLE). Using the information from classroom observations, the OLOS system will generate charts and reports to provide feedback to practitioners to improve practice.

Research Design and Methods: The Assessment Team will, in collaboration with other Network members, develop and validate the OLOS observation system over a five-year period. In years 1 and 2, the Assessment Team will develop a fully functional prototype of the OLOS observation system that will work on a computer or tablet. In Year 1, the researchers will use existing databases of student outcomes and classroom observations to test and validate the codes. The Assessment Team will also collect primary data in prekindergarten classrooms to inform the development of observational codes for early childhood settings. In Year 3, the Assessment Team will field test the OLOS observation system at local preschool programs and elementary schools. The Assessment Team will also work with the other Network members to field test OLOS in prekindergarten programs and elementary schools at the various research sites. In Year 4, the Assessment Team will use data collected in year 3 to refine and improve the psychometrics, reports, and user interfaces. The research team will also examine how much training observers will need to achieve reliability when using the system. By the beginning of Year 5, the Assessment Team will have developed a fully functioning version of the OLOS observation system that can be used live or with video. After a final round of testing in prekindergarten classrooms and elementary schools, the OLOS observation system will be a fully developed, reliable, and valid observation system that can be used by practitioners.

Control Condition: Due to the nature of the research design, there is no control condition.

Key Measures: In year 1, data sources will include archive classroom observation data from preschool and elementary schools in Arizona, Florida, and Michigan. For primary data collection, several web-based, researcher-developed measures will be used to assess children language and literacy and early math skills. The research team will use the Head-Toes-Knees-Shoulders measure to assess children's social behavioral skills. Researchers will use the ISI/Pathways, COLT, and Q-CLE measures to collect classroom observation data. The research team will include archival assessment and observation data, which includes a comprehensive battery of language, literacy, math, and social-behavioral assessments, as well as classroom observations during the fall, winter, and spring of the school year. In addition, researchers will include a core set of web-based assessments conducted a minimum of three times per year, as well as assessments proposed by the other Early Learning Network teams.

Data Analytic Strategy: The assessment team will conduct a series of generalizability and decision studies to evaluate the reliability and validity of OLOS both in lab prototypes and the full version in schools.