Project Activities
Using an iterative design process, the researchers worked with community college faculty and staff, students, and local employers to refine and study the New World of Work program. The researchers examined the NWoW curriculum, work-based learning component, and assessments to determine its promise for improving student outcomes and delivered recommendations for program improvement and refinement based on the research conducted. The original research plan included a pilot study of the intervention. However, due to complications arising from statewide NWoW management and maintenance changes and disruptions caused by the COVID-19 pandemic, the researchers and IES revised the project's scope of work to focus on reviewing the intervention and disseminating information to practitioners.
Structured Abstract
Setting
The research took place in select community colleges in California and through interviews with local employers.
Sample
From 2016 to 2022, the research team conducted four rounds of site visits (virtual and in person). In all, the team interviewed 16 instructors across 5 community colleges pilot testing the program, 37 students in focus group settings that were enrolled in Cooperative Work Experience and Education (CWEE) classes, and 15 employers. To conduct the psychometric analyses, the research team used assessment scores and item responses (from which individual identification data had been removed) from NWoW assessments that were administered from 2016 through 2018 across 16 postsecondary institutions in California. The data set had an average of 3,000 student respondents per assessment.
The New World of Work (NWoW) is a program that promoted teaching and learning soft skills and was designed in 2012 and operated in over 75 community colleges in California. The program consisted of a classroom component, a work-based learning component, and a credential-granting component. During the 2015-2016 academic year, community college faculty began piloting New World of Work (NWoW) by integrating NWoW courses into Cooperative Work Experience and Education (CWEE) classes throughout the state. CWEE is a state-approved program for postsecondary career and technical education (CTE) in which students learn technical and academic skills through direct instruction and a work requirement (either a job, a volunteer position, or an internship position). The specific skills the NWoW course covered include adaptability, analysis/solution mindsets, entrepreneurial mindsets, collaboration, communication, digital fluency, empathy, resilience, self-awareness, and social/diversity awareness. The course materials included lesson plans, PowerPoint presentations, and videos. NWoW also included guidelines for a companion work-based experience and a credentialing system (i.e., a self- and employer-assessment of students' 21st century skills and integration with a digital badging platform).
Research design and methods
Using a two-cycle, four-step per cycle iterative design process, the development team worked with instructors, community college staff, local employers, and students to refine and improve the existing core components of NWoW. This development work focused on strengthening several model elements including more uniform delivery of the curriculum and possible streamlining of content, enrichment of the program and adaptation specifically for the course integration, guidelines and tools to support the scaling of quality work-based learning experiences and their alignment with 21st century skills course modules, and the refinement of a skill assessment and digital badge system. Due to statewide NWoW management and maintenance changes and disruptions caused by the COVID-19 pandemic, the researchers revised the scope of the study and shifted focus to comprehensively analyze new and extant data from the NWoW development work and to produce and disseminate the findings. This included completing data collection, analyzing and compiling results, creating and posting both practitioner-oriented and researcher-oriented materials, and continuing dissemination and promotion of NWoW publications through MDRC dissemination channels.
Control condition
Not applicable.
Key measures
The team used assessment and badging data, interview and focus group data, curricula and training materials, and classroom observations.
Data analytic strategy
The researchers conducted a thorough analysis of the qualitative data collected from the study using a pre-specified analytic framework and coding schema. The research team also used psychometric analysis as a test of each of the program's ten 21st-century skill assessments to produce recommendations on improving the assessments. For the psychometric analysis, the research team ran descriptive statistics and conducted preliminary analyses using item response theory, and examined psychometric properties such as item difficulty, item-total correlation, total score distributions, total score correlations, Cronbach's alpha, and likelihood ratio tests.
Key outcomes
The main findings of this project are as follows:
Due to the aforementioned changes in scope, the project team focused on translating their work for practitioners. In particular, they highlighted a few notable issues for practitioners (see also the list under Additional online resources below):
As noted in Dalporto and Lepe (2022), the researchers recommend the following:
- Focus on a few skills to master when designing or selecting soft-skills curricula,.
- Weigh tradeoffs between stand-alone soft skills courses and embedding skill instruction in existing courses when determining where to house the curricula,.
- Get employers involved by having students set up soft-skills-centered check-in meetings with their work supervisors.
As noted in Yang, Halpin, and Handy (2022), the researchers recommend the following regarding skill assessment:
- Preliminary psychometric analysis on a soft-skills curriculum can be an opportunity to identify areas for refinement while preparing for a larger rollout of a program.
- Psychometric analysis, in combination with other information (such as the perspectives of instructors, employers, and students), can help determine how well assessments capture the mastery of the soft skills students need to thrive in today's labor market.
- Psychometric analyses can be used to demonstrate that assessments are measuring skill mastery and that credentials represent skill mastery, allowing employers and educators to better gauge the credentials' value.
People and institutions involved
IES program contact(s)
Products and publications
Project website:
Publications:
ERIC Citations: Find available citations in ERIC for this award here.
Select Publications:
Visher, M. & Larson, M. (2018, February 27) Teaching 21st Century Skills to Community College Students: An Innovative Approach Under Development in California. Inside IES Research.
Additional project information
Additional online resources and information:
- Using Psychometric Analysis to Improve Soft-Skill Assessments Yang, Halpin, and Handy (2022) Full text
- Implementing Soft-Skills Programs in a Postsecondary Setting: Lessons from the New World of Work Dalporto and Lepe (2022) Full text
- What Do Employers Want to See from Soft-Skills Credentials? Cellura and Lepe (2022) Full text
- How to Implement Soft-Skills Programs, Curricula, and Instruction in a Postsecondary Setting Otasowie and Dalporto (2023) Full text
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
