Research References

Claxton, G., Costa, A. L., & Kallick, B. (2016). Hard thinking about soft skills. Educational Leadership, 73(6), 60–64. Abstract retrieved from https://eric.ed.gov/?id=ED520815 ; full text available at http://www.learningpersonalized.com/wp-content/uploads/2016/09/Hard-Thinking-about-Soft-Skills.pdf

From the executive summary:

People use various terms to refer to traits and tendencies connected to social-emotional behavior and ways of thinking or approaching problems—from 21st century skills to mindsets to habits of mind. Such traits are also often called soft skills or non-cognitive skills. The authors contend that these latter terms imply that these traits and abilities are less cognitively challenging, less measurable, and less important to teach students as we prepare them for life. They discuss how the terms soft, non-cognitive, and even skills don't do justice to the qualities and the importance of practices like drawing on past knowledge and empathizing or traits like resilience. Claxton, Costa, and Kallick argue for using the term dispositions to refer to these essential learnings and discuss both questions and strategies connected to teaching students such dispositions.

Duckworth, A. L., & Yeager, D. S. (2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44(4), 23–251. Abstract retrieved from https://eric.ed.gov/?id=ED520815 ; full text available at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4849415 .

From the abstract:

There has been perennial interest in personal qualities other than cognitive ability that determine success, including self-control, grit, growth mindset, and many others. Attempts to measure such qualities for the purposes of educational policy and practice, however, are more recent. In this article, we identify serious challenges to doing so. We first address confusion over terminology, including the descriptor ‘non-cognitive.’ We conclude that debate over the optimal name for this broad category of personal qualities obscures substantial agreement about the specific attributes worth measuring. Next, we discuss advantages and limitations of different measures. In particular, we compare self-report questionnaires, teacher-report questionnaires, and performance tasks, using self-control as an illustrative case study to make the general point that each approach is imperfect in its own way. Finally, we discuss how each measure's imperfections can affect its suitability for program evaluation, accountability, individual diagnosis, and practice improvement. For example, we do not believe any available measure is suitable for between-school accountability judgments. In addition to urging caution among policymakers and practitioners, we highlight medium-term innovations that may make measures of these personal qualities more suitable for educational purposes.

Egalite, A. J., Mills, J. N., & Greene, J. P. (2016). The softer side of learning: Measuring students' non-cognitive skills. Improving Schools, 19(1), 27–40. Abstract retrieved from https://eric.ed.gov/?id=EJ1092212; full text of 2014 Working Paper available at http://www.uaedreform.org/wp-content/uploads/EDRE-WP-2014-03.pdf .

From the abstract:

With an abundance of datasets of standardized test score data, researchers and education policymakers run the risk of focusing exclusively on the measurement of cognitive outcomes in key academic subjects such as math and reading at the expense of important non-cognitive outcomes. We use behavioral measures of conscientiousness, perseverance, and delay of gratification as well as a self-reported measure of student grit—defined as student perseverance and passion for long-term goals—to assess the non-cognitive skills of 174 16- to 18-year-old students attending a residential public high school in Arkansas, United States. Analysis shows that 11th grade students (16- to 17-year-olds) rate themselves lower on self-reported grit but outperform their 12th grade counterparts (17- to 18-year-olds) on behavioral measures of persistence, delay of gratification, and conscientiousness. These findings point to the strengths and limitations of existing tools for measuring non-cognitive skills and the need for more measures to be developed and tested with diverse populations.

Eisner, S. (2010). Grave new world? Workplace skills for today's college graduates. American Journal of Business Education, 3(9), 27–50. Retrieved from https://eric.ed.gov/?id=EJ1058556

From the abstract:

Today's college graduates face an uncertain and demanding job market in which they are likely to encounter evolving skill needs, reduced hiring, and heightened competition from experienced laid off workers and globally available labor. These realities underscore the importance expressed by educators and practitioners of identifying attributes new college graduates should have to meet the challenges and access opportunities of the 21st century workplace. This paper identifies those core competencies through comprehensive literature review, original studies using primary data, and contextual analysis of the contemporary workplace in which the skills will be valued.

Gutman, L. M., & Schoon, I. (2013). The impact of non-cognitive skills on outcomes for young people. London, England: Institute of Education, University of London and the Education Endowment Foundation. Retrieved from https://v1.educationendowmentfoundation.org.uk/uploads/pdf/Non-cognitive_skills_literature_review_1.pdf

From the executive summary:

The term ‘non-cognitive skills’ refers to a set of attitudes, behaviours, and strategies that are thought to underpin success in school and at work, such as motivation, perseverance, and self-control. They are usually contrasted with the ‘hard skills’ of cognitive ability in areas such as literacy and numeracy, which are measured by academic tests. Non-cognitive skills are increasingly considered to be as important as—or even more important than—cognitive skills or IQ in determining academic and employment outcomes. Indeed, there is now growing attention from policymakers on how such ‘character’ or ‘soft’ skills can be developed in children and young people. However, despite growing interest in this topic, the causal relationship between non-cognitive skills and later outcomes is not well established. This rapid literature review is intended to summarize the existing evidence on how ‘non-cognitive skills’ can be defined and measured; assess the evidence that such skills have a causal impact on later outcomes; and the role of select interventions that aim to improve non-cognitive skills in children and young people. It has been jointly funded by the Education Endowment Foundation and Cabinet Office to inform future work in this area.

Humphries, J. E., & Kosse, F. (2016). On the interpretation of non-cognitive skills: What is being measured and why it matters. (SOEPpapers on Multidisciplinary Panel Data Research, Working Paper No. 876). Berlin, Germany: German Institute for Economic Research. Retrieved from https://www.econstor.eu/bitstream/10419/148554/1/875032826.pdf

From the abstract:

Across academic sub-fields such as labor, education, and behavioral economics, the measurement and interpretation of non-cognitive skills varies widely. As a result, it is difficult to compare results on the importance of non-cognitive skills across literatures. Drawing from these literatures, this paper systematically relates various prototypical non-cognitive measures within one data set. Specifically, we estimate and compare several different strategies for measuring non-cognitive skills. For each strategy, we compare their relative effectiveness at predicting educational success and decompose what is being measured into underlying personality traits and economic preferences. We demonstrate that the construction of the non-cognitive factor greatly influences what is actually measured, how it relates to more standard taxonomies and what conclusions are reached about the role of non-cognitive skills in life-outcomes such as educational attainment. Furthermore, we demonstrate that, while sometimes difficult to interpret, factors extracted from self-reported behaviors can have predictive power similar to well established taxonomies, such as the Big Five.

Kautz, T., Heckman, J. J., Diris, R., Ter Weel, B., & Borghans, L. (2014). Fostering and measuring skills: Improving cognitive and non-cognitive skills to promote lifetime success. (NBER Working Paper No. 20749). Cambridge, MA: National Bureau of Economic Research http://www.nber.org/papers/w20749 .

From the abstract:

This paper reviews the recent literature on measuring and boosting cognitive and non-cognitive skills. The literature establishes that achievement tests do not adequately capture character skills: personality traits, goals, motivations, and preferences that are valued in the labor market, in school, and in many other domains. Their predictive power rivals that of cognitive skills. Reliable measures of character have been developed. All measures of character and cognition are measures of performance on some task. In order to reliably estimate skills from tasks, it is necessary to standardize for incentives, effort, and other skills when measuring any particular skill. Character is a skill, not a trait. At any age, character skills are stable across different tasks, but skills can change over the life cycle. Character is shaped by families, schools, and social environments. Skill development is a dynamic process, in which the early years lay the foundation for successful investment in later years. High-quality early childhood and elementary school programs improve character skills in a lasting and cost-effective way. Many of them beneficially affect later-life outcomes without improving cognition. There are fewer long-term evaluations of adolescent interventions, but workplace-based programs that teach character skills are promising. The common feature of successful interventions across all stages of the life cycle through adulthood is that they promote attachment and provide a secure base for exploration and learning for the child. Successful interventions emulate the mentoring environments offered by successful families.

Kautz, T., & Zanoni, W. (2014). Measuring and fostering non-cognitive skills in adolescence: Evidence from Chicago Public Schools and the OneGoal Program. Chicago, IL: University of Chicago. Retrieved from https://www.onegoalgraduation.org/wp-content/uploads/2018/04/Measuring-and-Fostering-Non-Cognitive
-Skills-in-Adolescence-Evidence-from-Chicago-Public-Schools-and-the-OneGoal-Program.pdf .

From the abstract:

Recent evidence has established that non-cognitive skills (e.g., persistence and self-control) are valuable in the labor market and are malleable throughout adolescence. Some recent high school interventions have been developed to foster these skills, but there is little evidence on whether they are effective. Using administrative data, we apply two methods to evaluate an intervention called OneGoal, which attempts to help disadvantaged students attend and complete college in part by teaching non-cognitive skills. First, we compare the outcomes of participants and non-participants with similar pre-program cognitive and non-cognitive skills. In doing so, we develop and validate a measure of non-cognitive skill that is based on readily available data and rivals standard measures of cognitive skill in predicting educational attainment. Second, we use an instrumental variable difference-in-difference approach that exploits the fact that OneGoal was introduced into different schools at different times. We estimate that OneGoal improves academic indicators, increases college enrollment by 10–20 percentage points, and reduces arrest rates by 5 percentage points for males. We demonstrate that improvements in non-cognitive skill account for 15–30 percent of the treatment effects.

Kyllonen, P. C. (2012). Measurement of 21st century skills within the Common Core State Standards. Paper presented at the Invitational Research Symposium on Technology Enhanced Assessments. Retrieved from http://www.ets.org/research/policy_research_reports/publications/paper/2012/jvhx .

From the abstract:

For the past 10 years or so, Educational Testing Service has supported the Center for New Constructs, which has had as its mission the identification and evaluation of constructs and measures other than those typically measured by content tests, such as mathematics, English language arts, and science achievement tests. Since then the center has been renamed the Center for Academic and Workforce Readiness and Success (CAWRS) to highlight ETS's recent foray into the workplace, but the work of the center has remained the same—to explore the foundations, rationale, and feasibility of measuring personal characteristics that are important for educational success but that are typically ignored in educational assessment. During this period CAWRS staff have explored this area through literature reviews, collaborations, and data collections ranging from a few students to more than 100,000, including some in K–12 education settings, in community college and college settings, in graduate and professional school, and more recently, in the workplace. CAWRS staff have explored a wide array of constructs, ranging from ones that overlap considerably with what is already tested on standardized tests, such as critical thinking, to ones that are quite distinct, such as teamwork and work ethic. CAWRS also has explored a wide variety of measurement methods, such as self-rating and others' rating scales, situational judgment tests, anchoring vignettes, conditional reasoning, the implicit association test, and others. The purpose of this paper is to review some of this work, particularly as it pertains to the measurement of 21st century skills in K–12 education. I will touch on other areas, but that is the focus. This paper begins with a brief history and justification for why there is interest especially now in 21st century skills, and what is meant by that term.

Robles, M. M. (2012). Executive perceptions of the top 10 soft skills needed in today's workplace. Business Communication Quarterly, 75(4), 453–465. Retrieved from https://www.researchgate.net/profile/Marcel_Robles/publication/258126575_Executive_Perceptions_of_the_
Top_10_Soft_Skills_Needed_in_Today's_Workplace/links/56095e8908ae4d86bb11d036/Executive-Perceptions
-of-the-Top-10-Soft-Skills-Needed-in-Todays-Workplace.pdf .

From the abstract:

Hard skills are the technical expertise and knowledge needed for a job. Soft skills are interpersonal qualities, also known as people skills, and personal attributes that one possesses. Business executives consider soft skills a very important attribute in job applicants. Employers want new employees to have strong soft skills, as well as hard skills. This study identified the top 10 soft skills as perceived the most important by business executives: integrity, communication, courtesy, responsibility, social skills, positive attitude, professionalism, flexibility, teamwork, and work ethic.

West, M. R. (2016). Should non-cognitive skills be included in school accountability systems? Preliminary evidence from California's CORE districts. (Evidence Speaks Reports, Volume 1, #13). Washington, DC: Center on Children and Families at Brookings. Abstract retrieved from https://eric.ed.gov/?id=ED568257; full text available at https://www.brookings.edu/wp-content/uploads/2016/07/EvidenceSpeaksWest031716.pdf .

From the executive summary:

Evidence confirms that student skills other than academic achievement and ability predict a broad range of academic and life outcomes. This evidence, along with a new federal requirement that state accountability systems include an indicator of school quality or student success not based on test scores, has sparked interest in incorporating such ‘non-cognitive’ or ‘social-emotional’ skills into school accountability systems. Yet important questions have been raised about the suitability of extant measures of non-cognitive skills, most of which rely on asking students to assess their own abilities, for accountability purposes. Key concerns include the possibility of misleading information due to reference bias in students' self-reports and that students may simply inflate their self-ratings to improve their school's standing once stakes have been attached. The most ambitious effort to deploy common measures of non-cognitive skills as part of a performance management system is unfolding in California's CORE Districts, a consortium of nine school districts that collectively serve over one million students. In the 2014–15 school year, CORE conducted a field test of measures of four social-emotional skills involving more than 450,000 students in grades 3–12. Starting this year, information from these measures will be publicly reported and is expected to play a modest role in schools' performance ratings, comprising eight percent of overall scores. Analysis of data from the CORE field test indicates that the scales used to measure student skills demonstrate strong reliability and are positively correlated with key indicators of academic performance and behavior, both across and within schools. These findings provide a broadly encouraging view of the potential for self-reports of social-emotional skills as an input into its system for evaluating school performance. However, they do not address how self-report measures of social-emotional skills would perform in a high-stakes setting—or even with the modest weight that will be attached to them within CORE. The data currently being gathered by CORE provide a unique opportunity for researchers to study this question and others related to the role of schools in developing student skills and the design of educational accountability systems

Wilson-Ahlstrom, A., Yohalem, N., DuBois, D., Ji, P., Hillaker, B., & Weikart, D. P. (2014). From soft skills to hard data: Measuring youth program outcomes. Washington, DC: Forum for Youth Investment. Abstract retrieved from https://eric.ed.gov/?id=ED557763; full text available at http://forumfyi.org/files/soft_skills_hard_data_0.pdf .

From the abstract:

Everyone who runs a youth program believes in their hearts that their program helps kids, but in their heads, they know they need convincing data to prove it. This guide—updated from 2011—is here to help them get the data they need. The guide addresses a common problem throughout the youth field: Out-of-school time (OST) programs can help youth develop skills and attributes they need to be ready for college, work, and life—skills and attributes like communications, relationships and collaboration, critical thinking and decision making, and initiative and self-direction. Few OST programs have the tools to effectively measure those outcomes. Finding the right one can be daunting. This document reviews ten youth outcome measurement tools that are appropriate for use in after-school and other settings. For each tool, it provides sample items and crucial information about usability, cost, and evidence of reliability and validity. This guide—a companion to the lsquo;Forum's Measuring Youth Program Quality’—can help providers select conceptually grounded, psychometrically sound measures appropriate for programs that serve upper-elementary—through high school-aged youth.

Additional Ask A REL Responses to Consult

Ask A REL Mid-Atlantic at Mathematica Policy Research. (2018). What classroom strategies represent best practices for helping students learn about and build soft skills (e.g. communication, collaboration, and critical thinking)? Retrieved from https://ies.ed.gov/ncee/edlabs/regions/midatlantic/askarel_39.asp

Methods

Keywords and Search Strings

The following keywords and search strings were used to search the reference databases and other sources:

• (“soft skills” OR “non-cognitive skills” OR “non-academic skills” OR “21st century skills” OR “workplace communication”) AND (measure* OR define*)
• (“soft skills” OR “non-cognitive skills” OR “non-academic skills” OR “21st century skills” OR “workplace communication”) AND (work* OR job OR employ* OR success)

Databases and Resources

We searched ERIC, a free online library of more than 1.6 million citations of education research sponsored by the Institute of Education Sciences (IES), for relevant resources. Additionally, we searched the academic database ProQuest, Google Scholar, and the commercial search engine Google.

Reference Search and Selection Criteria

In reviewing resources, Reference Desk researchers consider—among other things—these four factors:

• Date of the publication: Searches cover the most current information (i.e., within the last ten years), except in the case of nationally known seminal resources.
• Search priorities of reference sources: Search priorities include IES, nationally funded, and certain other vetted sources known for strict attention to research protocols. Applicable resources must be publicly available online and in English.
• Methodology: The following methodological priorities/considerations guide the review and selection of the references: (a) study types—randomized controlled trials, quasi experiments, surveys, descriptive data analyses, literature reviews, policy briefs, etc., generally in this order; (b) target population, samples (representativeness of the target population, sample size, volunteered or randomly selected), study duration, etc.; (c) limitations, generalizability of the findings and conclusions, etc.
• Existing knowledge base: Vetted resources (e.g., peer-reviewed research journals) are the primary focus, but the research base is occasionally slim or nonexistent. In those cases, the best resources available may include, for example, reports, white papers, guides, reviews in non-peer-reviewed journals, newspaper articles, interviews with content specialists, and organization websites.

Resources included in this document were last accessed on September 11, 2018. URLs, descriptions, and content included here were current at that time.

This memorandum is one in a series of quick-turnaround responses to specific questions posed by educational stakeholders in the Appalachian Region (Kentucky, Tennessee, Virginia, and West Virginia), which is served by the Regional Educational Laboratory Appalachia (REL AP) at SRI International. This Ask A REL response was developed by REL AP under Contract ED-IES-17-C-0004 from the U.S. Department of Education, Institute of Education Sciences, administered by SRI International. The content does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government.