Do Live versus Audio-Recorded Narrative Stimuli Influence Young Children's Narrative Comprehension and Retell Quality?
Kim, Young-Suk Grace (2016). Grantee Submission. Retrieved from: https://eric.ed.gov/?id=ED575732
-
examining193Students, gradesK-4
Single Study Review
Review Details
Reviewed: January 2023
- Single Study Review (findings for Live narrative stimuli - Kim (2016))
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Test of Narrative Language (TNL) Comprehension: Age-standardized |
Live narrative stimuli - Kim (2016) vs. Audio recorded stimuli |
0 Days |
Full sample (Grades K, 2, and 4);
|
9.17 |
8.32 |
No |
-- | ||
|
Researcher-developed Narrative Retell Quality measure |
Live narrative stimuli - Kim (2016) vs. Audio recorded stimuli |
0 Days |
Full sample (Grades K, 2, and 4);
|
31.30 |
29.77 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Test of Narrative Language (TNL) Comprehension: Age-standardized |
Live narrative stimuli - Kim (2016) vs. Audio recorded stimuli |
0 Days |
Grade K;
|
9.30 |
7.93 |
No |
-- | ||
|
Test of Narrative Language (TNL) Comprehension |
Live narrative stimuli - Kim (2016) vs. Audio recorded stimuli |
0 Days |
Grade K;
|
21.07 |
17.78 |
No |
-- | ||
|
Test of Narrative Language (TNL) Comprehension |
Live narrative stimuli - Kim (2016) vs. Audio recorded stimuli |
0 Days |
Grade 2;
|
26.72 |
24.21 |
No |
-- | ||
|
Researcher-developed Narrative Retell Quality measure |
Live narrative stimuli - Kim (2016) vs. Audio recorded stimuli |
0 Days |
Grade K;
|
20.67 |
17.18 |
No |
-- | ||
|
Test of Narrative Language (TNL) Comprehension: Age-standardized |
Live narrative stimuli - Kim (2016) vs. Audio recorded stimuli |
0 Days |
Grade 2;
|
8.53 |
7.55 |
No |
-- | ||
|
Test of Narrative Language (TNL) Comprehension |
Live narrative stimuli - Kim (2016) vs. Audio recorded stimuli |
0 Days |
Full sample (Grades K, 2, and 4);
|
26.41 |
24.45 |
No |
-- | ||
|
Researcher-developed Narrative Retell Quality measure |
Live narrative stimuli - Kim (2016) vs. Audio recorded stimuli |
0 Days |
Grade 2;
|
32.08 |
30.18 |
No |
-- | ||
|
Test of Narrative Language (TNL) Comprehension |
Live narrative stimuli - Kim (2016) vs. Audio recorded stimuli |
0 Days |
Grade 4;
|
30.56 |
30.18 |
No |
-- | ||
|
Test of Narrative Language (TNL) Comprehension: Age-standardized |
Live narrative stimuli - Kim (2016) vs. Audio recorded stimuli |
0 Days |
Grade 4;
|
9.78 |
9.52 |
No |
-- | ||
|
Researcher-developed Narrative Retell Quality measure |
Live narrative stimuli - Kim (2016) vs. Audio recorded stimuli |
0 Days |
Grade 4;
|
39.38 |
39.61 |
No |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
1% English language learners -
Female: 55%
Male: 45% -
Town
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Florida
-
Race Black 54% Other or unknown 8% White 38% -
Ethnicity Other or unknown 100% -
Eligible for Free and Reduced Price Lunch Free or reduced price lunch (FRPL) 57% No FRPL 43%
Study Details
Setting
This study took place in two elementary schools, one public and one private, in a mid-size city in Florida.
Study sample
A total of 193 students in kindergarten and grades 2 and 4 from two elementary schools were included in the study. The majority of students were African American (54%), 38% were White, and 8% were another race. Approximately 45% of the students were male and 57% were eligible for free or reduced-price lunch. Two students (or 1%) were English learners.
Intervention Group
The live narrative stimuli intervention is a practice in which a trained interventionist reads stories aloud to a student during English language and literacy testing. Students in the live narrative group completed Tasks 1, 3, and 5 of the Test of Narrative Language (TNL) with a live narrative stimuli. A trained graduate student read aloud each component of the TNL to each student in an individual setting. In Task 1, the student listened to a story read by the graduate student without visual stimuli. In Task 3, the student was presented with five pictures along with a story. In Task 5, a single picture was presented to the student along with a story. Immediately after listening to each story, the student was asked to retell the story and answer a series of comprehension questions. Each student participated in one testing session. The study did not indicate how long each testing session lasted.
Comparison Group
Students in the comparison group completed Tasks 1, 3, and 5 of the TNL using audio-recorded narrative stimuli. Students listened to each story pre-recorded by a female in a natural storytelling voice via a CD player. For “ecological validity,” students did not use headphones and the volume was adjusted to each student's comfort level. Assessment procedures and locations were the same as the intervention group, except for the delivery method of the stories (live versus audio-recoded).
Support for implementation
Trained graduate students carried out the administration and scoring of the assessments. The same assessors (graduate students) administered all the tasks to students in both stimuli conditions in each grade. The assessors were trained to read stories in standard way according to a manual associated with the TNL. An assessor with extensive experience working with children and the TNL assessments observed each mock assessment. Research assistants assessed the audio recordings of stories used in the comparison condition for clarity and evaluated whether the story was read in a natural storytelling voice appropriate to the geographical region. Other graduate students, unaware of the study purpose and condition, were trained to code transcripts and score students' comprehension and retell responses using a scoring rubric. Assessors had to meet a minimum of 95% fidelity prior to implementing the procedures used for the study.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).