ShaleNET Round 2 TAACCCT Grant Third-Party Evaluation Final Report.
Dunham, K., Hebbar, L., Khemani, D., Comeaux, A., Diaz, H., Folsom, L., & Kuang, S. (2016). Social Policy Research Associates.
-
examining353Students, gradePS
Practice Guide
Review Details
Reviewed: February 2020
- Practice Guide (findings for ShaleNET)
- Quasi-Experimental Design
- Meets WWC standards with reservations because it uses a quasi-experimental design in which the analytic intervention and comparison groups satisfy the baseline equivalence requirement.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Average quarterly earnings |
ShaleNET vs. Business as usual |
3 Months |
Full sample;
|
5929.70 |
3391.09 |
Yes |
|
|
|
| Show Supplemental Findings | |||||||||
|
Average quarterly earnings |
ShaleNET vs. Business as usual |
3 Months |
This population, a subset of the Non-Training Group, excluded any individuals who accessed services without assistance from program staff members;
|
5880.10 |
3523.25 |
Yes |
|
||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Likelihood of employment after program completion |
ShaleNET vs. Business as usual |
3 Months |
Full sample;
|
N/A |
N/A |
Yes |
|
|
|
| Show Supplemental Findings | |||||||||
|
Likelihood of employment after program completion |
ShaleNET vs. Business as usual |
3 Months |
This population, a subset of the Non-Training Group, also excluded any individuals who accessed services without assistance from program staff members;
|
N/A |
N/A |
No |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 4%
Male: 97% -
Rural
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Ohio, Pennsylvania, Texas
-
Race Asian 2% Black 14% Other or unknown 5% White 80%
Study Details
Setting
This study was set in a consortium of four technical colleges, considered hubs, in three states (Pennsylvania, Ohio, and Texas) but ultimately also served students in adjacent areas such as West Virginia and New York. The program was set up under a TAACCCT grant to develop stackable credential training programs and establish career pathways in the shale oil and gas industry. The program offerings included: industry entry level certifications (short-term, noncredit), industry certificate programs (up to one year, for-credit programs), industry associate's degree (2-year, for-credit programs), and credits that could be applied towards an industry bachelor's degree in technology management.
Study sample
The sample was mostly (96%) male, white (83%), and non-Hispanic (96%). Data were not available on Pell grant receipt, but program participants were eligible for participation in TAACCCT, and comparison group participants were receiving or had received employment-related services from other federally-funded employment programs. The full treatment sample had an average age of 35.2 years and the comparison group had an average age of 33.5 years. No other sample characteristics were available for this group. For the sub-sample of the Staff Assisted group, which represented 305 of the 353 students in the total sample: 69.2 percent of the treatment students had received a high school diploma or GED, 6.5 percent had received an Associates degree, and 12.4 percent had received their Bachelor's degree. 64.2 percent of the comparison students had received a high school diploma or GED, 6.7 percent had received an Associates degree, and 12.5 percent had received their Bachelor's degree.
Intervention Group
ShaleNET is a Community Based Job Training grant funded by funded most recently by a Round 2 Trade Adjustment Assistance and Community College and Career Training (TAACCCT) grant from the U.S. Department of Labor (USDOL). A primary activity of ShaleNET hub staff members was to develop and enhance a series of stackable credential training programs that allowed individuals to follow career pathways in the shale oil and gas industry. This iteration of the ShaleNET initiative focused primarily on developing new curricula for industry certificate programs. Intervention participants were completers in the Tier 1/Tier 2 industry entry level certifications in areas such as roustabout, floorhand, service unit operator, and welder's helper. These certifications required less than one year of training (non-credit). The program also included staff who brokered relationships and supported partnerships between the colleges and the oil and gas industry. The sample included only students who exited prior to March 31, 2015 and for whom four quarters of outcome data were available. There were 3,947 such individuals. The intervention was especially strong because of the high level of integration with local industry partners, who created many hands-on opportunities and employment after credential attainment. Another strength was the multiple possible entry and exit points - a person could enter with some credit, build on it to attain a credential and then either continue for a better/different credential or degree or leave with marketable skills.
Comparison Group
The comparison group was drawn from the population of individuals from the surrounding counties who received assistance with finding employment from federally-funded workforce programs in Pennsylvania during the same time period. The comparison sample excluded all individuals who received any training because training received through these alternative programs was almost always much longer than the 2-3-week ShaleNET noncredit programs.
Support for implementation
Each of the two sites had at least one career counselor who served as the primary liaison with ShaleNET students. Local industry partners worked closely with the colleges in developing programming. These partnerships made this an excellent example both because they contributed funding to the hubs, in-kind donations, and funding to support post-TAACCCT grant sustainability.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).