Evaluation in Practice

Northern Kentucky University (NKU) and Ferris State University (FSU) joined forces to increase recruitment, retention, and graduation of students in STEM fields (science, technology, engineering, and mathematics). NKU wanted to build on past success in recruiting academically talented STEM students with demonstrated financial need, and FSU wanted to increase the number of female and minority students who obtain baccalaureate degrees in STEM.

UCESC joined the project at the proposal stage to help the applicants design an evaluation plan for an S-STEM grant from the National Science Foundation. The evaluation reflected the complexity of the project involving two universities and multiple interventions, including specialized recruiting, orientation, living communities, support services, and mentorship. The universities also wanted to determine whether adding a job-shadowing opportunity for first year STEM students helped improve retention rates between their first and sophomore years.

UCESC takes a collaborative approach to evaluation and worked closely with the project team to develop the proposed evaluation plan, which was based on a cohort-sequential design with a retrospective cohort and a concurrent comparison group. The plan included evaluation questions, appropriate measures, evaluation activities, data collection methods, and a data analysis plan.

Using a mixed methods approach, the proposed evaluation followed four cohorts of first year students exposed to varying levels of program components through the grant period using institutional data (grades, major, and enrollment data) as well as quantitative and qualitative data. Each cohort included three groups of students (all receiving scholarships): two groups were assigned to receive a shadowing experience either in fall or spring and a third group did not experience shadowing (comparison group).

To track progress over time, longitudinal analyses and cross-cohort comparisons were conducted, yielding understanding of the relationship between program components and student retention with a special emphasis on how varying research or shadowing experiences were related to retention.

By comparing processes and outcomes between two universities, the evaluation was designed to provide insights into factors that enhance and hinder program transferability and scalability. These data will inform future efforts to recruit, retain, and graduate low-income, academically talented students in STEM majors and careers.

Quantitative data collection and analyses documented student enrollment and retention, students’ experiences with research and shadowing, and students’ career interests and perceived school connectedness. Appropriate descriptive and inferential statistics (paired t-tests, repeated measures ANOVA) were used to analyze quantitative data.

Qualitative data collection and analyses documented project implementation at the two institutions and helped explain successes, challenges, adaptations, and opportunities. Thematic analyses were conducted as well as a triangulated analysis of quantitative and qualitative data to examine the extent to which project objectives were met.

The proposal was successfully funded by the National Science Foundation and started implementation in the 2019-2020 academic year and will continue through 2025. Through the 6-year project, UCESC conducted multiple evaluation activities to attain baseline data as well as to provide formative and summative results to foster continuous project improvement. Activities included conducting focus groups and administering surveys for multiple groups (e.g., students, administration, project team members, and industry partners), analyzing Institutional Research data from both universities, and reviewing documentation and processes. Results from the evaluation have informed project activities and improvements to retain STEM students.