Evaluation in Practice

Northern Kentucky University (NKU) and Ferris State University (FSU) have joined forces to increase recruitment, retention, and graduation of students in STEM fields (science, technology, engineering, and mathematics). NKU wants to build on past success in recruiting academically talented STEM students with demonstrated financial need, and FSU wants to increase the number of female and minority students who obtain baccalaureate degrees in STEM.

UCESC joined the project at the proposal stage to help the applicants design an evaluation plan for an S-STEM grant from the National Science Foundation. The project is complex, involving two universities and multiple interventions, including specialized recruiting, orientation, living communities, support services, and mentorship. The universities also want to determine whether adding the opportunity for freshman STEM students to job shadow with STEM professionals will help improve retention rates between their freshman and sophomore years.

UCESC takes a collaborative approach to evaluation and worked closely with the project team to develop the proposed evaluation plan, which was based on a cohort-sequential design with a retrospective cohort and a concurrent comparison group. The plan includes evaluation questions, appropriate measures, evaluation activities, data collection methods, and a data analysis plan.

Using a mixed methods approach, the proposed evaluation follows four cohorts of freshman students exposed to varying levels of program components through the grant period using institutional data (grades, major, and enrollment data) as well as additional quantitative and qualitative data. Each cohort includes three groups of students (all receiving scholarships): two groups are assigned to receive a shadowing experience either in fall or spring and a third group does not experience shadowing (comparison group).

To track progress over time, longitudinal analyses and cross-cohort comparisons will be conducted, yielding understanding of the relationship between program components and student retention with a special emphasis on how varying research or shadowing experiences are related to retention.

By comparing processes and outcomes between two universities, the evaluation is designed to provide insights into factors that enhance and hinder program transferability and scalability. These data will inform future efforts to recruit, retain, and graduate low-income, academically talented students in STEM majors and careers.

Quantitative data collection and analyses will document student enrollment and retention, students’ experiences with research and shadowing, and students’ career interests and perceived school connectedness. Appropriate descriptive and inferential statistics (paired t-tests, repeated measures ANOVA) are used to analyze quantitative data.

Qualitative data collection and analyses will document project implementation at the two institutions and help explain successes, challenges, adaptations, and opportunities. Thematic analyses are conducted as well as a triangulated analysis of quantitative and qualitative data to examine the extent to which project objectives are met.

The proposal was successfully funded by the National Science Foundation and started implementation in the 2019-2020 academic year.

In the first year of the project, UCESC conducted multiple evaluation activities to attain baseline data as well as to provide formative results to foster continuous project improvement. Activities included conducting focus groups with and administering surveys to stakeholders (e.g. students, administration, project team members and industry partners), analyzing Institutional Research data from both universities, and reviewing documentation and processes. Preliminary results from the first year of the evaluation have informed project activities for the second year, and the project team is working collaboratively to navigate new challenges from the impact of the COVID-19 pandemic.