Assessment of Departmental Study Abroad Programs

Purdue West Lafayette is transitioning campus-wide from previously required intercultural learning action plans and reports to a new assessment initiative for faculty- and staff-led study abroad. Please read on for an invitation to the pilot phase of the newly designed process.

The Invitation

There is a wide array of potential student developmental outcomes from faculty- and staff-led study abroad (FLSA), including disciplinary, destination-specific, and generalizable intercultural/global learning outcomes. The breadth and depth of these outcomes, in addition to the intensive nature of the experience, are why study abroad is considered a “high impact” practice in higher education. Recent scholarship demonstrates, however, that study abroad is not automatically or necessarily a high impact practice. 

The outcomes

FLSA needs to be designed intentionally to foster these three outcome areas:

1. Replicating coursework abroad exactly as it is implemented on campus likely will only accomplish disciplinary outcomes. 
2. Focusing on exploration of the local context might support achievement of destination-specific objectives, but scholarship has noted that some ways of interacting with the local environment seem more impactful than others. 
3. Research also suggests that mentor-guided reflection on experiences of cultural difference is one of the most effective ways for students to develop intercultural competence in the study abroad context.

Support for faculty by Global Partnerships and Programs (GPP)

Purdue's GPP wants to support FLSA program leaders in thoughtfully building towards desired outcomes in these three areas by partnering to gather and reflect on data that can inform curriculum development. In order to offer high quality programs, educators need evidence of what students are taking away from their study abroad experiences, and how those outcomes are related to who students are and what program design decisions faculty make. Faculty time is limited, and assessment processes can be logistically challenging. Purdue can institutionalize data collection for student outcomes in FLSA in such a way that:

• Program leaders are not burdened with clunky processes of administering pre- and post-tests to their program participants.
• Faculty and staff have access to the analysis of their program level data so that in the process of proposing the next iteration of their program they can consider this valuable information in their program design. This would not add to the current workload of completing an intercultural learning plan during program proposal and submitting a reflective report of data afterwards; it would instead replace those existing processes. 
• Program leaders and students benefit from formative assessment results—faculty, with group level reports to understand who their learners are and what perspectives they bring to program experiences; and students, with individual level reports to motivate developmental work and identify personalized learning goals.
• Purdue collects a large dataset at the institutional level for big data analysis exploring the connections between program design and student outcomes. We can answer questions about which program elements (for example, the opportunities students have to dialogue with local peers or the extent to which they encounter difference or the ways in which reflection is embedded in the experience) are most impactful for outcomes such as critical thinking, tolerance for ambiguity, openness to cultural diversity, communication skills, adaptability, self-awareness, empathy, and more. 

The Process

This plan pairs two instruments—one for faculty/staff and one for students—that are theoretically aligned and have been deployed together successfully in the past at other institutions.

Faculty/Staff:

During the program proposal process, program leaders will complete the Global Engaged Learning (GEL) index, an instrument developed by CoreCollaborative International in a project funded by the Colonial Academic Alliance to measure “high impact practices” in global learning experiences. The survey asks faculty/staff to reflect and report on design elements of their programs that have been demonstrated in research to engage students and enhance their learning outcomes in all three areas (disciplinary, location-specific, and intercultural/global learning). One benefit is that the GEL is broader than the current intercultural learning plan process that focuses only on the third area. Another is that it streamlines the process for faculty by eliminating data reporting after they return from abroad, instead asking faculty renewing a program to reflect on data from its previous iteration during their planning process for the next offering. Finally, program leaders can partner with CILMAR if they would like to submit an exempt IRB proposal and publish on study abroad programs using existing institutional data.

Students:

CILMAR will partner with the Study Abroad office to administer the Beliefs, Events, and Values Inventory (BEVI) as a pre- and post-test, relieving faculty of this responsibility (although of course program leaders may also still collect and analyze any additional data, intercultural or otherwise, that will benefit their students, their program, or their own research). The BEVI is a well-researched whole-person transformative learning measure with strong demonstrated validity and reliability that offers many advantages, including:

1) formative assessment potential via automated individual narrative reports returned directly to students and aggregate reports delivered to faculty before departure,
2) cost effectiveness via institutional licensing, and
3) a wide array of constructs measured so to meet needs in diverse disciplines.

• The BEVI has also already been paired successfully with the GEL index, suggesting that triangulating GEL and BEVI data will offer us insights into the relationship between design elements and outcomes. From CILMAR, faculty will receive anonymized program-level pre-test BEVI data prior to their upcoming programs, as well as post-test data before they submit a proposal to renew a program. College administrators will receive aggregated college-level data, and CILMAR and IDA+A will also analyze aggregate data at the institutional level. Note: Because not every student may feel comfortable taking the BEVI, at the individual level any student may opt to take an alternate qualitative assessment instead of the BEVI before and after their study abroad program.

Timeline

Assessment initiative timeline

	Who is involved	What they do	Student Data
Pilot phase—Spring, Summer, Fall, Winter 2024	A. Exploratory SAIL grant awardees	Meet with CILMAR, Study Abroad, and/or college staff for support with GEL and strategic program planning	No
	B. Voluntarily participtating program leaders	Fill out the GEL before departure. Support meetings available but not required.	Yes, unless another instrument is already planned
Moving forward—Spring 2025 and beyond	A. All leaders of new program proposals	Meet with CILMAR, Study Abroad, and/or college staff for support with GEL and strategic program planning early in proposal process.	Yes
	B. All leaders of existing program proposals	Consider available data from previous program. Fill out GEL during proposal process. Support meetings available but not required.	Yes

 

                                                                                                                           Updated March 11, 2024