|
April Ginther, Jeanne Lee, Rui Yang
MwALT 2006
Integration of Test Scores and Instruction in a Local Testing System
The Oral English
Proficiency Program at Purdue University developed a local,
computer-based, semi-direct test of oral English proficiency that
became operational in 2001. Last year, the test was placed into a
web-based network system that links registration, administration,
and rating. The information gathered from all input into the system
is saved as a database that can be used for score reporting,
research, and instructional purposes.
A critical gap for
score users – especially for test takers and researchers -- is the
one that exists between a test score and its meaning. Local testing
systems in which a test is linked to instruction and test data are
accessible for research can be used to bridge this gap by allowing
data to be leveraged for uses beyond the provision of a score.
In the rating
component of the system, raters have access to the prompt, the
scale, and sample benchmark performances. In addition, raters not
only assign scores but also provide a justification for the scores
they assign. These justifications serve as extra interpretative
information to the score users (examinees, departments, and
researchers).
The raters are also
instructors in the program. For those examinees who are placed into
an OEPP class, instruction begins with a review of each item
response by the student with the instructor/rater. Access to the
raw performance data, the review of the item responses, scores, and
rater comments allows the students to understand the reasons/meaning
for their scores. These negotiated interpretations of actual
performance serve as the foundation for each student’s instructional
plan, midterm evaluation, and final evaluation. For many of the
students, it’s the first time that they have had the opportunity to
examine their actual performance in relation to their assigned
score.
All data generated
by the test and subsequent evaluations are accessible to
researchers. Studies generated by these data will be reviewed.
This presentation
will demonstrate the components of the system and how they are
accessed by its users. The local system has many advantages that
are not possible when large scale assessments and tests focus
primarily on placement and not on the extended and system-integrated
opportunities for test score use.
(Back
to Top)
Nancy Kauper, April Ginther
MwALT 2006
Assessing Interaction and Conversation Skills: Bridging a Gap
This
paper describes a pilot project to develop an assessment of
conversation and interaction skills for ITAs in Purdue University’s
Oral English Proficiency Program (OEPP).
Until recently, our instruction and assessment in OEPP have focused
on presentation and teaching skills, but this left a gap in terms of
other types of skills students need, namely interpersonal
interaction and conversation skills. These skills are very important
for our international graduate students, many of whom teach
undergraduate students one-on-one in labs, office hours, and help
rooms, and all of whom must meet regularly to talk with advisors and
others in their programs, not to mention other social interactions
which require the ability to carry on small talk and conversations.
For these reasons, a new classroom component was developed to help
students acquire and practice conversation skills in a more focused
way, with a formal assessment done twice, at midterm and the end of
the semester.
Formal assessment criteria are in three categories: comprehension,
management, and feedback. The format of the formal assessment is
paired student conversations. Each pair of students is digitally
recorded on a video camera having two successive conversations on
two different topics. Students do not receive the topics in advance
of the assessment, but are given a choice of topics at the time of
assessment.
This
paper will describe the assessment criteria, preparation, and
procedures; discuss the choice of a paired student format; present
information from student self-assessments as well as instructor
feedback and comments on the pilot semester of the assessment.
(Back
to Top)
Jeanne (Yu-Chen) Lee
MwALT 2006
English Intonation as a Descriptor for Evaluating Oral English
Proficiency
Intonation is
often listed as a descriptor in oral English proficiency scales,
which demonstrates that it is a part of a performance that
influences people's perception of proficiency. However, there
is no explanation or guideline of how to evaluate intonation in oral
English performance. This study proposes an acoustic approach
to reliably measure English intonation for the analysis of pitch
patters by native speakers (NS) of English and English as a second
language (ESL) speakers who speak Mandarin as their first language
(L1). Speech samples of the five NS of English from the
Midwest, USA, and five Chinese speakers on their performance in
reading aloud and leaving a telephone message are analyzed.
Intonation is measured instrumentally with PRAAT, a speech analysis
computer program, on sentence nonfinal and final positions, where
sentence units are reliably determined by syntax.
The preliminary
findings indicate that native English speakers may use different
intonation patterns for different discourse situations: the
five English NS of this study prefer to use level or rising contours
in sentence final positions for leaving voicemails, whereas only
falling contours are used to mark sentence endings for reading
aloud. The Chinese ESL speakers, on the other hand, do not
make use of intonation for different discourse functions:
there is a prominent use of the level and falling contours for both
reading aloud and leaving a telephone message. It is argued
that Chinese learners of English tend to impose the prosodic patters
of their L1 on their second language (L2) that results in the
typical Chinese accent that can be characterized with a monosyllabic
rhythm with constant high level and falling tones. The results
of this study provide potential guidelines on how intonation may be
systematically used as a descriptor for oral English proficiency.
(Back
to Top)
Rui Yang
MwALT 2006
Factor Structure of the Oral English Proficiency Test
The purpose of this preliminary
study is to examine the factor structure of the Oral English
Proficiency Test (OEPT). The OEPT is used at Purdue University
to assess the oral English proficiency of prospective international
teaching assistants (ITAs). Only ITAs that pass the test are
allowed to conduct direct classroom teaching at the university.
A Confirmatory Factor Analysis (CFA) was used to examine the
structure of the test since a one-factor model is hypothesized
underlying the test structure. The sample for the factor
structure study includes 641 subjects from 40 different countries
and regions. For the analysis of factor invariance across
test-takers from different native language backgrounds, 180 Chinese
subjects and 110 Korean subjects were chosen from the overall sample
of 641 subjects.
The result of the CFA confirms
the hypothesized one-factor model (chi-square=87.45, GFI=0.973,
CFI=0.996, RESEA=0.0523). However, errors of three items were
found to covary. Further analysis using the Modification
Indices provided in the LISREL program suggest an effect of test
prompts. Compared to the other item prompts, the item prompts
of the three items with correlated errors are less text-based.
Thus, the prompts provide minimal information that examinees could
take advantage of in forming their responses. The one-factor
model with covariance among three items was invariant across the
Chinese and Korean samples chosen for the study. The
covariance among the three items in the one-factor test structure
implies that prompt revision is necessary.
(Back
to Top)
Christopher Grant Blake, PhD,
Purdue University, August, 2006
The Potential of Text-Based Internet Chats for Improving ESL Oral
Fluency
Major Professor: April Ginther.
Text-based Internet
chats have become a popular component of second language classrooms,
making it possible for students to communicate with native speakers
and second language learners across the globe. While a number of
studies have reported on the positive affects that chat discourse
can have on the learning environment, few studies have examined
whether participation in chat discourse can help learners improve
their proficiency in a second language. To the best of knowledge, no
studies to date have examined whether second language learners can
improve their oral fluency through participating in a text-based
chat learning environment.
This dissertation
addresses the above question by examining the oral fluency
development of 34 ESL learners who participated in the same six week
course but in separate instructional environments: a text-based
Internet chat environment, a traditional face-to-face environment,
and a control environment that involved independent learning with no
student interaction. A fluency pretest was administered prior to
the study and a posttest was administered at the end. Speech
samples collected from these tests were analyzed for fluency at five
temporal variable levels: speaking rate (SR), phonation time ratio
(PTR), articulation rate (AR), mean length of run (MLR), and average
length of pauses (ALP). Improvement in fluency was measured in
terms of the pretest to posttest gain scores on each of these
measures.
The study found that the gain scores of participants in the
text-based Internet chat environment were significantly higher on
the PTR and MLR measures than the gain scores of participants in the
face-to-face and control environments. Gain scores on the three
other measures were not significant. The author discusses these
findings in relationship to Levelt’s (1989) model of language
production and argues that text-based Internet chat environments can
be a useful way of building oral fluency by facilitating the
automatization of lexical and grammatical knowledge at the
formulator level.
(Back
to Top)
Christopher Blake
AILA, July 2005
Revision of an ITA Curriculum: A Case Study
By definition, ITA
preparation courses have traditionally focused on preparing ITAs for
the tasks of teaching in an undergraduate classroom setting. Skills
such as using visual aids, answering questions, organizing lessons,
and understanding American classroom culture are areas typically
covered in an ITA classroom. Because it is generally assumed that
prospective ITAs can make only limited progress in language
proficiency over the course of a semester, compensatory strategies
such as paraphrasing student questions and using discourse markers
are typically emphasized over language specific features such as
pronunciation and grammar.
An analysis of the
Purdue ITA curriculum revealed that only 8 percent of the activities
focused on language proficiency areas while the majority related to
teaching and compensatory strategies. This paper discusses the
curriculum revision process and explains how the changes reflect the
current language needs of the ITA population that it serves. The
author also describes how moving the curriculum onto a web-based
platform with links to on-line resources has created an environment
that facilitates student autonomy and life-long learning after the
course has been completed.
(Back
to Top)
Edie Cassell
AILA, July, 2005
Toward an Ecological Approach to International Teaching Assistant
Preparation
Most large research
universities in the United States rely in great part on
international graduate students, for whom English is not their
native language, as teaching and research assistants.
As universities
must ensure that International Teaching Assistants’ (ITA’s) oral
English skills are sufficient to teach in classrooms and otherwise
interact with Americans, many require
ITA’s to pass a spoken English test and if they do not pass the
test, to undergo an ITA training program.
In order to have
realistic expectations for ITA’s and to better inform the
development of ITA screening and training programs, it is necessary
to attempt to identify ITA’s needs and this requires collection of
information about their actual patterns of language use, both within
and outside their working environment. For this study, the
perspective of an ecological approach was adopted to gain insight
into ITA’s participation in a larger social sphere. This approach
was selected as a guide in conducting this research because it
considers: (1) all domains of language use; (2) the potential of
linguistic diversity; (3) the limitations of natural and human
resources; (4) the importance of support within the community; (4)
the need for a long term view - into the past and the future.
With the
considerations of this approach as guide, a questionnaire was
developed and administered to 200 ITA’s. ITA’s responded about
their demographic make-up, their participation in extra-curricular
and community activities, the amount of time they spend engaged in
activities for which they use their first language vs. English, and
the use of their first language vs. English to communicate (oral and
written) in their work and social environments. Analysis of
responses indicate their use and need of first language maintenance
as well as English and participation in activities spanning from
home country traditions to mainstream American events.
(Back
to Top)
Jennifer Haan
AILA, July, 2005
History, Policies, and Community Values in ITA Training
Within any language
ecosystem, the historical values of institutions within the
community influence language instruction and use. In ITA training
programs, the values set forth by the policies of the institution
and the attitudes of those involved in the community often times
affect the type of program that is put into place, the curriculum
that is offered, the language that is used, and the type of learning
that takes place. By analyzing the history and policies of the
institution, one can better understand the values that the community
brings to the instruction and integration of ITAs, and can compare
these values and policies with the needs and motivations of the
students themselves.
Using archival
research from 15 years of university senate documents, community and
university publications, interviews, and other university and
departmental public documents, this study examines the history of
policies and attitudes toward ITA training at a large Midwestern
university. The researcher investigates 1) what policies are in
place for the instruction of ITAs, 2) how these policies have
evolved over time, 3) what factors have influenced the
implementation of these policies, 4) and how these policies reflect
the values and attitudes of the community. The analysis reports a
shift from punitive policies which view ITAs simply in terms of
their language deficiency to more positive policies which focus on
the globalization and internationalization of the campus.
(Back
to Top)
Nancy Kauper
AILA, July, 2005
A Lexical-based Instructional Component for Developing Oral Fluency
Studies on lexical
coverage of spoken language (Adolphs & Schmitt, 2003) and on the
role of short term memory in lexical acquisition (Ellis & Sinclair,
1996) have shown that a focus on vocabulary development and lexical
sequence acquisition for L2 learners is essential for developing
fluency, especially in oral-based language programs.
According to Ellis
and Sinclair, a focus on lexical development should include
familiarizing the learner with word families and lexical sequences,
followed by repetitive oral practice to establish the material in
short term memory, use of the lexis in meaningful contexts, and
frequent periodic recycling of content and practices to promote
long-term learning.
This paper will
describe initial attempts to apply information from these and other
studies on vocabulary development (Schmitt & Meara, 1997), lexical
sequencing, and corpus linguistics (Simpson & Mendis, 2003) in the
design of a lexically-focused instructional component for developing
the oral fluency of intermediate to advanced ESL learners who are
graduate student teaching assistants-in-training. The curriculum
includes the use of lexical notebooks (Schmitt & Schmitt, 1995), and
subsequent recycling activities designed to give students multiple
opportunities in the instructional setting to take in and use
targeted chunks of language.
A key question for
any lexically-based program is how to choose content. For the
instructional component discussed here, some content, including
words, idioms, and collocations, were chosen from word frequency
lists and the MICASE corpus. Because learners have differing
academic, professional, and personal situations and needs, in
addition to instructor-chosen material, each student added
individualized material to their lexical notebooks, including items
that are difficult for the learner to pronounce due to unfamiliar
phonological structure, items that represent unfamiliar concepts,
critical vocabulary associated with academic, professional, and
personal domains of the individual student, and unfamiliar
colloquial expressions.
(Back
to Top)
Rui Yang
AILA, July, 2005
Conjunctions in International Teaching Assistants’ Oral Discourse
Despite the high
scores international students often receive on the TOEFL or GRE,
these same students can be challenged by the oral communication
requirements associated with employment as teaching assistants (TAs)
in North American universities. Previous analyses of international
teaching assistant (ITA) discourse have shown that listeners’
interpretation of discourse is not only influenced by pronunciation
but also by discourse-level patterns of language use, e.g. the use
of discourse markers. Researchers agree that discourse markers play
an active role in discourse coherence; however, they argue that the
use and misuse of different types of discourse markers have
differential impact upon the perceived comprehensibility of the
discourse. This paper examines four types of conjunctions in order
to investigate whether and to what extent the use of conjunctions
can be argued to influence scores on a semi-direct test used for
screening the oral proficiency of prospective ITAs. Item responses
by 40 Chinese examinees across 4 levels of proficiency and 10 native
English speakers to the “Compare and Contrast” item on the test
(OEPT) were examined and coded for the use and “misuse” of
conjunctions. The total number of words produced by individual
speakers ranged from 89-306 words. The Chinese examinees were found
to use conjunctions in some unexpected ways: the overextended use of
the simple coordinating AND to signal relations that should be cued
by SO or BUT, the inconsistent use of temporal conjunctions, and the
unexpected collocation of conjunctions are characteristic of
nonnative response. Native speakers’ use of AND was often
accompanied by parallel grammatical structures – a combination of
coordination and syntactic features absent in the nonnative
speakers’ responses. The characteristics of the ITAs’ use of
conjunctions can be argued to contribute to the distribution of
scores.
(Back
to Top)
Slobodanka Dimova
MwALT,
October, 2004
Test Prep, Washback, Washing Forward, What Else?
Traditionally, the
term test preparation triggers the idea of drilling, teaching
testing strategies and tricks, and focusing on test items. In the
area of large standardized testing, test preparation has been
investigated only in terms of its effects on test scores and test
validity. In education, test preparation is viewed as washback, or
the influence of tests on teaching. Even though test preparation and
washback are believed to have negative influences on teaching and
testing, the term washing forward (Pearson 1988) was created to
account for their benefits. In this paper, test preparation is not
viewed as a score-boosting method but rather as an instructional
tool that enhances learning.
In August 2002,
Purdue University developed a set of electronic preparatory
materials for the Purdue’s Oral English Test (POET), known as the
POET Tutorial. The purpose of the POET Tutorial is to direct
prospective examinees’ attention to the components of the construct
being measured, rather than to the particular items used in the
test. Even though POET’s test format and method are designed to
elicit adequate criterion performances, the Tutorial expands,
justifies, and completes the connection between the test methods and
the construct.
The present study
was designed to analyze (1) the relationship between the use of the
POET Tutorial and the examinees’ knowledge of the testing context,
the testing method, and the consequences of the test, and (2)
examinee’s opinions about the usefulness of the set of preparatory
materials included in the POET Tutorial. Two questionnaires were
administered to 600 examinees during the academic years 2002-04.
Most examinees (92%) found the tutorial helpful. Examinees who use
the tutorial are significantly more likely to report they are
familiar with the testing context, consequences, and methods.
(Back
to Top)
Shigetake Ushigusa
MwALT,
October, 2004
The relationship between Silent Pauses and Idiom Use
Idioms are
considered to be stored and retrieved from memory rather than
preferentially composed at the time of use, on the basis of the idea
that they are “lexicalized,” “institutionalized,” and
“non-compositional.” The literature pertaining to phraseological
units, formulaic language, and fixed expressions including idioms (FEIs)
indicates that idioms are regarded as a subset of prefabs.
This study examines
the hypothesis that there is a correlation between the amount of
silent pauses and the amount of idioms used in spontaneous speech by
ESL speakers by observing the presence of pauses and use of idioms
in examinee responses to Purdue’s Oral English Test (POET). The
idioms examined are the ninety-seven idioms identified as the most
frequently used idioms on the basis of corpora research of spoken
American English. The silent pauses are silences that are longer
than .2 seconds. In this pilot, twenty-four examinees’ responses to
the test item, Compare and Contrast of POET were analyzed to address
the following research question: What is the correlation between
the total silent pause time as percent of total time of delivery and
the total number of the idioms as percent of total number of words
in the delivery?
(Back
to Top)
Slobodanka Dimova
ECOLT,
March, 2003
POET Rater-Training Program
Performance-based
tests for oral language proficiency are recognized as important for
measuring certain aspects of language ability. However, the
establishment of valid and reliable rating procedures poses
challenges for practitioners and researchers. The purpose of this
paper is to analyze the rater- training procedure for the Purdue’s
Oral English Test (POET), an oral English screening method for
international teaching assistants at Purdue University. Seven
raters-in-training used the POET ten-hour self rater-training
program. During the program, the raters-in-training were introduced
to the five-level holistic scale. Besides the traditional band
descriptors, the scale contained performance descriptors based on
experienced raters’ perceptions. The score bands were introduced one
at a time (starting from the highest), and each band was closely
compared to the characteristics of the higher or lower band. The
raters-in-training were encouraged to focus on different aspects of
the speaking performances: organization and coherence, articulation,
pronunciation, grammar and syntax, delivery, and vocabulary range.
During practice rating, the raters-in-training were requested to
assign and justify their scores using perceptive description of
examinees’ performances. Percentages of agreement of the
raters’-in-training scores were calculated against the original
scores. The raters’-in-training perceptive descriptors were analyzed
in terms of frequency of variable perception, frequency of using
performance characteristics presented in the scale descriptors, and
frequency and type of perceptive descriptions not given in the
scale. Results suggest that the raters-in-training had high
percentages of score agreement even though they used different
characteristics in their performance descriptions. The
rater-training procedure analyzed in this paper not only informs
raters-in-training about the rating scale, it also gives an
opportunity for trainers to follow raters’ conceptualization of the
construct, their scale interpretation, and their performance
variable perception from the very beginning. Well-trained raters are
an essential component of performance-based testing because they are
one of the main factors that influence examinees’ scores. If raters
are trained to use the scale effectively, score variance due to
rating error may be minimized.
(Back
to Top)
| 
