CHALLENGES
TO USE AN INTERDISCIPLINARY TASK IN LOGISTICS AND ENGLISH TEACHING: AN ANALYSIS
OF A PERFORMANCE TEST
Eliana
Kobayashi
Federal
Institute of Education, Science and Technology of São Paulo, Brazil
E-mail: likobayashi@ifsp.edu.br
David
Eugênio Angelo Lima
Federal
Institute of Education, Science and Technology of São Paulo, Brazil
E-mail: professordavid.associados@hotmail.com
Adriano
Maniçoba Silva
Federal
Institute of Education, Science and Technology of São Paulo, Brazil
E-mail: adrianoms@ifsp.edu.br
Submission: 1/27/2020
Accept: 2/22/2020
ABSTRACT
In language performance assessment it is
assumed that the best way to assess one’s proficiency is to make her/him show such proficiency in a direct way in a situation
close to the reality. Such view contrasts with knowledge testing which assesses
language in an indirect way. This research objective is to discuss the
development and application of an interdisciplinary task connecting two school
subjects – English Language and Logistics – in a performance language test,
investigating the correlation between these two subjects’ scores. This study
uses qualitative method in order to build the test task and criteria and
quantitative one to analyze the test scores. The research was developed in a
public educational institution located in São Paulo metropolitan region and the
results show that there is no correlation of the mean score between English
language and Logistics subjects and students face more difficulties in using
English than approaching the task topic content.
Keywords: English language teaching; Logistics; language performance assessment
1.
INTRODUCTION
The English hegemony, which is present in 85% of
international companies as one of the most used languages, and is
also the main language in science and technology (GRADDOL, 2000), contributes
to the argument that educational institutions roles should cover the
development of the language skills of its students. In addition, companies that
seek professionals in the market use specific assessment tools to analyze the
English proficiency level of their applicants in recruitment processes
(KOBAYASHI, 2010).
Specifically in Management and Business area,
Logistics Technology is the second course more offered, representing a total of
more than 22% of the courses in the area, including baccalaureate ones. Due to
its applicability in international trade areas, Logistics professionals are
supposed to know how to communicate in English in order to meet the market
demands. Thus the language should be effectively taught, tested and assessed.
Language testing is used to many purposes and among
them to identify how far students have achieved the objectives of a course and
also to diagnose their strengths and weakness (HUGHES, 1989). Having said it,
teachers play an important role as they are usually the ones who build the
achievement tests. Building a fair and valid test can be seen as a challenge
especially considering the specificities of a language uses. However, one of
the most relevant aspects when building a test should be its similarity to the
authentic language use.
One of
the ways to approach such use when building achievement tests is through
interdisciplinary tasks which allow more integration between English language
and other subjects. One
of the most important point is to create situations in which students can show
their language skills in situations close to real life.
This research was developed in a
class of a technical degree course in Business integrated to high school
curriculum at a public educational institution located in São Paulo
metropolitan area. Such course leads to two diplomas: a high school one and a
technical degree course in Business diploma. The objective was to build and
apply an English language achievement test based on a Logistics subject task
and investigate the correlation between these two
subjects’ scores. Although it is
known that teaching and testing are two related activities and they have to be
planned and developed together, this article focusses on describing the
latter.
The students must take around eighteen subjects every
semester in the three year technical course in Business integrated to high
school curriculum. Among them there are English language (mandatory subject in
the three years of the course) and Logistics (mandatory subject in the second
year of the course).
In English subject, students are expected to develop among
communicative skills and strategies to express abilities, intentions,
predictions, conditions and hypothetical situations, among several other
topics. Logistics content is taught in Logistics and covers distribution
channels, supply chain management, supplier management, packaging management. Concerning specifically Logistics field, by the
end of the course, students are expected to be able to understand how the
following areas work: retail and wholesale; integrated management between
marketing and supply chain and physical and visual packaging.
The group who participated in this investigation was
composed of 38 students aged from 15 to 16 years old and were attending the
second year of the course. English language and Logistics subjects were each taught
weekly in a 100 minute-lesson.
2. LITERATURE REVIEW
Testing
is an activity which is required in several contexts and not only in academic
settings. The main purpose is to check if the applicants are able to perform
according to some established criteria and as a consequence the test results
may impact their lives in different ways. Testing is a process which always
involves consequences (STOBART, 2003).
In
language testing there four types of tests, classified according to their
purposes: proficiency tests, achievement tests, diagnostic tests and placement
tests. As this research investigates how to build and score a test according to
what is taught in two course subjects, the suitable test is the achievement
one.
The
definition of achievement tests, for HUGHES (1989) are those “directly related
to language courses, their purpose being to establish how successful individual
students, groups of students, or the courses themselves have been in achieving
objectives”. Hughes also emphasized that tests should be based on objectives as
it can provide accurate information about individual and group achievement.
The
main purpose of studying a language is to become able to communicate using it.
Thus English in academic settings should be taught and tested as a specific
purpose language. The connection between English language and Logistics content
is a key aspect in this research and for this reason English language subject
teacher researcher and Logistics researcher teachers worked in this
collaborative study. Douglas (2000) emphasizes such relation in the definition
of specific purpose language test.
A specific purpose language
test is one in which test content and methods are derived from an analysis of a
specific purpose target language use situation, so that test tasks and content
are authentically representative of tasks in the target situation, allowing for
an interaction between the test taker’s language ability and specific purpose
content knowledge, on the one hand, and the test tasks on the other. Such a
test allows us to make inferences about a test taker’s capacity to use language
in the specific purpose domain (DOUGLAS, 2000, p. 19).
The
relation between target language and content was also discussed by Bachman and
Palmer (1996) who presented three possibilities for what they called the
relationship between language ability and topical knowledge. In the first one,
the inferences should be only about language knowledge when test takers vary
widely in topical knowledge. On the other hand, in situations where such
variety is considered minimum, language and topical knowledges should be
tested. The last possibility would be a separation between language construct
and topical knowledge construct in cases where test developers and scores users
are uncertain about the strength of the test taker’s topical knowledge.
The importance of topical knowledge
has been emphasized for a long time and by other authors like Jones (1979, p. 51)
claimed: “it is impossible for a language test to predict task-oriented
proficiency unless it includes or approximates actual samples of the tasks”.
Having said it, this work relies on the alignment between English language
subject and Logistics (Logistics) subject in an interdisciplinary testing task.
Besides
purpose, tests differ in terms of methods. McNamara (2000) distinguishes
paper-and-pencil language tests from performance tests. The former is related
to more traditional and familiar examination paper which usually separates
components of language knowledge like grammar, vocabulary or skills like
reading, listening etc. The author also points out that this kind of test
usually has a fixed response format whose most important type is the multiple
choice one.
On
the other hand, performance test aims to assess language skills in an act of
communication in a situation close to real-life context as described by
McNamara (2000) “(…) most commonly tests of speaking and writing, in which a
more or less extended sample of speech or writing is elicited from the test
taker and judged by one or more trained raters using an agreed rating
procedure”.
According
to McNamara (2000), testing is about making inferences since a test provides
indicators of how a person would probably perform in real life. For this
reason, the author claims that “understanding testing involves recognizing a
distinction between the criterion (relevant communicative behavior in the
target situation” and the test.
Table 1: Test and criterion
|
Test |
|
Criterion |
|
A performance or series of performances,
simulating representing or sampled from the criterion. |
|
A series of performances subsequent to the
test; the target |
|
(observed) |
|
(unobservable) |
Source: McNamara (2000, p.9)
The
figure shows that the criterion consists of the target performance expected to
happen in real life situations and which the applicants are supposed to be able
to perform. For this reason, it is not possible to be observed in the test. On
the other hand, the test is a performance that simulates the ones specified by
the criterion.
McNamara
(2000, p.9) claims that “although this criterion behavior, as relevant to the
appropriate communicative role (as nurse, for example, or student), is the real
object of interest, it cannot be accounted for as such by the test”. Thus, the
criterion behavior is elusive because it cannot be actually observed. The
author points out testers must check the empirical evidence for their position
when assessing candidates, thus it is necessary to analyze candidates in the
light of their actual performance on test tasks.
For
these reasons, in this study we aim to balance the topical knowledge and the
English language knowledge according to the objectives of the subjects (English
and Logistics) building a test which reflects the proficiency level and content
expected from the participants.
Interdisciplinary
testing in this study involves collaboration to analyze students’ performance
so test content and method have to be set in the test specifications which work
as a blueprint for its construction.
In
terms of oral testing task, this research follows the suggestions presented by
Hughes (1989) who makes use of the framework introduced by the Royal Society of
Arts (RSA). Their oral test specifications cover: Operations, Types of texts,
Addressees, and Topics.
Table 2: Oral test specifications
|
Operations |
The
operation is to take part in oral interaction which may involve the following
language functions: |
|
Expressing: |
Thanks,
requirements, opinions, comment, attitude, confirmation apology, want/need, information, complaints, reasons/justifications. |
|
Narrating: |
Sequence
of events |
|
Eliciting: |
Information,
directions, service, clarification, help, permission (all areas above) |
|
Directing: |
Ordering,
instructions (how to), persuading, advising, warning |
|
Reporting: |
Description,
comment, decisions |
|
Text types |
Dialogue
and multi-participant interactions normally of face-to-face nature but
telephone conversations not excluded. |
|
Addressees and topics |
Not
specified except as under “Topics for writing”. |
Source: Hughes (1989, p. 101)
On the other hand, it is also
necessary to combine such content elements with the expected proficiency level.
There are frameworks and guidelines worldwide used which describe what
candidate should be able to do in different levels such as ACTFL Guidelines and
the Common European Framework for Language. In this research, the latter is
used in order to set the proficiency ratings. Such level is also specified in
the course program as an objective to be reached along the year.
Specifically
concerning the oral skill, the CEF A2 level establishes the expected “can do”
for speaking divided in spoken interaction and spoken production.
Table 3: Spoken interaction and spoken
production
|
Spoken
interaction |
I can communicate in simple and routine tasks
requiring a simple and direct exchange of information on familiar topics and
activities. I can handle very short social exchanges, even though I can’t
usually understand enough to keep the conversation going myself. |
|
Spoken
production |
I can use a series of phrases and sentences
to describe in simple terms my family and other people living conditions, my
educational background and my present or most recent job. |
Source: Common European
Framework (2000, p.26)
These
research participants are expected to be moving up from this level to B1. On
the other hand, as the task purposed in this research focusses on a specific
topical knowledge reflecting a particular domain, CEF level description would
not match the target level. Thus, although many aspects were kept, others have
to be included due to the target language specificities.
3.
METHODOLOGY
This
research is a qualitative work as it aims to develop an investigation in depth
to better understand how a phenomena works in the context. Shank (2002, p.5)
defines this kind of research as “a form of systematic empirical inquiry into
meaning”.
Yin
(1984, p.23) claims it is “an empirical enquiry that investigates a
contemporary phenomenon within its real-life context; when the boundaries
between phenomenon and context are not clearly evident; and in which multiple
sources of evidence are used”.
In
this type of research, a very used technique is the triangulation since there
are usually multiple sources of data (CRESWELL, 1998). According to Denzin
(1978), there are four types of triangulation techniques: sources, methods,
researchers and theories. The most used ones pointed by the authors were also
applied in this investigation the sources and methods. The objective of using
different sources of data and methods is to decrease the bias of one technique
to compensate with another (LINCOLN; GUBA, 1985).
3.1.
Setting and participants
This
study was developed in a public educational institution located in São Paulo
metropolitan area. In order to study there, students must go through a
selection process administered by the institution. There are usually more
applicants than available places which makes entrance competitive.
The
participants of this research, a total of 40 students, attend a technical
degree course in business integrated with high school curriculum. The subjects
including English language and Logistics subjects are taught and tested
separately. In this investigation, these subjects’ teachers, planned the
classes, the tests and applied the oral test together but the lessons were
delivered separately.
3.2.
The test task
The
task built for the test has two main objectives: analyze the expected English
level (CEF A2) and the topical knowledge in Logistics. The main skill to be
assessed is the speaking in a group (four students) presentation. Participants
are scored individually based on the criteria shared with the students by both
researchers.
Table 4: Spoken production description
|
Spoken Production |
I
can describe the company I work for: its mission and objectives. I
can describe the service or products offered by the company I work for. I
can communicate the actions that could be taken in a Logistics area to
improve the service or products of the company I work for. |
3.3.
Task instructions
Group
activity
You
are the managers of your company and have to work together to raise the
productivity to 15%, focusing on distribution channel, retail and wholesale
activities.
Prepare
an oral presentation to the board of directors explaining your proposal. The
use of slides and visual communication such as banners, catalogues etc can help your work. Each of you has to deliver part of
the presentation.
Step
1: Describe your company
Step
2: Select the problems to be solved
Step
3: Focus on the distribution channel, retail and wholesale
Step
4: Raise the possibilities and expected results (hypothesis)
The chart below can help you develop your
ideas.
|
Action Plan |
Current scenario |
What woud happen IF... |
Predicted results |
|
Distribution channel |
Suppliers delay |
Changed the raw material
suppliers |
Improve delivery and
finished product quality |
3.4.
The test criteria
The
performance assessment task designed for this study was used as an achievement
test in order to collect evidence to analyze learning progress for both
subjects. The assessment criteria were also built after the task design.
Table 5: Assessment criteria – Logistics
subject and English Language subjects Logistics (topical knowledge) assessment
criteria
|
Criteria |
Description |
Scale |
|
Distribution channel |
Describe
supply chain and marketing tool used |
0-20 |
|
Retail |
Describe
sale, primary and secondary packaging |
0-30 |
|
Wholesale |
Describe
batches, transport packaging |
0-30 |
|
Service level |
Describe
customers’ satisfaction and how it will be measured. |
0-20 |
English Language criteria
|
Criteria |
Description |
Scale |
|
Task
achievement |
Accomplish
what was proposed in the task |
0-25 |
|
Content |
Cover
the specified aspects |
0-25 |
|
Grammar
and vocabulary |
Use
forms and specific lexis to the proposed business situation |
0-25 |
|
Pronunciation
|
Pronounce
words intelligibly |
0-25 |
4.
RESULTS
In
Logistics subject, the participants of this study have worked on creating a
fictional company covering aspects like: name, product, mission, objectives,
target, market and so on. Students had also learned topics related to
Logistics: distribution channel, supply chain, retail, wholesale and how to
measure the company service satisfaction.
Concerning
English language, students had already learned how to describe current
situations and past events, and express hypothetical situation and advice.
Thus, the task designed in this interdisciplinary work aims to join what
participants have studied allowing them to connect the acquired knowledge and
skills which have been developed. For this reason, the task instructions were
set specifying such connection and students are expected to work in the same
group that developed a company in Logistics subject.
The
Logistics subject teacher assessed the topical knowledge based on the students’
delivery and also on a handout summarizing the presentation in Portuguese. An oral presentation even with visual
resources but only in English could hinder communication for those who are not
used to assessing topical knowledge in situations where language errors, slips
and cohesion usually happen.
First
of all, to better understand how students performed in each subject (English
language and Logistics) the mean scores were analyzed separately. In addition,
the paired t-test was used to check the correlation between the two scores
samples.
The
histogram of English shows that most of the students got scores between 6 and
6.5 and the highest one was 8.5. However all students were approved since the
passing score is 6.
Chart 1:
Histogram of English
The
histograms of English scores (Chart 1) and Logistics ones (Chart 2) show that scores
had different distributions according to the observed frequencies. Table 6
supports such difference showing the data descriptive measures which indicate
differences between parameters like mean, median, standard deviation, minimum,
maximum and quartiles. It is possible to notice that Logistics score was higher
than the English score.
On
the other hand, in Logistics subject the results were virtually the opposite as
most of the students achieved the highest scores, between 9 and 10. However
there were also students who were not approved, with scores below 6.
Chart 2: Historigram of Logistics
Thus
as the descriptive statistics below show the scores given by the English
teacher are lower than the Logistics one. As a consequence there’s a weak negative
correlation of -0.2917 between the two variables with significance level of
10%. (Table 6, Chart 3 and Table 7)
Table 6: Descriptive statistics
|
|
English |
Logistics |
|
Mean |
67.76 |
82.10 |
|
Standard deviation |
7.23 |
17.88 |
|
Median |
70 |
90 |
|
Maximum |
85 |
100 |
|
Minimum |
60 |
50 |
|
1st quartile |
60 |
60 |
|
3rd quartile |
70 |
95 |
The
correlation analysis between English score and Logistics score (Chart 3) show a
weak negative correlation between the variables.
Chart 3: Correlation between
English and Logistics scores
In order to check the difference
between the scores, the Pearson test correlation was used and the results
strengthened the hypothesis of difference.
Table 7: Pearson's product-moment correlation
|
t
test |
-1.83 |
|
Degrees
of freedom |
36 |
|
p-value |
0.0754 |
|
Correlation
coefficient |
-0.2917 |
Therefore, comparing the two
samples, the hypothesis that the two means are equal is rejected.
Table 8: Results of paired t-test
|
Value
of t |
4.17 |
|
p-value |
<0.01 |
|
Degrees
of freedom |
37 |
5.
DISCUSSION
The
achievement test task applied in this investigation was built aligning English
language and Logistics subject in an interdisciplinary attempt to assess
language as it is expected to be performed in a real life situation. The
criteria were described by both teachers in order to assure what students had
to present in the designed performance test (MCNAMARA, 2000) as well as the
score scales.
This
study emphasizes the importance of performance tests to increase the students’
ability to use the language and more specifically the speaking skill which is
usually neglected in Brazilian regular schools. In addition, performance tests
are especially relevant to technical course students who usually aim to find a
job soon after getting the diploma as they can better prepare students for professional
situations like job selections tests which involve several other specific
knowledge apart from English.
The
construction and application of this test task allowed students to better
connect two subjects that may be taught separately but can have much in common
as many other subjects. English for Specific Purposes in particular have
special connection with technical subjects and should for this reason be taught
and tested in an interdisciplinary way so the language specificities could be
better understood (DOUGLAS, 2000). Thus it is paramount to build activities
which can raise such awareness in students and teachers as well as.
The
score results indicate that students face considerable more difficulties in
dealing with English than with the Logistics subject since both subject scores
could not be correlated and the language ones were rather lower on
average. Although people in general fear
public speaking and the situation may be hypothetically worse in a foreign
language, the scores show that the students knew what to talk in the context
but they faced obstacles to express it.
As a
first time experience for teachers and students, the fact that participants
were able to carry out the whole task may be seem as a positive outcome. This
can increase their ability to deal with other performance tests in the future.
On the other hand, there is room for improvement in language use so they can
better know how to do it.
6.
CONCLUSION
This paper presents an
interdisciplinary task used in performance language test. Aspects like test
task construction, its objectives, expected “can do” in the language, expected
topical knowledge content, criteria descriptions and scoring were discussed.
Results show that there is no correlation between English language and
Logistics subjects mean scores. Besides, students had on average lower scores
in English, the highest score (10) was given in Logistics but also the lowest
one (5). The results indicate that there is room for improvement in language
use.
This study result can contribute to
reflections on how teachers can build achievement tests that can be more
meaningful and close to real situations considering the wide range of subjects
that students take every term. On the other hand, one limitation of this
research is the low number of participants and the fact that they were all from
the same class. For this reason, it would be important to develop further
investigations involving more students and classes in order to identify
possible similarities or differences.
REFERENCES
BACHMAN, L.; PALMER, A. S. (1996) The construct validation of some components
of communicative proficiency. TESOL Quarterly, v. 16, n. 4, p. 449-65.
Common European (2001) Framework of Reference
for Languages: Learning, Teaching, Assessment Cambridge
University Press: Cambridge.
CRESWELL,
J. W. (1998) Qualitative Inquiry and Research Design: Choosing Among Five Traditions.
London: SAGE Publications Ltd
DENZIN, N. K. (1978) Sociological methods.
Nova York: McGraw-Hill.
DENZIN, N.;
LINCOLN, Y. (2006) O planejamento da pesquisa qualitativa:
teorias e práticas. Tradução Sandra Regina. Netz.
Porto Alegre: Armed.
DOUGLAS,
D. (2000) Assessing Language for Specific Purpose. Cambridge:
Cambridge University Press.
GRADDOL, D.
(2000) The future of English? The
British Council.
HUGHES, A. (1989) Testing for language teachers. Cambridge:
Cambridge University Press.
JONES, R.
L. (1979) Performance testing of second language proficiency. In Briere E.J. and Hinofotis F.B. (eds) Concepts in language testing: some recent
studies, TESOL, Washington DC p. 50-7.
KOBAYASHI, E. (2010) Processos
avaliativos em língua estrangeira (inglês): Um estudo de caso em contexto
empresarial. Dissertação (Mestrado em Linguística Aplicada) – Instituto de
Estudos da Linguagem, Universidade Estadual de Campinas, Campinas.
LINCOLN, Y. S.; GUBA, E. G. (1985) Naturalistic
Inquiry. Beverly Hills: Sage.
MCNAMARA, T. (2000) Language Testing. Oxford:
Oxford University Press.
Shank, G.
(2002) Qualitative Research. A
Personal Skills Aproach. New Jersey: Merril Prentice Hall.
STOBART,
G. (2003) Alternative Assessment. In: KELLAGHAN, T.; STUFFLEBEAM, D. (eds). International Handbook of Educational
Evaluation. Kluwer Academic Press, Dordrecht, Netherlands, p.
549-576, 2003.
YIN, R. K. (1984) Case study research:
design and methods. California: Sage publications.