Transcript Document

Needs analysis
and evaluation
Advisor
: Dr. Patricia Su
Presenter
: Lisa Yin
Number
Date
: 10022606
: April 12th,2012
Aims
The key stages in ESP are needs analysis, course (and
syllabus) design, materials selection (and production),
teaching and learning, and evaluation.
These are not separate, linearly-related activities, rather
they represent phases which overlap and are
interdependent. The simplicity and clarity of figure7.1 is in
reality more like figure 7.2
Needs analysis is the process of establishing the what and
how of a course; evaluation is the process of establishing
the effectiveness.
2
Figure 7.1 Stages in the ESP process: theory
Needs
analysis
evaluation
assessment
course
design
teaching learning
3
Figure 7.2 Stages in the ESP process: reality
Needs
analysis
evaluation
assessment
course
design
teaching learning
4
Needs analysis
Needs analysis is neither unique to language
teaching nor within language training.
However, needs analysis is the corner stone of
ESP and leads to
a very focused course.
5
Needs analysis
The information obtained from clients and students will only
be as good as (a) the questions asked and (b) the analysis
of the answers.
A crucial point, whenever data is being collected, is to know
beforehand what will happen to the raw data and to the
information derived from it.
As ESP practitioners we need to know exactly what we are
trying to find out and what we will do with the answers
before we start. (Berwick,1989:62)
6
What is meant by needs?
A confusing plethora of terms exists: needs are described
as objective and subjective, perceived and felt , target
situation /goal-oriented and learning, process-oriented
and product-oriented ; in addition, there are necessities,
wants and lacks. These terms have been introduced to
describe the different factors and perspectives which have
helped the concept of needs to grow. Each of these terms
represents a different philosophy or educational value.
Briefly, objective and perceived needs are seen as
derived by outsiders from facts, from what is known and
can be verified, while subjective and felt needs are
derived from insiders and correspond to cognitive and
affective factors.
7
What is meant by needs?
Target situation analysis(TSA)
Learning situation analysis(LSA)
Present situation analysis(PSA )
Means analysis (Holliday and Cooke, 1982:133)
A TSA includes objective, perceived and
product-oriented needs;
An LSA includes subjective, felt and processoriented needs;
A PSA estimates strengths and weaknesses in
language, skills, learning experiences.
8
What is meant by needs?
To establish a workable course design, means analysis is
suggested (Holliday and Cooke, 1982:133)as an adjunct to
needs analysis.
Means analysis looks at the environment in which a course
will be run or, as in the original metaphor that generated the
term, the environment in which a project will take root, grow
healthily and survive.
The two key factors considered for Means analysis are the
classroom culture and the management infrastructure and
culture.
9
A current concept of needs analysis
Needs analysis in ESP now encompasses determining:
A. professional information about the learners : the tasks and
activities learners are/will be using English for – target
situation analysis and objective needs
B. personal information about the learners : factors which
may affect the way they learn such as previous learning
experiences, cultural information, reasons for attending the
course and expectations of it, attitude to English – wants,
means , subjective needs
C. English language information about the learners : what
their current skills and language use are – present situation
analysis – which allows us to assess (D)
D. the learners’ lacks : the gap between (C) and (A) – lacks
10
A current concept of needs analysis
E. Language learning information : effective ways of
learning the skills and language in (D) – learning
needs
F. professional communication information about (A) :
knowledge of how language and skills are used
in the target situation – linguistic analysis,
discourse analysis, genre analysis
G. What is wanted from the course
H. Information about the environment in which the
course will be run – means analysis
11
A current concept of needs analysis
Personal
information
about
learners
professional
information
about
learners
how to
communicate
in the target
situation
Language
information
about target
situations
environmental
situation
Learners’
lacks
Language
learning
needs
learners’
needs from
course
• What needs analysis establishes
12
Matching needs analysis to situation
A course outline, materials and other resources can
be in place before teaching begins.
We must distinguish between overall needs and
course needs.
Course needs :
What do you need/ want from the course?
Overall needs :
What do you need English for?
13
Evaluation
What is evaluation?
There are many definitions of evaluation ; fundamentally
evaluation is asking questions and acting on the responses.
We accept the following : evaluation is a whole process which
begins with determining what information to gather and ends with
bringing about change in current activities or influencing future
ones.
Evaluation must be more than collecting and analyzing data: to
have value the evaluation process must include action.
Evaluation is usually described as formative or summative.
14
What is evaluation?
Evaluation is usually described as formative or summative.
Formative evaluation
Summative evaluation
 which takes place during the
lifetime of an activity(a course)
and the findings help to shape the
course during its life-time.
 is typically undertaken at
intervals and will consist of a
series of ‘mini-evaluations’.
 Testing the arrangement of
lessons in a primer before its
publication
 Collecting continuous feedback
from participants in a program in
order to revise the program as
needed
 takes place at (or after) the end
of an activity and so does not
influence that version of the
activity.
 Its purpose is to assess impact
and to provide information that
can be fed into repeat versions or
related activities.
 Determining attitudes and
achievement related to using a
primer after it has been used in a
training course
 Collecting data on the impact of a
program operating in a
community for a period of time
15
What is evaluation?
Evaluation can be both qualitative and quantitative.
Evaluation can be very threatening; it suggests change
and change is often resisted.
Evaluation is a very constructive and powerful activity and
a very stimulating one.
Evaluation will also show weaknesses or features that
were just not suitable for the particular group of learners.
A good evaluation emphasizes the successes and
discusses less successful aspects. It also addresses the
crucial how and why issues.
Knowing how well something has worked is not significant
on its own.
Understanding why will enable us to repeat success and
avoid the less successful.
16
What do we evaluate?
Evaluation in ESP situation is concerned with
the effectiveness and efficiency of learning.
Evaluation results can be used to influence
decisions and bring about long-term change.
One-off courses
Durable courses
1. The important questions are
1. Evaluation questions may be
those asked part-way through
asked for a range of different
2. Focus on change that is feasible
purposes.
and immediately
implementable.
17
What do we evaluate? -1
Some of the questions to ask before an evaluation are:
 Audience and purpose
 Who are the stakeholders? The term ‘stakeholder’ is used
to cover all those who have an interest or concern with
the course. The client who requests the course, th learners
and the teachers are the main stakeholders but there can
be others (sponsors, organisers).
What do you want to evaluate?
What do you want to change?
18
What do we evaluate?-2
Criteria for evaluation
 What are the objectives you are evaluating against? In
some situations
Criteria for analysis of results
 What will you do with the answers? What can you change?
What requires the authority of others? And what will
convince them?
Sources of information
 Who can provide useful information?
 When Would it be appropriate for them to do that?
19
Collecting data for needs analysis
and
evaluation purposes
Who collects the data?
For both needs analysis and evaluation, outsiders or insiders could be
involved.
Outsiders, they do not know the situation and the environment so they may
miss or misinterpret data.
Insiders will have a feel for the situation but can be too close and involve, or
lack expertise.
A known, respected outsider working together with insiders is one alternative
(Alderson and Scott, 1992:36/7) The outsiders can be others in the same
institution who are not involved with the course that is being evaluated.
Complete outsiders are most likely to be used both for needs analysis and
evaluation on large-scale projects with external funding.
20
Who provides the data and how?
Checklists and questionnaires
Checklists are narrower in scope and more commonly used
for a qualitative feel. They can determine facts or attitudes.
Of the attitude scales the easiest to construct is an adjective
checklist.
The Likert scale consists of statements that respondents
agree or disagree with.
Rating scales are easier to use and useful for broad
distinctions.
21
Structured interviews
Structured interviews consist of questions which have been
carefully thought out and selected in advance.
Structured interviews should be recorded so that the
interviewer can really listen rather than take lots of notes. The
art is to gain the maximum relevant information in the minimum
time. Key skills for interviewers are
active listening, summarising and asking open questions.
22
Observation
Observation can cover a range of activities from watching a
particular task being performed to shadowing individuals at
work.
EAP examples of observation for needs analysis include
sitting in on subject lectures or practical sessions.
Observations have led to an understanding of how and
when English and the L1 are intertwined, of the codeswitching which often goes on.
In EOP situations, any relevant work processes may be
observed.
Most observation or shadowing is for Target Situation
Analysis(TSA) purposes but it could be for the Present
Situation Analysis (PSA) of a particular individual.
23
Observation
Most observation for evaluation is of classroom activity , but
it could also include seeing how well a learner was coping
with using the language in their work or studies.
Classroom observation requires careful groundwork and
handling.
After the session, observer and teacher should share their
perceptions of what was happening.
The early comments must focus on positive features.
Beginning with negative comments sets up the wrong
perspective.
24
Analysis of authentic texts
The texts can be written documents or audio and video
recordings of events such as lectures, meetings, telephone
interactions, classroom activities.
The analysis may be for TSA purposes, to determine the key
linguistic features of a communicative event or genre that is
new to us.
Authentic texts are invaluable for learning about real and
carrier content.
25
Assessment
Assessment includes formal and informal judgements of
students’ performance and progress through classwork,
assignments and tests.
Testing or assignments may form part of a pre-course PSA
or evaluate progress. Issues of assessment and testing are
covered in chapter 11.
26
Discussions
Discussions can pave the way in both needs
analysis and evaluation to other methods such
as interviewing and observing or be an end in
themselves.
Record keeping
What is needed in record keeping is systematicity:
records must be made immediately.
27
How do we analyse the data?
First, the raw data must be converted into information. With
small amounts of quantitative data manual methods can be
employed to process the data into information.
Jones(1991) described how Principal Component
Analysis(PCA)
By using PCA, which is a type of correlational analysis that
can resolve data into underlying factors, he was able to
identify clusters of variables.
LANA(LAnguage Needs Analysis) is another tool intended
either for individual needs analysis or as part of an audit.
Needs analysis must result in an understanding of a target
situation such that we, the ESP practitioners, could be
efficient communicators in it.
28
What happens to the results?
The feedback may be how a text is to be used,
how the design of a course component has been
influenced, the overall results of a minievaluation.
Feedback is good PR(public relations), good for
the quantity and quality of future cooperation.
29
Summary
Behind successful ESP courses is a continuous process of
questioning.
A. Ascertaining what a course should contain, how it should
be run;
B. Checking throughout how valid the original answers
were and how effective the ideas they led to are;
C. Discovering what works best and why.
Initial needs analysis
On-going needs analysis
Formative evaluation
Summative
evaluation
set objectives
revise objectives
inform future
determine approach
modify teaching and
materials
justify
measure
30
Thank you !
31