Transcript Slide 1

Behavioral Scientists in Action
[email protected]
CCMC
Corporate Comprehensive Management
Consultants
File-1
“People Drive the Process of
Excellence”
© CCMC Method Engineers Training Manual 1996 (2010 update)
1
Behavioral Scientists in Action
[email protected]
Workshop on Method Engineering for
Assessment Centers
at the Pre-conference workshop of the
Annual Conference of ACSG, Stellenbosch- South Africa
Dr. S. Pandey
Corporate Comprehensive Management Consultants
© CCMC Method Engineers Training Manual 1996 (2010 update)
India, Bombay
16th March, 2011
2
Behavioral Scientists in Action
[email protected]
Objective
To provide familiarity with relevant
concepts, techniques and skills
pertaining to Method Engineering for
Assessment center (AC) design.
© CCMC Method Engineers Training Manual 1996 (2010 update)
3
Behavioral Scientists in Action
[email protected]
Goals (expectation)

An appreciation program to acquire new AC-community
member and refresher to existing members (members
include both buyers and suppliers of AC)

Participants should be able to make informed decision
while selecting or designing a set of methods for an
Assessment center. This implies being able to differentiate
good methods from bad methods for a given
implementation context. Getting out of the myth that there
are good methods irrespective of implementation context

Participants should be able to identify needs for
customization and some ideas to customize a method

Ultimately, some participants may chose method
engineering as a profession and commit personal energy to
advance research and design of AC-Methods, thereby
contribute to the community
© CCMC Method Engineers Training Manual 1996 (2010 update)
4
Behavioral Scientists in Action
[email protected]
Most essential element of the goal to me
today is
 Your inquisitiveness (literally, the state
of active curiosity)
 Our assumption is that for Adult
learner “if inquisitiveness is there
everything else can be arranged”
© CCMC Method Engineers Training Manual 1996 (2010 update)
5
Behavioral Scientists in Action
[email protected]
Session-1
AS practitioner you know the fundamental strength of ACapproach; reduction of error in human attribute
measurement.
Concepts of errors in the context of Human attribute
measurement:




Sources of errors and controls of errors
Method related errors; when your Method is imperfect
your measurement data is contaminated by
measurement variance and not the construct variance.
Essence of reliability (specially, inter-rater reliability)
validity (specially, content and predictive validity)
Measurement of deterministic vs. probabilistic
(stochastic) entities – single data point vs. expectancy
of a distribution - limits of error tolerance.
© CCMC Method Engineers Training Manual 1996 (2010 update)
6
Behavioral Scientists in Action
[email protected]
Session-1: Role of methods in AC
 Role of methods in AC
 (Methods) Stimulus-Response
(behavior)
 Concept of behavior variance
 Introduction to relevant concepts
of experimental design and
naturally occurring experiments
© CCMC Method Engineers Training Manual 1996 (2010 update)
7
Behavioral Scientists in Action
[email protected]
Who is Method Engineer?



For a detail understanding of Method Engineer’s role refer to
‘Method Engineer’s task flow chart elsewhere in this workshop.
For the timebeing let us note the following.
 Method Engineer has the responsibility of design & development
of Methods, and also the entire implementation of the AC/DC
design. They convert scientific principles into pragmatic/ field
application.
 Methods & Processes are their domain that mediate between
AC-design and AC-goals
Like any other Engineers they often conduct field tests of
prototype of Methods & processes of a design to validate
earlier laboratory tests and to obtain broader feedback for
refinement
© CCMC Method Engineers Training Manual 1996 (2010 update)
8
Sources of Errors in Human Assessment
Behavioral Scientists in Action
[email protected]
Assessment Error =
ƒ (Assessor x Assessment methods x Assessment Factors)
 Trained Assessors (ORCE, Factor & Method familiarity)
 Accredited (for both Knowledge & skills) for Quality
 Standardized methods
 Adequate Quality in data generation. [A Method
Engineer by design can compensate Assessor’s skill
deficiency]
 Standard administration process with approved
flexibility for customization
 Simple in structure but profound in eliciting behavior
 Unbiased to Assessees irrelevant background e.g., age,
race, gender.
 Competency research based Assessment factors/criteria
© CCMC Method Engineers Training Manual 1996 (2010 update)
9
Behavioral Scientists in Action
[email protected]
Common Assessment Errors: vis-à-vis
Method Engineers responsibility
Assessment errors are To reduce the errors
a function of not
Method Engineer can
knowing the following: do the following:
Not knowing: What to
Observe
Attention to Method
specific Behaviors &
Factors
Not knowing: How to
Observe
Attention to Methods’
process compliance
Not knowing: Who should Attention to Assessors’
Observe
skill specification for each
Method
© CCMC Method Engineers Training Manual 1996 (2010 update)
10
Assessment Errors
Behavioral Scientists in Action
[email protected]
1. Not knowing What to Observe –The
answer is Factors
Assessment Centre approach ensures relevant factors
through:
 Analysis of the Job
 Deriving a standardized set of Competency Factors to be
assessed
 Identifying observable behaviors linked to the factors.
 Observing, recording, diagnosing the pattern and rating
of behaviour
© CCMC Method Engineers Training Manual 1996 (2010 update)
11
Behavioral Scientists in Action
[email protected]
Assessment Errors….contd.
2. Not knowing How to Observe- The Methods
 Assessment Centre follows rigorously, simple
standardized methods that simulate the job
partially/fully.
3. Not knowing Who should Observe – The
Assessor
 Assessment Centre approach helps developing
guidelines for assessor selection and training
© CCMC Method Engineers Training Manual 1996 (2010 update)
12
Behavioral Scientists in Action
[email protected]
Common Assessment Errors
Method
Factor
Assessor
Minimizing errors



Linking to job
Defining
Relating to
observable
behaviors
© CCMC Method Engineers Training Manual 1996 (2010 update)
• Simplification  Personal level
capability;
• Rigor (design)
Namely,
• Standardization
Knowledge,
Attitude, Skill
 Rating errors
13
Behavioral Scientists in Action
[email protected]
Hi
2. Conscious
3. Conscious
incompetence
competence
Energy
Level
1. Unconscious 4. Unconscious
incompetence
competence
Lo
Lo
Effectiveness / Results
© CCMC Method Engineers Training Manual 1996 (2010 update)
Hi
14
Behavioral Scientists in Action
[email protected]
Session-2
Design elements of a battery of
Methods - Methods taxonomy and
illustration – types of competencies visà-vis fitting Methods – types of AC
outcome and Method fit – brief ideas of
subtler types, as for example, based on
degree of rigor or skill levels of
Assessors and so on.
© CCMC Method Engineers Training Manual 1996 (2010 update)
15
Behavioral Scientists in Action
[email protected]
Session-3
 Design elements of Methods – method
specifications elements – Method content
design (lead, casting etc.) - Method process
design (control) – pilot and standardization
(reliability, security, content validity and true
validity etc.) – Method testing
 Concepts of single stage vs. multi-stage
Methods - parallel Methods – sequential
supplementary Methods
© CCMC Method Engineers Training Manual 1996 (2010 update)
16
Behavioral Scientists in Action
[email protected]
Hands on exercise
Design of a prototype on the spot
(Any one of the following: case
method/ role play method/ on the spot
exercise method/ advance method/
simple work simulation)
© CCMC Method Engineers Training Manual 1996 (2010 update)
17
Behavioral Scientists in Action
[email protected]
Session-4
 Practice tips – Take home assignment
suggestions
 Suggestions for applied research –
suggestions for basic research
© CCMC Method Engineers Training Manual 1996 (2010 update)
18
Behavioral Scientists in Action
[email protected]
Simple criteria to improve Content
Validity Exercise
 Candidates’ Tasks on the Method should reflect the
critical activities of the job
 The choice of Methods batteries should reflect
Organization’s Values, principles and culture
 The Format of the Method and the Process in which
participants are asked to respond should match the
reality on the job
 The difficulty level and complexity of Methods must
fit the ‘difficulty level and complexity of the job.
 The Task on the Method must be aligned to intrinsic
Strengths of the Method (not weakness)
© CCMC Method Engineers Training Manual 1996 (2010 update)
19