Transcript Computational Approaches to Studying Design Teams

Team Thinking
Time Variation of Design
“Story Telling” in Engineering
Design Teams
Andy Dong
Key Centre of Design Computing and Cognition
University of Sydney
Shuang Song and Alice M Agogino
Department of Mechanical Engineering
University of California, Berkeley
Team Thinking
Design Teams: Research
Questions
• How can we identify, represent, and visualize
“story telling” of engineering design teams?
• How does the “story telling” evolve over time?
• What insights into emergent design
processes could quantitatively depicting
patterns in “story telling” provide?
Team Thinking
“Story Telling” in Engineering
Design Teams
• Engineering design is a complex technical
pursuit mediated by social processes such as
communication, negotiation, and shared
agreements.
• We define “story telling” in design as
establishing a coherent “story” about the
design process and the designed artifact by
bringing coherence to the perspectives and
interests of each design team member.
Team Thinking
Methods for Studying
Design Teams
• Protocol analysis
– Discourse analysis and
“Think aloud”
• Ethnography
– Observations, interviews
& case studies
• Computational
– Latent Semantic Analysis
– Social Network Theory
– Speech Act Theory
Team Thinking
What are “good” design
teams?
• Balanced teams consisting of members with
complementary roles, a plurality of
viewpoints, a neutral manager and a “wild
card”. (D. Wilde)
• Establishes a social contract among
members that relate to their purpose and
guides and obligates how they must work
together. (J. Katzenbach)
• Design strategies, including reflection, that
lead to the creation of shared understanding
result in effective design outcomes. (R.
Valkenburg)
Team Thinking
Product
Development StageGate Model
Idea Validation
Conceptual Design
Specification and
Design
Prototype production
and testing
Manufacturing rampup
Technical Process
Technical Design Methods
Determine customer and market
needs
Customer interviews, surveys, and focus
groups; Ethnographic studies; Patent search;
Affinity diagrams; TRIZ/ARIZ
Determine broad product and
business objectives
Identify essential problems
Product portfolio planning
Propose function structures
Bond graphs; Design structure matrix;
Mechanical design compiler
Search for and propose solution
principles
Case-based information retrieval; Method of
imprecision
Select, combine and refine into
concepts
Evaluate concepts against design
criteria
Pugh charts, pairwise comparison charts,
Borda charts
QFD; Design Compatibility Analysis;
Compromise Decision Support Problem
Develop preliminary configurations
Shape annealing; evolutionary and coevolutionary design
Select best preliminary design(s)
Refine designs and configurations
and evaluate against technical and
economic criteria
Detailed analysis of refined
design(s)
Review for errors,
manufacturability, and cost
Prepare a preliminary parts list and
manufacturing and assembly
drawings
Final design analysis and
verification
Complete detailed drawings and
production documents
Multi-objective concept selection
Design and system configuration optimization
Prototype
Test and evaluate
Design solution
Design freedom and uncertainty
Finite element modeling and analysis; Designrule checking
DFM/DFA/DFx
Social Processes
Formation of core product development team;
Assign team roles; Create “social network”
(i.e., who including external suppliers should
be involved)
Define and order product objectives (e.g.,
must-have features and nice-to-have features)
Development of team jargon and vocabulary;
Decide how to handle ambiguity, uncertainty
and imprecision; Distinguish specifications
from constraints
Division of design tasks among team
members; Division of larger design teams into
smaller teams by product sub-system
Brainstorming; Consultation with experts
within company, communities of practice, etc.;
Source transactive memory
Shared Understanding
References
(Wilde, 2000)
Shared team-work mental
model
(Simpson et al., 1998)
(Ward and Seering, 1993)
(Wood and Agogino, 1996)
(Antonsson and Otto, 1995)
(Dym, Wood and Scott, 2002)
Agreement on method(s) for evaluating and
selecting concepts against criteria; Establish
norms for critiquing designs
Establish accepted design procedure (e.g.,
adopt appropriate design codes);
Communication and sharing of technical
design issues to all team members (distributed
cognition)
Reconciliation of design objective conflicts
Team learning of design trade-offs, potentials
of new technologies, incorrect assumptions,
etc.
Acceptance of testing methodologies, test
environments, lead customers, etc.
Shared agreement of design
process (task mental model)
(Hausing and Clauser, 1998)
(Ishii et al., 1989) (Bras and
Mistree, 1995)
(Cagan and Mitchell, 1993)
(Maher, 2000)
(Iyengar et al., 1992)
(Chen et al.,1996)
(Boothroyd et al., 2002)
Finite element modeling and analysis
“Sign-off” by all team members
Computer-aided drafting and visualization
(e.g., solid modeling; surface modeling;
rendering; animation)
Rapid prototype; “desktop machining”
Team post-mortem
Shared understanding of the
function, behavior and
structure of the design
Transfer of design knowledge to
manufacturing, sales, operations and
maintenance personnel
Existence of a socially shared
narrative of the design
Taguchi quality
(Taguchi, 1996)
Team Thinking
Measuring Time Variation
of Design “Story Telling”
• Capture design documentation
• Text processing (keyword extraction,
frequency counts)
• Text analysis (latent semantic analysis)
• Calculate intra-stage and cross-stage
semantic coherence
• Compare predicted team performance
to human assessments
Team Thinking
Metrics
cos(d i , d j ) 
di d j
di
dj
1. Intra-Stage Semantic Coherence
2. Cross-Stage Semantic Coherence
Team Thinking
Data Set: Product Design
Teams
• Managing the New Product
Development Process: Design Theory
and Methodology
• Product definition to alpha prototype
• Self-managing design teams of students
from engineering, business and
information science
Team Thinking
Assessment
• Professional Product Designers (IDEO,
frog design, Lunar Design and others)
• Judging Criteria (Scale 1-5)
–
–
–
–
mission statement
customer and user needs
concept generation
concept selection
– prototype feedback
– financial analysis
– final prototype
• Correlate computational results to
judges’ assessments
Team Thinking
Design Coherence Over
Time
• The design process is characterized by an
iterative broadening and narrowing of design
possibilities, and an iterative reconciliation of
design interests and conflicts towards a set of
shared agreements.
• Increasing coherence with cycles of
divergence during the design process is
desirable. Decreasing coherence in design
“story telling” is likely disruptive and
increasingly dysfunctional.
Team Thinking
Semantic Coherence
Variation of Average
Semantic Coherence
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
-0.1
-0.2
-0.3
A
A
C
B
C
FG
D E
S1
A B
D
B
D E FGH
H
B
F
G
C
S2
E F
S3
H
A
CD E
S4
Design Stages
s1: Preliminary Investigation
s2: Detailed Investigation
s3: Development
s4: Testing and Validation
H
G
Team Thinking
Variation of Intra-Stage
Semantic Coherence
Semantic Coherence
1.2
1
S1
S2
S3
S4
0.8
0.6
0.4
Avg=0.51
Avg=0.48
Avg=0.53
Avg=0.45
0.2
Avg=0.43
Avg=0.21
Avg=0.25
Avg=0.07
0
-0.2
Sep -16
Oct -8
Oct -10
Oct -16
Oct -28
Nov -21
Date
High performing Team B
Low Performing Team C
Dec -7
Team Thinking
Cross-stage Semantic
Variations
Cross-Stage Semantic
Coherence Variations
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Faculty
Rank
E
F
B
Team A
Team A
8
Team B
Team B
1
Team C
6
Team E
Team D
5
Team F
Team E
2
Team F
3
Team G
4
Team H
7
Team C
Team D
H
C
D
A
Team G
Team H
G
S1
S2
S3
Design Stages
S4
Team Thinking
Number of E-mail
Messages
Patterns of E-mail Usages
250
200
150
100
50
0
B
C
E
F
G
H
Team
Preliminary and detailed investigation
Development, testing and validation
Team Thinking
Patterns of Semantic
Coherence from Two Sources
Semantic Coherence
1.2
1
0.8
0.6
0.4
0.2
0
-0.2
Sep -16
Oct - 7
Oct -15
Oct -24
Nov - 18
Nov -20
Dec -5
Date
Team G Design Documents
Team G E-mail Messages
Dec -15
Team Thinking
Summary
• This research establishes a formal methodology
for providing a real-time window into the design
process and coherence of design thinking of the
design teams.
• Evidence to suggest a link between patterns of
the semantic coherence of design documentation
(accuracy of language in explaining thoughts)
and design performance.
• Latent semantic analysis shows great promise as
a tool for modelling and evaluating the cognitive
and psychosocial behaviour of design teams.
Team Thinking
Future Research
• How do we distinguish between “creativity” and
“lack of vision”?
• What is the role of individuals (personalities,
leadership, gender,etc.) in the “team thinking”
process?
• How information technology (information
source, information accessibility,
communication, etc.) influences design team
cognition?