Transcript Software Engineering Ethics

University of Southern California
Center for Systems and Software Engineering
Software Engineering Ethics
Barry Boehm
CS 577a
Fall 2010
© USC-CSSE
1
University of Southern California
Center for Systems and Software Engineering
Outline
• Definitions and context
– Power to do public harm or good
– ACM/IEEE Software Engineering Code of Ethics
• Principles and examples
– Rawls’ Theory of Justice
– Relation to stakeholder win-win
– Case study: Mercy Hospital
• Integrating ethics into daily software
engineering practices
– ICM/VBSE/MBASE
– CS 577 ethics situations
© USC-CSSE
2
University of Southern California
Center for Systems and Software Engineering
Definition of “Ethics”
-Webster, 1993
• The discipline dealing with what is good
and bad
– And with moral duty and obligation
• A theory, system, or set of moral principles
or values
• The principles of conduct governing an
individual or group
– Professional ethics
© USC-CSSE
3
University of Southern California
Center for Systems and Software Engineering
Context
• Software engineers have increasing power to
do public harm or good
– Intellectual property, privacy, confidentiality, quality
of work, fairness, liability, risk disclosure, conflict of
interest, unauthorized access
• Professional societies have developed codes of
ethics
• Hard to integrate value-based ethics into valueneutral software engineering practices
• ICM/VBSE/MBASE enable ethics integration
– And buildup of trust
© USC-CSSE
4
University of Southern California
Center for Systems and Software Engineering
Shared Commitments are Needed to Build Trust
• New partnerships are increasingly frequent
– They start with relatively little built-up trust
• Group performance is built on a bedrock of trust
– Without trust, partners must specify and verify details
• Trust is built on a bedrock of honored commitments
• Once trust is built up, processes can become more
fluid
– But need to be monitored as situations change
© USC-CSSE
5
University of Southern California
Center for Systems and Software Engineering
Elements of Commitments
adapted from [Humphrey, 1989]
• Made willingly, subject to assumptions
• Not made lightly
• Agreement on what is to be done, by whom, and
when
• Openly and publicly stated
• Committers make every effort to meet commitments
• Unforeseen conflicts and invalidity of assumptions
are addressed early, and new commitments
negotiated
© USC-CSSE
6
University of Southern California
Center for Systems and Software Engineering
Power to Do Public Harm or Good – I
• Intellectual Property: use without credit; use
copyrighted material
• Privacy: credit, health, personal information
• Confidentiality: competitive information,
political sensitivity
• Quality of work: many dimensions; see table
© USC-CSSE
7
University of Southern California
Center for Systems and Software Engineering
Example: Confidentiality
• Government agency hires company to support SW
procurement
– Provides data under nondisclosure agreement
• Employee and company consultant prepare cost
estimate
– Employee: “ I don’t see how anyone can do all this
for $8M”
• Consultant provides $8M target cost to some
bidders
• Government agency angry with company for leak
– Whose fault? How could it be avoided?
© USC-CSSE
8
University of Southern California
Center for Systems and Software Engineering
Information
Consumers
Acquirers
indirect
Administrators
0 Insignificant or
System Controllers
*Significant
Info Brokers
Info Suppliers
Stakeholder
Classes
System Dependents
**Critical
Developers, Maintainers
Quality Concerns Vary by Stakeholders Role
Mission Protection
critical
Safety
**
Security
*
Privacy
**
**
**
uncrit.
**
**
**
**
*
Robustness
Reliability
**
**
**
*
*
Availability
**
**
**
*
*
**
**
**
**
**
*
**
*
*
Survivability
**
Quality of Service
Performance
Accuracy, Consistency
**
**
**
*
**
*
Accessibility, ease of use;
*
**
**
**
**
*
Evolvability
**
**
*
*
Interoperability
**
difficulty of misuse
**
**
**
*
**
Correctness
**
**
Cost
*
**
**
**
Schedule
Reusability
*
© USC-CSSE
**
*
*
9
University of Southern California
Center for Systems and Software Engineering
Power to Do Public Harm or Good - II
• Fairness: equality of opportunity/treatment; fair
reward system
• Liability: accountability; parity of authority and
responsibility
• Risk Disclosure: safety tests, COTS
capabilities; schedule slips
• Conflict of Interest: source selection; personnel
or product reviews
• Unauthorized Access: reading, copying,
modifying; denial of service
© USC-CSSE
10
University of Southern California
Center for Systems and Software Engineering
Examples: Fairness
• Enron software to schedule power outages,
raise prices
– Suppose you had been asked to develop it?
• Urban fire dispatching system
– Inefficient old system caused $700M property loss
– New-system spec. includes dispatching algorithm to
minimize property loss
• Any fairness issues?
© USC-CSSE
11
University of Southern California
Center for Systems and Software Engineering
Fairness and Accountability Issues:
Fire Dispatching System
• Dispatch to minimize value of property loss
– Neglect safety, least-advantaged property owners
• English-only dispatcher service
– Neglect least-advantaged immigrants
• Minimal recordkeeping
– Reduced accountability
• Tight budget; design for nominal case
– Neglect reliability, safety, crisis performance
© USC-CSSE
12
University of Southern California
Center for Systems and Software Engineering
CS 577 Ethics Accountability
• Honoring commitments to CS 577b
– Team DCR Life Cycle Plan for 577b should identify
577b continuing team members and roles.
– If you signed that you will continue in 577b in the
basic 577a questionnaire, we are expecting you to
honor your commitment.
– If you are considering not honoring your
commitment, please meet with me as soon as
possible.
© USC-CSSE
13
University of Southern California
Center for Systems and Software Engineering
Example: Safety Tests
• Your company is delivering a drug prescription
fulfillment system
– Reusing software from a warehouse inventory system
• You are the quality assurance manager
– With company responsibility for certifying product safety
• The software has passed all the contracted tests
– But many off-nominal conditions untested
– Some have shown unsafe outcomes
– You feel more off-nominal testing if necessary
• Company president says if you don’t certify safety by
delivery date, company may go out of business
– What should you do?
© USC-CSSE
14
University of Southern California
Center for Systems and Software Engineering
ACM/IEE Software Engineering Code of Ethics
-Table of Contents
1. Products: achievable goals, realistic estimates, high
quality
2. Public: safety, respect of diversity, public interest first
3. Judgment: objectivity, no bribes or conflicts of interest
4. Client and Employer: no employer-adverse interests,
surface problems
5. Management: fair, ethical work rules, due process for
violations
6. Profession: support profession and ethics code, don’t
misrepresent software
7. Colleagues: credit colleagues’ work, give colleagues a
fair hearing
8. Self: improve your technical and ethical knowledge and
© USC-CSSE
15
practices
University of Southern California
Center for Systems and Software Engineering
Code of Ethics 2. Public
2.01
2.02
2.03
2.04
2.05
2.06
2.07
2.08
2.10
Disclose any software-related dangers
Approve only safe, well tested software
Only sign documents in area of competence
Cooperate on matters of public concern
Produce software that respects diversity
Be fair and truthful in all matters
Always put the public’s interest first
Donate professional skills to good causes
Accept responsibility for your own work
© USC-CSSE
16
University of Southern California
Center for Systems and Software Engineering
Code of Ethics 4. Client and
Employer
4.01
4.02
4.03
4.04
4.05
4.06
4.07
4.08
4.09
Provide services only where competent
Ensure resources are authentically approved
Only use property as authorized by the owner
Do not use illegally obtained software
Honor confidentiality of information
Raise matters of social concern
Inform when a project becomes problematic
Accept no detrimental outside work
Represent no interests adverse to your employer
© USC-CSSE
17
University of Southern California
Center for Systems and Software Engineering
Outline
• Definitions and context
– Power to do public harm or good
– ACM/IEEE Software Engineering Code of Ethics
• Principles and examples
– Rawls’ Theory of Justice
– Relation to stakeholder win-win
– Case study: Mercy Hospital
• Integrating ethics into daily software
engineering practices
– ICM/VBSE/MBASE
– CS 577 ethics situations
© USC-CSSE
18
University of Southern California
Center for Systems and Software Engineering
Rawls’ Theory of Justice (1971)
-Following Collins et al., “How Good Is Good Enough?”
Comm.ACM, Jan. 1994
• Fair rules of conduct
• Principles of justice
• Participants and obligations
–
–
–
–
Provider (developer)
Buyer (acquirer)
User(s)
Penumbra (general public)
• Negotiate mutually satisfactory (win-win)
agreements
© USC-CSSE
19
University of Southern California
Center for Systems and Software Engineering
Rawls’ Theory of Justice - II
• Fair rules of conduct
– Negotiation among interested parties
– Veil of ignorance (about what affects whom)
– Rationality
• Principles
– Least Advantaged - don’t increase harm to them
• Harm = probability x magnitude (~risk exposure)
– Risking harm - don’t risk increasing harm
• Don’t use “low-threat” software in “high-threat” context
– Publicity test - defensible with honor before an
informed public
• Use for difficult cost-benefit tradeoffs
© USC-CSSE
20
University of Southern California
Center for Systems and Software Engineering
Obligations of the Software Provider
To the provider To the buyer To the user
 Make a
reasonable
profit
To the penumbra
 reasonable
 clear operating  reasonable
use warrantee
instructions
protections against
physical, emotional,
and economic harm
 informed
 reasonable
from applications
consent about
protections
testing
from and
process and
informative
 open about software
potential
responses to
development process
shortcomings
use and abuse
and limits of
correctness
 provide
reasonable
technical
support
© USC-CSSE
21
University of Southern California
Center for Systems and Software Engineering
Obligations of the Software Buyer
To the provider
To the buyer To the user
 negotiate in good
faith, recognizing the
importance of
provider’s fair profit
 provide quality
software
appropriate to
users’ needs within
reasonable budget
constraints
 learn enough about
the software to make
an informed decision
 prudent
introduction of
automation
 facilitate adequate
communication users
To the penumbra
 buy software only
with reasonable
safeguards for the
public
 open about software
capabilities and
limitations
 informed consent to
using software
 represent user’s
interests with
providers
© USC-CSSE
22
University of Southern California
Center for Systems and Software Engineering
Obligations of the Software User
To the provider To the buyer
 respect
ownership rights
To other users
 active
 willing
communication
cooperation in
with the buyer
learning and
using software
 good faith
effort to learn
and use
software
responsibly
To the penumbra
 conscientious effort
to reduce any risks
to the public
 encourage
reasonable
expectations about
software
capabilities and
limitations
 reasonable
requests for
computing
power
© USC-CSSE
23
University of Southern California
Center for Systems and Software Engineering
Obligations of the Software Penumbra
To the provider To the buyer To other users To the penumbra
 become aware of  advocate a
 expect only
the capabilities
suitable
reasonable
and limitations of
economic and
service from
software
statutory
users
environment
for quality
 advocate a
 become aware
software
suitable
of the
economic and
capabilities and
statutory
limitations of
environment for
software
quality software
© USC-CSSE
 become aware of
the capabilities and
limitations of
software
 advocate a suitable
economic and
statutory
environment for
quality software
24
University of Southern California
Center for Systems and Software Engineering
Case Study: Mercy Hospital Pharmacy System
-Collins et al., 1994
• Growing hospital
– Manual pharmacy information system reaching overload
• Spec developed for PC-based information system
– Rachel: VP, Records & Automation
– George: Chief Pharmacist
• System developed by consultants
–
–
–
–
Hired by George
Rachel: test procedures
Based on mature warehouse inventory system
Budgeted 50% more testing than other bidders
• Installation & Training discovers problems
– Helen: consultant in charge of installation & training
– Ann: skeptical nurse cross-checking computer outputs
© USC-CSSE
25
University of Southern California
Center for Systems and Software Engineering
Mercy Hospital Pharmacy System: Problems
• Dosage problems from data entry errors
– 10x dosage; wrong patient
• Cross-checking incomplete; not trusted by some
doctors
• Heavier data-entry load
– Formalizing automated procedures  more info. needed
– Pharmacy info > warehouse info
• Helen: Should go back to old system during cleanup
• George: - Is old system less risky?
- How do we ensure cleanup will get it right?
- How much will cleanup cost?
• Future practice: How to anticipate, avoid similar
problems?
© USC-CSSE
26
University of Southern California
Center for Systems and Software Engineering
Outline
• Definitions and context
– Power to do public harm or good
– ACM/IEEE Software Engineering Code of Ethics
• Principles and examples
– Rawls’ Theory of Justice
– Relation to stakeholder win-win
– Case study: Mercy Hospital
• Integrating ethics into daily software
engineering practices
– ICM/VBSE/MBASE
– CS 577 ethics situations
© USC-CSSE
27
University of Southern California
Center for Systems and Software Engineering
Mercy Hospital : Use of VBSE/MBASE/ICM
Win Win Spiral
• Results chain
– Add patient safety outcome, patient stakeholder representative
– Rework-business-workflows initiative, including safety checks;
add clerical-staff stakeholder
• Stakeholder Win Win
– Patient representative: safety criteria; parallel-operation phasein
– Clerical staff: prototype GUI, including safety-check support
• Business Case: includes added safety costs and
benefits
• Risk Management: assess warehouse package safety,
effects of workflow changes.
© USC-CSSE
28
University of Southern California
Center for Systems and Software Engineering
Use of VBSE/MBASE/ICM/Win Win Spiral-II
• Concurrent Engineering
– Concurrently address business workflows, GUI
prototypes, COTS alternatives, feature prioritization,
cost/schedule/benefits analysis, other risks
– Prepare to pass VCR, FCR, DCR, CCD, and IOC
anchor point milestone reviews
• Monitoring and Control: Use Balanced
Scorecard to track progress with respect to
plans; apply corrective actions as necessary
• Change as Opportunity: Look for emerging
COTS pharmacy-related fulfillment systems
© USC-CSSE
29
University of Southern California
Center for Systems and Software Engineering
CS 577 Ethics Situations
• Assuming your priorities match those of other
stakeholders
– Users: GUI; quality factor priorities
– Maintainers: programming language, reuse, documentation
– Customers/Owners: legacy compatibility, advanced vs. mature
technology, full business case
• Favoring stakeholders who agree with you
– Excessive privacy protection: customers vs. users
• Weighting stakeholders equally on each issue
– Users on GUI; owners on legacy compatibility: developers on
cost/schedule/risk
• Promising more than you can deliver
• Borrowing from other projects without credit
• Suppressing or delaying bad news
© USC-CSSE
30
University of Southern California
Center for Systems and Software Engineering
Conclusions
• Software engineers have increasing power to
do public harm or good
• Value-based codes of ethics are hard to
integrate with value-neutral software
engineering practices
• Rawls’ Theory of Justice enables constructive
approach for integrating ethics into daily
software engineering practice
– Stakeholder win-win with least-advantaged system
dependents as success-critical stakeholders
– ICM/VBSE/MBASE provides daily-practice
framework, way to build trust
© USC-CSSE
31