Transcript Document

Lecture 4b: Risks and
Liabilities of Computer-based
Systems
CSCI102 - Introduction to Information Technology B
ITCS905 - Fundamentals of Information Technology
Overview
Historical examples of software
risks
 Implications of software
complexity
 Risk assessment and management

Historical Examples

Software errors

Can KILL
 Indirectly
 Directly

Cost MONEY
 Loss
of equipment
 Loss of business
Software Aids and Abets
Murder: 1992

A New Jersey inmate escaped
from computer-monitored house
arrest in the spring of 1992

He simply removed the rivets holding
his electronic anklet together and
went off to commit a murder
Software Aids and Abets
Murder: 1992

A computer detected the
tampering

when it called a second computer to
report the incident, the first
computer received a busy signal and
never called back
Radiation Machine Kills Four:
1985 to 1987

Faulty software in a Therac-25
radiation-treatment machine
resulted in several cancer
patients receiving lethal
overdoses of radiation
Radiation Machine Kills Four:
1985 to 1987

Four patients died
Radiation Machine Kills
Four: 1985 to 1987
When their families sued, all the
cases were settled out of court
 There were several errors,


among them the failure of the
programmer to detect a race
condition (i.e., miscoordination
between concurrent tasks)
Radiation Machine Kills
Four: 1985 to 1987

It was found that found that accidents
occurred even after AECL thought it
had fixed particular bugs


"A lesson to be learned from the Therac25 story is that focusing on particular
software bugs is not the way to make a
safe system”
"The basic mistakes here involved poor
software-engineering practices and
building a machine that relies on the
software for safe operation”
Hyphen Costs $80 Million:
1962

A probe launched from Cape
Canaveral was set to go to Venus
After takeoff, the unmanned rocket
carrying the probe went off course
 NASA had to blow up the rocket to
avoid endangering lives on earth

Hyphen Costs $80 Million:
1962

NASA later attributed the error to a
faulty line of Fortran code


“Somehow a hyphen had been dropped
from the guidance program loaded aboard
the computer, allowing the flawed signals
to command the rocket to veer left and
nose down
...Suffice it to say, the first U.S. attempt
at interplanetary flight failed for want of
a hyphen”
Hyphen Costs $80 Million:
1962

The vehicle cost more than $80
million, prompting Arthur C.
Clarke to refer to the mission as
“the most expensive hyphen in
history”
AT&T Long Distance Service
Fails: 1991
In the summer of 1991, telephone
outages occurred in local
telephone systems in California
and along the Eastern seaboard
 These breakdowns were all the
fault of an error in signalling
software

AT&T Long Distance Service
Fails: 1991

Right before the outages
DSC Communications introduced a
bug when it changed three lines of
code in the several-million-line
signalling program
 After this tiny change, nobody
thought it necessary to retest the
program

AT&T Long Distance Service
Fails: 1991
These switching errors in AT&T's
call-handling computers caused
the company's long-distance
network to go down for nine hours
 The meltdown affected thousands
of services and was eventually
traced to a single faulty line of
code

There’s a Hole in the Bucket

Small systems


…form part of larger systems
A fault within a small part could
result in a catastrophe later on
There’s a Hole in the Bucket

Designers have an ethical
responsibility to design the best
system possible
Bugs

Bugs exist because



…humans aren't perfect
Since humans design and program
hardware and software, mistakes are
inevitable
That's what computer and software
vendors tell us, and it's partly true

What they don't say is that software is
buggier than it has to be
Bugs

Why?

Because time is money, especially in
the software industry
Bugs

This is how bugs are born
Bugs
A software or hardware company
sees a business opportunity and
starts building a product to take
advantage of that
 Long before development is
finished, the company announces
that the product is on the way

Bugs

All the while pressuring the
software engineers to add more
and more features
Bugs
Shareholders and venture
capitalists clamour for quick
delivery because that's when the
company will see the biggest surge
in sales
 Meanwhile, the quality-assurance
division has to battle for
sufficient bug-testing time

Bugs


“The simple fact is that you get the
most revenues at the release of
software,”
“The faster you bring it out, the more
money you make. You can always fix it
later, when people howl. It's a fine line
when to release something, and the
industry accepts defects“
What Is Risk Assessment
and Management?

Risk and uncertainty are fundamental
elements of modern life


They are ever present in the actions of
human beings and they are frequently
magnified in large-scale technological
systems
Risk and uncertainty must be managed
effectively to protect people from injury
and to permit the development of reliable,
high-quality products
What Is Risk Assessment
and Management?

Risk is often defined as a measure
of the probability and severity of
adverse effects
What Is Risk Assessment
and Management?

In risk assessment, the analyst
often attempts to answer the
following set of triplet questions
What can go wrong?
 What is the likelihood that it would
go wrong?
 What are the consequences?

What Is Risk Assessment
and Management?

Answers to these questions help
risk analysts identify, measure,
quantify, and evaluate risks and
their consequences and impacts
What Is Risk Assessment
and Management?

Risk management builds on the risk
assessment process by seeking answers
to a second set of three questions



What can be done?
What options are available and what are
their associated trade-offs in terms of all
costs, benefits, and risks?
What are the impacts of current
management decisions on future options?
What Is Risk Assessment
and Management?


To be effective and meaningful, risk
management must be an integral part
of the overall management of a system
This is particularly important in the
management of technological systems,
where the failure of the system can be
caused by the failure of the hardware,
the software, the organization, or the
humans