Trust, Safety and Reliability CS 340, Spring 2013

Ch. 5 in Ethics in a Computing Culture

Computer errors

• • A range of possible consequences – – Seconds lost, other inconvenience Data lost – – $ - $$$ Injuries and deaths Examples: – Disfranchised voters – – False arrests Price & billing errors – Air traffic control, airplane control

The Apocalypses That Might Have Been

Apocolypse cont’d

Ch. 5 Topics

• • • • Computer reliability can be a matter of life or death Exploring the various ways people trust computer and technology to perform tasks for them.

Examining failures Approaches to minimize risks

Causes of Computer Failure

• A computer might fail to meet expectations by: – Hardware errors (malfunction) – Software errors (bugs) – Being programmed to solve the wrong problem (programmers fail to deliver client expectations) – Misuse (a computer is provided erroneous data) – Communication failure (human misunderstands a computer prompt) – Malice (hackers) Ethics in a Computing Culture 6

Hardware errors

• Physical components of the system

Software Errors

• “Bugs” • Bug v. mistake debate

Bugs and Public Safety

•

Safety-critical software:

properly software that may affect someone’s safety if it fails to work •

Decision point:

on input data a place in computer code where the next instruction executed depends •

Control programs:

programs that control some sort of machinery Ethics in a Computing Culture 9

Bugs and Public Safety (continued)

•

Real time:

a program must do something within a specific amount of time •

Multiprocess:

programs that execute at the same time as one or more other programs Ethics in a Computing Culture 10

Bugs and Public Safety (continued)

• Many executives of software production companies, as well as several famous computer scientists, have asserted that the most important skill needed by a software developer is the ability to communicate. • Would more extensive testing of the software have prevented the Ariane flight failure?

Ethics in a Computing Culture 11

Bugs and Public Safety (continued)

•

Brooks’s Law:

“Adding people to a late project makes it later.” – Adding new people adds new lines of communication, which increase the chances of miscommunication or missed communication Ethics in a Computing Culture 12

Software errors

• Pages 152- 154 – In 2005, Ariane 5 • ariane 5 explosion – YouTube • Floating point number storage – Arithmetic overflow • Pages 155-156 – In 1994, Intel P5 Pentium chip • Certain floating point numbers when divided produced invalid results

Questions for the Pentium Floating Point Divide

• Did Intel have a moral obligation to replace the defective Pentium 5 chips, even for those users who had no need for high precision? • When employees of Intel first discovered the bug, did they have an ethical responsibility to make the problem public? Ethics in a Computing Culture 14

Computer Solves the Wrong Problem

• • • Complexity of programming: garbage in, garbage out • Gemini 5, 105 miles off landing target: Earth’s rotation is not 360° in 24 hr but 360.98° • Friendship 7, 40 miles off landing target: Failure to take into account weight loss in aircraft due to use of consumables

Misuse

• • • The Hernandez situation: – Emiliano Hernandez was pulled over running a stop sign – When the officer ran his license the officer was convinced that he was dealing with a wanted man, “Enrique” Hernandez Same birthday, similar height, weight, tattoo.

Officer was convinced that the name difference was a mistake in the system. It was not.

Communication Failure

• • Misunderstandings – Of what the system is capable of or how it works • Book example -refueling

Electronic voting machines:

– After the "hanging chad" controversy of the 2000 election, Congress passed a federal law that gave states funding to replace their punch card and lever voting systems with electronic voting machines

Electronic voting problems/concerns

• • North Carolina 2004 – Voting machine manufacturer said the electronic voting machine could handle 10,500 votes each • Reality – could only hold 3,005 votes • Result – lost 4,530 votes Many voting machines do not create a paper trail – Voting machines used by as many as a quarter of American voters in 2012 can be hacked with just $26 in parts and an 8th grade science education – Vulnerability Assessment Team at Argonne National Laboratory – The experts say the hack could change voting results while leaving absolutely no trace of the manipulation behind

Malice

• • With ill intent people destroy or modify computer systems.

Can be for-profit crime, terrorism, or warfare (this topic will be covered in Tuesday’s class)

Intrinsic & Extrinsic software

• Intrinsic: – Software that is part of a completed product • Extrinsic: – Loaded onto the computer or machine of the user, user directly encounters

Important Case Study:

The Therac-25

• • Radiation therapy machine Typical system malfunctions number 40x day

The Therac-25 case cont’d

• • • 20 month period, overdoses to 6 patients, directly killing 3.

Previous models, 6 & 20 – Differences to the 25?

Chronology of accidents – Please see http://computingcases.org/case_materials/therac /supporting_docs/therac_case_narr/therac_toc.ht

ml

What was wrong with the Therac-25 programming?

• • • 2 modes – X-ray: high intensity beam deflected by tungsten target – Electron: removes tungsten & reduces beam intensity by factor of 100 Quickly changing* (data entry editing) between mode resulted in electron mode not dropping the beam intensity – * If the operator was able to edit and start < 8 seconds Use of a Race condition – 2 or more tasks sharing a variable, order that each is encountered can affect behavior of the program – The Therac-25 no longer had the hardware safety feature

Therac 25

• Problems – No fail safe – No dose reporting – Complicated programming – Re-use of code

• • •

Who had moral and/or legal responsibility?

Harm clearly shown. Was there intent? No – So we consider tort of negligence Defining negligence from law.com

– n. failure to exercise the care toward others which a reasonable or prudent person would do in the circumstances, or taking action which such a reasonable person would not Must prove: – “a) that the party alleged to be negligent had a duty to the injured party-specifically to the one injured or to the general public, b) that the defendant's action (or failure to act) was negligent-not what a reasonably prudent person would have done, c) that the damages were caused ("proximately caused") by the negligence. An added factor in the formula for determining negligence is whether the damages were "reasonably foreseeable" at the time of the alleged carelessness”

Extrinsic Software Failures

• With these examples, companies or individuals are buying software for what it can do for them.

– Disclaiming liability for problems through warranties

Software Warranties

• • • Limiting liability to: – A refund of the purchase price – Repair of the software product Accepting no liability for – Business losses arising out of the use of the product Enforceability of these disclaimers?

– UCC & the Magnuson-Moss Warranty Act – Mix of case law

Extrinsic Business Software Failures

• • • NCR’s Warehouse Manager Warehouse Manager was an inventory program. It was developed for a different operating system than it was deployed for.

The “deadly embrace” NCR continued selling it, claiming 200 successful installs, but that was actually on other op sys.

– When problems reported, told customers that was “unique”

NCR cont’d

• Hopper bought the system. – Did not get honest disclosure from NCR. • NCR sold it to him after product had been discontinued.

– Errors about inventory and pricing resulted.

• $114 item listed for 54 cents; $17 item listed as on sale for $30.

– Hopper was operating a successful co., but after adoption of WM income was half. • Processes took to long & were inaccurate. Inventory inaccuracies

NCR conclusion

• • • Hopper tried to sue NCR for $4.2 million.

However, the sales agreement signed by Hopper stated that in the event of problems, NCR was only responsible for the original cost of the software minus the depreciation of the equipment.

The agreement also had an arbitration clause, so the court refused the case.

ProCD v. Zeidenberg

• • • • ProCD selling mailing list generating software – 2 prices, 1 for personal 1 for business Zeidenberg bought it as personal and created a mailing list sales business using the product.

ProCD sued saying this violated terms of the license • Terms not found on box but on click thru agmt Ct found that Zeidenberg could be held to those terms of the click through agreement and was in violation of the software license.

Mortenson v. Timberline Software

• • • • • Precison Bid software Used it and created bid $1.95 million too low Licensing disclaimed business losses related to use of the software in excess of licensing fee Timberline was aware of bug, did not send fix to Mortenson.

Ct. found the Timberline was not liable b/c licensing agmt. properly limited liability.