Transcript Some “facts” about software…

Embedded System Testing
Testing
• Consider how testing techniques may
have made your state-machine project
implementation go smoother
– Test button circuit separate
– Test software with debugger (instead of
hardware)
– etc.
Testing
• You have all [undoubtedly] done
software testing
• Much of what we talk about here will be
similar
• Although the differences may be subtle,
they can have huge effects on system
implementation
Differences
• As much as we hate to admit it, more
often than not embedded system
software is ugly
• It’s not that the programmer’s are subpar, but due to the fact that there are
more constraints on the system
• What are some of the constraints?
Differences
• There’s more to the system than just software
• In a pure software system (i.e. PC based
application) we can safely assume that the
hardware is working properly
• In an embedded system we’re never quite
sure
• Thus, we must devise “isolation” techniques
Differences
• Pure software systems (PC based
applications) are, for the most part,
deterministic systems
– They [pretty much] know what to expect and
when to expect it
• Embedded systems generally must react
to events at arbitrary times
– Example?
• How does this affect testing?
Differences
• PC based applications spend a lot of
time sitting around waiting for the user
to do something
• Many embedded systems must operate
in real-time
Differences
• PC based applications have an
operating system to guide them along
– This can be good in that the OS does a lot
of bookkeeping for us
– This can be bad in that the application is
sharing the underlying OS and hardware
with other applications
• Embedded systems may or may not
have an OS
Difficulties
• Real-time and concurrency are often
difficult to design and difficult to test
• Resource constraints (speed, size, …)
are often difficult to design and test
• Testing tools are intrusive
• Reliability is absolutely critical
• All of these “traits” exist within an
embedded system design
Bottom Line…
PC Based Testing
Embedded Systems Testing
System under test
In-Circuit
Emulator and Logic analyzer
monitor
Oscilloscope
Why Test?
• Find bugs in both software and
hardware
• Reduce product development and
maintenance costs
• Improve system performance
• Ensure product acceptance/success in
the market place
Finding Bugs
• It is theoretically impossible to prove [in the
general case] that a program is correct
– Proved with “The Halting Problem”
• But, testing can prove that a program is
incorrect
• The goal of testing is to detect all the possible
ways that a system might do the wrong thing
• Don’t fall into the trap of telling yourself “the
system is simple, there’s no need to test it”
Cost Reduction
• We saw this chart previously…
System specification
and design
Hardware and software
design/debug
Prototype
debug
System
test
Cost to fix
Design cycle
• And you can now tack on “after
deployment” to the end of the graph
Performance Improvement
• Embedded systems have numerous
constraints of which two primary ones
are
– Time
– Space
• Testing for performance addresses
these constraints by
– Optimizing code
– Driving the partitioning decision
Market Acceptance
• Embedded systems don’t sit in a sterile lab
environment
• By their vary nature the exist in the harsh,
real world
• The systems must work as specified
• Thus, many agencies demand thorough
testing prior to introduction into the market
– DOD, FAA, FCC, QVC…
• And, if the agencies don’t get you, the
consumer will
When To Test?
• Testing is done in phases
– Design phase
• Prototypes
– Development phase
• Unit tests
– Integration phase
• Regression testing
• More often than not, testing within the design
and development phases is down played if it
exists at all
Design Phase
• Software developers write snippets of
code to test algorithms
• Hardware developers “burn” HDL
designs to FPGA’s to test
algorithms/circuits
• These tend to be early “proof of
concept” works and may or may not end
up in the final design
Development Phase
• Unit tests are generally designed and
implemented by the designer of the unit
(module) under test
• Focus is on the operation of the module
• Tests may or may not have anything to do
[directly] with the final integrated product
– The fact that a module works in isolation in no way
guarantees that it will work within the system
– Example?
• But, unit tests are important
– When integration begins it’s nice to know that the
pieces work individually
Integration Phase
• This is where “testing to specification” begins
– Tests relate directly to the customer specification and may
involve environmental testing, temperature range testing…as
well as “does it do what it’s supposed to do under all
circumstances” testing
• The goal is to find out if the system works the way the
customer expects it to work
• Method of choice is “regression testing”
– This involves a suite of tests that are all run from beginning
to end whenever a change is made
• Quite often these tests are run by an organization
other than the design team
– A method of “keeping everyone honest”
Testing Techniques
• Black-Box testing (Functional Testing)
– The test designer does not get to see how the system is
implemented when designing/running the tests
– These concentrate solely on the system (functional) specification
• White-Box testing (Coverage Testing)
– The test designer knows exactly how the system has been
implemented and designs tests with breaking it in mind
• Gray-Box testing
– These are tests where you have suspicions about how a system
may be implemented based on past experience
– You hammer away at what you know were weak spots in similar
systems
– Competitors will do this as a means of reverse engineering your
product
Testing Techniques
• Functional tests are the most difficult to
design (this is where the halting problem
comes to play)
– Unfortunately, they’re probably the most important
• Coverage tests are easy to implement as there are
automated tools for doing them
– But if the functional tests are truly complete and
the code is not fully covered, who cares?
When To Stop Testing?
• Coverage tests are easy (easier)…when all the code
has been executed through testing (and the tests
pass), you’re done.
• Functional tests get trickier
– For better or worse, your customer may tell you when you’re
done
– With each round of regression testing you may analyze the
trends and determine when you’re done
– Bottom line is you’ll never by 100% sure your system is
correct [in the general case] so you play the “statistics
game”…and hope you come out a winner
– It ultimately becomes a money issue
How To Test?
• Obviously, testing every possible input
and every possible combination of paths
through the system and under every
possible environmental [external]
condition will ensure success
• Equally obviously, you can’t do this
• So, once again you play the “statistics
game”
Functional Test Categories
• Stress tests
– Try to break the system by flooding input
channels with signals, overloading
memory, reducing power supply inputs,
dropping the device on a hard surface…
• Boundary value tests
– Check the extreme boundaries on input
systems
– Especially useful for input power supplies
Functional Test Categories
• Exception tests
– Provide inputs that you know will cause failure
conditions and see how your system handles them
– Remember, this is an embedded system and a
BSOD may not be a suitable option
• Error guessing
– Figure out how to break a competitor’s system (or
your previous version) then check to see how your
system handles the same situation
Functional Test Categories
• Random tests
– Use a Monte Carlo simulation to generate input
vectors then analyze the outputs/behaviors to
check for consistency
• Performance tests
– The definition of “a working system” may have
time/size/environmental constraints as well as
“getting the right answer” constraints
– These should be spelled out in the specification
and, therefore, included in the functional testing
Functional Test Categories
• And don’t forget to test those cases that
“you know will never occur in the real
world”…
• …because they will!
Coverage Test Categories
• Statement coverage
– Design tests to execute every line of code at least once
• Decision or branch coverage
– Make sure every condition of/path through a conditional
(if/then/else or switch) statement is executed
• Condition coverage
– Make sure every piece of a compound condition within a
conditional statement is tested
– Consider this gem…(if with no else)
• These clearly require white-box testing and may need
external devices to force certain conditions
Failure Modes
• Always crashes (does the wrong thing) in the same place
– Relatively easy to detect and correct
– “Standard” techniques usually suffice
• Crashes (does the wrong thing) at seemingly arbitrary places
– Very difficult to detect and correct
– Problems in this category are generally due to timing (“race
conditions”), sequencing (ordering) of events, or number (size) of
events
– Recreating these conditions is often very difficult and time
consuming, if not impossible
– H/W – S/W co-verification [discussed previously] may be helpful
here
– These often lead to redesigns in either the software, hardware, or
both
Instrumentation For Functional
Testing
• In general, you don’t get System.out.println()
functionality
– If you do [via a monitor program like the Stamp IDE]
then you’re not testing the “real” system
• You may resort to hardware devices (logic
analyzers, oscilloscopes, ROM emulators…)
but these things may add loading [i.e.
capacitance] to a system which either hides the
error or causes a different error
Instrumentation For Coverage
Testing
• Three categories (as specified previously)
– Statement Coverage – instrumentation to verify that all
statements are exercised
– Decision Coverage – instrumentation to verify that all paths
through all conditionals are exercised
– Modified Condition Decision Coverage – instrumentation to
verify that all terms of complex conditions are exercised
• Basically, the instrumentation consists of memory
“spies”, logic analyzers, and PC side software to
analyze the recorded results
• It’s all statistics gathering
• The amount of data gathered can be staggering –
consider a recursive function call
Instrumentation For Performance
Testing
• Again, it’s done with external devices
– Logic analyzers and oscilloscopes are set
to catch triggering events and measure the
time to respond, etc.
Testing Is Hard
• In all three cases (functional, coverage,
performance) we’re dependant on
external devices that may alter the
functionality of the embedded system
• In a PC based application, we see a
similar situation, but not as often, when
we run a debug build vs. a release build
Remember…
• It’s safe to assume that the intentions of
the designer were good
• It’s equally safe to assume that the
implementer fouled up the
implementation
• Even if they’re the same person
• Therefore, test even the simplest of
things!
An interesting example…
Button circuit
Servo circuit
• To save space, P14 was used for both the button and the servo
• It appeared to work (some/most of the time) and since the
buttons are fraught with issues such as bounce and
synchronization (with the software) it was easy to write off as an
issue with the BasicStamp subsystem
An interesting example
• But, what was the real problem?
– Why did it appear to work sometimes and not
others?