Transcript Testing Telecoms Software with Quviq QuickCheck
Testing a Media Proxy with … QuickCheck
Thomas Arts John Hughes Chalmers/ITU Joakim Johansson Ulf Wiger Ericsson
QuickCheck: Properties not Test Cases • Write
general properties
test cases of code instead of prop_reverse() -> ?FORALL(Xs,list(int()), ?FORALL(Ys,list(int()), lists:reverse(Xs++Ys)== lists:reverse(Xs)++lists:reverse(Ys))).
• Test in many, many randomly generated cases
QuickCheck Testing
3> eqc:quickcheck(test:prop_reverse()).
...........Failed! After 12 tests.
[-3,2] A random counter [-3,1] Shrinking..........(10 times) example: Xs and Ys [0] [1] false
NEW
: Automatic simplification of failing cases • Simplification is extremely important— separates the
signal
from the
noise!
Why Did it Fail?
prop_reverse() -> ?FORALL(Xs,list(int()), ?FORALL(Ys,list(int()), lists:reverse(
Xs
++
Ys
)== lists:reverse(
Xs
)++lists:reverse(
Ys
))).
• QuickCheck says: Xs=[0], Ys=[1] • reverse([0,1])==reverse([0])++reverse([1]) ?
• Xs, Ys the wrong way round
QuickCheck in a Nutshell
• Features – Properties, not test cases – Controlled random generation – Automated simplification • Result – Compact, reusable test code – Specification
checked against the code
– Shortcut from property to bug
Property First Development?
• Code code code… quickcheck A Bug!
• Fix it… quickcheck A Bug!
• Fix it… quickcheck… quickcheck quickcheck quickcheck • Code some more…
Ericsson’s Media Proxy
Media Gateway Controller IP network, servers, etc etc Media stream
How hard can it be?
• All we need to do is open and close ”media pinholes” when the controller says… • Megaco H.248 protocol – ITU standard… 212 pages!
– Ericsson Interwork Description… 183 pages!
– Control software… 150KLOC • 20-30K Megaco
Our Strategy
QuickCheck Sends random command sequences Results make sense?
Completely random commands are all just rejected —non-sensical!
Poor test data
Generating Sensible Messages
Message Generators •
Example:
A
media descriptor
contains a list of streams – ASN.1 from the standard (simplified):
MediaDescriptor { streams { ::= SEQUENCE CHOICE oneStream StreamParms, multiStream SEQUENCE OF StreamDescriptor } }
QuickCheck Generator mediadescriptor(Streams) when Streams=/=[]->
{mediaDescriptor, #'MediaDescriptor'{ streams =
case Streams of [{Id,Mode}] ->
oneof(
[
{oneStream,
Records generated by ASN.1 compiler streamParms(Mode)
}, {multiStream,[
stream(Id,Mode)
]}
]
)
; _ ->
{multiStream,
[stream(I,M) || {I,M}<-Streams]
}
end
}}.
stream(I,Mode) ->
#'StreamDescriptor'{ streamID =
I
, streamParms = Message construction
Logic from the IWD
QuickCheck
streamParms(Mode)
}.
Conditions in the IWD
Add Request Desc.
Desc.
Media Stream Desc.
Local control Properties, Package mode M Comment M M O O Multiple Stream descriptors can be included.
LocalControl will be included in all cases except when no media (m-line) is defined in the remote SDP.
The default value of the mode property is “Inactive”, the property is not mandatory if the wanted value is “Inactive”.
StreamParms { ::= SEQUENCE localControlDescriptor LocalControlDescriptor OPTIONAL, localDescriptor remoteDescriptor …, LocalRemoteDescriptor OPTIONAL, LocalRemoteDescriptor OPTIONAL, statisticsDescriptor StatisticsDescriptor OPTIONAL }
Remot e SDP in here
Two Cases: With and Without Remote Media streamParms(Mode) ->
?LET(RemoteMediaDefined, bool(),
if RemoteMediaDefined ->
#'StreamParms'{ localControlDescriptor =
localControl(Mode),
localDescriptor = remoteDescriptor =
Included in this case localDescriptor(RemoteMediaDefined), remoteDescriptor(RemoteMediaDefined)
}
; true -> …… end
)
.
Passed on to ensure an m-line is generated
Generating Sensible Commands
Megaco Commands
Context Context
Megaco Commands
Context Context Termination Context Add
Add
Megaco Commands
Context Context Termination Termination Context
Megaco Commands
Context Termination Context Termination Stream Stream Context Modify Uses termination ID
Megaco Commands
Context Subtract Termination Context Termination Stream Stream Context
Megaco Commands
Context Context Termination Stream Stream Context Subtract
Megaco Commands
Context Context
Sensible Command Sequences?
• Track the
state
of each test case – What calls in progress, which terminations in each call… – Simple to model in Erlang • State machine model –
Preconditions
say which commands are appropriate – –
Postconditions
check command results
Next state
function specifies how commands behave • QuickCheck library – generates valid sequences, tests, and shrinks them – (developed ”just in time”)
Faults
• Proxy was already well tested • 6 days of work, mostly writing generators • 5 faults found: – One in Megaco message encoding/decoding – Four command sequences crashed
Error-provoking Sequences
• Add Mod • Add Sub • Add Add Mod Streams must have two ends Here today, gone tomorrow… …with differing numbers of streams
The Best Error!
Shrinking reduced 160 commands to seven!
Add Add Sub Add Sub Add Sub No one in their right minds would test this!
Due to data corruption here
”Most Likely Bug” Effect
• Once a bug is found, almost
every
QuickCheck run finds the same one!
– (or at least, shrinks to it) • Add
bug preconditions
to the model to avoid it and find the next one – Formulate a ”bug hypothesis” – Test the hypothesis!
– Document it as a precondition
Real bugs or not?
• Most of these sequences are not sent by
Ericsson’s
Media Gateway Controller – Controller and Proxy were tested together!
• But the Interwork Description doesn’t say so!
•
One
of the IWD/the code was buggy!
Using QuickCheck Earlier?
• Testing an older version – Using the
same
generators and model • 9 errors found in 6 hours! – (2 TRs for this version) • Most of the work: writing bug preconditions
QuickCheck is Moreish…
• Follow-up projects under way • Testing non-Erlang systems (the big one!)
Conclusions
• QuickCheck + Erlang – Simple declarative models of the SUT – Concise, maintainable test code • Testing: a great application for FP – Performance irrelevant – No need to commit to Erlang in the product •
A winning combination?