Transcript Benchmarking Process as a Part of Risk Identification

Benchmarking process as a part of
risk identification
06.02.2007
Author: Marita Harju
Supervisor: Docent Timo O. Korhonen
Instructor: BSc. Jari A. Juntunen
1
© 2005 Nokia
Thesis_presentation.ppt / 2007-02-06
Content
• Thesis context and objectives
• Process definition
• Implementation
• Conclusion
2
© 2005 Nokia
Thesis_presentation.ppt / 2007-02-06
Thesis context
• Benchmarking
• Definition: Continuous process of measuring products, services, and practices against
the company’s toughest competitors (Juran 1999)
• Effective method of establishing performance goals and quality improvement projects
based on industry best practices
• Can be used to identifying gaps between the own product and the competitors,
helping to identify what must be changed and to showing what is possible
• Performance testing
• The goal of performance tests is to evaluate the compliance of a system or
component with specified performance requirements (BS 7925-1 1998). Level of
performance is the degree to which the needs are satisfied, represented by a specific
set of values for the quality characteristics (ISO/IEC 9126-1 2001).
• Usability
• The degree to which the software is easy to use (ISO 9126 standard 2001)
3
© 2005 Nokia
Thesis_presentation.ppt / 2007-02-06
Thesis objectives
•
The purpose of this Thesis was to
1. define benchmarking process for mobile devices
2. find out how the benchmarking process can be used as a part of risk and fault
identification
•
Research questions:
1. “How to select main use cases for benchmarking in question?” The follow-up for
this question is to consider how to create relevant test assets according to use
cases.
2. “How benchmarking can be used as a part of risk and fault identification?”
•
The objectives are discussed in scope of selected quality attributes:
1. performance (main attribute)
2. usability
4
© 2005 Nokia
Thesis_presentation.ppt / 2007-02-06
Process definition
•The process is defined for
benchmarking of S60 based
mobile devices against
competitors
•The process is based on
formal benchmarking
process and fundamental
testing process
5
© 2005 Nokia
Thesis_presentation.ppt / 2007-02-06
Implementation: Test setting
• Testing scope
• Whole phone
• Quality attributes: Performance & usability
• Phones
• Nokia E60
• Motorola RAZR V3
• i-mate JAMin
6
© 2005 Nokia
Thesis_presentation.ppt / 2007-02-06
Implementation: User data & test execution
• User profile was defined using data
from S60 device user panel
• Use cases were defined according to
user profile
• Main use cases = the most used 20
percent of all use cases
• Test cases were defined according to
use cases and user profile
• In this case, only PIM (Personal
Information Managements) and
messaging related cases are
developed
• Test execution
• Performance: Response times
• Usability: Heuristic evaluation
7
© 2005 Nokia
Thesis_presentation.ppt / 2007-02-06
voice calls
15%
productivity
1%
pim
29%
unknown
1%
utitity
4%
browsing
4%
config
6%
games
2%
multimedia
9%
infotainment
0%
messaging
29%
Phone usage activities by average S60 phone user
Implementation: Performance test results, PIM
Test ID
Test case name
Data
Unit
Nokia E60
Motorola V3
i-mate
JAMin
PIM
8
1
Opening calendar application
255 meetings
s
1.4
0.1
0.6
2
Opening clock application
-
s
1.5
0.1
0.8
3
Opening logs application
20 logs
s
0.1
0.1
0.5
4
Opening phonebook application
200 items
s
0.5
0.6
0.6
5
Saving a new contact to phonebook
200 items
s
0.5
4.7
0.6
6
Calendar view switch from month view
to day view
255 meetings
s
0.6
-
0.1
7
Calendar view switch from week view
to month view
255 meetings
s
0.6
0.8
0.1
8
Calendar view switch from day view to
week view
255 meetings
s
0.8
0.6
0.6
9
Opening calendar item
255 meetings
s
1.2
0.1
0.1
10
Initializing a phone call using
phonebook application
200 items
s
3.2
3.4
4.4
11
Initializing a phone call using logs
20 logs
s
3.4
2.9
4.1
12
Opening a contact item
200 items
s
1.0
0.6
0.6
© 2005 Nokia
Thesis_presentation.ppt / 2007-02-06
Notes
Implementation: Performance test results, messaging
Test ID
Test case name
Data
Unit
Nokia E60
Motorola V3
s
1.4
0.7
i-mate
JAMin
Notes
Messaging
13
Opening messaging application
Opening a received SMS from idle
14
Creating a new SMS message
15
Deleting the first SMS message from inbox
16
Deleting the first MMS message from inbox
17
Opening a received SMS message
18
Opening a received MMS message
19
Opening messaging inbox
20
Adding image to a MMS message
21
Creating a new MMS message.
22
9
© 2005 Nokia
Thesis_presentation.ppt / 2007-02-06
40 SMS + 5
MMS
0.1
new SMS and
other 40
SMS +
5MMS
s
3.0
1.0
0.7
40 SMS + 5
MMS
s
1.2
0.8
0.8
40 SMS + 5
MMS
s
0.6
2.5
0.1
40 SMS + 5
MMS
s
0.8
2.2
0.1
40 SMS + 5
MMS
s
1.4
0.8
1.0
40 SMS + 5
MMS
s
2.8
4.3
2.3
40 SMS + 5
MMS
s
0.1
1.1
0.1
40 SMS + 5
MMS
s
2.6
2.7
0.9
40 SMS + 5
MMS
s
1.7
0.7
0.8
i-mate opened
inbox view,
not the
SMS
Implementation: Result summary & next steps
• Performance results are summarized using
performance index (calculated from
measured response times)
• Usability evaluation was done using heuristic
evaluation (scores from 0 to 5). Summarizing
is done by calculating averages.
• Results in short:
• Poor message opening performance in S60
device (E60), especially SMS opening
performance is weak
• S60 device’s usability is in average level
Nokia
E60
Motorola
V3
0.6
0.8
0.7
Average
messaging
index
0.4
0.5
1.0
Overall
average
index
0.5
0.6
0.8
Average PIM
index
i-mate
JAMin
Usability, score scale 0-5
Feature
Nokia
E60
Motorola
V3
i-mate
JAMin
Score
Score
Score
Calendar
3.0
2.0
5.0
Clock
5.0
1.0
2.0
Phonebook
5.0
3.0
2.0
• Performance: More testing on SMS area
Logs
2.0
5.0
2.0
• Usability: More thorough evaluation with
several evaluators
PIM average
3.8
2.8
2.8
Messaging
3.0
4.0
2.0
Total (0.51 x
PIM+ 0.49 x
messaging)
3.4
3.4
2.4
• Next steps
10
Performance index, scale 0-1
© 2005 Nokia
Thesis_presentation.ppt / 2007-02-06
Conclusion
1/2
The purpose of this Thesis was to
1. define benchmarking process for mobile devices
2. find out how the benchmarking process can be used as a part of risk and fault identification
1. The test execution according to the defined benchmarking process proves that
in a general level the process is working
2. It seems that the process is able to fulfill its main target; it can find gaps and
therefore is able to identify improvement areas, risks and faults in S60 devices
11
© 2005 Nokia
Thesis_presentation.ppt / 2007-02-06
Conclusion
2/2
Research questions:
1. “How to select main use cases for benchmarking in question?”. The follow-up for this question is to consider
how to create relevant test assets according to use cases.
2. “How benchmarking can be used as a part of risk and fault identification?”
1. Main use cases are the most used 20 % of all use cases. Test asset, including
e.g. test cases, is defined according to use cases and user profile (end user
point of view).
•
The selection to use S60 device based user profiles as a basis for main use case
definition makes the cases a bit S60 device specific, but…
•
•
Results are easy to compare, because the same profile is used for all products
It is possible to establish quality goals to S60 devices in their main use areas according to
results obtained for other device.
2. Several iterations of the benchmarking process is the answer to the question
how the benchmarking process can be used as a part of risk and fault
identification.
•
•
12
In the first iteration the selected quality attribute of the device is tested as a whole
If there are some gaps found in the monitoring results- step, those gaps can be then
further investigated in the second iteration of the process
© 2005 Nokia
Thesis_presentation.ppt / 2007-02-06
Questions?
13
© 2005 Nokia
Thesis_presentation.ppt / 2007-02-06
References
• Juran, J.M., 1999. Juran’s Quality Handbook. McGrawHill.
• British Standard (BS) 7925-1:1998. Software testing Part 1: Vocabulary.
• ISO/IEC 9126-1: 2001. Information technology- Software Product Quality – Part
1: Quality model.
14
© 2005 Nokia
Thesis_presentation.ppt / 2007-02-06