Transcript Session Title

Testing and go-live best
practices for a successful
dashboard rollout
Dr. Berg
Comerit Inc.
© 2011 Wellesley Information Services. All rights reserved.
What We’ll Cover …
•
•
•
•
•
•
•
•
•
Background
Planning for the Xcelsius Dashboard Initiative
The JAD, RAD, Agile (XP) or ASAP methodologies
Getting the Right Dashboard Requirements
Upgrade and Tool Migrations
The Xcelsius Testing Approach
Big-Bang Vs. Gradual Go-lives
The BI Support Organization
Wrap-up
1
Go-Live Planning for the Xcelsius Dashboard Initiative
•
During the planning phase you should also start planning your go-live
strategy. This include answering the following questions:
•
Where are my users located?
 What
is the network capacity?
 Do I need support people for extended time periods - different time zones?
 Do I need multi-currency support on my dashboards?
 Do I need multi-language support?
 What type of users do I have in each region?
Create a user map as part of your
project planning. This will help you
understand your user base
2
User Training Options
There are four core options for the training strategy:
•
Classroom training

•
On-line training

•
Source: dashbaord insights, 2009
Best when users are dispersed, dashboards are simple or go-live over long time
period
Train the trainer

•
Best when users are similar and centrally located
Best when users resides in many locations, multiple languages are involved and
when there is a very high number of users
One-on-one training

Best for executives and senior management. Should be
done at each user's office.
Communicate and schedule training early in
the project, so that everyone will be available
3
User Training for Complex Dashbaords
•
•
If users need training you have often failed to create a good dashboards.
However, there are times when training are needed this include:

Interactive dashboards and dashboards with complex graphing
In this dashboard,
users can budget for
travel categories, for
each month, and also
save scenarios.
Some training is
therefore required
and should be
planned early.
4
Plan for an On-Line Help System for Your Dashboard Go-Live
•
On-line help should be created for each dashboard
The on-line help system should
explain
 how number are calculated,
 how to read graphs
 what functionality is embedded
5
How to Create On-Line Help Systems
•
Some ways to create a low-cost online help system Include:
1.
Flash files
Create a simple help dashboard & embed this
into the overall dashboard
2.
Word
Create a word document with screenshots.
Save it as HTML and store the web pages on a
web server.
You can then link the URL on your dashboard.
3.
Custom Application
Use a tool like front-page or any web authoring
tool and create a complex on-line help system
with menus, search functionality, movies that
shows demonstrations etc.
On-line help centers can include
contact information, training schedules.
They can also be used to communicate
information on future projects
6
What We’ll Cover …
•
•
•
•
•
•
•
•
•
Background
Planning for the Xcelsius Dashboard Initiative
The JAD, RAD, Agile (XP) or ASAP methodologies
Getting the Right Dashboard Requirements
Upgrade and Tool Migrations
The Xcelsius Testing Approach
Big-Bang Vs. Gradual Go-lives
The BI Support Organization
Wrap-up
7
The JAD, RAD, Agile (XP) or ASAP methodologies
•
There are several options for the Xcelsius Dashboard project




•
Joint Application Design (JAD)
Rapid Application Development (RAD)
Agile, or Extreme Programming (XP)
Accelerated SAP Methodology/ System Development Life-Cycle
(SLDC)
Many of the methodologies are not appropriate for the
dashboard development effort
Pick your Xcelsius methodology carefully.
Do not use ASAP unless you project is part of a
budgeting, consolidation or planning effort.
8
The "Waterfall Methodologies" are Not Good for Dashboards
The System Development Life Cycle (SDLC) methodologies, such as
ASAP, are known collectively as "waterfall methodologies".
They give a false sense of clear cut stages and does not address
substantial functionality changes during development. It is hard to fix
missing functionality during integration test.
The waterfall
Examples for Accelerators:
• Project Plan, Estimating
• Design Strategies, Scope Definition
• Documentation, Issues Db
Fill
Fillin
inthe
theBlank
Blank
Versus
Versus
Start
from
Scratch
Start from Scratch
• Workshop Agenda
• Questionnaires
• End-User Procedures
• Test Plans
• Technical Procedures
• Made Easy guidebooks (printout, data transfer, system
administration…)
The challenge with ASAP is that users don't know
what they want until they see it...
9
The SAP ASAP Methodology Overview
Integration
Testing
Create Technical
specs
No
Create Functional
specs
System Testing
Complete?
No
Yes
Unit Testing
Complete?
Yes
Configuration
Yes
Peer Review
No
Approved?
Peer Review
Yes
No
Complete?
Yes
Approved?
Structured
walkthrough
No
No
Complete?
Yes
Structured
walkthrough
10
Joint Application Design (JAD) - Who participates?
1.
2.
3.
4.
5.
6.
Facilitator – Facilitates discussions, enforces rules
End users – 3 to 5, attend all sessions
Developers – 2 or 3, question for clarity
Tie Breaker – Senior manager. Breaks end user ties, don’t attend
Observers – 2 or 3, do not speak
SMEs
– A few subject matter experts (SME) for understanding
business & technology
Keep it very focused and explore the interfaces. How do the user want
to see the screen layouts and functionality?
A study of 60 development projects and found that
without JAD, 35% of the functionality was missed
(Source: Caper Jones)
11
Rapid Application Development - RAD
RAD has an abbreviated blueprinting phase where meetings are executed in short
succession to get the requirements. Most of the blueprinting and realization
phase of the project are combined.
The first meeting: a one or two days work session with
uninterrupted time
Who: Power users, casual users, people who today interact with the current
system and managers who have a stake in the outcome of the dashboards
How many: A rapid pace is kept in these meetings and the number of attendees is
kept at no more than 7 people in attendance.
The coordinators should focus on shared information needs and conduct multiple
sessions (typically once a week)
Why RAD?.. Increase involvement, less business disruption, less
opinions, more consensus, information sharing and an education event.
12
Agile and Extreme Programming (XP) for Xcelsius Dashboards
XP was started by programmers who decided that the
traditional requirements gathering sessions took too much
time and often just verified what they already knew.
The argument for XP is that other methodologies were
developed to build software for low levels of change and reasonably predictable
outcomes.
But, the business world is no longer very predictable, and software
requirements change at extremely high rates.
Development can be completed faster with collaborative efforts of
paired programmers with small 'sprint' timelines and many go-lives'.
The core premise of XP is that you can only pick 3 out
of these 4 dimensions: cost, quality, scope, time.
13
Framework for picking your Dashboard Methodology
When to Select Different Methodologies
High
Joint Application Design
(JAD)
System development Life-Cycle
based methodologies
(SDLC)
Extreme Programming
(EP)
Rapid Application Development
(RAD)
Time to
Delivery
Low
Low
High
Impact of Failure
Source: Dr. Berg,
DM Review, 2006
14
What We’ll Cover …
•
•
•
•
•
•
•
•
•
Background
Planning for the Xcelsius Dashboard Initiative
The JAD, RAD, Agile (XP) or ASAP methodologies
Getting the Right Dashboard Requirements
Upgrade and Tool Migrations
The Xcelsius Testing Approach
Big-Bang Vs. Gradual Go-lives
The BI Support Organization
Wrap-up
15
The Xcelsius Business Requirements
Business requirements can be collected in a variety of ways based on
the methodology that the company employs. It is a complex process
and involves a period:




Discovery and Education,
Formal communication,
Prototypes and Reviews
Final approvals.
A dashboard implementation does not simply
involve a series of black-and-white technical
decisions; just because something is
technically feasible does not mean it is wise
or desirable from a business perspective.
Source: Gooy_GUI, 2007 16
Where to you start? - First Alternative
You can start with
start with a blank
template and fill-in
the capabilities
Focus on graphs,
layout, measures
& navigation.
One method is to
write story-boards
from a user
perspective and
add needed
functionality to
support this
Where to you start? - Second Alternative
Get a group of 5-7 people for a
brainstorming session.
Draw the solution, knowing
that it may look somewhat
different once developed.
Focus of the use of space, graphs,
navigation, available data and the
purpose of the dashboards.
Do not design fixed format 'reports'
Building a Mockup in Excel
•
If you can make a 'mockup' in Excel, users can see what it may look like
very quickly. You do not need to have any BOBJ tools installed.
This can be done in 30-60 minutes
19
Prototyping The Dashboard Requirements
•
Once the first day of brainstorming is completed, you can create data in
Excel and prototype the solution in Xcelsisus. Focus on layout, space
management, colors and basic formatting.
Plan for multiple weekly prototypes before you get
the solution that everyone can agree on.
20
Flexible Options to Meet Many Requirements
There are often disagreement on how to present numbers and graphs.
Make your
dashboard flexible
and present data
in many
interactive ways.
Amount Vs. Percentages
Different graphs
Users can select what
they want graphed
21
What We’ll Cover …
•
•
•
•
•
•
•
•
•
Background
Planning for the Xcelsius Dashboard Initiative
The JAD, RAD, Agile (XP) or ASAP methodologies
Getting the Right Dashboard Requirements
Upgrade and Tool Migrations
The Xcelsius Testing Approach
Big-Bang Vs. Gradual Go-lives
The BI Support Organization
Wrap-up
22
Tool Upgrades and Deployment of New Xcelsius Functionality
•
When you upgrade your Xcelsisus dashboards consider a two step
approach:
 Conduct a developer training or workshop session to learn the new
functionality and agree on the upgrade timeline
 Do a technical upgrade over the weekend
 Copy all of the dashboards
 Implement new functionality on the copied dashboards
 Have a formal feedback session with user involvement to see how
they like the changes
 Implement the new changes 2-6 weeks after the technical upgrade
Stability is the key to success in all end user systems.
Even if the new functionality is better, users do not like
a system that changes look and feel frequently.
23
Tool Migration Strategy
•
If you are migrating users from a legacy reporting tool to your new
dashboards you need a formal migration strategy. This could include:




Maintaining two systems (not recommended)
Running two systems in parallel for short time to reconcile results
Remove legacy reporting tool as part of go-live (recommended)
When Vikings settled new lands, they always burned the boats so that
the settlers was 100% committed to the new situation. While it caused
some anxiety, this is a great migration strategy that can be used for
system migration efforts as well.
A "burn the boats" reporting migration strategy assures high
commitment to the new solution and possibly a high number of
new requirements after go-live. Be prepared to support this...
24
What We’ll Cover …
•
•
•
•
•
•
•
•
•
Background
Planning for the Xcelsius Dashboard Initiative
The JAD, RAD, Agile (XP) or ASAP methodologies
Getting the Right Dashboard Requirements
Upgrade and Tool Migrations
The Xcelsius Testing Approach
Big-Bang Vs. Gradual Go-lives
The BI Support Organization
Wrap-up
25
The Dashboard Test Methodology
•
Dashboards testing should follow a formal
methodology
Test Strategy
Test Plan
Test Execution
Problem Resolution
The test strategy is written at the
beginning of the project and should
include:
 What will be tested
 Who will test it and approve it
 Where will the test occur
 When will the test be scheduled
The test plan is more detailed and should be
written once the first prototype is built
26
Area
Theme
The Xcelsius User Acceptance
Testing (UAT) Form
Dashboard Name:
By requiring each of the 5-7
UAT members to complete
a form for each dashboard,
you get solid feedback that
you can use in the next
RAD development cycle
Change
Wanted
Description
Background color &
Image
Button layout &
locations
Link names &
locations
Chart and Table
names & location
Table location
Table and font size
Tables
•
Approved
Colum order
Sort ability
Summary row
layout & colors
Graph type
Graph location
During each UAT test cycle,
you should solicit detailed
feedback on layout, graphs,
theme, tables and
navigation.
Graph colors
Ability to change
graph type
Ability to select
fields
Usefulness of fields
displayed
Drilldown to more
detail
On-line help
Navigation
•
Graphs
Graph labels & size
Print
Save
Single sign-on
Other
Comments:
This form is also on
your memory stick
Approvers Name:
Approvers Signature:
27
The Xcelsius Load Testing Form
•
The load testing is
intended to show any
performance issues prior
to the go-live.
Dashboard Name:
Estimated number of named users
Concurrent
User Type
Number
Factor
Users
Casual Users (less then 2-5 times monthly)
20%
Power Users (daily)
40%
Executive Users (2-10 times monthly)
25%
Total:
Load Testing
•
•
While all areas cannot be
tested, this will give you
an idea on bottlenecks.
For large scale go-lives,
you should consider
having test PCs in
multiple locations
This form is also on
your memory stick
Test lab setup
Number of PCs required
(concurrent users divided by 5)
Batch script created (i.e. Java script)
5 copied of script installed on each PC
Execution
Start scripts on each PC
(continuous loop)
Pass
Issue
Findings
Monitor BOBJ BI Server Load
Monitor BW App sever load
Monitor BW Server load
PC load time
(average of 10 dashboard loads)
Comments
28
The Xcelsius Stress Testing Form
•
•
•
•
Stress testing is very similar
to load testing
The difference is that the
number of concurrent users
are expected to be doubled
Not all companies will use a
stress test, nor pass it. This is
due to the unrealistic
concurrent load and the high
hardware costs of meeting
them.
However, it provides very
useful information
Dashboard Name:
Estimated number of named users
Concurrent
User Type
Number
Factor
Users
Casual Users (less then 2-5 times monthly)
40%
Power Users (daily)
80%
Executive Users (2-10 times monthly)
50%
Total:
Stress Testing
Test lab setup
Number of PCs required
(concurrent users divided by 5)
Batch script created (i.e. Java script)
5 copied of script installed on each PC
Execution
Start scripts on each PC
(continuous loop)
Pass
Issue
Findings
Monitor BOBJ BI Server Load
Monitor BW App sever load
Monitor BW Server load
PC load time
(average of 10 dashboard loads)
Comments
This form is also on
your memory stick
29
Xcelsisus Dashboard Test Planning
Key Activities:
Tasks
1
2
3
4
5
Create test script
Identify roles to be used
Documentation on using test tools
Procedure for documenting test results
Training sessions for using test scripts
6
7
8
9
Identify key contacts
Communicate about transports
Arrange time for progress control
Schedule facilities
The business analysts are responsible for planning,
coordinating, and executing the testing of dashboards
30
Deliver
Cost and Profitability
Order
Manufacturing
Plan and scheduling
Demand planning
Source
Environment
preparation
4/2
4/1
3/31
3/30
3/29
3/28
3/27
3/26
3/25
3/24
3/23
3/22
3/21
3/20
3/19
3/18
3/17
3/16
3/15
3/14
3/13
3/12
3/11
3/10
3/9
3/8
3/7
3/6
3/5
3/4
3/3
3/2
3/1
Large-Scale Xcelsisus Test Scheduling: Example
Resolving
outstanding
issues and retesting
= Morning session 8:30 - noon
= Evening session 12:30 - 5:00
•
Each UAT team could have dedicated time in the test room
•
Provide food and snacks
•
At least two testers (preferably more) should be assigned to
test each functionality
All test results should be logged so that fixes do not
impact other dashboards and consistency is maintained
31
What We’ll Cover …
•
•
•
•
•
•
•
•
•
Background
Planning for the Xcelsius Dashboard Initiative
The JAD, RAD, Agile (XP) or ASAP methodologies
Getting the Right Dashboard Requirements
Upgrade and Tool Migrations
The Xcelsius Testing Approach
Big-Bang Vs. Gradual Go-lives
The BI Support Organization
Wrap-up
32
Big-Bang Vs. Gradual Go-lives
•
There are many ways to do a gradual rollout of the dashboards
•
The simplest way is:



Bring all content into the production box,
Setup the roles in production environment
Gradually release roles to the end users.
•
The benefit of doing this is that you can see how the system performs
and solicit feedback from an increasingly large user group, before you
release the dashboards to many users.
•
A gradual go-live can reduce the potentially negative impact on the
organization.
Unless the dashboards are for a single department, or very
few users, you should always plan for a gradual go-live
33
Gradual Go-lives by Region
•
A gradual go-live by region makes sense if:
 Training is needed
 Multi-language support is required
 A high number of users are expected
 Dashboards are tailored to local needs
•
When you use a regional go-live strategy, you can also roll out the
dashboards to power users first (i.e. one month before go-live) to see
how the system works in a real production setting
Global dashboards require serious attention to language support, user
support, 24/7 availability and attention to non-native English speakers.
34
Xcelsius Dashboard Go-lives by Organization Units
•
A gradual go-live by Organization Units makes sense if:
 Organization is very large
 Departments have very different needs
 Dashboards are tailored to dept. needs
 Data must be secured between organizations
•
When you use an organizational go-live strategy, you need to focus on
the branding of each dashboard (i.e. logos of subsidiaries) and on
integration of support functions in their organizations (i.e. helpdesk).
Departmental dashboards should have first level
support in their respective business units.
35
What We’ll Cover …
•
•
•
•
•
•
•
•
•
Background
Planning for the Xcelsius Dashboard Initiative
The JAD, RAD, Agile (XP) or ASAP methodologies
Getting the Right Dashboard Requirements
Upgrade and Tool Migrations
The Xcelsius Testing Approach
Big-Bang Vs. Gradual Go-lives
The BI Support Organization
Wrap-up
36
BI Support Organization — Big Picture
You need to separate the operations of BI systems from the project work
If there is no support organization, the BI system quickly becomes an
orphan when the project ends
Without a
support org. there
is a risk that
future BI projects
are delayed since
the project team
has to support
previous
projects
An Example of a Large BI Support Team
Note: Job areas are meant for
illustrations, and will vary
depending on the BI
applications supported
Dashboards
i.e. Xcelsius
Full-time
BI Query
i.e. WebI, BEx
Full-time
Support leader
Full time
Helpdesk,
user support
SAP BI Basis
Data loads & fixes
Full-time
Full-time
Data quality &
data resource mgmt.
Data loads & fixes
Full-time
Training,
user support
Full-time
Full-time
Full-time
Portal, collaboration,
KM, security
Full-time
This large team can support complex applications, cockpits, BI
portals, and broadcasting while providing training and help desk
support as well as on-going data warehousing production support.
38
What to Include in a BI Dashboard Service Level Agreement (SLA)
1.
When must data stores be loaded by (time)
 What will happened if a persistent problem occurs (“swat” teams)?
 Who is responsible for fixing process chains and who pays?
 Do you get a discount for each DataStore that is not loaded in time?
2.
How should Service Packs and Fix Packs be applied
 When will SP, FP and SAP Notes be applied?
 Who pays for it?
 Who is responsible for testing them?
3.
When will the BOBJ system be upgraded
 When will upgrades occur, how is the pricing determined?
 Who pays for it and who is responsible for testing?
 How long can the system be off-line?
4.
Minimum uptime and target uptime
 What is uptime defined as (data store loaded vs. queries available vs. security
fixes applied vs. portal uptime vs. third-party reporting tool uptime vs. network
uptime vs. Xcelsius issues, etc.)?
 What are the penalties (money) for missing the dashboard uptime requirements?
39
What to Include in a Dashboard BI SLA (cont.)
5.
6.
7.
8.
Issues log
 What issues must be logged?
 Who owns the log? Do you have access?
 Can entries be updated, or must an audit trail be preserved?
Backup and disaster recovery
 What is included in the backup and when is it taken?
 When will restore abilities be tested?
 How fast must restore occur, and what data stores and users will first have
access (priority list)?
Who owns the data
 If you switch vendors, who owns the data?
 How will you get access to the data? Do you get full insights to all?
 Who, of the vendor’s employees, gets access to your data? Can they share it
with your competitor?
Service tickets
 When will service tickets be monitored?
 What are the categories and who will resolve them?
 What are the resolution process and timelines?
 How are customer and support satisfaction measured?
40
What to Include in a BI Dashboard SLA (cont.)
9.
Escalation process
What
will happened if an issue cannot be
resolved by the internal IT department/
vendor and your business SLA manager?
What
are the steps needed to terminate the
SLA contract and are there any
payments/fault payments or budget recourse
(i.e., move money from cost centers)?
The more details you put into the dashboard SLA upfront, the easier it will be to measure and the more
likely you are to have a successful relationship
41
Reasonable Xcelsius Dashboard SLA Performance
Examples of reasonable performance targets:
90% of all dashboards run under 20 seconds
System is available 98% of the time
Data loads are available at 8am — 99% of the time
User support tickets are answered within 30 minutes
(first response)
 User support tickets are closed within 48 hours — 95% of the time.
 System is never unavailable for more than 72 hrs — including
upgrades, fix packs, service packs, and disaster recovery
 Delta backups are done each 24 hrs cycle and system backups are
done every weekend




What We’ll Cover …
•
•
•
•
•
•
•
•
•
Background
Planning for the Xcelsius Dashboard Initiative
The JAD, RAD, Agile (XP) or ASAP methodologies
Getting the Right Dashboard Requirements
Upgrade and Tool Migrations
The Xcelsius Testing Approach
Big-Bang Vs. Gradual Go-lives
The BI Support Organization
Wrap-up
43
Resources
•
Project Management Metrics, KPIs, and Dashboards: A
Guide to Measuring and Monitoring Project Performance
by Harold Kerzner, Aug. 2011
•
Agile Project Dashboards - Bringing value to
Stakeholders and top management (What can you do for
your PO Today?) by Leopoldo Simini, July 2011
•
Key Performance Indicators: Developing, Implementing,
and Using Winning KPIs by David Parmenter, Jan 2007
44
7 Key Points to Take Home
•
Use a RAD, JAD or XP approach for your dashboards
•
Have multiple meetings with the user groups
•
Build interactive prototypes and expect requirements changes
•
Plan for a formal load testing of the dashboard
•
Have a rollout plan and a long-term vision of how to get there
•
Requirements gathering is interactive and users are
discovering what they want
•
Spend serious time planning for a support and on-going
enhancement of your dashboards, or they will become useless
in a very short time...
45
Your Turn!
How to contact me:
Dr. Berg
[email protected]
Disclaimer
SAP, R/3, mySAP, mySAP.com, SAP NetWeaver®, Duet®, PartnerEdge, and other SAP products and services mentioned herein as well as their respective
logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names
mentioned are the trademarks of their respective companies. Wellesley Information Services is neither owned nor controlled by SAP.
47