Website Evaluation in the Tourism Industry

Evaluating Quality of Customer Service in Airline B2C Systems

Prof. Niv Ahituv, Gil Amoray 8.6.09

, טנרטניאה רקחל 013 ן ' זיווטנ ןוכמ , ילבולגה רפכב טנרטניאו תוריית סנכ

Based on a Research performed for M.Sc. Thesis in Management Sciences - Technology and Information Systems, 2006

Agenda



Research Introduction, Objectives and Scope



Assumptions and Hypotheses



Methodology



Results and Best Practices

2

Research Introduction



WWW introduced new method of communication:

In-Store Service/Sales Phone Service/Sales Web-Based Service/Sales



Companies and non-profit organizations have recognized the benefits and values of online communication channels

   Reducing operating costs Increasing sales opportunities Improving customer service by:  Reducing wait times and expanding service hours (24/7/365)  Enabling self-sufficiency of customers

Research Introduction, Objectives and Scope

3

Research Objective



Define criteria for the evaluation and measurement of the Quality of Service (QoS) in airline B2C systems



Demonstrate the applicability of those criteria through an empirical study



Substantiate applicability of evaluation tool (AWET-- Airline

Website Evaluation Tool) in other industries

Research Introduction, Objectives and Scope

4

Highlights of Literature Review



Website Evaluation

Interface & Design 18 evaluation criteria:

• Graphic design and multimedia • Style & text • Flexibility

Navigation 20 evaluation criteria

• Logical structure • Ease of use • Search engine • Help functions • Navigation necessities

Reliability 28 evaluation criteria:

• Customer profile • Order process • After-sale support • Customer service

Content 15 evaluation criteria

• Prod/Service-related content • Company info • Information quality • Interactivity

Technical Aspects 12 evaluation criteria

• Speed • Security • Software and database

Highlights of Literature Review

5

Research Assumptions



Assumptions: tested using non-statistical sample



A1 – Streamlined process and improved website design are the main reasons for users to favor a website

 User group had to identify dominant criteria in favoring websites: Website design, Process, Content, Technical Aspects, Other 

A2 – Users do not consider technical aspects to be a significant factor in the quality of service provided by airline websites

 Technical aspects were not mentioned as a contributor for an award-winning website in previous studies

Assumptions and Hypotheses

6

Research Hypotheses



Hypotheses: tested by evaluating 33 airline websites



H1 – Web-based QoS level in the airline sector is higher in N.A compared to Europe and Israel

 Percentage of online sales: 40% in N.A vs. 14% in Europe and lower rate in IL  Purchasing patterns are expected to drive higher standards of service  QoS incurs costs, therefore investment decisions seek value and ROI 

H2 – Web-based QoS of competitors working in the same region and industry will be similar

 High competition is expected to result in resemblance of websites w/i industry 

H3 – QoS in websites of “low-fare” airlines is superior comparing to QoS of “Major” Airlines’ websites

 Direct sales (e.g.: online) are expected to impel outstanding service levels

Assumptions and Hypotheses

7

Hypotheses Analysis



H1 – Web-based QoS level in the airline sector is higher in N.A



compared to Europe and Israel



Validated

: Average total score of 67% vs. 64% vs. 59% respectively 

H2 – Web-based QoS of competitors working in the same



region and industry will be similar



Could not be categorically validated or refuted

: airlines differ by category 

H3 – QoS in websites of “low-fare” airlines in superior comparing to QoS of “Major” Airlines’ websites

X

 

Rejected

: Average total score of 67% for major airlines vs. 63% for low-fare Possible explanation: better capabilities to make significant investment by major airlines

Results & Best Practices

8

Methodology



Innovative multi-phased approach, embracing various techniques:

Phase 1 Preparations

• Literature review • Identification of evaluation criteria • Identification of participating websites

Phase 2 Evaluation Tool Development

• Preliminary questionnaire • Industry characteristic mapping • Development of AWET, v1 • SME review and feedback • Development of AWET, v2

Phase 3 Implementation

• Self evaluation of 33 websites

Phase 4 Analysis

• Results analysis • Hypotheses validation

Methodology

9

Evaluation Categories in AWET



Interface and Design



Navigation



Content



Technical



Reservation Process



Usability



Customer Support

Total of 73 Questions in 7 Categories 30 Yes/No Questions 43 Descriptive Questions Methodology

10

Implementation



Self evaluation of 33 airline websites

Major Airlines “Low-Fare” Airlines

Continental United American Airlines Delta U.S. Airways Air Canada Northwest Airlines Alaska Airlines Hawaiian America West JetBlue Southwest Spirit ATA Ted Song USA3000 Frontier Independence Air Alitalia British Airways Air France Lufthansa Iberia Olympic Swiss EasyJet Sky Europe Virgin Atlantic Air One

Methodology

El Al Arkia Israir

11

4 8 2 5 1 3 9

AWET Demonstration

Enter Search Parameters (Reservation Process)

• • 2 adults,

1 child (age: 12)

• NY (LGA) to San Diego

Prefer non-stop

• Date and time specified • Flexible plans for better deal • Assign seats together • Obtain flight info

6 7 11 12 13 10

Methodology

12

Results: Preliminary Questionnaire

User Group Demographics



19 questionnaires sent to non-random B2C system users

  Return rate = 68% (13 surveys) Total of 26 evaluations, for 21 airline websites:  12 American airlines (46%)  6 European airlines (23%)  5 Israeli airlines (19%)  3 Canadian airlines (12%) 

User group members average 11 trips per year

 Performing 58% of airline reservations online

Results & Best Practices

13

Results: Preliminary Questionnaire

Dominant Criteria Groups and Validation of Assumptions



Process and Website Design are most important for online QoS

 Average number of evaluation criteria mentioned by users per group:

Website design Navigation Process Contents Technical Avg # of criteria mentioned Airlines

2.38

1.69

3.08

1.31

0.54

9.00



Assumptions A1 & A2 have been validated:

 A1 – Streamlined

process

and improved

website design

are the main reasons for users to favor a website  A2 – Users do not consider

technical aspects

to be a significant factor in a quality of service provided by airline websites  

Results & Best Practices

14

Results: Self-Evaluation of 33 Airline Websites

Total Score

260 250 240 230 220 210 200 310 300 290 280 270

Best Performer 81% - American Airlines Group Avg = 252

190 180

N.A Major Carriers 69%

Results & Best Practices

Total Score - All Websites (Max Score = 365) Group Avg = 236 N.A Low-Fare Carriers 65% Average Total Score: 237 Points = 65% Overall Avg = 237.27

(65%) Group Avg = 233 EU Major Carriers 64% Group Avg = 229 Group Avg = 214 EU Low-Fare Carriers 63% 15 52% - Arkia Worst Performer Israeli Carriers 59% Avg Group Score

Evaluation Results: Relative Ranking



Results reveal significant performance gaps:

100% 95% 90% 85% 80% 75% 70% 65% 60%

Relative Score 16

Evaluation Results: Distribution



Results are in proximity to Normal Distribution:

Normal Distribution Study

0.0% 0.1% -3s 3.0% 2.1% 12.1% 13.6% -2s -1s 36.4% 34.1% 0s 30.3% 34.1% 1s 15.2% 13.6% 3.0% 2.1% 0.0% 0.1% 2s 3s 17

Best Practices



Based on evaluation results, best performance values have been established for every question within AWET

Question Criterion

1 Fast home page loading 2 3 4 5 6 7 8 9

Measurement

Seconds to fully load homepage

Best Practice

3 seconds speed Fun and matching colors Photos that inspire vacation atmosphere Easy to read typefaces (on homepage) Best dominant colors Number of photos Type of photos Font type Font size (headers) Font size (main text) Blue Could not be clearly determined Hawaiian beach Arial Above 8 Above 8 Page load Site avoids vertical and horizontal scrolling Amount of menu options and links over the text Number of words in home page Number of pictures and banners Spacing score (1-to-5) Percentage of home-page presented on screen (w/o scrolling) Featuring reservations from homepage One-click to most menu and sub-menu options, by rolling 132-226 2-5 3-5 100% Number of menu option and links 34-35 Yes Yes

Example of Best Practices for Questions 1-9

By

Air France 79% of websites N/A Hawaiian Airlines 45% of websites 88% of websites 58% of websites United, AA, Spirit, Alitalia 18% of websites Spirit, British Airways 85% of websites 39% of websites

Results & Best Practices

18

הבשקהה לע הדות

19

Appendix:

AWET: Airline Website Evaluation Tool

AWET (1/12)

Website: Evaluation Date: Section 1 - Website Entry Type the address of the website you wish to evaluate.

Please measure time from clicking “go” to full loading of home page (in seconds).

Rating # Criterion

1 Fast home page

loading speed Not at all 1 2 3 4 Very Much 5 Question Description

Time to fully load home page (seconds)

Answer Section 2 - Home Page Browsing Look at the home page, and answer the following questions: Rating # Criterion Not at all 1 2 3

2 3 How fun and matching are the

colors

?

To what extent the

photos

used inspire vacation atmosphere?

4 How easy to read are the

typefaces

on the home page?

4

5 How appropriate is the

page-load

(amount of text and pictures, spacing)

AWET: Airline Website Evaluation Tool

Very Much 5 Question Description

What are the dominant colors?

Number of photos Describe photos Font type (1) Font size – headers (1) Font size - main text (1) Number of words in home page (1), (2) If you rated 1 or 2 - do you think amount of home page text is lacking or excessive?

Number of pictures and banners Is the spacing sufficient?

Answer 21 Comments Comments

AWET (2/12)

6 7 Site avoids

scrolling

?

vertical and horizontal

To what extent is the

options and links amount of menu

appropriate?

Estimated percentage of home page presented on screen (w/o scrolling) Number of menu option and links (including banners) If you rated 1 or 2 - do you think amount of home page text is lacking or excessive?

(1) Copy sample of text from home page and paste to Word; If page features more than one size / type, refer to the dominant size / type (2) Count number of words by looking at “File >> Properties >> Statistics”.

#

8 9 10 11

Criterion

Site featuring

reservations from home page

One-click to most menu and sub-menu options, by

rolling over the text

Home page features

specials

on the home page

"Contact Us"

information easily accessible from the home page

Yes Section 3 - Site Structure & Sub-Pages Browse through the website and answer the following questions: #

12 13 14 15 16 17 18 19 20 21

Criterion

Site featuring easily accessible

site map Home page icon is consistent

across all site pages Is the

location of menus consistent

across all pages?

Site featuring

consistent frame

Site using

consistent font size

across pages Site using

consistent colors

across pages Site available in

foreign languages

Did the site feature easily-accessible

flight status information

(departure, arrival, gate)?

All the following

contact methods

are available: email, fax, phone, mail Availability of

Frequently Asked Questions (FAQ)

AWET: Airline Website Evaluation Tool

Yes 22 No No Comments Comments

AWET (3/12)

Rating # Criterion

22 23 24 25

Site structure

is clear and straightforward How interactive is the

route map

(1 = not existing, 5 = interactive) Does the site offer

cars, hotels and packages

?

(1 = none, 3 = just one, 4 = both separately, 5 = packages) Fast sub-pages

loading speed

(3) (3) Average loading time of three random pages

Not at all 1 2 3 4 Very Much 5 Question Description

What are the main menu options?

What type of map is featured?

What add-on service is featured?

Time to fully load sub-page (3)

Section 4 - Reservation This reservation section is comprised of seven phases

Basic search, View results, Modify search, Selecting flights, Reservation support, Error handling and Reservation summary

Note: Please read instructions & questions to all phases prior to starting the reservation process.

Answer Comments

You wish to make flight reservations for your family trip to San Diego. - Your wife, 10 year-old daughter and you would like to leave NY on Thursday, August 11, 2005 and return on Thursday, August 18, 2005. - Price is a major consideration in your plans.

- You prefer leaving NY between 7 and 10 am, but would consider leaving earlier or later if the price difference is significant (over $50 per tickets). - Preferred return time would be anytime after 12:00 noon, but could be earlier or later if necessary.

- You prefer flying out of LaGuardia airport. Second priority is JFK airport. - You prefer non-stop flights, but would consider maximum of one connection if cheaper by $50 or more. However, you would only agree to one connecting flight as long as the stop-over is not longer than 3 hours in each direction.

- You wish to assign all three seats together.

- You would like to know what meals are served during the flights.

- You would like to know how many miles you can earn for the round-trip.

AWET: Airline Website Evaluation Tool

23

AWET (4/12)

Phase I - Enter Basic Search Requirements Please return to the website home page. Record the current time: Start entering your flight requirements. Count the number of mouse clicks / fields filled. If needed, continue to the reservation page Stop this phase upon completion of your basic flight requirements - just before you click on "Search Flight" (or any equivalent button). Time to complete phase I :

26

Total duration of phase I (in seconds):

27

Number of clicks (phase I): Please answer the following questions: Rating # Criterion

28 How easy to operate did you find the

entry for dates and hours

of travel?

Not at all 1 2 3 4 Very Much 5 Description Question

Did the site feature calendar for date search?

Did the site feature type-in fields for date search?

Did the site feature day and month fields for date search?

Default search dates are in the future?

Return date automatically updated based on departing flight?

Site allowing part-of-day entry for time (e.g.: morning, evening)

Answer Comments

AWET: Airline Website Evaluation Tool

24

AWET (5/12)

29 How easy to operate did you find the

entry for airport

or route?

#

30 31 32 33 34 35

Criterion

Website offers ability to search flights based on

price or schedule

Website offers ability to search flights based on

number of connections

Search entry differentiates

passengers

: adult, children and seniors Could you

complete the reservations from the home page

(without continuing to the reservation page)?

Upon moving to reservation page, could you avoid data

re-entry?

Website offers ability to

navigate across fields using the tab and arrow keys

Did the site feature type-in fields for airports?

Did the site feature drop down list of airports?

Did the site feature search for airport codes?

Did the site feature an interactive map to select the departure/desination airports?

Did the provide search of nearby airports?

Yes No Comments

AWET: Airline Website Evaluation Tool

25

AWET (6/12)

Phase II - View Search Results Record the current time: Click on "Search Flight" (or "Go", or any equivalent button). Time to load search results (in seconds) (4) Please view results and answer the following questions: Rating # Criterion Not at all 1 2 3 4 Very Much 5 Description Question

Outbound and inbound flight presented on one page?

Outbound and inbound flights are paired (as opposed to showing separately)?

Answer

36

Presentation

of all possible itineraries was

clear and comfortable

37 Fast

loading speed

(4) 38

Abundance

of flights was sufficient and appropriate Did site feature matrix for combinations of departing and returning flight?

Time to present flight search results (as recorded above) (4) Total number of itineraries presented (5) Number of itineraries fully accommodating your request Percentage of matching itineraries (calculated from the data above) Did the site clearly indicate when the no matches were available?

(4) If inbound and outbound flights presented on separate pages, consider the first search only (5) If inbound and outbound flights presented on separate pages, count the number of possible combinations (Outbound x Inbound)

AWET: Airline Website Evaluation Tool

26 Comments

AWET (7/12)

#

39 40 41 42 43 44 45

Criterion Requested itinerary

(route, time) showing next to search results Did the site feature clear presentation of

departure and arrival times?

Did the site feature

meal information

during the flight?

Did the site feature clear

child pricing

for your flight?

Pricing information

available in U.S. dollars Site features ability to

filter results based on number of connections

Site features ability to

sort results by price Identify the least expensive flights accommodating your request (please do not select flight yet) Total Price (two adults and one child): Yes No Comments

AWET: Airline Website Evaluation Tool

27

AWET (8/12)

Phase III - Modify Search

You are still not sure you got the best deal, and would like to search for cheaper flights… You might be more flexible with your travel dates if you would get better prices.

Please try to find cheaper flights for a 7-day vacation on the same route #

47

Criterion

46 Search for

cheaper flights for flexible dates

was easy and comfortable Site prevented

unnecessary data re entry Not at all 1 2 Rating 3 4 Very Much 5 Description Question

Search results page features a link for flexible flights Flexible search suggested alternative dates and time of day Site providing fields to enter range of possible travel dates (e.g.: +/- 3 days) Did you have to browse back and forth in order to search your flights?

Did you have to re-enter airports (from/to)?

Did you have to re-enter number of passengers?

Identify the least expensive flights accommodating your request Total Price (two adults and one child): Answer Comments

AWET: Airline Website Evaluation Tool

28

AWET (9/12)

Phase IV - Selecting Your Flights Stop this phase upon placing your flights on hold, or prior to entering credit card information (6)

(6) If the website does not allow placing reservation on hold, please write "N/A" in question 52 and other relevant questions

Please answer the following questions: Rating #

48 49

Criterion

Comfortable entry of

passenger details

Clear presentation of all

order charges Not at all 1 2 3 4 Very Much 5 Description Question

Number of field requiring entry (per person, not including optional fields) Price information shows breakdown by person and total Price information shows breakdown of price, surcharges and taxes?

50 51 Clear and straightforward presentation of

Terms & Conditions

Site allows alternative

methods of payment

Terms & conditions clearly presented (not hidden) prior to purchase?

Number of credit cards accepted

#

52 53 54 55 56

Criterion

Site allow

“Hold”

on tickets prior to purchase

Seat availability

clearly presented prior to purchase

Seat selection

available online Site allows you to

return to previous process step

(without an error)

Help available

for process-related questions

Yes Answer No Comments Comments

AWET: Airline Website Evaluation Tool

29

AWET (10/12)

Phase V - Reservation Support Select your preferred flights. Record the current time: Try to contact a customer service / sales representative online, to ask them if you could change your reservation within 2 hours #

57 58

Criteria

Site offering

online, in-person support?

Looking up contact info was available without

losing my reservation info?

Yes No Comments Send an email regarding your question via the website Date and time email was sent: Time you received response to your question (7) :

(7) Refers to "final" response to your question (not to immediate response promising future follow-up)

Rating #

59

Criterion

User-friendly

contact form

(via email) 60

Quick reply

to email enquiries 61

Email response

was courteous, accurate and relevant

Not at all 1 2 3 4 Very Much 5 Question Description

Number of required fields Are all required fields are applicable for your enquiry?

Did you receive immediate response informing you that your enquiry has been received (within 5 minutes from sending your question)?

Did the website (or immediate email response) specify expected length to reply to your question?

Time to reply to your email enquiry (in hours) Business hours to reply (assuming 8 hours/day) Did final response meet time specified?

Were you satisfied with the answer provided to you? Reponse was courteous?

Answer Comments

AWET: Airline Website Evaluation Tool

30

AWET (11/12)

Phase VI - Error Handling Return to the reservation page / home page. Enter departure date of Aug 11, 2005 and return date of Aug 8, 2005 (same route) #

62

Criterion

System provides

error warning message

when return date is earlier than departure date.

Yes No Comments

63 System provides

instructions for fixing the error Return to the reservation page / home page. Enter departure date of Aug 11 (morning) and return date of Aug 11 (afternoon) #

64

Criterion

System

alerts for possible error

when return time is adjacent to departure time.

Yes No Comments Return to the reservation page / home page. Enter departure city "Nwe York" (deliberately misspelled) #

65

Criterion

System suggests ways to handle

spelling mistakes

?

Yes No Comments

AWET: Airline Website Evaluation Tool

31

AWET (12/12)

Phase VII - Reservation Summary Please answer the following questions regarding the reservation process you just experienced Rating # Criterion

66 The reservation

process was transparent, easy and clear

67

Task was fully completed

68 Task completed

without errors and broken links

69 Is the

transaction perceived as secure

?

70 Avoidance of

unnecessary security alerts Not at all 1 2 3 4 Very Much 5 Description Question

Was progress of the reservation process indicated?

Percentage of the completion, based on the following five tasks: - Found desired flight?

- Found seat assignment?

- Found child pricing?

- Found mileage information?

- Found meal information?

Number of error messages Was a security alert or message containing security presented prior to entering credit card info?

Number of security alerts during visit

Section 5 - Post Sale and Summary Please answer the following questions: #

71 72 73

Criterion

Site allows the user to

change an existing reservation

Site allows the user to

cancel reservation

You were able to

avoid pop-ups

during the visit (8)

Yes

(8) Assuming use of same computer, to eliminate impact of different settings.

AWET: Airline Website Evaluation Tool

32 Answer No Comments Comments