Transcript SportsStuff.com discussion
The Importance of the COBIT Framework IT Processes For Effective Internal Control over the Reliability of Financial Reporting: An International Survey
David S. Kerr
University of North Carolina at Charlotte
Uday Murthy
University of South Florida UWCISA Symposium, October 11-13, 2007, Toronto, Canada
Introduction and Background
Publicly held companies must have a system of internal controls, per regulatory requirements Internal controls are heavily “IT-dependent”
Need for strong IT governance
COBIT – a framework for IT governance
Specifies “best practices” for IT processes
Conformance to COBIT IT processes should result in better internal control
2
Motivation
To understand the extent to which the COBIT IT processes contribute to effective internal control over the reliability of financial reporting Given limited resources, are there certain “key” processes that organizations should focus on from the viewpoint of reliability of financial reporting? To determine whether demographic variations in IT auditors explain differences in perceptions regarding the value of COBIT 3
COBIT
C
ontrol
OB
jectives for
I
nformation and related
T
echnology Focus of COBIT is on the management and control of IT Comprises 34 IT processes organized into 4 domains Plan and Organize (plan) Acquire and Implement (build) Deliver and Support (run) Monitor and Evaluate (monitor) 4
Figure 1: COBIT Framework
5
Prior Work
COBIT usage survey by Guldentops and De Haes (2002) Profile of COBIT adopters (n=182) Almost half of the respondents were from the Americas Most over 1,000 employees with 1/3 rd > 10,000 employees 90% of responding organizations used COBIT Uses: audit planning and audit program development, validate current IT controls, to evaluate IT risks, to reduce IT risks, and as a framework for improving IT ~ 40% of respondents indicated that their control framework and audit process was partly COBIT-based; less than 5% of respondents indicated that COBIT had been formally adopted and was enforced as corporate policy 6
Research Questions
RQ1: In the context of the reliability of financial reporting, what is the relative importance of each of the 34 IT control and security processes?
RQ2: In the context of the reliability of financial reporting, to what extent does the relative importance of each of the 34 IT control and security processes vary as a function of characteristics of the IT professionals within the organization? 7
Method
Web survey of IT professionals ISACA members targeted through local chapters Sections of survey instrument Demographics Background information COBIT familiarity Importance rating for each process, top 10 processes 8
Respondents
189 respondents from 21 countries Average age: 40.1 years Gender: 71% were male. Working in… industry: 66% public accounting: 18% government: 16% Average time with current employer: 5.8 years Degrees: 38% masters; 57% bachelors Certifications: 58% CISAs 9
Selected Demographics
TIME SPENT REVIEWING IT CONTROLS
Less than 10% 10% - 25% 26% - 50% 51% - 75% Greater than 75%
Frequency
18 39 34 33 65
Percent
9.5 20.6 18.0 17.5 34.4
Familiarity with COBIT* Frequency Percent
1 2 3 4 5 2 11 60 55 61 1.1 5.8 31.7 29.1 32.3 * 1 = Not at all familiar; 3 = Somewhat familiar; 5 = Very familiar 10
Table 2 COBIT Processes Sorted by Mean Importance Ratings COBIT Process*
DS5 AI6 PO9 DS11 M2 PO8 DS10 AI4 M1 PO11 DS4 M4 DS7 PO10 M3 DS9 PO2
Description of process
Ensure System Security Manage Changes Assess Risk Manage Data Assess Internal Control Adequacy Ensure Compliance with External Requirements Manage Problems and Incidents Develop and Maintain Procedures Monitor the Process Manage Quality Ensure Continuous Service Provide for Independent Audit Educate and Train Users Manage Projects Obtain Independent Assurance Manage the Configuration Define the Information Architecture
Mean importance rating
4.661 4.487 4.413 4.333 4.328 4.222 4.101 4.085 4.079 4.074 4.048 4.021 4.005 3.952 3.947 3.931 3.884 11
Table 2 (contd.) COBIT Processes Sorted by Mean Importance Ratings COBIT Process*
PO5 PO7 AI1 PO3 DS6 DS8 DS13 PO1 AI5 PO6 AI3 AI2 DS2 PO4 DS12 DS1 DS3
Description of process
Manage Operations Define a strategic IT plan Install and Accredit Systems Communicate Management Aims and Directions Acquire and Maintain Technology Infrastructure Acquire and Maintain Application Software Manage Third-party Services Define the IT Organization and Relationship Manage Facilities Define and Manage Service Levels Manage Performance and Capacity Manage the Information Technology and Relationships Manage Human Resources Identify Automated Solutions Determine the Technological Direction Identify and Allocate Costs Assist and Advise Consumers
Mean importance rating
3.884 3.878 3.873 3.825 3.815 3.799 3.783 3.746 3.730 3.714 3.714 3.709 3.640 3.566 3.545 3.407 3.238 12
Table 3 Number of times each IT process was selected as a “Top 10” process COBIT process
DS5 AI6 PO9 M2 DS11 PO1 M1 AI4 DS10 DS7 PO8 M4 M3 DS4 DS9 PO10 PO2
Description of process
Ensure System Security Manage Changes Assess Risk Assess Internal Control Adequacy Manage Data Define a strategic IT plan Monitor the Process Develop and Maintain Procedures Manage Problems and Incidents Educate and Train Users Ensure Compliance with External Requirements Provide for Independent Audit Obtain Independent Assurance Ensure Continuous Service Manage the Configuration Manage Projects Define the Information Architecture 55 51 50 49 48 98 97 91 81 74 70 66 64 58
Top 10 count
147 133 122 13
Table 3 (contd.) Number of times each IT process was selected as a “Top 10” process
PO7 DS2 DS3 PO3 DS6 AI1 DS12 DS8
COBIT process
AI2 PO11 PO6 AI3 PO4 DS1 DS13 PO5 AI5
Description of process
Acquire and Maintain Application Software Manage Quality Communicate Management Aims and Directions Acquire and Maintain Technology Infrastructure Define the IT Organization and Relationship Define and Manage Service Levels Manage Operations Manage the Information Technology and Relationships Install and Accredit Systems Manage Human Resources Manage Third-party Services Manage Performance and Capacity Determine the Technological Direction Identify and Allocate Costs Identify Automated Solutions Manage Facilities Assist and Advise Consumers 34 31 29 24 20 19 17 6
Top 10 count
46 45 44 39 38 38 36 35 35 14
Table 4 Factor Analysis Results: Rotated Component Matrix Factor 3 Factor 4 Factor 5 Factor 6 Factor 1 Factor 2 COBIT process
AI6: Manage Changes DS5: Ensure System Security DS11: Manage Data M2: Assess Internal Control Adequacy AI4: Develop and Maintain Procedures PO9: Assess Risk DS10: Manage Problems and Incidents M1: Monitor the Process DS7: Educate and Train Users AI5: Install and Accredit Systems DS4: Ensure Continuous Service Key processes: General & application controls .787
Planning and IT mgmt processes .121
Organization and relationships processes .104
.755
.734
.673
.196
.197
.067
.145
-.050
.239
Technology processes .120
.175
.153
-.011
Operations and facilities processes .117
Independent audit processes .098
.130
.294
.056
.114
.033
.390
.643
.624
.584
.551
.533
.041
.092
.431
.254
.326
.176
.417
.166
.244
.171
.348
.093
.000
-.056
.274
.050
.138
.354
.245
.102
.143
.221
.074
.318
.081
.497
.280
.083
.764
.057
-.002
.458
.149
.262
.178
.165
-.023
DS3: Manage Performance and Capacity DS1: Define and Manage Service Levels PO3: Determine the Technological Direction PO1: Define a strategic IT plan .186
.202
-.011
.711
.693
.649
.249
.242
.511
.132
.066
.251
.269
.235
-.011
.131
.197
-.004
DS8: Assist and Advise Consumers DS6: Identify and Allocate Costs PO10: Manage Projects PO11: Manage Quality .158
-.010
-.015
.282
.484
.648
.616
.590
.543
.485
.428
.210
.252
.181
.164
-.011
.177
.148
.300
.159
.036
.343
.217
.019
-.064
.040
.263
.478
.039
.063
15
Table 4 (continued) Factor Analysis Results: Rotated Component Matrix Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 Factor 6 COBIT process
PO4: Define the IT Organization and Relationship Key processes: General & application controls .138
Planning and IT mgmt processes .285
Organization and relationships processes .759
PO5: Manage the Information Technology and Relationships .148
.209
.711
PO6: Communicate Management Aims and Directions .133
.510
.587
PO7: Manage Human Resources .167
.280
.572
PO2: Define the Information Architecture PO8: Ensure Compliance with External Requirements AI2: Acquire and Maintain Application Software AI3: Acquire and Maintain Technology Infrastructure AI1: Identify Automated Solutions .266
.388
.283
.140
.454
.032
.126
.242
.566
.520
.188
.256
.160
.408
-.008
DS13: Manage Operations DS2: Manage Third-party Services DS12: Manage Facilities DS9: Manage the Configuration .444
.247
.293
.409
.185
.351
.338
.042
.272
.223
.109
.393
M3: Obtain Independent Assurance .441
.109
.158
M4: Provide for Independent Audit .458
.156
.086
Technology processes Operations and facilities processes Independent audit processes .063
.147
.131
.091
.071
.165
.148
.179
.800
.707
.656
.185
.084
.167
.277
.086
.086
.171
.083
.245
-.095
.263
.131
.306
-.085
.606
.577
.570
.546
.048
.163
.180
.086
.134
-.067
-.002
-.023
-.012
.285
.252
-.043
.315
.075
.765
.672
16
Table 5 Ratings of Technology Processes by Employment Type Panel A: Descriptive Statistics
* 1=Not at all important; 5=Very important.
Employment type Public accounting Industry Government Total Mean* 4.0505
3.6640
3.6022
3.7216
Std. Deviation .67762
.83671
.65783
.79507
N 33 124 31 188
Panel B: Tests of Between-Subjects Effects
Source Corrected Model Intercept Employment Error Total Corrected Total Type III Sum of Squares 4.424(a) 1813.311
4.424
113.786
2722.111
118.210
df 2 1 2 185 188 187 Mean Square 2.212
1813.311
2.212
.615
a R Squared = .037 (Adjusted R Squared = .027) F 3.596
Sig.
.029
2948.191
3.596
.000
.029
17
Table 6 Ratings of Technology Processes: North America vs. Rest of the World Panel A: Descriptive Statistics
* 1=Not at all important; 5=Very important.
Country USA or Canada All other countries Mean* 3.8197
Std. Deviation .80259
3.6148
.77719
Total 3.7216
.79507
N 98 90 188
Panel B: Tests of Between-Subjects Effects
Source Corrected Model Type III Sum of Squares 1.970(a) df 1 Mean Square 1.970
Intercept Country Error Total Corrected Total 2593.100
1.970
116.240
2722.111
118.210
1 1 186 188 187 2593.100
1.970
.625
a R Squared = .017 (Adjusted R Squared = .011) F 3.152
Sig.
.077
4149.321
3.152
.000
.077
18
Table 7 Extent of COBIT familiarity by Audit Experience Panel A: Descriptive Statistics
* 1=Not at all familiar; 5=Very familiar.
Extent of audit experience Relatively less (four years or less) Relatively more (more than 4 years) Total Mean* 3.65
4.08
3.86
Std. Deviation .902
1.014
.979
N 97 91 188
Panel B: Tests of Between-Subjects Effects
Source Corrected Model Intercept Audit Exp Error Total Corrected Total Type III Sum of Squares 8.578(a) 2802.919
8.578
170.544
2975.000
179.122
df 1 1 1 186 188 187 Mean Square 8.578
2802.919
8.578
.917
a R Squared = .048 (Adjusted R Squared = .043) F 9.356
3056.940
9.356
Sig.
.003
.000
.003
19
Table 8 Extent of work relating to task of reviewing/evaluating IT controls: North America vs. Rest of the World
.
Panel A: Descriptive Statistics
+ Scale 1 = less than 10%; 2 = 10% - 25%; 3 = 26% - 50%; 4 = 51% - 75%; 5 = greater than 75% Country USA or Canada Mean + 3.69
Std. Deviation 1.380
N 98 All other countries 3.23
1.366
90 Total 3.47
1.389
188
Panel B: Tests of Between-Subjects Effects
Source Corrected Model Intercept Country Error Total Corrected Total Type III Sum of Squares 9.951(a) 2251.270
9.951
350.916
2629.000
360.867
df 1 186 188 1 1 187 Mean Square 9.951
2251.270
9.951
1.887
a R Squared = .028 (Adjusted R Squared = .022) F 5.274
1193.265
5.274
Sig.
.023
.000
.023
20
Table 9 Extent of Work Relating to Task of Reviewing/Evaluating IT Controls by Employment Type
.
Panel A: Descriptive Statistics
+ Scale 1 = less than 10%; 2 = 10% - 25%; 3 = 26% - 50%; 4 = 51% - 75%; 5 = greater than 75% Std. Employment type Public accounting Mean + 3.94
Deviation 1.345
N 33 Industry 3.46
1.428
124 Government 3.03
1.140
31 Total 3.47
1.390
188
Panel B: Crosstabulation Employment type
Public accounting Count Expected Count Industry Government Count Expected Count Count Expected Count Total Count Expected Count Less Percentage of work relating to task of reviewing/evaluating IT controls Greater than 10% 2 3.2
10% 25% 4 6.7
26% 50% 6 6.0
51% 75% 3 5.8
than 75% 18 11.4
Total 33 33.0
14 11.9
2 3.0
18 18.0
26 25.1
8 6.3
38 38.0
15 22.4
13 5.6
34 34.0
27 21.8
3 5.4
33 33.0
42 42.9
5 10.7
65 65.0
124 124.0
31 31.0
188 188.0
21
Figure 2: Dendrogram
Rescaled Distance Cluster Combine C A S E 0 5 10 15 20 25 Label Num +---------+---------+---------+---------+---------+ AI6_imp 17 DS5_Imp 22 PO9_imp 9 DS11_Imp 28 M2_imp 32 PO8_imp 8 AI4_imp 15 AI5_imp 16 DS12_Imp 29 DS13_Imp 30 DS9_Imp 26 DS10_Imp 27 M1_imp 31 DS7_Imp 24 M3_imp 33 M4_imp 34 AI2_imp 13 AI3_imp 14 AI1_imp 12 PO2_imp 2 PO3_imp 3 PO4_imp 4 PO5_imp 5 PO7_imp 7 PO6_imp 6 DS1_Imp 18 DS3_Imp 20 DS2_Imp 19 PO10_imp 10 PO11_imp 11 DS4_Imp 21 PO1_imp 1 DS6_Imp 23 DS8_Imp 25 22
Overview of Results
Of the 34 IT processes, results reveal that some are more important than others from the viewpoint of the reliability of financial reporting In particular, five processes stood out as being critical: Ensure System Security (DS5); Manage Changes (AI6), Assess Risk (PO9), Assess Internal Control Adequacy (M2), and Manage Data (DS11) Factor analysis results revealed six distinct factors, with the “general and application controls” factor being the most prominent 23
Limitations
True response rate and hence extent of non-response bias is unknown Extent to which importance ratings were affected by the length of the instrument is unknown (the “fatigue factor”) Order of 34 processes was not randomized Despite instructions, it is possible that respondents were not attuned to the focus on the effect of the COBIT IT processes on the reliability of financial reporting Lack of a “reference point” or context for assessing importance of IT processes 24
Conclusion and Future Research
Some COBIT IT processes are deemed more critical than others from the standpoint of the reliability of financial reporting Internal and external auditors can focus their attention on the “Top 10” most critical COBIT processes Future research could focus on the
why
question – why some IT processes are deemed more critical than others Also worth investigating the extent to which COBIT processes contribute to other organizational objectives 25