Transcript Behavioral AIS research - University of Waterloo

Information Fusion in
Continuous Assurance
Johan Perols
University of San Diego
Uday Murthy
University of South Florida
UWCISA Symposium
October 2, 2009
1
Introduction and Motivation


Drivers for continuous assurance (CA) have been in place
for some time
 A number of CA implementations have been described in
prior research (Groomer & Murthy 1989; Vasarhelyi &
Halper 1991; Alles et al. 2006)
 Technology vendors providing CA functionality (SAP,
ACL)
Existing continuous assurance architectures focus on the
detection of exceptions and do not address the important
problem of processing detected exceptions.
2
Introduction and Motivation

CA systems can lead to information overload, with a large
number of detected exceptions

Dealing with detected exceptions requires aggregating,
processing, and analyzing exceptions

Behavioral research in psychology and auditing has shown
that humans are not good at these tasks
3
Research Objective

We introduce Continuous Assurance Fusion (CAF):
an architecture for CA that uses the concept of
information fusion for aggregation and analysis of
detected exceptions.

CAF is grounded in prior CA literature and
computer science information fusion research.
4
Information Fusion

Information fusion involves gathering data about an object of
interest from multiple sources and integrating these data in
order to arrive at a holistic conclusion about the object.

Application fields: defense, geoscience, robotics, medicine,
and industrial engineering

Goals of information fusion—
 Dimensionality reduction
 Improving precision / reducing uncertainty
 Improving robustness when some data are noisy
5
CAF Overview

The data gathering component of the architecture draws from
prior CA research

REA ontology concepts are used in the first data integration
step

Artificial intelligence and machine learning algorithms are
used in the subsequent integration and evaluation steps
6
Background: Exception Detection

Detection of exceptions can be data-oriented or controloriented

Two architectures have been proposed to accomplish the
exception detection task

Embedded audit modules (EAM)

Monitoring and control layer (MCL)
7
Embedded Audit Modules

Embedded audit modules (EAM)
 Examine transactions and controls for exceptions in
real-time
 Relatively intrusive as detection code is implemented
at the application or database level
 Offers better detection capabilities, but consumes more
resources, so EAMs are better suited for controloriented exception detection, in real-time
 Control over EAM resides with transaction processing
system owners
8
Monitoring and Control Layer

Monitoring and control layer (MCL)
 Examines transactions and controls for exceptions on a
periodic basis
 Implemented as a stand-alone system that periodically
queries transaction and control data
 Better suited for large-volume data-oriented exception
detection, since it does not affect operational TPS
 Control over EAM resides with EAM owners: auditors
9
Alarm Overload

Both data exceptions and control exceptions can result in
alarm overload

Dealing with alarm overload
 Manually turning off groups of controls to minimize the
degree of exception flooding (Alles et al. 2006)
 Adjust CA monitoring parameters such that fewer alarms
are generated
10
Continuous Assurance Fusion (CAF)


CAF provides a method for aggregating
and analyzing detected exceptions to draw
audit-relevant conclusions
Basic idea: It is possible to get a more
complete and accurate assessment of
objects and situations if data from many
sensors are combined and multiple models
are used to evaluate these data.
11
CAF Architecture
Accounting Information System
Business
Process
Internal Data
CAF
Monitoring Layer
DAI-DAO
External Data
Aggregation Layer
CAF Data
DAI-FEO
FEI-FEO
Evaluation Layer
FEI-DEO
Decision Layer
DEI-DEO
Information Users
Figure 1 - CAF Conceptualization
12
Layers in CAF Architecture




Monitoring layer – same functionality as extant CA
architectures
Aggregation layer – generates object features by grouping
exceptions based on their association to specific objects and
computing additional object features
Evaluation layer – invokes classifiers (e.g., logistic
regression, ANN) on object features to make decisions about
objects’ class membership
Decision layer – combines the individual classifier object
classifications into an overall CAF object class membership
decision.
13
Aggregation Layer

Data In  Feature Out fusion

Use McCarthy’s REA ontology as the basis for
aggregation layer processing


Automatically discerning features of exceptions
Processing done is primarily grouping of exceptions
in terms of severity, from Level 1 to Level 5
14
REA Ontology
15
Purchasing subsystem REA model
16
Aggregation Layer





Level 1: Exceptions that relate to changes affecting a single
R, E, or A object.
Level 2: Exceptions grouped in terms of event-resource
relationships.
Level 3: Exceptions grouped in terms of event-agent
relationships.
Level 4: Exceptions grouped in terms of event-event
relationships.
Level 5: Exceptions grouped in terms of resource-eventagent relationships.
17
Exceptions – Purchasing Scenario

Level 1: Invalid date/time stamp on purchasing order (exception
involving a single ‘E’).

Level 2: Purchase order for an inventory item with excessive
quantity on hand (exception involving ‘E’ – ‘R’ relationship).

Level 3: High dollar value purchase order placed by low level
purchasing agent (exception involving ‘E’ – ‘A’ relationship).

Level 4: Purchase order placed without existing purchase
requisition (exception involving ‘E’ – ‘E’ relationship).

Level 5: P.O. placed by low level purchasing agent for high value
item (exception involving ‘R’ – ‘E’ – ‘A’ relationship).
18
Evaluation Layer



Feature In  Decision Out fusion
Use artificial neural network (ANN) technology
 Input = features from aggregation layer
 Output = probabilistic decision regarding “state”
of the subsystem
Purchasing business process example
 Based on Level 1 – Level 5 features input, is the
purchasing subsystem “in control” or “out of
control”?
19
Decision Layer



Decision In  Decision Out fusion
Could use Bayesian control concepts, ANN, or
RBES
 Input = decisions regarding the state of individual
subsystems from evaluation layer
 Output = probabilistic decision regarding “state”
of the overall information system
Level of analysis for Evaluation and Decision layers
is a design choice – modules / subsystems / system
20
Future Work




Development of a functioning CAF prototype
following architecture presented in this paper
Use of agent-based technologies for implementing
CAF
Empirically evaluate and compare the utility of
different classification algorithms at the Evaluation
Layer
Use of information fusion concepts in areas such as
fraud detection
21
Summary and Conclusion




Using concepts from information fusion, CAF is an
architecture for dealing with audit exceptions
CAF is not a functioning system but an approach
that represents a way forward in addressing the
problem of dealing with detected exceptions
Aggregation layer applies REA ontology concepts
to group exceptions by severity
Evaluation and decision layers apply ANN or
Bayesian concepts to draw conclusions about the
state of a subsystem or the system as a whole
22