Transcript IS2010 Model Curriculum Core Assessment Examination

Supporting Institutional and
Professional Development
Lynn McKell – Brigham Young U
Kewal Dhariwal – Excelsior College & ICCP
Tom Hilton – U Wisconsin (Eau Clare)
Bruce White – Quinnipiac U



Introduction/Background – McKell (moderator)
ICCP Certification Council – Dhariwal
IS2010 Model Curriculum Assessment – Hilton
 ISA




Examination
Institutional Assessment/Improvement – White
Individual Certification – McKell
ISA Exam Procedures – Dhariwal
Questions

ICCP - Org. 1971 by AITP (DPMA), ACM, etc.


ISECON 2001 panel



2002 Workshop : ICCP and U of South Alabama
Representatives from 18 Institutions
NCC 2003 Calibration
Uses:



Possible role of ICCP exams in curriculum assessment
Conclusion: A new assessment instrument was needed
IS2002 Model Curriculum – Exam Development




Professional certifications governed by industry associations
Institutional Assessment/Improvement (Accreditation)
Individual Assessment (Certification: ISA)
IS2010 Model Curriculum

Exam Update in process






Members: Academic & Industry Professionals
Exam Item creation/validation
Benchmarking and Vetting
Item Statistics and Tracking
Item updating and retiring
Future: ABET, SIGITE

John Whitehouse Jr., PhD, CCP
 Director of Certification (Cleveland, OH)











Rick Seaman, VP Financial systems (Virginia)
Terry Linkletter, Professor (Seattle)
David Horvath, Lecturer, Author (Philadelphia)
Lynn McKell, PhD, Professor (Provo)
Pat Cupoli, DAMA International, (Philadelphia)
Jim Woolen, CIO, PhD, Professor (Michigan)
Bob Grenier, PhD, Professor (Illowa, IW)
Tom Hilton, PhD, Professor (Eau Claire, WI)
Bruce White, PhD, Professor (Hamden, CT)
Kewal Dhariwal, PhD, Professor (Albany, NY)
Ken Bainey, PhD, CIO, Author (Edmonton, AB)
 Many (>40) others who have contributed to the examinations


Exam Writer’s guide
Complexity of items







Blooms taxonomy of learning objectives
Depth
Subject outline classification
Items reviewed by the Certification Council before
being included into the ICCP test bank
Items’ updating and retirement practice
ICCP test banks are conservatively valued between
$7-$9million
29 examinations available across a broad spectrum
of information and communications technologies
(ICT)

Alpha Exam development








ACM, AIS , IEEE and AITP – Model Curriculum
Standards
Subject outline developed (Test Management Council)
Individual Subject Matter Experts develop test items
(SMEs)
Group of reviewers modify, wordsmith and classify items
to a subject outline, set initial complexity level.
Beta exam is offered to select colleges with students
Beta results analyzed, Items assessed and updated if
needed, scores for beta exam takers adjusted if needed
Reporting system finalized
Examination released to the market

EXAM – Information Systems




Version=2009304,
Number of exam takers=1026
Age=0.64
Pass rates



Practitioner (>50%) = 83.3%,
Mastery(>70% ) = 15.1%
Range of Scores
 Low 30,
 High 82
 Mean=59
 Standard Deviation=9.900

Difficulty of items
 Hardest 12% get correct
 Easiest 99% get correct
 Point BiSeRial PBSR Correlation of item
with Mastery level exam passers
 Lowest -0.05, low discriminant item
 Highest 0.48), high discriminant item
 Split Half Rho=.77
 KR20=.79
 Skew=-.53
 Kurtosis=-4.20













V=2007118,
N=1498
Age=3.15
Pass(P,M)=(81.7%,20.0%)
Range=(2,85)
Mean=60
SD=11.405
Dif=(0.16,0.99)
PBSR=(-0.05,0.48),
Split Half Rho=.80
KR20=.85
Skew=-.49
Kurtosis=-4.07













V=2009304,
N=1026
Age=0.64
Pass(P,M)=(83.3%,15.1%)
Range=(30,82)
Mean=59
SD=9.900
Dif=(0.12,0.99)
PBSR=(-0.05,0.48),
Split Half Rho=.77
KR20=.79
Skew=-.53
Kurtosis=-4.20






Item Number 8325
Creation date: February 2009
Author: Certification Council
Number of exam takers: 1623
Percentages choosing{A,B,C,D,Skip}:
{.15 .06 .20 .59 .00}, correct - D
PBSR correlation with mastery level exam
passers {ABCD}: (-.13 -.25 -.14 .33)

Model Curriculum Structure




“The structure has become more flexible and
adaptable to local conditions while keeping a
national/international alignment” – Hilton, 2010
Model Curriculum Learning Objectives
ISA Examination Content
ISA Examination Structure and Administration
IS Minor
IS 2002.P0
Personal
Productivity
w/ IS Technology
IS 2002.2
E- Business Strat,
Architecture, &
Design
IS 2002.3
Information
Systems Theory &
Practice
I S 2002.1
Fundamentals of
Information
Systems
IS 2002.4
IT Hardware &
System Software
IS 2002.5
Programming,
Data, File & Object
Structures
IS 2002.6
Networks &
Telecomm
IS 2002.8
Physical Design &
Implementation
with DBMS
http://www.acm.org/education/is2002.pdf
IS 2002.9
Physical Design &
Implementation in
Emerging Envs
IS 2002.7
Analysis & Logical
Design
IS 2002.10
Project Mgt
& Practice
I S 2010.C1
Foundations of
Information
Systems
IS Minor
IS 2010.C2
Data &
Information
Management
IS 2010.C3
Enterprise
Architecture
IS 2010.C4
IS Project
Management
IS 2010.C5
IT Infrastructure
IS 2010.C6
Systems Analysis
& Design
IS 2010.E2
Business Process
Management
IS 2010.C7
IS Strategy,
Management &
Acquisition
IS 2010.E1
Application
Development
IS 2010.E3
Collaborative
Computing
IS 2010.E4
Data Mining /
Bus Intelligence
IS 2010.E5
Enterprise Systems
IS 2010.E6
Human-Computer
Interaction
IS 2010.E7
Information
Search & Retrieval
IS 2010.E8
IT Audit &
Controls
IS 2010.E9
IT Security & Risk
Management
IS 2010.E10
Knowledge
Management
IS 2010.E11
Social Informatics
http://www.acm.org/education/curricula/IS%202010%20ACM%20final.pdf
Business
Fundamentals
Information
Technology
Information Systems
“Technology-Empowered
Business Development”
Analytical and Critical
Thinking
Interpersonal,
Communication,
and Team Skills
Engineering
Fundamentals
Information
Technology
Information Systems
“Technology-Empowered
Engineering Development”
Analytical and Critical
Thinking
Interpersonal,
Communication,
and Team Skills
IS-Specific
Knowledge &
Skills
Domain
Fundamentals
Foundational
Knowledge &
Skills




Identify and design opportunities for ITenabled organizational improvement
Analyze trade-offs in solution & sourcing
alternatives
Design and implement information systems
solutions
Manage ongoing information technology
operations





Lead and collaborate
Communicate
Negotiate
Think analytically, critically, creatively, ethically
Apply statistics and probability

General models of the domain





Key specializations within the domain







Business models
Business process design and management
Organizational theory
Business strategy
Finance
Accounting
Marketing
Operations management; service science and management
Organizational behavior
Business law
Evaluation of performance within the domain




Analysis of organizational performance
Analysis of individual and team performance
Business analytics
Business intelligence










Foundations of Information Systems
Data and Information Management
Enterprise Architecture
IT Infrastructure
IS Project Management
Systems Analysis & Design
IS Strategy, Management & Acquisition
258 Randomly Selected Items
In Two 105-Minute Sessions with a 10-Minute Break
At Program-Offering Institution












Application Development
Business Process Management
Enterprise Systems
Human-Computer Interaction
IT Audit and Controls
IS Innovation and New Technologies
IT Security and Risk Management
Knowledge Management
Social Informatics
Available
Available
Available
Future
Integrated
Future
Available
Future
Future
110 Randomly Selected Items per Examination
In One 90 minute Session per Examination
At Program-Offering Institution

1.0 Foundations of Information Systems









2.0 Data & Information Management









1.1 Information Systems in Organizations
1.2 Valuing Information Systems
1.3 Information Systems Components and Infrastructure
1.4 Systems Development and Acquisition
1.5 The Internet and World Wide Web
1.6 Security of Information Systems
1.7 Business Intelligence
1.8 Globalization
2.1 Database Approach
2.2 Data Modeling
2.3 Database Languages
2.4 Data and Database Administration
2.5 Transaction Processing
2.6 Data / Information Architecture
2.7 Data Security Management
2.8 Data Quality Management
3.0 Enterprise Architecture




3.1 Enterprise Architecture Frameworks
3.2 Monitoring and Metrics for Infrastructure and Business Processes
3.3 Green Computing
3.4 Business Continuity

4.0 IT Infrastructure





5.0 Project Management










5.1 Introduction to Project Management
5.2 The Project Management Lifecycle
5.3 Managing Project Teams
5.4 Project Initiation and Planning
5.5 Managing Project Scheduling
5.6 Managing Project Resources
5.7 Managing Project Quality and Risk
5.8 Systems Procurement
5.9 Project Execution, Control and Closure
6.0 Systems Analysis & Design






4.1 Systems Concepts
4.2 Operating Systems
4.3 Networking
4.4 IT Control and Service Management Frameworks (COBIT, ITIL, etc.)
6.1 Business Process Management
6.2 Structuring of IT-based Opportunities into Projects
6.3 Analysis and Specification of System Requirements
6.4 Implementation Strategies
6.5 Design and Analysis
7.0 IS Strategy, Management & Acquisition



7.1 The Role of IS in the Organization
7.2 Structuring the IS Organization
7.3 Acquiring Information Technology Resources and Capabilities


1.0 Foundations of Information Systems (20%)
Information systems are an integral part of all business activities
and careers. This section covers the key components of
information systems - people, software, hardware, data, and
communication technologies, and how these components can be
integrated and managed to create competitive advantage.

1.1 Information Systems in Organizations
1.1.1. Characteristics of IS professionals
1.1.2. IS Career Paths
1.1.3. Cost, Value and Quality of Information
1.1.4. IS and Organizational Strategy
1.1.5 Value Chains and Networks

1.2 Valuing Information Systems
1.2.1. Making a Business Case for Information Systems
1.2.2. Multi-Criteria Analysis ·
1.2.3. Total Cost of Ownership and Return on Investment
1.2.4. Purchasing Systems and IT Infrastructure
1.2.5. Identifying and Implementing Innovations

Analytical Reports

Closing the Loop

Useful in Accreditation




Some Background:
The Information Systems Management (ISM)
program at Quinnipiac University is accredited
by ABET and is an AACSB accredited School of
Business
We do have an assessment program
incorporating several data items
For our most important direct assessment, we
have used the Information Systems Assessment
test since 2004



IS Assessment Test
Taken as part of ISM 440 Project Management, our
undergraduate capstone class.
While the IS Assessment test has three main
sections, we concentrate on sections 2 and 3:






2.0 Organizational and Professional Skills (which maps to our
Role of IS in Organizations goal)
- 2.1 Individual and Team Interpersonal Skill
- 2.2 Business Fundamentals
3.0 Strategic Organizational Systems Development with IS
- 3.1 Organizational Systems Development
- 3.2 Project Management
Area
2.1 Individual and Team
interpersonal skills
QU
2008
QU
2010
National
67
69
69
69
69
66
50
54
50
58
65
59
2.2 Business Fundamentals
3.1 Organizational Systems
Development
3.2 Project Management



ABET expects ‘direct’ measurement of learning.
This test gives us the best analysis
It also helps us identify areas of weaknesses
and places to improve. (see next pages)
We encourage students to take the test
seriously by making it part of their final grade
Analysis and Design
60
50
40
30
20
10
0
2004
2005
2006
2007
2008
2009
Data Management
60
50
40
30
20
10
0
2004
2005
2006
2007
2008
2009
IS in Organizations
70
60
50
40
30
20
10
0
2004
2005
2006
2007
2008
2009
Programming Languages
60
50
40
30
20
10
0
2004
2005
2006
2007
2008
2009
Hardware
50
45
40
35
30
25
20
15
10
5
0
2004
2005
2006
2007
2008
2009
Networking
70
60
50
40
30
20
10
0
2004
2005
2006
2007
2008
2009
Report:
Item #
12143
12144
12145
12146
12147
12148
12149
12150
12151
12152
12153
12154
12155
Model Curriculum Skill Analysis
by course
IS2010 Section 4. IT Infrastructure
National Local
82
82
44
46
29
9
24
27
51
46
71
91
35
27
83
91
38
36
32
36
56
64
39
9
53
64
Course
4
4
4
4
4
4
4
4
4
4
4
4
4
Exam Date:
College:
LU
1
1
2
2
3
3
3
3
3
3
3
4
4
20-Oct-10
Any University College
Question Objective
Understand and assess volume and the need and application for automated procedures (stored procedures)
Understand applications of distributed databases
Understand the role of data security and scalability in web applications and corporate databases
Understand the purpose of layered systems standards
Understand, assess and apply client server database transactions capacity and flow operations
Understand the capacity and role of mobile devices and corporate databases in client server applications
Apply information requirements, logical and capacity processing including lockouts for concurrent systems
Understand how task switching works in computer systems
Understand how operating systems process database commands in multiuser systems
Understand key terminology of interprocess communications and operation
Understand how to troubleshoot problems with network capacity and collisions
Understand the role and usage of DBMS in enterprise systems
Understand capacities for data communications of different physical media
ABET Program Educational Outcomes

Objectives and Assessments
 INTENT
 That the program has documented, measurable objectives, including
expected outcomes for graduates. The program regularly assesses its
progress against its objectives and uses the results of the assessments to
identify program improvements and to modify the program’s
objectives.
 STANDARDS
 I-1. The program must have documented, measurable objectives.
 I-2. The program’s objectives must include expected outcomes for
graduating students.
 I-3. Data relative to the objectives must be routinely collected and
documented, and used in program assessments.
 I-4. The extent to which each program objective is being met must be
periodically assessed.
 I-5. The results of the program’s periodic assessments must be used to
help identify opportunities for program improvement.
 I-6. The results of the program’s assessments and the actions taken based
on the results must be documented.
A.
B.
C.
D.
E.
F.
G.
H.
An ability to apply knowledge of computing and mathematics
appropriate to the discipline
An ability to analyze a problem, and identify and define the
computing requirements appropriate to its solution
An ability to design, implement, and evaluate a computerbased system, process, component, or program to meet desired
needs
An ability to function effectively on teams to accomplish a
common goal
An understanding of professional, ethical, legal, security and
social issues and responsibilities
An ability to communicate effectively with a range of audiences
An ability to analyze the local and global impact of computing
on individuals, organizations, and society
Recognition of the need for and an ability to engage in
continuing professional development
I.
An ability to use current techniques, skills, and
tools necessary for computing practice.

Coverage of the fundamentals of







J.
i.1 a modern programming language
i.2 data management
i.3 networking
i.4 data communications
i.5 systems analysis and design
i.6 the role of Information Systems in organizations
i.7 project management
An understanding of processes that support the
delivery and management of information systems
within business application environments
Report:
ABET Program Outcomes Report
ICCP's ISA Part A Exam
Exam Date:
College:
Naitonal
Local
Course #
LU
83
91
0
19
a
38
36
4
7
a
44
46
5
5
a
71
91
7
4
a
35
27
8
4
a
32
36
9
0
a
51
46
0
9
b
82
82
2
3
b
29
9
5
10
b
24
27
5
18
b
39
9
7
7
b
53
64
9
0
b
56
64
10
5
b
63
82
0
9
c
22
18
2
6
c
19
27
5
18
c
47
46
8
5
c
44
46
9
2
c
ABET PO Item #
20-Oct-10
Your University College
Question Objectives
Given a spreadsheet application, choose the best technique to solve a
problem.
Identify mechanisms that allow multitasking in operating
10514
systems
Apply the order of precedence rules to evaluate an arithmetic expression. In this case, the
10395
steps are:
Given a high impact project with a short time line, choose the best development
7244
methodology.
Given a business need for source data automation, choose the method that best captures the
7116
necessary data.
Given a web page programming task, choose the appropriate type of
6969
tool.
The SWBAT specify the output components necessary for a computer system, given a specific business need, such as a point of
7102
sale terminal.
Given client/server Web applications, such as e-commerce shopping carts, students should understand characteristics
7143
of the technology.
Choose appropriate development tools for web site
7214
development
Given the need to maintain user information during a web-application session, choose the
7218
appropriate method.
Given unclear interview results the analyst selects an alternative
10444
course of action
Given a reported problem in a system module, such as a web form, evaluate various approaches to identify
6965
the error.
Given a system problem, the SWBAT determine the first place to begin the trouble7176
shooting process.
Given a system need, such as access control to a database, identify the necessary
6973
information.
Differentiate among computer components on the basis of functional
7105
use.
Given a new system, identify the best acceptance testing
6938
method.
Given a client-server database system, determine a thin-client
7207
solution.
Given a project, such as a new software component, develop an appropriate test
7029
plan.
7074

IS Model Curriculum Exam



Managed/Maintained by the ICCP
Individual Performance Measure
Basis for Information Systems Analyst (ISA) certification
 Several other certification requirements
 Proficiency Levels

Sample of other ICCP Certifications:



Associate Computing Professional (ACP)
Certified Computing Professional (CCP)
Certified Data Management Professional (CDMP)

Examination Administration Procedure






Application form
Proctor agreements
USB or networked USB
Cost to college: $50/student to a max of $2,000 per college
Cost to student: $250 to receive their ISA credential upon graduation from
their degree and must join recertification program
Results Reporting



Unofficial student performance profile
Office student performance profile
Institutional reports





National and Local scores by subject outline
National and Local scores by IS2010 Courses and learning units
Students ranked by order of performance best to worst, by college
ABET Program Outcomes report on request
Points of Contact

ICCP office : 800.843.8227; email: [email protected] website: www.iccp.org





Adopt the ISA exam at your institution
Use it in curriculum development
Use it in the accreditation process
Help create a database to support development
of national/international norms.
Encourage certification