Transcript Criteria 2000 Evaluator Training

1
Copyright © 2008 by ABET, Inc.
Conducting a Visit Using
the New Criteria
1
2
1.
2.
3.
4.
5.
6.
7.
8.
9.
Introduction
Criteria
Self Study
Visit Preparation and Conduct
Exit Meeting
Visit Forms
Draft Statement
Final Statement
Wrap Up
Copyright © 2008 by ABET, Inc.
New Criteria Visits Outline
Copyright © 2008 by ABET, Inc.
3
General Criteria
4
ABET Definitions (1)
• Program Outcomes – Program outcomes are narrower
statements that describe what students are expected to
know and be able to do by the time of graduation. These
relate to the skills, knowledge, and behaviors that students
acquire in their matriculation through the program.
Copyright © 2008 by ABET, Inc.
• Program Educational Objectives – Program educational
objectives are broad statements that describe the career
and professional accomplishments that the program is
preparing graduates to achieve.
5
ABET Definitions (2)
• Evaluation – Evaluation is one or more processes for
interpreting the data and evidence accumulated through
assessment practices. Evaluation determines the extent to
which program outcomes or program educational objectives
are being achieved, and results in decisions and actions to
improve the program.
Copyright © 2008 by ABET, Inc.
• Assessment – Assessment is one or more processes that
identify, collect, and prepare data to evaluate the
achievement of program outcomes and program
educational objectives.
6
Notes on definitions
• There is no requirement that the institution
use these terms as defined by ABET in the
self study. It is important to determine
what terms the institution uses and the
correspondence with the ABET definitions.
Copyright © 2008 by ABET, Inc.
• These are very important: you (and your
PEVs) should know them well.
7
Criterion 1 – Students
Comment: Similar to previous criteria.
Copyright © 2008 by ABET, Inc.
Students can complete the program in a
reasonable amount of time. They have ample
opportunity to interact with their instructors.
Students are offered timely advising, by qualified
individuals, about the program’s requirements
and their career alternatives. Students who
graduate from the program meet all program
requirements.
8
Criterion 2 – Program Educational
Objectives
Comment: Appropriate constituencies (e.g., faculty,
students, alumni, employers) should be used in
determining the constituency needs.
Copyright © 2008 by ABET, Inc.
The program has documented, measurable
educational objectives, based on the
needs of the program’s constituencies.
9
Criterion 3 – Program Outcomes
Comment: If there is broad constituency input to
objectives and outcomes are derived from objectives,
then broad constituency input to outcomes development
may not be needed. (The faculty is the primary
constituency involved in development of outcomes.)
Copyright © 2008 by ABET, Inc.
The program has documented, measurable
outcomes, based on the needs of the
program’s constituencies.
10
Criterion 3 – (continued)
a) An ability to apply knowledge of computing and
mathematics appropriate to the discipline
b) An ability to analyze a problem, and identify
and define the computing requirements
appropriate to its solution
c) An ability to design, implement and evaluate a
computer-based system, process, component,
or program to meet desired needs
Copyright © 2008 by ABET, Inc.
The program enables students to achieve, by the
time of graduation:
11
d) An ability to function effectively on teams to accomplish
a common goal
e) An understanding of professional, ethical, legal,
security and social issues and responsibilities
f) An ability to communicate effectively with a range of
audiences
g) An ability to analyze the local and global impact of
computing on individuals, organizations and society
h) Recognition of the need for, and an ability to engage in,
continuing professional development
i) An ability to use current techniques, skills, and tools
necessary for computing practice.
Copyright © 2008 by ABET, Inc.
Criterion 3 – (cont.)
12
Notes on Criterion 3 – (1)
• Institution is expected to define its own
outcomes.
Copyright © 2008 by ABET, Inc.
• Additional elements will appear in program
criteria.
13
Notes on Criterion 3 – (2)
• Institution will need to show how a-i are enabled.
• Notice that it says enables students to achieve;
doesn’t demand that all students have achieved
them.
– How can this be evaluated?
Copyright © 2008 by ABET, Inc.
– How can this be done?
– What about partial compliance?
– Is it sufficient for the institution to have an effective
assessment process that covers a-i?
14
Notes on Criterion 3 – (3)
Criterion 3 (c) should be interpreted as
appropriate for the nature and objectives of the
program.
–
–
3(c): An ability to design, implement and evaluate a
computer-based system, process, component, or
program to meet desired needs
Normally not all of “system, process, component, or
program” are required.
•
•
I.e., the “or” is not inclusive
One could be enough
Copyright © 2008 by ABET, Inc.
•
15
The program uses a documented process
incorporating relevant data to regularly assess
its educational objectives and outcomes, and to
evaluate the extent to which they are being met.
The results of the evaluation are documented
and used to effect continuous improvement of
the program through a documented plan.
Copyright © 2008 by ABET, Inc.
Criterion 4 – Continuous
Improvement
16
Notes on Criterion 4 (1)
• Must assess the extent to which outcomes are
being achieved by students.
• Effective assessment usually implies:
• No requirement that program improvements
must have been made.
– What is satisfactory evidence?
Copyright © 2008 by ABET, Inc.
– Measures based on actual student performance for
outcomes
– A priori establishment of performance goals
17
Notes on Criterion 4 (2)
• Must assess the extent to which objectives are being
achieved by graduates.
• Effective assessment for objectives usually implies:
– Collection of data relative to the achievement of each objective
• Surveys are normally used.
• No requirement that program improvements must have
been made.
– But assessment and evaluation results must be documented,
including decisions as to whether improvement might be needed.
Copyright © 2008 by ABET, Inc.
– A priori establishment of performance goals
18
Notes on Criterion 4 (3)
– This should not be discouraged, but it is not by itself
evidence of program improvement referred to in the
2008-09 CAC criteria.
• Proposed harmonized criteria recognize additional sources to
suggest improvements.
Copyright © 2008 by ABET, Inc.
• Note that the use of data/ideas from sources
other than the assessment of objectives and
outcomes achievement can be used to improve
the program.
19
The program’s requirements are consistent with its educational
objectives and are designed in such a way that each of the program
outcomes can be achieved. The curriculum combines technical and
professional requirements with general education requirements and
electives to prepare students for a professional career and further
study in the computing discipline associated with the program, and
for functioning in modern society. The technical and professional
requirements include at least one year of up-to-date coverage of
fundamental and advanced topics in the computing discipline
associated with the program. In addition, the program includes
mathematics appropriate to the discipline beyond the pre-calculus
level. For each course in the major required of all students, its
content, expected performance criteria, and place in the overall
program of study are published.
Copyright © 2008 by ABET, Inc.
Criterion 5 - Curriculum
20
Notes on Criterion 5
• Program criteria provide further specificity.
– No mention of enabling a-i.
– When might a-i enter into the picture relative
to the program’s curriculum?
• What does “… performance criteria …
must be published” mean?
Copyright © 2008 by ABET, Inc.
• Note that the curriculum must be designed
so that its outcomes can be achieved.
21
A. Faculty Qualifications
Faculty members teaching in the program are current
and active in the associated computing discipline. They
each have the educational backgrounds or expertise
consistent with their expected contributions to the
program. Each has a level of competence that normally
would be obtained through graduate work in the
discipline, relevant experience, or relevant scholarship.
Collectively, they have the technical breadth and depth
necessary to support the program.
Comment: Similar to present criteria.
Copyright © 2008 by ABET, Inc.
Criterion 6 – Faculty
22
B. Faculty Size and Workload
There are enough full time faculty members to provide
continuity, oversight and stability, to cover the curriculum
reasonably, and to allow an appropriate mix of teaching,
professional development, scholarly activities, and
service for each faculty member. The faculty assigned
to the program has appropriate authority for the creation,
delivery, evaluation and modification of the program, and
responsibility for the consistency and quality of its
courses.
Comment: Similar to present criteria.
Copyright © 2008 by ABET, Inc.
Criterion 6 – (cont.)
23
Criterion 7 – Facilities
Computing resources are available, accessible,
systematically maintained and upgraded, and otherwise
adequately supported to enable students to achieve the
program’s outcomes and to support faculty teaching
needs and scholarly activities. Students and faculty
members receive appropriate guidance regarding the
computing resources and laboratories available to the
program.
Comment: Similar to previous criteria.
Copyright © 2008 by ABET, Inc.
Institutional facilities including the library, other electronic
information retrieval systems, computer networks,
classes, and offices are adequate to support the
educational objectives and outcomes of the program.
24
The institution’s support for the program and the financial
resources available to the program are sufficient to
attract and retain qualified faculty members, administer
the program effectively, acquire and maintain computing
resources and laboratories, and otherwise provide an
environment in which the program can achieve its
educational objectives and outcomes. Support and
resources are sufficient to provide assurance that the
program will retain its strength throughout the period of
accreditation.
Comment: Similar to existing criteria.
Copyright © 2008 by ABET, Inc.
Criterion 8 –Support
25
Criterion 9 – Program
Criteria
Note that this criterion will disappear (as a criterion: its requirements will
remain) if the proposed harmonized criteria are adopted.
Copyright © 2008 by ABET, Inc.
Each program must satisfy applicable Program Criteria (if any).
Program Criteria provide the specificity needed for interpretation of the
General Criteria as applicable to a given discipline. If a program, by
virtue of its title, becomes subject to two or more sets of Program
Criteria, then that program must satisfy each set of Program Criteria;
however, overlapping requirements need to be satisfied only once.
26
(End of General Criteria)
Copyright © 2008 by ABET, Inc.
• Questions/comments?
27
Existing CAC Program Criteria
• Computer Science
• Information Technology
Copyright © 2008 by ABET, Inc.
• Information Systems
Program Criteria Notes (1)
28
• Applicable program criteria are determined
by the title (name) of the program being
evaluated.
• Check program requirements for consistency with
assigned criteria.
• Do this well BEFORE the visit.
– Graceful recovery from a problem is almost impossible
after the visit.
Copyright © 2008 by ABET, Inc.
– Determined by HQ based on the program title
listed on the RFE.
– Be alert to a possible mismatch between
expectations of program and HQ program
criteria assignment.
29
Program Criteria Notes (2)
– Examples: Information Science, Digital
Forensics, Computational Science
Copyright © 2008 by ABET, Inc.
• If a program title does not imply one or
more existing sets of CAC program
criteria, it is evaluated under the General
Criteria only.
Additional criteria added to General
Criteria:
• Capabilities that must be enabled.
• Curriculum requirements.
• Faculty qualification requirement.
Copyright © 2008 by ABET, Inc.
Computer Science Program
Criteria Additions
30
31
CS Program Criteria (1)
3. Program Outcomes
The program enables students to achieve, by the time of
graduation:
k) an ability to apply design and development principles
in the construction of software systems of varying
complexity.
Copyright © 2008 by ABET, Inc.
j) an ability to apply mathematical foundations,
algorithmic principles, and computer science theory in
the modeling and design of computer-based systems in
a way that demonstrates comprehension of the tradeoffs
involved in design choices;
32
CS Program Criteria (2)
5. Curriculum
Students have the following amounts of course work or
equivalent educational experience:
1. coverage of the fundamentals of algorithms, data structures,
software design, concepts of programming languages, and
computer organization and architecture.
2. An exposure to a variety of programming languages and systems.
3. proficiency in at least one higher-level language.
4. advanced course work that builds on the fundamental course work
to provide depth.
Copyright © 2008 by ABET, Inc.
a. Computer science: One and one-third years that
includes:
33
CS Program Criteria (3)
5. Curriculum (cont’d)
b. One year of science and mathematics:
2. Science: A science component that develops an
understanding of the scientific method and provides students
with an opportunity to experience this mode of inquiry in
courses for science or engineering majors that provide some
exposure to laboratory work.
Copyright © 2008 by ABET, Inc.
1. Mathematics: At least one-half year that must include discrete
mathematics. The additional mathematics might consist of
courses in areas such as calculus, linear algebra, numerical
methods, probability, statistics, number theory, geometry or
symbolic logic.
34
Notes on CS Curriculum
• No probability and statistics requirement.
• No comments about communication skills,
ethics, design, etc. since these are subsumed by
the abilities in the program outcomes criterion.
Copyright © 2008 by ABET, Inc.
• No specific number of credits of science.
35
CS Program Criteria (4)
Copyright © 2008 by ABET, Inc.
6. Faculty Qualifications
Some full time faculty members have a Ph.D. in
computer science.
Copyright © 2008 by ABET, Inc.
36
Visit Forms
36
37
•
•
•
•
•
Program Evaluation Worksheet (PEW)
PEV Visit Report (PER)
Program Audit Form (PAF)
Program Evaluation Tracking Form (PET)
Short Form (SF)
Copyright © 2008 by ABET, Inc.
Visit Forms to be Discussed
38
Program Evaluation Worksheet
(PEW)
• Completed by each PEV.
• Columns for before visit, after day 0 and day 1,
and at the exit meeting.
• Used by the TC to track the evaluation status
and as consistency check for the Draft
Statement.
– Collect a copy before the visit and at the end of the
visit.
• (View form.)
Copyright © 2008 by ABET, Inc.
– Updated by PEV as the visit progresses.
• Replaces the previous VRF.
• Much shorter than before.
• Includes curriculum and transcript evaluation
that is completed before the visit.
• Includes specific information about OO&A.
• Principles for completion are the same as
before.
• Includes the draft of the PEV’s section(s) of the
Draft Statement (exit meeting statement).
• TC combines all PERs into a single file for
submission to HQ.
• (View form.)
Copyright © 2008 by ABET, Inc.
PEV Visit Report (PER)
39
40
Program Audit Form (PAF)
• Similar to previous form.
• TC works with PEVs to produce one form for
each program.
• TC leaves forms with dean after exit meeting.
• TC combines forms for all programs into a single
file for submission to HQ.
Copyright © 2008 by ABET, Inc.
– Consistency with exit meeting statement/draft
statement is essential.
41
Program Evaluation Tracking
(PET) Form
• Similar to the previous PEA.
• Created by TC for each program from the
PEWs.
• Use the template.
• Used to track the evaluation status from the Exit
Meeting to the Final Statement, including editing.
• (View form.)
Copyright © 2008 by ABET, Inc.
– Eliminate Program Criteria sections not used.
– Combine all PETs into a single file, in alphabetical
order of program name.
42
Short Form (SF)
• One form for all programs at an institution.
– Essentially the same as what is used by other
commissions.
• (View form.)
Copyright © 2008 by ABET, Inc.
• TC prepares SF after the visit.
43
Deliverables
• From PEVs to TC (at end of visit)
Copyright © 2008 by ABET, Inc.
– Program Evaluator Worksheet (PEW)
– Program Evaluator Report (PER)
– Program Audit Form (PAF)
44
Deliverables (cont.)
• From TC to ABET HQ ([email protected])
•
•
•
•
•
Short Form (SF)
Draft Statement (DS)
PET for each program (one file)
PAF for each program (one file)
PER from each PEV (one file)
Copyright © 2008 by ABET, Inc.
– Within 2 days after visit: Initial SF
– Within 30 days after visit (5 files):
Copyright © 2008 by ABET, Inc.
45
Draft Statement
45
46
Draft Statement (2)
• There can be at most one each of deficiency,
weakness, and concern per criterion.
• Note that shortcomings in program criteria are
listed under Criterion 9 (Program Criteria), not
merged into other criteria.
Copyright © 2008 by ABET, Inc.
– A shortcoming may have multiple factors that
contribute to the shortcoming.
– See example statements.
47
Draft Statement (3)
• Do not use “category”.
• Use “criterion” instead.
Copyright © 2008 by ABET, Inc.
• Applies to old (as well as new) criteria.
48
Section for each program:
1. General description (faculty, students,
etc.)
2. Strengths
3. Shortcomings
4. Observations
Copyright © 2008 by ABET, Inc.
Draft Statement (5)
49
Draft Statement (6)
Copyright © 2008 by ABET, Inc.
Strength statements:
• What was observed.
• What makes it stand out above the norm.
• What positive impact it has on the
program.
50
Shortcomings
• Criterion citation
• What was observed that causes a
shortcoming
• Negative impact as a result of the
observed characteristics
Copyright © 2008 by ABET, Inc.
Draft Statements (7)
51
Copyright © 2008 by ABET, Inc.
Statement and Forms
Processing Summary
51
52
Statement and Forms
Processing Summary (1)
1. TC collects PEWs, PERs, and PAFs
from PEVs.
2. TC creates SF and PETs.
Submits SF to HQ.
3. TC creates draft DS.
4. TC submits files to HQ.
•
DS, SF, PERs, PETs, PAFs
Copyright © 2008 by ABET, Inc.
•