Transcript synthesis map - color - ::Gateway Engineering Education
Faculty-Industry Collaborations for Assessing Student Learning
Flora McMartin
University of California - Berkeley
Jack McGourty
Columbia University
Gateway Engineering Education Coalition
Why Involve Industry?
Insures what is assessed is valued
Strengthens accountability
Addresses accreditation mandate
Gateway Engineering Education Coalition
Campus-Industry Relationship
% that have formal ties to business or industry for: Advisory panels Credit/non credit courses for employees Equipment donated, loaned, or shared Jointly sponsored programs Scholarship or loan programs All Institutions Research funding ACE Campus Trends Survey - 1996 87 72 71 66 62 31 Gateway Engineering Education Coalition
Synthesis Coalition
Berkeley Stanford Northern Cal Poly Arizona Synthesis Coalition Goals
•
Institutionalize reforms
• • •
Assess & disseminate reforms Develop NEEDS Institutionalize K-12 outreach Cornell Iowa State Hampton Southern Tuskegee Gateway Engineering Education Coalition
Gateway Coalition
•
Major Gateway Goals
•
Curriculum
•
Assessment
•
Underrepresented Populations
•
Instructional Technology
•
Professional Development
•
Linking & Sharing Gateway Engineering Education Coalition
Assessment Components
Stakeholder participation (faculty, students, & industry)
Faculty assessment training program
Campus (local) control of assessment process
Flexible, valid, and reliable tools Gateway Engineering Education Coalition
Assessment Goals
A. Involve stakeholders to measure effects of coalition program reforms B. Develop/promote valid & reliable performance assessment tools C. Promote the institutionalization of assessment at participating campuses D. Satisfy needs of: individual faculty, colleges, & coalition program evaluation E. Provide a foundation for ABET accreditation Gateway Engineering Education Coalition
ABET 2000 Learning Outcomes
1. math, science & engineering knowledge 2. experiment, analyze & interpret data 3. design a system, component or process 4. function on multi-disciplinary teams 5. identify, formulate & solve engineering problems 6. understand professional ethics 7. communicate effectively 8. understand global/societal impact of engineering 9. engage in life-long learning 10. knowledge of contemporary issues 11. use modern engineering techniques, tools & skills Gateway Engineering Education Coalition
Synthesis Case Study
Gateway Engineering Education Coalition
Student Learning Outcomes
Open Ended problem solving
Teamwork
Multi-disciplinary design
Hands-on facility with hardware
Communication skills - (oral, written, & interpersonal) Gateway Engineering Education Coalition
Goals for Involving Industry
Determine common learning outcomes valued by industry & engineering education
Articulate student learning outcomes in a common language
Bridge industry & engineering educational needs
Energize the assessment process through active participation of stakeholders Gateway Engineering Education Coalition
Mechatronics Industrial Board
Berkeley Process Control Motorola Maytag Rockwell International Xerox PARC Ford Raychem Hewlett-Packard
Gateway Engineering Education Coalition
The MIB, continued
Membership:
primarily project engineers
observed needs vs. self reports Role:
define what industry values in educational outcomes
try out & review assessment tools
try out & review assessment rubrics & measures
evaluate assessment results Gateway Engineering Education Coalition
Planning Process: Track 1
ALL stakeholders define & articulate Student learning outcomes (SLOs) & classroom activities related to Synthessis goals SLOs reduced & refined
Identifying Learning Outcomes
SLOs reviewed by ALL stakeholders -
draft of Synthesis Assessment Framework (SAF)
Draft SAF compared to scenario activities Overlap between SLOs & activities = MOST VALUED OUTCOMES = Final version of SAF Gateway Engineering Education Coalition
Track 1: Learning Outcomes
Industry Engineering Ed .
Most valued student learning outcomes = Synthesis Assessment Framework
Gateway Engineering Education Coalition
Planning Process: Track 2
Validating the Outcomes
MIB writes scenarios Scenarios edited, reviewed by MIB for accuracy Scenarios analyzed to ID activities related to SLOs Draft SAF compared to scenario activities Overlap between SLOs & activities = MOST VALUED OUTCOMES = Final version of SAF Gateway Engineering Education Coalition
Track 2: Validating with Scenarios
Scenario
a description of common or critical situations faced by an engineer
actual past, present, or future situation experienced by the MIB member
includes the context and actions of the engineer Gateway Engineering Education Coalition
Validation
approximately 80% overlap
Differences
ability to estimate resources (costs, time)
ability to supervise others
ability to take risks
ability to deal with ambiguity
ability to decide if project is worth pursuing Gateway Engineering Education Coalition
Assessment Planning Process
Track 1: Identification ALL stakeholders define & articulate Student learning outcomes (SLOs) & classroom activities related to Synthesis goals SLOs reduced & refined Track 2: Validation MIB writes scenarios Scenarios edited, reviewed by MIB for accuracy SLOs reviewed by ALL stakeholders -
draft of Synthesis Assessment Framework (SAF)
Scenarios analyzed to ID activities related to SLOs Draft SAF compared to scenario activities Overlap between SLOs & activities = MOST VALUED OUTCOMES = Final version of SAF Gateway Engineering Education Coalition
Assessment Tools
1) Institutional effect
• •
Course portfolio Course
• •
questionnaire Degree map Syllabi 2) Individual course & faculty pedagogy effect on learning outcomes
•
Self/peer
• •
assessment Design project report Scenario assignments
•
Audio/video observations 3) Exposure to Synthesis over time
•
Track learning outcomes assessments
•
Student portfolios
•
Alumni interviews Gateway Engineering Education Coalition
Scenario-Based Assessment
Assignment developed from MIB scenarios
Scoring rubric based on SAF
Assignment & rubric tested by MIB
MIB responses established baseline expert score
MIB analysis of rubric used to refine measurement criteria Gateway Engineering Education Coalition
Scenario Assessment Process
• •
Faculty select from 1 of 4 scenarios Faculty administer pre & post test Faculty evaluate diagnostic course grade
• •
Syn. HQ assess holistic analytic faculty revise course Gateway Engineering Education Coalition dept/col - revise curriculum Syn. - longitudinal study
Synthesis - Lessons Learned
Collaboration created a bridge between industry & faculty regarding engineering education
Clarified what is/is not possible to teach & learn
validated measurable outcomes
Identifying & articulating learning outcomes valued by ALL stakeholders built commitment to the assessment process & results.
Vitalized assessment process for all stakeholders Gateway Engineering Education Coalition
Gateway Case Study
Gateway Engineering Education Coalition
NJIT/NCE Assessment Statement
Institute an assessment plan and process that moves from the current anecdotally based assessment to a formal, rigorous, valid, and useful assessment and continuous improvement process.
Gateway Engineering Education Coalition
NJIT/ NCE Industrial Advisory Board
Lucent Technologies P SE&G PrimeMedia SIAC US Army Foster Wheeler Raytheon Curtiss Wright Becton Dickinson CompUSA Gateway Engineering Education Coalition
Industrial Sub-Committees
NJIT/NCE Advisory Board Strategic Planning Assessment Curriculum Review Gateway National Visiting Committee Gateway Engineering Education Coalition
Industrial Assessment Committee
Charter
To support the development of a comprehensive qualitative and quantitative curriculum assessment process that incorporates inputs and feedback from the employer and takes into account the fast pace of technological change.
Gateway Engineering Education Coalition
Industrial Assessment Committee
Short-term objectives
Identify/prioritize the core competencies required for entry-level engineers
Work with faculty to identify educational requirements based on core competencies
Assess current curricula as to its effectiveness in providing students with the required core competencies Establish employer level metrics to measure graduate’s skill proficiency and provide a baseline for continuous improvement Gateway Engineering Education Coalition
Industry Partner Role
Participating member of faculty assessment team Bring in assessment best practices from industry such as planning, TQM processes, metrics, multi source feedback, assessment centers, etc Form relationships between school and engineering/human resources organizations Act as liaison with Gateway Coalition National Visiting Committee Gateway Engineering Education Coalition
Assessment Design Steps
Step #5 Apply Results Step #1 Define Objectives, Strategies, & Outcomes Continuous Improvement Step #2 Identify Assessment Methods Step #4 Implement/Expand Assessment Processes Gateway Engineering Education Coalition Step #3 Develop/Pilot Assessment Processes
Focus Group Objectives
Design Step #1
Identify outcomes, objectives & performance criteria for each course and program represented Discuss existing assessment tools Review potential use for continuous improvement Gateway Engineering Education Coalition
Participating Programs
Curriculum Reform
FE/FED - Freshman course in design & manufacturing
General University Requirements Learning Tools
Computer-aided learning
Industrial Interaction
Intern program
Advanced Manufacturing Laboratory (AML) Seminars Series “What Do Engineers Do?”
Advance Design Engineering in Product Teams (ADEPT) Gateway Engineering Education Coalition
People Involved
35-45 faculty members involved Multidisciplinary - 3 of 4 colleges involved Across all levels with enthusiastic involvement from tenured faculty Covers courses & programs in all four years Deans and department chairs Members of NJIT/NCE Advisory Board Gateway Engineering Education Coalition
Learning Outcomes
Question:
What critical knowledge, skills, and behaviors must students acquire in this course?
Example:
Student will demonstrate an ability to actively participate, listen and collaborate with other team members while working on a technical design project.
Gateway Engineering Education Coalition
Results: 9 Core Learning Outcomes
Analytical Thinking Communication Skills Creative Problem Solving Project Management
Research Skills Self-Learning Systems Thinking Teamwork Technical Competence Gateway Engineering Education Coalition
Learning Outcome Definitions
Analytical Skills
Applies logic in solving problems and analyzes problems from different points of views. Translates academic theory into practical applications and recognizes interrelationships among problems and issues.
Gateway Engineering Education Coalition
Learning Outcome Survey
NCE Learning Outcomes Instructor/Coordinator The purpose of this survey is to gather your ratings on the importance of each of the nine broad learning outcomes and their specific objectives as identified by you during the focus group sessions conducted recently by the NCE Assessment Team. Please review each statement carefully and rate it using the scale provided. In addition, you are provided with extra space to add and rate objectives within each broad learning outcome.
Technical Competence How important are the following student learning objectives to the satisfactory completion of your course or program: Not at all important Minor importance Moderate importance Considerable importance Critical importance 2 3 4 5 Demonstrates a basic knowledge of fundamental engineering principles in the specific disciplines focused on in this course/program 1 Integrates basic knowledge of other 1 engineering disciplines within the scope of the course’s project 2 3 4 5
Gateway Engineering Education Coalition
Ratings by Faculty and Industry Partners
Analytical Thinking (3.83, 3.52) Communication Skills (3.99, 4.62) Problem Solving (3.62, 4.20) Project Management (4.10, 4.44)
Research Skills (3.82, 3.54) Systems Thinking (3.84, 4.65) Self-Learning (3.82, 3.22) Teamwork (4.23, 4.67) Technical Competence (3.61, 3.54)
Other analyses show that learning outcome emphasis changes as one moves to upper division curricula.
Gateway Engineering Education Coalition
Advisory Board - Post Review
Discuss/answer the following questions:
Think about an engineer that has recently entered your organization and is regarded as highly effective (up to speed quickly, little retraining, etc.)
Does he or she demonstrate the selected competencies? Give examples of how they demonstrate each selected competency. Would you add specific statements to the definition of a selected competency?
What performance criteria would you use to evaluate successful demonstration of the selected competencies in your organization?
Would you add any other competencies that new engineers should have based on your experience with the successful engineer above?
Gateway Engineering Education Coalition
Other Outcomes
Creation of several competency-based assessment processes
Revised portfolio process for freshman design experience
Team Developer version for engineering learning outcomes
Senior Exit, Alumni, and Employer surveys Development of new assessment planning process diffused to all Coalition Schools and beyond Gateway Engineering Education Coalition
Gateway - Lessons Learned
Industrial partners add value to assessment process, especially in providing a real-word perspective on educational objectives and learning outcomes Process fostering comparison helps to promote new thinking Important for sub-group of industrial partners to work closely with faculty throughout assessment cycle Must keep all participants actively involved in planning and decisions Gateway Engineering Education Coalition
Conclusions
INPUT Fresh perspectives & real world insights to the assessment enterprise. New ways of thinking about education Commitment to improving education OUTCOME Vitalizes planning process & boosts faculty commitment/involvement to assessment Increased campus support for innovative teaching & assessment practices Industry partnerships focused on education in both sectors Gateway Engineering Education Coalition
The Future of Collaboration
Current research shows:
In most cases, industry kept to the periphery of assessment
Faculty & campus administrators fear industry partners will make unreasonable demands because they don’t understand the challenges facing higher education
Colleges do not know how to capitalize on the expertise of industrial partners
Assessment is still in fledgling stages The conclusions from case studies confirmed by Colleges who integrated industry into their assessment processes Gateway Engineering Education Coalition