Transcript Assessment in ITOM and Beyond - Institutional Effectiveness
Assessment in ITOM and Beyond: Defining, Assessing, and Documenting Student Learning Outcomes
G A I L W I S A N U N I V E R S I T Y D I R E C T O R O F A S S E S S M E N T O F F I C E O F I N S T I T U T I O N A L E F F E C T I V E N E S S A N D A N A L Y S I S G W I S A N @ F A U . E D U X 7 1 0 0 6 ( 5 6 1 ) 2 9 7 - 1 0 0 6 M A R C R H O R E R A S S I S T A N T D E A N C O L L E G E O F B U S I N E S S M R H O R E R @ F A U . E D U X 7 0 2 1 0 S E P T E M B E R 2 0 1 0
Overview :
Describe the Nature and Purposes of Assessment Relate Assessment in ITOM to Larger Context Explain the uses of a Curriculum Matrix and how it relates to the Academic Learning Compact (ALC) Writing better learning outcomes Assessing and Documenting Student Learning Outcomes Close the Assessment Loop: Identify the Do’s and Don’ts of how you will USE your assessment results
Assessment in FAU ITOM Programs: The Context 1.
2.
3.
4.
5.
National: Department of Education and Public Opinion Regional Context: SACS Accreditation Florida Context: Academic Learning Compacts with required annual reports to the state BOG on continuous learning improvement Institutional Context: Strategic Plan The College of Business Context: o o Core Curriculum Business programs are accredited by of Business (AACSB International) The Association to Advance Collegiate Schools
Assessment in FAU ITOM Programs: The SACS Accreditor Context The Commission on Colleges of the Southern Association of Colleges and Schools (SACS) is the recognized regional accrediting body in the eleven U.S. Southern states (Alabama, Florida, Georgia, Kentucky, Louisiana, Mississippi, North Carolina, South Carolina, Tennessee, Texas and Virginia) and in Latin America for those institutions of higher education that award associate, baccalaureate, master's or doctoral degrees. It is charged with carrying out the accreditation process.
SACS Accreditation (2012-2013): Need 3 years of good data on learning outcomes assessment and continuous improvement. o 2009-2010 o o 2010-2011 2011-2012
Assessment in FAU ITOM Programs: The SACS Accreditor Context 1.
2.
3.
4.
Accreditation standards requires all institutions of higher learning to establish college level learning goals Assess student achievement against those goals Provide evidence of student achievement Use results of assessment to improve student learning outcomes
Assessment in ITOM and College of Business: The AACSB Context Accreditation standards requires that: “The school specifies learning goals and demonstrates achievement of learning goals for key management-specific, and/or appropriate discipline-specific knowledge and skills that its student achieve in each undergraduate degree program.”
Assessment in ITOM and College of Business: The AACSB Context Accreditation standards requires that: “…the school specifies learning goals and demonstrates achievement of learning goals in
each specialized master’s degree
program.”
The AACSB Context
AACSB, at the undergraduate level, is concerned about “degree programs” Academic Learning Compact (ALC) required by Florida for each degree program and could be aligned with AACSB /COB learning goals. (ALCs will have some unique content and skills learning goals for degree programs.) The COB learning goals (AACSB) for the BBA/BS program are universal across all majors
The AACSB Context BBA/BS Learning Goals (approved 2006) – our graduates will: Demonstrate functional knowledge specified in the college core Understand the impact of global economic and multicultural issues as they related to business Demonstrate the ability to utilize computing or telecommunications technologies in business practice and making business decisions Demonstrate communication skills appropriate for a business professional
The AACSB Context
BBA/BS Learning Goals (approved 2006) – our graduates will: Demonstrate analytical critical thinking Understand ethical and legal implications of business decisions and practices Understand managerial and legal aspects of conducting business in a diverse environment
The AACSB Context
Perhaps FAU COB learning goals need slight revision or combining for conciseness – examples: 2 goals with similar/same skill of computer technology 2 goals with similar/same issue of understanding legal aspects 2 goals with similar/same issue of multiculturalism and diversity Several learning goals (AACSB) overlap Academic Learning Compacts (BOG) – recommend simplification and reduce duplication of effort
The AACSB Context
Example of concise learning goals from the undergraduate degree program (Virginia Tech – Pamplin COB) Students will be effective communicators Students will be analytical problem solvers Students will be proficient with a wide range of relevant business-related technology Students will have business skills that include an international emphasis
The AACSB Context
AACSB emphasizes assurance of learning (AOL) now, as much as faculty sufficiency Key areas of emphasis for AACSB in AOL Actively engaged participants Faculty involvement and understanding of the process Assessment of learning, not faculty methods or teaching Meaningful discussion of improvement among stakeholders, NOT collection of data and the process of assessment methods Alignment of learning goals, what faculty teach, and what students experience/learn Group work cannot be used for AOL assessment unless specific contribution/authorship is attributed to each student
The AACSB Context
The essence of AACSB Assurance of Learning in 3 questions What are students supposed to learn in your program?
How do you know that they are learning these?
What can you do to improve learning?
If these cannot be answered, there will be problems with the reaffirmation of accreditation
The AACSB Context
Assurance of Learning research and Relationship to Faculty Sufficiency AACSB values and recognizes research in AOL Research in student learning in business programs (AOL) is applicable and valued by AACSB Publications from assessment and AOL are applicable to attainment / maintenance of AQ status for terminally qualified (PHD) faculty Journal outlets / examples
What Is Student Learning Assessment?
Assessment is a process in the learning cycle in which faculty measure and document the degree to which students are attaining defined (should be valued and important) learning outcomes Assessment is a student learning-centered and faculty-guided approach to improving student learning in the classroom and in degree programs
Defining Assessment: The Three Steps of Assessment
1.
2.
3.
Articulate the program/course goals for student learning Learning Outcomes: Clear Measurable Expected Outcomes When completing the program, students will be able to ….
Systematically gather evidence about whether goals are being met. Where are the learning opportunities in the program?
Measuring outcomes: gathering, analyzing, & interpreting data Use Information for Improving Student Learning Outcomes.
Defining Assessment: The Three Questions to Ask
1.
2.
3.
What are the program student learning outcome goals?
When completing the program, What should students know? What should students be able to do? What skills should they have?
How will the program/course gather evidence about whether students have achieved the expected learning outcomes?
In what courses should students have learned the skill? How was the skill measured? Was it measured at different times in the program?
How will the program/class gathering, analyzing, & interpret results?
How will the program/course Use the Information gathered to Improving Student Learning Outcomes?
What Assessment Isn’t
Assessment is not just giving grades Assessment is not an evaluation of faculty Assessment is not an infringement of academic freedom Assessment is not an invasion of student privacy Assessment is not punitive
The Value of Assessment: What It Does Best…
“Much of the value of assessment comes from the systematic way it makes educators question, discuss, share and observe.” Palomba and Banta (1999). Assessment essentials: Planning, implementing, and improving assessment in higher education. San Francisco: Jossey-Bass.
The Purposes of Assessment: Why Assess Student Learning?
To provide evidence of institutional effectiveness To demonstrate the effectiveness of ITOM, College of Business and FAU courses/programs to all stakeholders (internal & external) To know your students’ weakness and strengths and use those results to improve overall quality of teaching and learning within your program To provide critical feedback to students and faculty To provide students with clear statements about how they can expect to improve skills and abilities
Types of Assessment: Indirect
Learning is subtle, often supporting direct methods Grades not based on explicit criteria Course evaluations Student hours spent on active learning and/or service learning Focus group interviews with students Employer or alumni surveys Student perception surveys Employer and internship supervisor ratings
Types of Assessment: Direct (Preferred)
Learning is tangible and visible Course assignments Examinations Classroom Assessment Techniques (CATs) Term papers and reports Research projects Case study analysis Portfolios – body of work Capstone projects Licensure exams
ANALYTIC Rubrics Used to score performance
Use…
Results… To improve teaching and learning
Assess…
Student achievement against those goals or outcomes
So, The Cycle Continues
Plan…
Establish Learning Goals Learning Outcomes
The Assessment Cycle Provides Feedback for Continuous Improvement Implement…
Provide Opportunities for students to achieve those goals
Why Aren’t Grades Sufficient?
Grades (A’s, B’s & C’s) in and of themselves do not give you enough information about students’ strengths and weaknesses Increasing emphasis is now being put on students: thinking critically acquiring life-long learning and business professional skills acquiring social values Assessment can build on to the grading process, but grades alone do not provide the kind of feedback we want
What are key Student Learning Outcomes in FAU’s MIS B.B.A/B.S. Degrees and how are they assessed ?
?
Critical Thinking in MIS
Man. 4720: Students will demonstrate analytical critical thinking through their ability to apply relevant business models and/or strategic concepts and tools in a research project on a firm and/or industry. (ALC) Fin. 3403: Students will apply financial math and valuation principles to value securities, capital projects and other assets. Students will interpret financial information and use it in decision making. (ALC)
Some Key Student Learning Outcomes in ITOM Programs and their Assessment
Written Communication in ITOM
ENC 3213:Demonstrate Communication Skills appropriate to a professional (basic level) (ALC) ISM 4133 Adv. Design & Anal.: ALC (more advanced )
Oral Communication in ITOM
ISM 4133: The presenter effectively communicates the organizations’ problem(s), solution(s), and methodologies used to obtain solution(s). (Outcome stated in rubric provided in ALC.)
Writing Better Learning Outcomes
DO
•
Better: Students will locate information, evaluating its validity and appropriateness for a project.
•
Better: Students will: 1.) accurately describe and analyze two major management approaches , and 2.) evaluate each perspectives strengths and weaknesses.
DON’T
•
Too Vague:
Students will demonstrate information literacy skills. •
Too Specific:
institutional databases to demonstrate information literacy.
Students will be able to use a specific •
Too Vague:
Students will demonstrate critical thinking skills. •
Too Specific:
Students will be able to describe the X management perspective.
•
Better: Students will write clearly organized position papers, providing financial and spreadsheet support.
•
Too Vague:
Students will write proficiently. •
Too Specific:
Students will write a five page paper with no more than 3 grammatical errors.
Writing Better Learning Outcomes
DO
•
Better: Students will systematically analyze and solve problems, advocate and defend their views, and refute opposing views.
•
Better: Student teams will analyze and evaluate case studies about ethical business dilemmas.
•
Better: to identify the major periods in 19 th context.
Students will be able and 20 th century European art, describing the art styles and their cultural DON’T
• •
Too Vague:
Students will solve problems.
Too Specific:
problem x.
Students will find a solution to •
Too Vague:
Students will demonstrate ethics. •
Too Specific:
Students will value the x point of view of ethical business??? (arguable?) •
Too Vague:
Students will appreciate art. •
Too Specific:
Students will be able to identify 7 of 8 artists from their paintings.
What is a Rubric and Why Use One?
A Systematic
Scoring Guideline for
evaluating performance (e.g., case study analysis, research paper, business plan, book review, exam) via descriptions of criteria for different levels of achievement Rubrics provide structure for consistent and unbiased grading
What are the 2 Main Types of Rubrics and their Use?
Holistic Rubrics
performance. provide a single score based upon multiple factors for each level of
Analytic Rubrics
provide multiple scores based upon distinct achievement criteria/factors from which they can then create a single score.
What are Holistic Rubrics?
Holistic Rubrics
standards provide a single score for a whole performance (e.g., research paper or case study analysis is given one score) : Excellent, Good, Fair, Poor or Exceeds standards, Meets standards, Below Each Grading Level using MANY CRITERIA so
you cannot identify specific strengths/weaknesses
Holistic Rubric’s Value and Purposes:
Easy to Grade Quick Picture of Performance Single Dimension is Sufficient for Grading Helpful tool for calibrating grading standards across multiple classes Not useful for improving student learning outcomes standards across multiple classes
What are Analytic Rubrics? Why Use?
Analytic Rubric
provide specific feedback on multiple criteria.
Value
: Performance (e.g., research paper or case study analysis) is assigned many scores on key outcome criteria so that student and class outcomes can be assessed as to strengths and weaknesses.(See ITOM rubric in ALC.) Assessment Grading uses MANY Specific CRITERIA which can then be used to calculate (automatically if desired) total grade.
Analytic Rubric’s Value and Purposes:
Detailed feedback to learners, professors, and programs More detailed scoring can be used to guide improvement Rubric, when provided to students, can help them identify key factors in their performance and thus guide achievement Useful feedback to faculty and the ITOM Program on students’ strengths and weaknesses on specific criteria Assessment results can be used to modify learning opportunities in the curriculum and improve the program and document continuous improvement cycle in assessment.
Association of American Colleges and Universities (AACU): VALUE Project VALUE: Valid Assessment of Learning in Undergraduate Education Valid data needed to guide planning, learning, teaching, and improvement Good Practice in assessment requires multiple assessments over time http://www.aacu.org/value/rubrics Developed Tested Rubrics Critical Thinking Problem Solving Written Communication Oral Communication Others
Keep It Simple
Utilize the same student work (artifact) for various assessments of a learning outcome Example: A Case Study Analytic and company or industry problem solving paper on improving profits Multiple outcomes assessed using this paper Critical Thinking Written Communication Team Work Oral Communication
Curriculum Outcomes Matrix: What is it?
Curriculum mapping is a method to align instruction with desired goals and program outcomes. It can also be used to explore what is taught and how. The map or matrix: Documents what is taught and when Reveals gaps in the curriculum Helps design an assessment plan http://manoa.hawaii.edu/assessment/howto/mapping.
htm
Curriculum Outcomes Matrix: Benefits
Why do it? Benefits
Improves communication among faculty Improves program coherence Increases the likelihood that students achieve program-level outcomes Encourages reflective practice http://manoa.hawaii.edu/assessment/howto/mapping.
htm
Assessment In Multiple Courses
Introduce the learning outcome skill (I) Reinforce the learning outcome skill (R) Master the learning outcome skill (M)
Department matrix : More Benefits
Process more important than product Thinking in skills, not only content knowledge Tool for constructing a degree program building in redundancy Plays to strengths of individual faculty Each person need not cover all skills in a single course
Community and Communication within department Working with matrix Requires mutual intellectual respect Requires willingness to find out what colleagues are doing and why Requires opening oneself up to similar scrutiny e.g. “Why do you think students need to understand spreadsheet applications?” Helps Further Develop Sharing, Discussion and Community within the Department
Additional Skills: Learning Outcomes for Matrix Information Literacy skills Statistical analysis skills (broken down in another document to identify strengths and weaknesses) Oral Presentation skill Projection, Simulation and computer modeling skills Specific language skills Microsoft office skills Problem Solving skills: Analysis, evaluation, creative solutions
The Outcomes Matrix and Assessment Observations of student success and difficulty in the capstone course or other core courses Use the skills/learning outcome categories on the matrix to frame specific questions to a class in exit interviews/surveys Formative assessment in various courses How Can Assessment Results be Used to Improve Student Learning Outcomes for program graduates.
Timetable
When you will start collecting the assessment data?
How often you will collect it?
When will you plan to report on the
results?
Reporting on the Results
Now, What Are You Going To Do With All These Good Data?
Collaborate with your departmental colleagues on the findings Determine how the results can be used to increase students’ learning and success Put assessment results into action Re-assess to measure the effectiveness of your course/program modifications
Explaining your use of results
DO
•
DO
focus on making specific improvements based on faculty consensus.
•
DO
address specific program improvements that will impact student learning.
DON’T
•
DON’T
focus on simply planning for improvements or making improvements without faculty feedback. •
DON’T
address assessment improvement plans such as revising the rubric.
•
DO
use concrete ideas.
•
DON’T
write vague ideas.
• •
DO
state strategies that are sustainable and feasible.
DO
use strategies that can improve the curriculum and also help improve student learning outcomes (e.g. solve problems).
•
DON’T
use strategies that are impossible to complete in one year considering your resources.
•
DON’T
focus only on improving the curriculum.
Suggestions
Review and Revise Academic Learning Compact Coordinate Assessment Plans with ALC Set priorities as a department “Hand off” specific learning outcomes assessment to course/professor best suited to task but share and use results Consider using Analytic Rubrics Be clear as to how assessment results will be used to revise pedagogy, curriculum matrix, and teaching/learning strategies in program/courses
Conclusions?
Align Assessment Activities to make them Useful Don’t duplicate work: Align the work Create An Outcomes Curriculum Matrix Align assessment for AACSB, ALC, and FAU Assessment Database Document and Report Results Identify Specific ways in which assessment results will be shared Identify Specif ways to USE data to improve learning
The Flow of the Assurance of Learning - AACSB
Questions?
A S S E S S M E N T I S A T O O L T O B R I N G
About better teaching and learning!
www.fau.edu/iea/assessment [email protected]
Appendix
ITOM ALC on remaining slides
B.B.A. & B.S. Management Information Systems College of Business Dep’t. of Information Technology And Operations Management