TVET Australia Assessment, validation and moderation

A power point presentation developed by the NQC to support information sessions on assessment, validation and moderation

Contact

NQC Secretariat TVET Australia Level 22/ 390 St Kilda Road Melbourne Vic 3004 Telephone: +61 3 9832 8100 Email: [email protected]

Web: www.nqc.tvetaustralia.com.au

Disclaimer

This work has been produced on behalf of the National Quality Council with funding provided through the Australian Government Department of Education, Employment and Workplace Relations and state and territory governments. The views expressed in this work are not necessarily those of the Australian Government or state and territory governments

Acknowledgement

This presentation was designed to support the interactive information sessions that formed part of the NQC’s communication and dissemination strategy: NQC products: validation and communication. Reports and materials which focus on validation and moderation may be downloaded from the NQC website at http:www.nqwc.tvetaustralia.com.au/nqc_publications This work was produced for the National Quality Council by Andrea Bateman, Quorum QA Australia Pty Ltd Chloe Dyson, Quorum QA Australia Pty Ltd

Quality of assessment

Setting the Scene

 Concerns about the quality of assessments and comparability of standards across the VET sector    OECD (2008) Reviews of VET (Australia) NQC (2008) Industry Expectations of VET Service Skills SA (2010) VETiS Project

Today’s workshop

     Assessment Developing Assessment Tools Competency Mapping Simulated Assessment  Engaging industry Assessment Quality Management Framework    Validation and moderation System considerations Diverse settings

NQC resources

• Guide for Developing Assessment Tools • Assessment Facts Sheets • Simulated Assessment • Making Assessment Decisions • Peer Assessment and Feedback • Quality Assuring Assessment Tools • Assessor Partnerships • Systematic validation • Assessor Guide: Validation and Moderation

Session 1: What is assessment?

 Purposeful process of systematically gathering, interpreting, recording and communicating to stakeholders, information on student performance.

Assessment Purposes

    Evaluative 

designed to provide information to evaluate institutions and curriculum/standards – primary purpose is accountability

Diagnostic 

Produce information about the candidate’s learning

Formative 

Produce evidence concerning how and where improvements in learning and competency acquisition are required

Summative 

Used to certify or recognise candidate achievement or potential

Assessment Purposes

 Assessment for learning occurs when teachers use inferences about student progress to inform their teaching (formative)  Assessment as learning occurs when students reflect on and monitor their progress to inform their future learning goals (formative)  Assessment of learning occurs when teachers use evidence of student learning to make judgements on student achievement against goals and standards (summative)  http://www.education.vic.gov.au/studentlearning/assessment/preptoyear10/default.htm

Session 2: Developing Assessment Tools

NQC Products

• Guide for Developing Assessment Tools • Assessment Facts Sheets • Simulated Assessment • Making Assessment Decisions • Peer Assessment and Feedback • Quality Assuring Assessment Tools • Assessor Guide: Validation and Moderation http://www.nqc.tvetaustralia.com.au/nqc_publications

Impact

Changes to definitions within the NQC publications   AQTF 2010 User Guide documentation; and the Training Package Development Handbook Reliability Validity Assessment tool Validation Moderation

Key Stages – developing assessment tools

       identify and describe the purposes for the assessment identify the assessment information that can be used as evidence of competence/learning identify a range of possible methods that might be used to collect assessment information define the contexts for interpreting assessment information in ways that are meaningful for both assessor and candidate determine the decision making rules define procedures for coding and recording assessment information identify stakeholders in the assessment and define their reporting needs.

Essential Characteristics Assessment Tool

An assessment tool includes the following components:  The context and conditions for the assessment  The tasks to be administered to the candidate    An outline of the evidence to be gathered from the candidate The evidence criteria used to judge the quality of performance (i.e., the assessment decision making rules); as well as the The administration, recording and reporting requirements.

Ideal Characteristics

            

The context Competency mapping The information to be provided to the candidate The evidence to be collected from the candidate Decision making rules Range and conditions Materials/resources required Assessor intervention Reasonable adjustments Validity evidence Reliability evidence Recording requirements Reporting Requirements

Competency Mapping

 The components of the Unit(s) of Competency that the tool should cover should be described. This could be as simple as a mapping exercise between the components within a task (eg each structured interview question) and components within a Unit or cluster of Units of Competency. The mapping will help determine the sufficiency of the evidence to be collected as well as the content validity. Advice regarding competency mapping can be found in the NQC Assessor Guide:

Validation and Moderation

Competency Mapping

TAEASS401A Workplace Project: 14 tasks to be performed by the candidate

Plan Assessment Activities and Processes Elements

  Determine Assessme nt Approach Prepare the Assessme nt Plan  Develop Assessme nt Instrumen ts

Required skills

    Cognitive Interperso nal skills Research and evaluation skills Communi cation skills Interperso nal skills                                        1 2 3

Methods

Practical demonstration Third party report Workplace documents

Elements Component of Unit(s) of Competency BSBITU201A Produce simple word processed documents Required Skill and Knowledge Evidence Guide

Elements 1.1 RS3, RK3 Elements 1 – 3 RS1-5, RK 1-4 CA1, CA2 Elements 1-3 especially 1.1, 2, 3 RS1-5, RK 1-4 CA2

Competency Mapping: Steps in the process

   Step 1: Unpack the unit of competency to identify its critical components. Step 2: For each assessment method, list the tasks to be performed by the candidate.

Step 3: For each assessment method, map the critical components of the unit to each assessment task. Refer to NQC Assessor Guide: Validation and Moderation

Level of specificity in mapping – Risk Assessment

Risk can be determined by consideration of:  Safety (eg potential danger to clients from an incorrect judgement)    Purpose and use of the outcomes (eg selection purposes) Human capacity (eg level of expertise and experience of the assessors) Contextual (eg changes in technology, workplace processes, legislation, licensing requirements and/or training packages)

Decision Making Rules



The rules to be used to:



Check the quality of the evidence (i.e. the rules of evidence)



Judge how well the candidate performed on the task according to the standard expected



Synthesise evidence from multiple sources to make an overall judgement

Additional advice – refer to Fact Sheet 2

Reasonable Adjustments

 This section of the assessment tool should describe the guidelines for making reasonable adjustments to the way in which evidence of performance is gathered without altering the expected performance standards (as outlined in the decision making rules).

Simulated assessment

  For the purposes of assessment, a simulated workplace is one in which all of the required skills are performed with respect to the provision of paid services to an employer or the public can be demonstrated as though the business was actually operating.

In order to be valid and reliable, the simulation must closely resemble what occurs in a real work environment.

 The simulated workplace should involve a range of activities that reflect real work experience. It should allow the performance of all of the required skills and demonstration of the required knowledge.

Ref: AQTF definition (refer to Activity Handout), Assessment Fact Sheet 1

Activity 1: Engaging Industry

 In your groups discuss what input employers (you might wish to specify a vocational area) could provide to develop valid assessment tools and processes.  For the following scenarios, note down 2/3 questions you could ask employers and how the responses will inform the development or review of assessment tools and/or processes.

Relevant NQC support materials: • Industry Enterprise & RTO Partnership • Assessment Fact Sheets: Assessor Partnerships • Assessor Guide: Validation and Moderation

Activity 2: Self Assessment

 In groups of 3, review the assessment tool using the self assessment checklist from the NQC (2009) Implementation Guide (Template A.1, p. 45).

 Identify any gaps in the tool?

 Discuss the pros and cons of including such additional information within the tool?

Tool Review

      Has clear, documented evidence of the procedures for collecting, synthesising, judging and recording outcomes (i.e., to help improve the consistency of assessments across assessors [inter-rater reliability]). Has evidence of content validity (i.e., whether the assessment task(s) as a whole, represents the full range of knowledge and skills specified within the Unit(s) of competency.

Reflect work-based contexts, specific enterprise language and job-tasks and meets industry requirements (i.e., face validity).

Adheres to the literacy and numeracy requirements of the Unit(s) of Competency (construct validity).

Has been designed to assess a variety of evidence over time and contexts (predictive validity).

Has been designed to minimise the influence of extraneous factors (i.e., factors that are not related to the unit of competency) on candidate performance (construct validity).

Tool Review

       Has clear decision making rules to ensure consistency of judgements across assessors (inter-rater reliability) as well as consistency of judgements within an assessor (intra-rater reliability).

Has a clear instruction on how to synthesise multiple sources of evidence to make an overall judgement of performance (inter-rater reliability).

Has evidence that the principles of fairness and flexibility have been adhered to.

Has been designed to produce sufficient, current and authentic evidence.

Is appropriate in terms of the level of difficulty of the task(s) to be performed in relation to the skills and knowledge specified within the relevant unit(s) of Competency.

Has outlined appropriate reasonable adjustments that could be made to the gathering of assessment evidence for specific individuals and/or groups.

Has adhered to the relevant organisation assessment policy.

Quality Checks

  

Panel Pilot Trial

Refer to Fact Sheet 4, Quality assuring assessment tools

Session 3: Assessment Quality Management

NQC Products

• Code of Professional Practice: Validation & Moderation • Implementation Guide: Validation and Moderation • Assessment Facts Sheets • Quality Assuring Assessment Tools • Systematic Validation • Assessor Partnerships • Assessor Guide: Validation and Moderation http://www.nqc.tvetaustralia.com.au/nqc_publications

Validation

 Validation is a quality review process. It involves checking that the assessment tool produced valid, reliable, sufficient, current and authentic evidence to enable reasonable judgements to be made as to whether the requirements of the relevant aspects of the Training Package or accredited course had been met. It includes reviewing and making recommendations for future improvements to the assessment tool, process and/or outcomes.

NQC Implementation Guide: Validation and Moderation 2009

Outcomes of validation

Recommendations for future improvements       Context and conditions for the assessment Task/s to be administered to the candidates Administration instructions Criteria used for judging the quality of performance (e.g. the decision making rules, evidence requirements etc) Guidelines for making reasonable adjustments to the way in which the evidence of performance was gathered to ensure that the expected standard of performance specified within the Unit(s) of Competency has not been altered Recording and reporting requirements.

Moderation

 Moderation is the process of bringing assessment judgements and standards into alignment. It is a process that ensures the same standards are applied to all assessment results within the same Unit(s) of Competency. It is an active process in the sense that adjustments to assessor judgements are made to overcome differences in the difficulty of the tool and/or the severity of judgements.

NQC Implementation Guide: Validation and Moderation 2009

Outcomes of moderation

    Recommendations for future improvement and adjustments to assessor judgements (if required) and Recommendations for improvement to the assessment tools Adjusting the results of a specific cohort of candidates prior to the finalisation of results and Requesting copies of final candidate assessment results in accordance with recommended actions.

Validation vs Moderation

Features

Assessment Quality Management Type Primary Purpose Timing Focus Type of Approaches Outcomes

Validation

Quality Review Continuous improvement On-going Assessment tools; and Candidate Evidence (including assessor judgements) (desirable only) Assessor Partnerships Consensus Meetings External (validators or panels) Recommendations for future improvements

Moderation

Quality Control Bring judgements and standards into alignment.

Prior to the finalisation of candidate results Assessment tools, and; Candidate Evidence, including assessor judgements (mandatory) Consensus Meetings External (moderators or panels) Statistical Recommendations for improvements; and Adjustments to assessor judgements (if required)

Types of Approaches - Statistical

    Limited to moderation Yet to be pursued at the national level in VET Requires some form of common assessment task at the national level Adjusts level and spread of RTO based assessments to match the level and spread of the same candidates scores on a common assessment task  Maintains RTO-based rank ordering but brings the distribution of scores across groups of candidates into alignment Strength  Strongest form of quality control Weakness  Lacks face validity, may have limited content validity

Types of Approaches - External

 Types   Site Visit Versus Central Agency Strengths  Offer authoritative interpretations of standards   Improve consistency of standards across locations by identifying local bias and/or misconceptions (if any) Educative Weakness   Expensive Less control than statistical

Types of Approaches – Assessor Partnerships

      Validation only Informal, self-managed, collegial Smal l group of assessors May involve:  Sharing, discussing and/or reviewing one another’s tools and/or judgements Benefit   Low costs, personally empowering, non-threatening May be easily organised Weakness  Potential to reinforce misconceptions and mistakes Ref: Implementation Guide, Assessment Fact Sheet 5

Types of Approaches - Consensus

 Typically involves reviewing their own & colleagues assessment tools and judgements as a group  Can occur within and/or across organisations  Strength  Professional development, networking, promotes collegiality and sharing  Weakness  Less quality control than external and statistical approaches as they can also be influenced by local values and expectations  Requires a culture of sharing

Systematic Validation (consensus)

Indicators

Is there a plan for assessment validation (including validation of RPL assessment) in place?

Does your plan: • Determine the sample of units of competency to be validated over a set period of time • Provide dates for proposed validation activities • Include details about who will participate in assessment validation, including the Chair of consensus panels, if relevant • Include a strategy to ensure that all relevant staff are involved • Identify what processes and materials will be used for implementing and recording the outcomes of assessment validation Does your RTO have terms of reference in place to guide the work of consensus panels?

Does your RTO have validation materials (policy, procedure, forms) in place that cause participants to engage effectively in validation?

Does your RTO have a process for monitoring the action taken as a result of validation?

Does your RTO have a process and plan in place for reviewing the effectiveness of assessment validation?

Yes/No Action

Ref: Assessor Guide

System considerations

 What is the most appropriate approach to validation?

Condition

Whenever my RTO conducts internal validation few opportunities for improvement arise Our assessors are contractors and cannot come to validation consensus meetings because my RTO can’t afford to pay for their time and some are located interstate Our RTO conducts high risk units related to licensing, where the licensing authority has mandated the use of assessment tools it provides Our RTO is new and assessors do not have a lot of experience

Suggested approach

Consider including external representation on your validation panel Consider establishing assessor validation partnerships at your local level, but ensure that improvements identified are recorded and fed back to other assessors and formalised Consider consensus moderation, ideally with external representation on your panel.

Consider inviting an external person with expertise in assessment tool design to validation consensus meetings

Assessment Quality Management

   

Quality Assurance (Input approach)

Examples include:  Industry competency standards as the benchmarks for assessment  National assessment principles  Minimum qualifications for assessors (i.e., TAE40110)  Development of a Professional Code of Practice  Standardisation of reporting formats  Assessment Guidelines and Policy Documents  Benchmark examples of varying levels of performances Assessment tool banks Common assessment tasks Exemplar assessment tools Panelling, Piloting and/or Trialling of assessment tools.  Professional development programs/workshops for assessors

Quality Control (Outcome approach)

Examples include: 

Moderation

in which adjustments to assessor judgements are made to overcome differences in the difficulty of the assessment tool and/or severity of the judgement.

Quality Review (Retrospective approach)

Examples Include:  Monitoring and auditing of registered training organisations  Review and

validation

of assessment tools, processes and outcomes to identify future improvements.  Follow-up surveys with key stakeholders (e.g., student destination surveys, employer feedback on how well the assessment outcomes predicted workplace performance).

Quality management in diverse settings

  Identified barriers:  Structural (i.e., the organizational and resource aspects) – financial, variations of definitions across key documents  Process (i.e., the practices and activities that take place) – rolling enrolments, partnering arrangements, workloads  Personal factors (i.e., the attitudinal, assessment literacy and expectations of the key players).

Strategies deployed by RTOs Refer to Handout – Quality management processes in diverse settings.

Activity 3: Assessment Quality Management

Approach



Strategies implemented Activity 4: Assessment Quality Man

Quality assurance Quality control Quality management Ref: Implementation Guide

Andrea Bateman Director

Education Consultant Quorum QA Australia Pty Ltd Email: [email protected]

Phone: 0418 585 754

Chloe Dyson

Principal author

Associate Professor Shelley Gillis Deputy Director

Work-based Education Research Centre Victoria University Email: [email protected]

Phone: 0432 756 638

Director

Education Consultant Quorum QA Australia Pty Ltd Email: [email protected]

Phone: 0408124825