Portfolios for Lifelong Competence Assessment

1

Assessment of Competence More than One Approach

Traditional (primarily external scrutiny) Emerging (more internal, self-directed) Research (educationally-driven) 2

Assessment of Competence The Traditional Approach

Tests standardized where possible to maximize reliability (e.g. simulations - MCQ/ virtual, SP) Narrowly focused on knowledge and reasoning High stakes, summative May focus on individuals (“assessment”) or programs (“evaluation”) All about minimal competence Preferred approach of regulators (viz. societal protection) 3

Assessment of Competence The Emerging Approach

Tests tailored to the actual practice and learning needs of the individual Assessment beyond knowledge (e.g. includes skills and behaviors) Low stakes, formative, abundant feedback Usually focuses on the individual All about driving continuous improvement Impelled by innate professionalism (internal locus of accountability), not just external scrutiny 4

Assessment of Competence - The Research Approach Seeking better pedagogies, best practices Broadening base of assessment, shaping learning Finding effective balance between traditional and emerging approaches Focus may be on individual, but more often programmatic Seeking balance between baseline competence and continuing improvement Primarily the realm of professional educators Driven by need to define effective educational approaches 5

Availability of Data Sources and Methodologies

For the Traditional approach - Good For Emerging & Research approaches - Poor 6

Needed: Access to Many New Data Sets - Some Examples

Work learning experience - scope of clinical practice and changes over time Clinical performance - processes of care, outcomes, management Behaviors and skills Achievement of competency milestones Longitudinal research data set 7

Comes the Portfolio aka eFolio - Definition

An mechanism to collect electronic data sets in near real time (i.e. A Process) A particular collection of data and data sets specific to purpose - “Assessment of competence” (i.e. A Thing) Both a Process and a Thing (i.e. A “Proing”) 8

The Good News: Bad News Debate

Healthcare professionals leave a massive contrail of electronic data in their daily work BUT relevant data are very fragmented, scattered across a galaxy of repositories HIT now allows the potential for sourcing and compilation of data in near real time BUT the relevant repositories have scant tradition nor appetite for data-sharing 9

The eFolio Connector Vision

eFolio Connector Suite of apps compiling data reports for the learner, and for other viewers authorized by the learner Data Sourcing Data Commons A collection of Portfolios and Repositories that can provide data for many different applications 10

eFolio Connector: Initial Data Sets Planned

Suite of apps compiling data reports for the learner, and for other viewers authorized by the learner Data Sourcing Data Commons A collection of Portfolios and Repositories that can provide data for many different applications 11

Some Other Important Issues

Reporting out to authorized viewers specifications eFolio Connector Suite of apps compiling data reports for the learner, and for other viewers authorized by the learner Trusted Agent Hub Data Sourcing “Virtual Repository” Data Commons A collection of Portfolios and Repositories that can provide data for many different applications 12

The “Virtual Repository”: Affiliated Repositories Sign standard data-sharing agreements (Commons likely to become separate entity) Generically willing to share data, but decide on case by case basis Follow data custodian best practices Expose data in XML, comply with technical specifications for each data set May consume data, with authorization of the learner identified Might charge fee to Apps for data provided 13

More on Trusted Agent Hub

Provides a backbone of connectivity across Data Commons infrastructure Includes array of System Tools (Apps) for ID matching De-identification Authentication Business rules Permissions Transaction tracking and auditing 14

More on Report-Out to Viewers Channels of Distribution Viewers (authorization by learner or de identification for identified data) Report Out Apps and Data Compilations Data Commons

Compiling Data Sets: How Sausage Really Gets Made

Clearly define purpose (e.g. Understanding my practice) Devise necessary schema (e.g. to codify narrative) Do use cases to define content, context, use of data set Find reliable sources of data required Execute agreement with repositories to share data Build technical specification to facilitate interoperability Source data from repositories using web services Pay any agreed-upon fees to repositories providing data 16

Examples of New Data Reports In Development

eFolio Data Sourcing Data Commons

What If?

There is no current source for data required?

Add needed data to existing repository(ies) Build a new repository, affiliate with Commons Permanent access to data required, but repository(ies) unwilling or unable to maintain Arrange with repository to provide archiving Build central archiving function outside Commons Learner says that data in a report is wrong?

Build central process for reconciliation of data 18

What if? -2

Legal discovery is attempted on one of the data sets generated?

Compiled data sets are expunged by policy immediately after report-out (cf. Mandalas) and no data values are stored in the system A request is received for identified data from someone other than the relevant learner No date are moved, compiled or reported out without authorization of the learner 19

Evolution of the eFolio Connector - Feasibility Work

2007 Invitational Conference (sponsored by AAMC, ACGME, FSMB and NBME) 2008 Pilot Organizational Meeting; User advisory group established 2010 Feasibility Pilot - built ANSI-conformant international technical specification for “Educational Trajectory” data set 20

Evolution of the eFolio Connector: Pre-Implementation 2011 At least two more specifications (likely “Assessment of Milestones” and “My Practice”) Marketing, pricing, and business model Prototype featuring 3-4 reports for feedback 21

Conclusions

The eFolio Connector can source data from a diverse galaxy of repositories affiliated with the Data Commons and provide potentially useful data reports that bridge the continuum provide feedback on competence and inform lifelong learning Reports are available in near real time to learner and any other viewers authorized 22