EML Alchemy NIST Voting Data Formats Workshop Gaithersburg October, 2009 David RR Webber Technical Editor OASIS Election Services TC E-mail: [email protected].
Download ReportTranscript EML Alchemy NIST Voting Data Formats Workshop Gaithersburg October, 2009 David RR Webber Technical Editor OASIS Election Services TC E-mail: [email protected].
EML Alchemy NIST Voting Data Formats Workshop Gaithersburg October, 2009 David RR Webber Technical Editor OASIS Election Services TC E-mail: [email protected] Overview Change is constant (even in the conservative voting world) how do we make trusted, cost effective information technology solutions, that improve democratic processes? How do we leverage and align standards based mechanisms and tools within our voting processes to make them more consistent, predictable, interoperable and verifiable? How does this fit into modern software engineering techniques, familiar best practices and popular development environments? Provide voting management staff the ability to vet and operate standard open processes without constant hand holding from technical staff. slide 2 What makes something interoperable? Clearly defined open processes and exchanges Based on open standards and approach; not proprietary Adaptive approach with consistent methods This allows tailoring to specific local profiles and use patterns When requirements change can be adjusted consistently Fault tolerant and not brittle Ability to ignore non-critical interchange items and especially not to fail for trivial technical reasons not related to actual information details Leveraging XML capabilities to make self-describing transactions and exchanges possible Support new uses without extensive reprogramming Usage patterns set via external configuration allowing broad but controlled uses slide 3 Agenda EML Technical Overview Introduction to transaction sets Typical simple illustrative use path OASIS interoperability event Recent US related EML work California Election Results Reporting Pew/Google VIP Project Geo-districting Massachusetts State Elections, IRV and EML Rapid EML localization tools Using OASIS CAM toolkit and EML templates Quick look – EML templates and NIEM IEPD process The Importance of Domain Dictionaries Summary / Review slide 4 EML Technical Overview Introduction to XML transaction sets Typical simple illustrative use path OASIS interoperability event The Voting Process Overview slide 6 Why XML? XML is multi-faceted – it can represent: data (information exchanges) rules workflows and processes forms and reports configuration and control semantics (metadata) documents and contracts multimedia and presentations XML pervasive across software systems slide 7 EML V6.0 What’s in the box? OASIS EML transactions – XSD Schemas EML XML dictionary – Excel spreadsheet Formal specification document EML templates – OASIS CAM templates* We will discuss EML transactions first… * CAM = Content Assembly Mechanism, OASIS standard template specification slide 8 Category Overview of EML transactions Pre-election Election details (100) Candidates (200) Voters (300) Ballot Options (600) Election One or more series of XML schemas are provided to support each general process area. Voting (400) Post election Results (500) Analysis (500) Audit (350, 400) slide 9 EML V6.0 Election Transactions Family Standard record structures to hold election information, ballot details, voting records and services Document Name Purpose EML 110 – election event Information about an election or set of elections. EML 150 – election boundaries Boundary geo-districting information and polling places. EML 210 – candidate nomination Used to nominate candidates or parties, consenting or withdrawing EML 230 – candidate list Contest and candidates details EML 310 – voter registration Used to register voters for an election EML 330 – voter election list Details of actual voters for an election EML 340 – polling information Notification to voter of an election, their eligibility and how to vote EML 410 – ballot Describes the actual ballot to be used for an election EML 420 – voter authentication Used for voter authentication during a voting process EML 440 – cast vote Actual record of vote cast EML 460 – votes group Group of votes being transferred for counting EML 480 – audit log Documents access to voting records and reason EML 510 - count Results of election contest(s) and counts EML 520 – results reporting Communicating specific result details on candidates and elections EML 530 – statistics reporting Statistics relating to the election results and votes cast slide 10 EML V6.0 – Supporting / Control transactions Used to manage security, audit control, access, system, setup and event tracking. Document Name Purpose EML 130 – response Confirm receipt and processing of EML transaction. EML 220 – response Candidate nomination response and confirmation. EML 350 – series of messages General system level of messages for operational control EML 360 – channel in/out detail Voting channel device selection and use EML 445 – retrieve vote Access and audit control to vote records, reason, event EML 450 – vote confirmation Acknowledges receipt of vote from voting channel EML 470 – voting token log Voting token digital certificate channel use log EML 610 – options nomination Nomination of option proposals for referendum EML 620 – options receipt confirm Confirm receipt of option proposal nomination details EML 630 – list of proposal options List of accepted option proposals for referendum ballot form slide 11 Example - EML 230 Candidate List <EML Id="230" SchemaVersion="6.0"> <TransactionId>OK-2007-09-1</TransactionId> <CandidateList> <Election> <ElectionIdentifier Id=“Oklahoma 2007" /> <Contest> <ContestIdentifier Id="State Governor" /> <Candidate>Brad Henry</Candidate> <Candidate>John Wayne</Candidate> <Candidate>Bill Okapi</Candidate> <Candidate>Jane Smith</Candidate> </Contest> </Election> </CandidateList> </EML> Example of an EML artefact in XML slide 12 Typical EML Process Path EML 230 candidate list EML 330 voter registration Pre-Election configuration EML 410 ballot layout EML 440 cast votes EML 460 vote group Polling activity EML 480 audit log activity EML 510 vote counts Tallying and Reporting EML 520 election results Also consider EML 360 voting channel use EML 470 voting channel digital token log EML 530 election statistics reporting slide 13 OASIS Demonstration – London, 2007 EML 330 voter registration records Voter selects voting channel preference EML 440 retained by voting channels EML 460 sent to IBM tabulator system in Brussels Reconciled with EML 330 totals per channel EML 510 count results EML 520 voting results Illustrates the reconciliation of the number of authorised voters with votes cast and votes counted by channel and in total. White Paper available describing event: http://www.oasis-open.org/committees/download.php/25992/EML%20Interop%20Demo%20Report.pdf slide 14 System interoperability testing Publish Structure Rules Context Examples Documentation Dictionary XML Templates Package Test Cases 2 3 1 Validate Report html XML Test Localization Content Hints Testing, Agreement / Alignment. Results Verify Result Outcomes System Uses & Creates XML 4 Pass / Fail slide 15 Technology Considerations Summary Use of XML in software systems is pervasive All platforms and all major developer environments support XML Everything from FaceBook™ to Microsoft Office™ to Oracle™ database is XML capable The OASIS EML transaction Schemas can be directly used by developer software tools to handle the associated XML transactions The OASIS EML templates provide support for generating additional XML artifacts to suit localization needs slide 16 Recent US related EML work California Election Results Reporting Pew/Google VIP Project Geo-districting Use of EML in State Elections with IRV California Election Results Reporting Audience – news media channels Developed in summer of 2007 Enhanced EML 510/520 New EML 530 statistics transaction Joint effort between CA SoS staff and OASIS EML TC slide 18 Outcomes Extend EML logical geographic constructs to handle USA district / ward structures Results for California 2006, 2007 and 2008 elections reported Validated ability of EML to report US election results Added extensible statistics reporting – same EML 530 now being used in Belgium as well slide 19 Sample of California Election Statistics EML 530 in web browser EML 530 XML slide 20 EML 150 - Pew / Google VIP Project Audience – voters District boundary, ballot and polling place information Mapping tool support EML 150 slide 21 EML 150 – Lessons Learned Re-usable components from EML allow rapid development of new transactions EML V6.0 dictionary provides core enabler EML 150 provides capabilities surpassing the original VIP XML layouts Includes GML standard mapping capabilities Polling place location, layout, access and facilities Aligned to OASIS EDXL for emergency management support OASIS EML technical committee able to rapidly respond to emerging needs and provide solutions VIP team is able to transition their existing data to new EML 150 format and gain benefits of broader information access and reuse (currently each state is new project). slide 22 Massachusetts State Elections, IRV, EML Feasibility study on use of EML to support IRV for state-wide elections Determine whether EML would be an appropriate way to convey partially aggregated ballots from an IRV (or other preferential ranking system) from a lower jurisdictional level up to a higher jurisdictional level where the ballots would be counted and the result of the election would be determined Currently changes being considered for Massachusetts law to use IRV for state-wide offices OASIS EML TC members supporting investigation into using EML v6.0 Possible enhancements envisioned during EML v6.0 review period to precisely cover identified needs slide 23 Rapid EML Localization tools Using OASIS CAM toolkit and EML templates Quick look – Templates + NIEM IEPD process Importance of Domain Dictionaries Localization Key Components XML Schema 1 EML Schema XSD aligns voting information model, processes and domain dictionary 2 Documentation 3 XML artifacts 4 Examples & Test Cases Readable information use details (HTML), EML cross-reference Spreadsheet The localized XSD Schema and picklist.xml Realistic test XML instances for conformance and interoperability testing Key components (aka IEPD*) *IEPD – Information Exchange Package Documentation slide 25 National Information Exchange Model Support for NIEM approach Builds components for IEPD (Information Exchange Package Documentation) Best practices template evaluation for NDR compliance http://www.niem.gov OASIS at NIEM training event: http://events.oasis-open.org/home/ei-summit/2009 slide 26 NIEM and OASIS work OASIS Emergency Management TC - joint MoU between NIEM and OASIS Development of NIEM “adaptor” for handling OASIS XSD schema and domain dictionary definitions within NIEM Core Integrated into NIEM 2.1 (latest release) Successful formula / election services overlaps Opportunity to follow same path for OASIS EML localization for USA use Potential benefits: NIEM is Federal, State, Local and Tribal facing - developers already using NIEM Broad community resources for training/facilitation slide 27 Localization Approach Steps EML 4 Voting Exchange 1 develop 2 XML XSD ingest Base CAM Template 3 localize Generate Revised CAM Template 5 Compare Manage 6 Steps: Local XML XSD Publish EML Dictionary Documentation + Examples 1. OASIS develops standard international XSD schema Implements the voting process, management and information exchange needs 2. Ingest the standard XSD into base CAM template 3. Apply localizations to CAM template Exclude unwanted optional components Align local exchange needs to standard Extend for local needs, add new components as needed 4. Generate new localized XSD schema that conform to EML standard 5. Build dictionary cross-reference of localization usage 6. Publish documentation and XML examples slide 28 EML V6.0 provides base CAM templates* Templates come pre-packaged in the EML V6 download * Open source Eclipse desktop tool: http://www.sourceforge.com/projects/camprocessor slide 29 How it works in a nutshell CAM Template OASIS EML XSD Collection XML Schema Common xsd Components XSLT tools process entire EML XSD collection CIQ xsd Structure Rules INGEST Namespace Localization EML xsd Documentation EML xml CAM = Content Assembly Mechanism slide 30 Example – Eclipse XML Structure Editor EML template is shown in the structure navigator panel for review and editing Template contains all the default content model and structure rules ingested from the XSD. All annotations and documentation from XSD also ingested (show as “paperclip” symbol). Code lists and typical content values inserted for easy visual reference. slide 31 Using Editor to mark Exclude items Structure Editor Tools Select focus on item Invoke action menu (right mouse click) Pick action Tip: exclude rule display is context sensitive and only available on optional items Tip: use “Add New Rule” mode to specify different XPath for exclude (quick mode assumes “current path”) slide 32 Extending the base EML schema In addition to excluding existing optional components you may need to add new pieces New pieces should be assigned their own extension namespace prefix – to identify them from the base EML components Typically the OASIS EML committee will review new pieces as candidates for adding to future versions of the standard • Send comments and suggestions to [email protected] slide 33 Adding New Domain Elements 1 2 Select root element node in structure; right mouse click for context menu; select “Add Namespace”; then enter prefix and URI Select element node in structure; right mouse click for context menu; select “Add Child Attribute / Element”; then enter prefix and name slide 34 Generating your EML use schema Once your localization details are completed you are ready to complete the remainder of the process This starts with generating your own new subset schema Then once schema details are OK – creating test cases, cross-reference spreadsheet and documentation slide 35 Generating subset use schema File Menu Option Select Export CAM as XSD menu option Confirm the location and filename, and namespace mode. 1 2 Select ‘default’ for Mode to write a regular import set of XSD schema for each namespace slide 36 Schema subset generated Set of XSD files with filename and namespace suffix Each namespace file is import for those specific type definitions Reviewing XSD results in a schema visual editor tool slide 37 Information Restriction Considerations The CAM template allows full use of XPath conditional expressions and a rich set of over 30 functions including: setNumberRange(), setLength(), setLimit(), setDateMask(), setNumberMask(), restrictValues(), lookup() Those that are compatible with XSD constraints will cause constraint schema assertions to be written out when exporting to schema Also option to set cross field validations using XPath conditional rules slide 38 Dictionary and Documentation Tools Dictionary Tools Create dictionary from CAM template Merge dictionaries together Compare to dictionary Excel spreadsheet compatible Documentation Tools Web interactive layout Business review tabular layout Developer raw template view Dictionary importance across process Discovery, reuse, consistency New development - library of components XML Tool support slide 39 EML v6.0 spreadsheet dictionary slide 40 Test Case Generator Feature Summary Make both Pass / Fail testing examples Content hinting so examples use real not fake data Test optional item logic with: all / random / none Uses exclude() assertions so does not include those items – makes realistic examples of your use pattern Can pass in seed value – use when adding and testing hints (each test case is labelled with its seed value) Make hundreds of test cases without manual editing Can link test case to XSD schema for structure tests You can modify XSLT tool for own testing needs slide 41 Run CAM Rules Check on Examples Run Menu Option 1 Pick Test Case Example to VALIDATE; click Finish to run validation rules 2 Review validation results 3 slide 42 Localization Approach Recapitulation EML 4 Voting Exchange 1 develop 2 XML XSD ingest Base CAM Template 3 localize Generate Revised CAM Template 5 Compare Manage 6 Steps: Local XML XSD Publish EML Dictionary Documentation + Examples 1. OASIS develops standard international XSD schema Implements the voting process, management and information exchange needs 2. Ingest the standard XSD into base CAM template 3. Apply localizations to CAM template Exclude unwanted optional components Align local exchange needs to standard Extend for local needs, add new components as needed 4. Generate new localized XSD schema that conform to EML standard 5. Build dictionary cross-reference of localization usage 6. Publish documentation and XML examples slide 43 Summary Review and take-aways EML V6.0 Technical Highlights Set of standard transactions for core processes in elections Extended control, event and audit tracking transactions Domain dictionary of common components used in EML v6.0 OASIS CAM templates allow rapid localizations Set of IEPD artifacts generation Compatible with NIEM.gov approach slide 45 Questions? slide 46 Localization Support Tools Quick Feature Review Sample Screenshots Tutorial Links Localization Support Tools Provides automated generation of need XML artifacts: Pick list development from template Schema subsets generation Dictionary cross-reference spreadsheet Use and rule documentation reports XML example instance and test cases Quick summary screenshots here Tutorial resources: http://oasis-open.org/committees/download.php/34489/EML%20and%20jCAM%20tutorial%20and%20primer.pdf slide 48 Test Case Generator Feature Summary Make both Pass / Fail testing examples Content hinting so examples use real not fake data Test optional item logic with: all / random / none Uses exclude() assertions so does not include those items – makes realistic examples of your use pattern Can pass in seed value – use when adding and testing hints (each test case is labelled with its seed value) Make hundreds of test cases without manual editing Can link test case to XSD schema for structure tests You can modify XSLT tool for own testing needs slide 49 Test Case Generation Quick Start File Menu Option Default directory to write examples into Name to be used for the examples How many examples to create Repeating elements count for Quick Test – just click “OK” to use default settings slide 50 Test Case Results Active links to view the generated examples slide 51 Run CAM Rules Check on Examples Run Menu Option 1 Pick Test Case Example to VALIDATE; click Finish to run validation rules 2 Review validation results 3 slide 52 Dictionary and Documentation Tools Dictionary Tools Create dictionary from CAM template Merge dictionaries together Compare to dictionary Excel spreadsheet compatible Documentation Tools Web interactive layout Business review tabular layout Developer raw template view slide 53 Dictionary Compare Tool Select compare tool menu option Browse and choose EML dictionary slide 54 Drag and Drop to Excel Spreadsheet Click OK slide 55 Tabular Documentation Documentation Menu Option Select Format, resulting HTML opens in browser viewer slide 56 Tabular HTML Content Details Clear use pattern Extended Code list handling Annotations XPath references and functions Enhanced Data type Logic slide 57 Resources Links to sites and examples Glossary IEPD - Information Exchange Package Documentation NIEM - National Information Exchange Model XSD - XML Schema Definition XSLT - XML style sheet language transformations slide 59 Simple EML 440 Example Example using the EML 440 cast vote schema Simplified to minimum requirements Example can be downloaded here: http://markmail.org/message/qyrpfq75mtctblyq slide 60 Resources: www.oasis-open.org/committees/election www.oasis-open.org/committees/cam www.oasis-open.org/committees/emergency www.niem.gov www.sourceforge.net/projects/camprocessor slide 61