Development of Accessible E documents and Programs for the Visually Impaired Creating accessible Windows programs

1. Today topics

• Most frequent problems and solutions • Light intro to windows in Windows • Back to principles of screen reader • MSAA • Accessibility in Java

2. Important from previous sessions

• Applications can be controlled only from keyboard • User has no information about screen as a whole • User has no access to graphical information • Standard applications are not served in special environments

3. Most frequent problems

• Keyboard navigation problems: There are some components which are not accessible from the keyboard • Icons without the text alternative (for example, Hint property) are often used • Non-standard focus representation: For example, caret in edits is often hard to track

4. Most frequent problems (2)

• Time limits on input activities: For example, Windows Balloons • Non-customizable user interface: Some users may have problems to see some colour combinations (contrast problems)...

• Non-standard user interface components without special support for screen readers • Lack of keyboard shortcuts: Kbd shortcuts are very important in standard applications. Provide shortcuts to all functions if possible

5. Shortly about Windows

• Each application has an application window • Each control on application window is a child of application window • Each control can have children • Windows in applications generates a tree • Each window provides some information (Controll ID, Class, handle...)

6. Screen reader techniques (1)

• Described windows represent user interface components • Screen readers have to solve how to access information in a window and use combinations of different strategies • Special "hacks" for particular application: not easy to develop, internal implementation of application user interface often changes...

7. Screen reader techniques (2)

• Access via DOM (document object model) of particular application: Not all applications provide DOM, Implementations are different (are not standardized), • Access via standardized methods such as MSAA

8. DOM examples

• Mathematical software LAMBDA provides a special COM (component object model), object which can be used by screen readers to access information in an active document • LAMBDA COM object provides functions such as GetChar to read a particular character on a specified column and row, GetLine to read a line, Getword, GetParagraph, GetAll...

9. DOM examples

• Screen reader developers have to write a special support to access a math in the opened document • Other ui components in LAMBDA are standard Windows components

10. MSAA (1)

• Microsoft Active Accessibility • Is a COM-based technology designed to improve the way accessibility aids work with applications running on MS Windows • Provides dynamic-link libraries that are incorporated into the operating system • Also provides COM interface and application programming elements that provide reliable methods for exposing information about user interface elements

11. MSAA (2)

• Architecture: client-server • client -> AAserver application • Client: screen reader or other assistive software • Server: contains key parts of MSAA implementation • Target application implements IAccessible interface for each custom user interface component

12. MSAA (3)

• Client can: • Query for information — for example, about a UI element at a particular location • Receive notifications when information changes — for example, when a control becomes grayed or when a text string changes • Carry out actions that affect user interface or document contents — for example, click a push button, drop down a menu, and choose a menu command

13. MSAA (4)

Server can: • Provide information about its custom user interface objects and the contents of its client windows • Send notifications when its user interface changes

14. MSAA: accessible objects

• UI elements are exposed to client as COM objects • These accessible objects maintain properties which describe object's name, screen location... • Also methods that clients call to cause object to perform actions

15. MSAA: accessible objects

• Accessible objects are implemented using IAccessible interface • IAccessible methods and properties enable client applications to get information about UI elements needed by users • For example, IAccessible::get_accParent returns an interface pointer to an accessible object's parent

16. MSAA: simple elements

• A simple element is a UI element that shares an IAccessible object with other elements and relies on that IAccessible object (typically its parent) • To be uniquely identified and exposed, a simple element requires an IAccessible object and child ID

17. MSAA: simple elements (2)

• The IAccessible object shared among simple elements often corresponds to a common parent object in the user interface. For example, system listboxes expose an accessible object for the overall listbox and simple elements for each listbox item

18. IAccessible: descriptive properties

• Name: object name (screen text of a button, label of a listbox...) • Role: Role of an object (button, checkbox, listbox, edit, menu...), there are predefined constants in IAccessible • State: state of object (maximized, checked...), there are predefined constants in IAccessible

19. IAccessible: descriptive properties (2)

• Value: textual representation of an object (not supported by all objects), e.g. contents of a text field, level information in tree view...

• Keyboard Shortcut: ...

• ...

20. MSAA: basic principle

• After activation of application client can register one or more hooks by sending query to application's server • Server notifies client when something happens • Client somehow gets a pointer to active ui component IAccessible interface • and starts the required action depending on type of event which occured in application

21. Examples of events

• EVENT_SYSTEM_MENUSTART, EVENT_SYSTEM_MENUEND: menu bar was activated or deactivated • EVENT_SYSTEM_MENUPOPUPSTART, EVENT_SYSTEM_MENUPOPUPEND: popup menu was activated or deactivated • EVENT_OBJECT_FOCUS: focused object changed

22. Examples of events (2)

• EVENT_OBJECT_SELECTION: something was selected • EVENT_OBJECT_SELECTIONADD, EVENT_OBJECT_SELECTIONREMOVE: Selection was expanded or reduced • EVENT_OBJECT_STATECHANGE: state changed (checkbox was (un)checked...) • EVENT_OBJECT_VALUECHANGE: value changed (progress bar was updated...)

23. Examples of methods

• get_accParent: Get the parent of an IAccessible • get_accChildCount: Get the number of children of an IAccessible • get_accChild: Get the child of an IAccessible • get_accName: Get the "name" of the IAccessible

24. Examples of methods

• get_accRole: Get an enumerated value representing what this IAccessible is used for • get_accFocus: Which child is focused?

• accLocation: Get the x,y coordinates, and the height and width of this IAccessible node

25. Examples of states

• STATE_UNAVAILABLE: grayed items • STATE_SELECTED: e.g. item in listbox • STATE_FOCUSED: object has focus • STATE_CHECKED: checkbox • STATE_READONLY: edit • STATE_EXPANDED, STATE_COLLAPSED: node in treeview

26. Examples of roles

• Roles marked with * are inserted by operating system and provided with IAccessible automatically • ROLE_MENUBAR , *ROLE_TITLEBAR, *ROLE_WINDOW, ROLE_MENUITEM, ROLE_DIALOG, ROLE_STATUSBAR, ROLE_TABLE, ROLE_CELL, ROLE_LIST, ROLE_LISTITEM, ROLE_PAGETAB, ROLE_STATICTEXT, ROLE_PUSHBUTTON, ROLE_CHECKBUTTON, ROLE_RADIOBUTTON, ROLE_COMBOBOX • ...

27. When to use and when not

• -If you use only standard windows UI components • -If you want to provide access to complicated (structured) documents (DOM is better alternative) • +Works well for dialogs and UI • +Use MSAA whenever you have created custom controls where you handle the drawing, mouse and keyboard accessibility in your own code. MSAA is the only way to let all ATs know what your code is doing.

28. Tools for testing

• There are three mostly used tools for testing IAccessible implementation • Accessible event watcher: logs events as they occur within the system • Accessible explorer: allows developer to examine the IAccessible properties of objects and objects relations • Inspect objects: Tool for inspection of object properties and hierarchy

29. IAccessible2

• New accessibility api • Initially developed by IBM • Free (opened) standard • Extends MSAA interface

30. IAccessible2 (2)

• Provides functionality which can be used to improve accessibility of rich document applications • Added new states and roles • One element can provide more actions • Developers can define own roles and states • server implementations: Used in Mozilla Firefox 3 • Client implementations: NVDA, JAWS,...

31. UI Automation

• is the new accessibility framework for Microsoft Windows, available on all operating systems that support Windows Presentation Foundation • provides programmatic access to most user interface (UI) elements on the desktop • enabling assistive technology products such as screen readers – to provide information about the UI – to manipulate the UI by means other than standard input • and automated testscripts to interact with the UI

32. UI Automation(2)

• Provider API: A set of interface definitions that are implemented by UI Automation providers, objects that provide information about UI elements and respond to programmatic input • Client API: A set of types for managed code that enables UI Automation client applications to obtain information about the UI and to send input to controls • Core: The underlying code (sometimes called the UI Automation core) that handles communication between providers and clients • UIAutomation Clientside Providers: A set of UI Automation providers for standard legacy controls. (WPF controls have native support for UI Automation.

33. UI Automation model

• Every piece of the UI is exposed to the client applications as a AutomationElement • Elements are contained in a tree structure (desktop is a root) • clients can filter the raw view of the tree as a control view or a content view • AUElements expose common properties of the UI elements they represent (such as control type) • and control patterns that provide properties specific to their control types

34. UIAutomation model(2)

• Control patterns also expose methods that enable clients to get further information about the element and to provide input • A control may expose multiple patterns (E.G. combobox: expand and collapse functionality, selection mechanism) • There are also events: client registers for a specific event notifications and can request that specific UI Automation properties and control pattern information be passed into their event handlers

35. Java accessibility

• User interface can be created with different technologies • swing • SWT: standard widget toolkit

36. Swing

• Widget toolkit for Java • part of JFC • API for providing a graphical user interface for Java programs • Implements Java accessibility API • Can be used by screen readers via Java Access Bridge

37. Java Access Bridge

• Is a class containing native methods • some of them are provided in dynamic library which can be used by accessibility technologies • Communication with Swing ui components is realized through this class • Java Access Bridge is supported by many screen readers and magnifiers • It is not a part of Windows os and must be installed

38. SWT

• Standard Widget Toolkit • Uses native ui components of target platform • Is fully accessible without special requirements

39. Resources

• MSAA SDK: http://www.microsoft.com/downloads/details.aspx?FamilyId=8BC82 E65-DBEB-4BC4-9F27-8AC7DF6B7B77&displaylang=en • MSAA SDK tools: http://www.microsoft.com/downloads/details.aspx?familyid=3755582 A-A707-460A-BF21-1373316E13F0&displaylang=en • Jen's MSAA resource page (excellent resource with implementation examples,...): http://www.cs.ubc.ca/~gluck/msaa.html

• Implementing an MSAA Server - How Mozilla Does It, and Practical Tips for Developers: http://www.mozilla.org/access/windows/msaa server • VAGNER, Peter; Alternative ways of accessing information. [Bachelor Thesis] (Slovak language); Matej Bel University in Banská Bystrica. Faculty of Natural Science; Department of Informatics: http://cezap.sk/~lecky/presentations/bakalarka.pdf

• UI Automation overview: http://msdn.microsoft.com/en us/library/ms747327.aspx