Transcript - Computer Science Degree Programs
5. Computer Science and Human-Computer Interaction
5.1. Motivation 5.2. User Interface Software Concepts
5.2.1. Design Process for Interactive Systems 5.2.2. Interaction Techniques
5.2.3. Interaction Styles 5.2.4. User Interface Management Systems 5.2.5. User Interface Description Languages 5.3. User Interface Software Tools
5.3.1. Window Systems 5.3.2. Toolkits 5.3.3. UIMS 5.4. Future Directions
5-1
5.1. Motivation
Easier to use, but harder to program [Myers]
Multiprogramming
Real-time programming Robustness Testability Modularization Need to support iterative design and testing Software tools
5-2
5.2. User Interface Software Concepts
Dialogue independence
Interface (syntax) vs. application (semantics, business logic)
Model-View-Controller architecture Interaction techniques Basic interaction styles Design levels: conceptual, semantic, syntactic, lexical Generate UI from abstract description Early research in user interface management system (UIMS) and user interface description languages (UIDL) User interface software tools New interaction styles (non-WIMP)
5-3
5.2.1. Design Process for Interactive Systems
Partition into Four Levels [Foley & Wallace]
Conceptual Semantic
Syntactic Lexical
5-4
5.2.2. Interaction Techniques
Interaction task Interaction device Interaction technique
A way of using a device to perform a generic HCI task Widget in a toolkit Two information processors communicating via constrained interface [Tufte]
Goal: Increase bandwidth across channel
5-5
5.2.3. Interaction Styles: Current
Command Language
Question and Answer
Form-based
Menu
Natural Language
Direct Manipulation
5-6
Interaction Styles: Emerging
Virtual and augmented reality
Tangible interfaces
Ubiquitous, pervasive, handheld
Lightweight, tacit, passive, non command, context-aware
Affective computing
Multi-modal interfaces
Reality-based interfaces
Brain-computer interaction
5-7
5.2.4. User Interface Management Systems Research
Dialogue Independence [Hartson & Hix] User Interface Management System (UIMS) Approach [Olsen]
Before USER Application and its own User Interface
After USER UIMS
(Syntactic+Lexical)
Application
(Semantic)
Levels of design:
Conceptual [Johnson & Henderson], Semantic, Syntactic, Lexical
5-8
Classes of Programmers and Tool Users
Application Programmer User Interface Designer
UIMS Tools
(UIDL) End User
Runtime UIMS Application
5-9
Dialogue Independence
Same UIMS for many applications USER UIMS Application 1 Application 2 Application 3
Different interfaces to same application USER User Interface 1 Application User Interface 2 User Interface 3
Interface to null application (rapid prototyping)
5-10
5.2.5. User Interface Description Languages
How does the user interface designer describe the desired user interface to the UIMS or toolkit?
Layout Behavior With a User Interface Description Language (UIDL)
May be interactive
Generate UI from abstract description
UIDL describes user-visible behavior, but not implementation
More in Section 5.3.3
Other uses of UIDL (evaluation)
5-11
5.3. User Interface Software Tools
Goal: Support dialogue independence and separation of programming roles [Foley, van Dam, Feiner, & Hughes] Application UIMS Toolkit Window system/graphics package Operating system Hardware
5-12
5.3.1. Window Systems
Manage shared resources (pixels, keyboard, mouse), provide clipping, translation Tells client when exposed, resized, or other damage Could implement in different places Single built-in window manager or allow separate Dispatcher AccessWindowMouseEvent (Event *e) { . . .
FileWindowMouseEvent (Event *e) { If (e->type==MS_LEFT) . . .
(N.B. Security classifications shown are simulated only.)
5-13
5.3.2. Toolkits
Library of reusable routines for implementing human-computer interface features Basic components
Set of widgets
"Intrinsics" Does not try to describe or support user interface as a whole Standard look and feel
Encourage consistency But imposes limits on interaction style
5-14
Toolkits (cont'd)
Examples
Visual Studio (Windows)
Interface Builder (Macintosh) GTK (X/Unix/Linux) Based on controls or "widgets" AccessCheckBox (Boolean val) ClassifTypein (char *val) DisplayMsgFile ()
(N.B. Security classifications shown are simulated only.)
5-15
5.3.3. UIMS
Higher level than procedure library May include main program, provide external control of application
Call semantics/business logic as subroutines
Message to component software Design time, run time, and evaluation time tools [Myers] Module partitioning: Runtime vs. interface design time partitioning
5-16
Model-View-Controller
Controller USER Model View
Design pattern, originally Smalltalk Found, in some form, in most UI toolkits Matches dialogue independence, syntactic/semantic split Extend to web browser
5-17
User Interface Description Language
Designer creates it, UIMS executes it
Specific language or technique suitable for describing user interfaces Describe user-visible behavior, but not implementation Goal: Generate a UI from high-level, abstract description
Platform-independent if possible (Windows, Mac, X) Platform-type-independent if possible (desktop, cellphone, voice)!
Model-based UI development [Paterno; Jacob, Limbourg, & Vanderdonckt] Current GUI practice
Has evolved from general purpose UIDL into small, stable set of widgets and classes Change with new UI styles, portability needs Choice of design language at heart of UIMS
5-18
Basic Classes of UIDLs
Based on techniques for specifying static languages
Both can be modified for interactive languages BNF send-cmd::= SEND DRAFTNAME RECIPNAME State Transition Diagram
start SEND getdr getre RECIPNAME
Alternate Form of State Transition Diagram start: SEND ->getdr getdr: DRAFTNAME ->getre getre: RECIPNAME ->end
5-19
Early UIMS/UIDL Research
Language-based
Menu Network [Kasik] BNF, grammar-based [Olsen] State transition diagram [Newman] Event language Declarative Non-language-based
Interactive builder (bottom up)
Current state of practice, use GUI techniques to specify GUI Plug together components with
pre-defined
interactive behavior Automatic generation (top down) [Foley]
By demonstration [Myers] Other categories
Constraint-based [Hudson]
User-oriented [Moran]
5-20
Example: Visual Programming for UI
Layout vs. Behavior
UIMS, with state diagrams for behavior
[Jacob, “ A state transition diagram language...
” ]
Visual Basic, pre-programmed behavior
5-21
Example: State Diagram as UIDL
Command-driven Graphical Report Editor
5-22 [Jacob, “ Using formal specifications...
” ]
UI Evaluation
Directly from UIDL
Evaluate interface without building system
Emerging area, but lack underlying formal knowledge of UI design Analyze user interface to detect specific properties [Darlington et al.] Apply human performance models [Card] Automated screen design critique [Tullis] Prove safety-critical properties Screen mockup, prototyping tools
“
Slide show" of canned screens based on user inputs
UIMS sans application (use stubs for semantics) Visual Basic, Powerpoint, plain HTML also used this way
5-23
UI Tools: Standardization
Early:
Many desktop platforms No standardization Current:
~3 major platforms, standard widgets Settled into stable toolkits, widget libraries, GUI class libraries UIMS seems like overkill Emerging:
Mobile, automobile Diverse platforms, no standardization Wider range of interaction techniques Model-based development, UIMS redux Future:
Same issue for new non-WIMP interaction styles Still wider range of possible interactions
5-24
UI Tools: Trends
UI toolkits, libraries
Become important, integral part of software development tool sets
e.g., MacOS UI features are well integrated
Good uptake despite unusual programming language Web browser as UI platform
Cloud computing Chrome-only OS New tools for web UIs, but still clumsy, poorly integrated Making cross platform toolkits less significant
5-25
UI Tools: Mobile Devices
Basic HCI development and design principles
Largely GUI, not new interaction style + gestures, GPS, ...
Use context: Full attention or not?
Development tools
Early: Clumsy tools, restricted UI designs
Much like early GUIs, function key-driven (pre-mouse) GUI Current: UIs and tools very similar to standard GUI
IOS: Interface Builder (Mac) Android: Java + Swing (minor differences) UI Consistency
Low level consistency High level models, UIDL could help Example: IMAP mail clients
5-26
5.4. Future Directions: Emerging Interaction Style
Parallel, Highly-interactive (non-WIMP) Lightweight, Context-aware, Non-command Continuous (+ Discrete) Tangible Interfaces
Microprocessors, 3D forms Ubicomp, Pervasive, Context-aware
Sensors, networks Reality-based Interaction Brain-computer Interaction
5-27
Implications for Software
Current Single thread I/O Discrete tokens Precise tokens Sequence, not time Explicit user commands Future Parallel, asynchronous dialogues; may be interrelated Continuous inputs and responses (plus discrete) Probabilistic input, not easily tokenized Real-time requirements, deadline-based Passive monitoring of user
5-28
Reality-Based Interaction
Connects several emerging interaction styles
Understand together as a new generation of HCI through RBI [Jacob et al.,
“
Reality-Based Interaction...
”
] Computer interaction more like rest of world
Exploit skills and expectations user already has about simple, everyday, non-digital world Reality + extensions
Design tradeoffs:
5-29
Tangible User Interfaces
Simple, transparent mechanical structures
Use knowledge of physical world to operate Augment physical objects with digital meaning Combine physical + digital representations
Exploit advantages of each
[Jacob et al., “ A Tangible Interface...
” ] [Zigelbaum et al., The Tangible Video Editor...
” ] 5-30
Technology for TUI
Sensors
Produce signal in response to change in surroundings Actuators
Produce physical change in response to signal Microcontroller
Communicates with devices and with main computer Other technologies
RFID Computer vision ...
Wider range of I/O events than GUI Development methodology, tools
None!
5-31
TUIML
A Visual Language for Modeling Tangible User Interfaces [Shaer] TAC paradigm
Each TUI consists of token within a constraint Same object may sometimes be token, sometimes constraint Two tier model fits well
Dialogue (states, storyboard) Interaction (especially continuous)
5-32
Conclusions: 1. Research Background
Background for following current research in user interface software
Interaction styles Dialogue independence Four levels of design User interface management system (UIMS) User interface description language (UIDL) Goal: Generate UI from abstract description Types of software tools New interaction styles (non-WIMP) and requirements Easier to use, harder to program?
5-33
Conclusions: 2. Practice
What research ideas can be applied in building user interfaces today?
Separate interface from application programming Model-View-Controller architecture Use specialized tools (UIMS) for leverage Use specialized languages (UIDL) for perspicuity and portability
5-34
Sources
Textbooks
Olsen Foley, van Dam, Feiner, & Hughes Shneiderman, Plaisant, Cohen, & Jacobs History
Early UIMS research [Buxton et al.; Foley and Wallace; Jacob; Kasik; Newman; Olsen] Classic surveys: ACM Transactions on Computer-Human Interaction [Myers, Hudson, & Pausch], Computing Surveys [Hartson & Hix] Current research
ACM UIST Conference on User Interface Software and Technology (annual conference) ACM EICS Symposium on Engineering Interactive Computing Systems (conference) CADUI Conference on Computer-Aided Design of User Interfaces (conference) ACM Transactions on Computer-Human Interaction (journal) interactions (ACM magazine)
5-35
References
W. Buxton, M.R. Lamb, D. Sherman, and K.C. Smith, ''Towards a Comprehensive User Interface Management System,'' Computer Graphics 17(3) pp. 35-42 (1983).
S.K. Card, T.P. Moran, and A. Newell, ''The Keystroke-Level Model for User Performance Time with Interactive Systems,'' Comm. ACM 23 pp. 396-410 (1980).
J. Darlington, W. Dzida, and S. Herda, ''The Role of Excursions in Interactive Systems,'' International Journal of Man-Machine Studies 18 pp. 101-112 (1983).
J. Foley, W.C. Kim, S. Kovacevic, and K. Murray, ''Defining Interfaces at a High Level of Abstraction,'' IEEE Software 6(1) pp. 25-32 (January 1989).
J.D. Foley, A. van Dam, S.K. Feiner, and J.F. Hughes, Computer Graphics: Principles and Practice, Addison Wesley, Reading, Mass. (1990). J.D. Foley and V.L. Wallace, ''The Art of Graphic Man-Machine Conversation,'' Proceedings of the IEEE 62(4) pp. 462-471 (1974).
H.R. Hartson and D. Hix, ''Human-computer Interface Development: Concepts and Systems for its Management,'' Computing Surveys 21(1) pp. 5-92 (1989).
S.E. Hudson, ''Graphical Specification of Flexible User Interface Displays,'' Proc. ACM CHI'89 Human Factors in Computing Systems Conference pp. 105-114 (1989).
R.J.K. Jacob, "A Specification Language for Direct Manipulation User Interfaces," ACM Transactions on Graphics, Vol. 5(4) pp. 283-317 (1986) http://www.cs.tufts.edu/~jacob/papers/tog.pdf].
R.J.K. Jacob, ''A State Transition Diagram Language for Visual Programming,'' IEEE Computer 18(8) pp. 51-59 (1985) [http://www.cs.tufts.edu/~jacob/papers/ieeecomputer.pdf].
R.J.K. Jacob, "Using Formal Specifications in the Design of a Human-Computer Interface," Communications of the ACM, Vol. 26(4) pp. 259-264 (1983) [ http://www.cs.tufts.edu/~jacob/papers/cacm.pdf
].
R.J.K. Jacob, A. Girouard, L.M. Hirshfield, M.S. Horn, O. Shaer, E.T. Solovey, and J. Zigelbaum, "Reality Based Interaction: A Framework for Post-WIMP Interfaces," Proc. ACM CHI 2008 Human Factors in Computing Systems Conference pp. 201-210, ACM Press (2008) [http://www.cs.tufts.edu/~jacob/papers/chi08.pdf].
5-36
R.J.K. Jacob, H. Ishii, G. Pangaro, and J. Patten, "A Tangible Interface for Organizing Information Using a Grid," Proc. ACM CHI 2002 Human Factors in Computing Systems Conference pp. 339-346, ACM Press (2002) [http://www.cs.tufts.edu/~jacob/papers/chi02.pdf].
R.J.K. Jacob, Q. Limbourg, and J. Vanderdonckt (Eds.), Computer- Aided Design of User Interfaces IV, Kluwer Academic Publishers, Dordrecht (2005).
J. Johnson and A. Henderson, "Conceptual Models: Begin by Designing What to Design," interactions, Vol. 9(1) pp. 25-32 (January 2002).
D.J. Kasik, ''A User Interface Management System,'' Computer Graphics 16(3) pp. 99-106 (1982).
T.P. Moran, ''The Command Language Grammar: A Representation for the User Interface of Interactive Computer Systems,'' International Journal of Man-Machine Studies 15 pp. 3-50 (1981).
B.A. Myers, ''User Interface Software Tools,'' ACM Transactions on Computer-Human Interaction 2(1) pp. 64 103 (March 1995). B.A. Myers, Creating User Interfaces by Demonstration, Academic Press, Boston (1988).
B. Myers, S.E. Hudson, and R. Pausch, "Past, Present, and Future of User Interface Software Tools," ACM Transactions on Computer- Human Interaction, Vol. 7(1) pp. 3-28 (March 2000). Also reprinted in Human Computer Interaction in the New Millenium, ed. J.M. Carroll, Addison-Wesley/ACM Press, Reading, Mass., 2002, pp. 213-233.
W.M. Newman, ''A System for Interactive Graphical Programming,'' Proc. Spring Joint Computer Conference pp. 47-54, AFIPS (1968).
D.R. Olsen, Building Interactive Systems: Principles for Human-Computer Interaction, Course Technology Inc. (2009).
D.R. Olsen, User Interface Management Systems: Models and Algorithms, Morgan Kaufmann, San Mateo, Calif. (1992).
F. Paterno, Model-Based Design and Evaluation of Interactive Applications,, Springer-Verlag, London (2000).
O. Shaer and R.J.K. Jacob, "A Specification Paradigm for the Design and Implementation of Tangible User Interfaces," ACM Transactions on Computer-Human Interaction (2009) [http://www.cs.tufts.edu/~jacob/papers/tochi.shaer.pdf].
B. Shneiderman, C. Plaisant, M. Cohen, and S. Jacobs, Designing the User Interface: Strategies for Effective Human-Computer Interaction, Addison-Wesley, Reading, Mass. (2009).
E.R. Tufte, ''Visual Design of the User Interface,'' IBM Corporation, Armonk, N.Y. (1989).
5-37
T.S. Tullis, ''A Computer-Based Tool for Evaluating Alphanumeric Displays,'' Proc. INTERACT '84 Conference, B. Shackel, ed. (1984).
J. Zigelbaum, M. Horn, O. Shaer, and R.J.K. Jacob, "The Tangible Video Editor: Collaborative Video Editing with Active Tokens," Proc. TEI 2007 First International Conference on Tangible and Embedded Interaction pp. 43-46 (2007) [http://www.cs.tufts.edu/~jacob/papers/tei07.zigelbaum.pdf].
5-38