Past, Present and Future of User Interface Software Tools

Brad A. Myers, Scott E. Hudson, and Randy Pausch Developed for HCIC’99 and TOCHI Updated 2009 1



Introduction

User Interface Software Tools  Help developers design and implement user interfaces   Focus on

Tools

, but influenced by future UIs Today’s tools are highly successful  Window Managers, Toolkits, Interface Builders ubiquitous   Most software built using them

Are

based on HCI research Brad A. Myers. “A Brief History of Human Computer Interaction Technology.”

ACM interactions

. Vol. 5, no. 2, March, 1998. pp. 44-54. http://www.cs.cmu.edu/~amulet/papers/uihistory.tr.html

2

Talk Outline

 Historical Perspective     What worked What didn’t catch on Why Lessons Learned  Future Prospects and Visions  UI Trends that will require new tools  Important issues 3

Historical Perspective



Themes

 Address the useful & important aspects of UIs  Tools that succeeded helped (

just

) where needed  Threshold / Ceiling  Threshold = How hard to get started  Ceiling = how much can be achieved  Path of Least Resistance  Tools influence user interfaces created  Predictability  If not predictable, then not accepted by programmers  Moving Targets  Changing user interface styles makes tools obsolete 4

What Worked

 Window Managers and Toolkits  Event Languages  Graphical, Interactive Tools  Component Architectures  Scripting Languages  Hypertext  Object Oriented Programming  Constraints 5

Window Managers

  Multiple (tiled) windows in research systems of 1960’s: NLS, etc.

Overlapping introduced in Alan Kay’s thesis (1969)  Smalltalk, 1974 at Xerox PARC  Successful because multiple windows help users manage scarce resources:   Screen space and input devices Attention of users  Affordances for reminding and finding other work 6

Toolkits

 A collection of widgets  Menus, scroll bars, text entry fields, buttons, etc.

 Toolkits help with programming  Help maintain consistency among UIs  Key insight of Macintosh toolbox  Path of least resistance translates into getting programmers to do the right thing  Successful partially because address common, low-level features for all UIs  Address the useful & important aspects of UIs 7

Event Languages

 Create programs by writing event handlers  Many UIMSs used this style  Univ. of Alberta (1985), Sassafras (1986), etc.

 Now used by HyperCard, Visual Basic, Lingo, etc.

 Toolkits with call-backs or action methods are related  Advantages:   Natural for GUIs since generate discrete events Flow of control in user’s hands rather than programmer’s  Discourages moded UIs  May not work well in future 8

Graphical Interactive Tools

 Create parts of user interface by laying out widgets with a mouse   Examples: Menulay (1983), Trillium (1986), Jean Marie Hullot from INRIA to NeXT Now: Interface Builders, Visual Basic’s layout editor, resource editors, “constructors”  Advantages:   Graphical parts done in an appropriate, graphical way  Address the useful & important aspects of UIs Accessible to non-programmers  Low threshold 9

Component Architectures

 Create applications out of

components

separately developed and compiled which are   In UI software, each component controls an area of the screen Example: drawing component handles picture inside a document  Invented by Andrew research project at CMU (1988)  1999: OLE, OpenDoc, ActiveX, Java Beans  Now: SOA  Address the useful & important aspects of UIs  Just the “glue” to hold together components 10

Scripting Languages

 First GUIs used interpreted languages  Smalltalk, InterLisp  Rapid development, supports prototyping  Low threshold  Then C and C++ became popular  Now, bringing back advantages in scripting languages   tcl/tk, Python, perl Visual Basic, Javascript, Ruby, …  But language

must

contain general-purpose control structures 11

Hypertext

 Ted Nelson named it in 1965 and developed Hypertext system at Brown University  Important systems: NLS (1967), Hyperties (1986)  World-Wide Web  Phenomenal success due to:  Ease of use of Mosaic browser  Support for embedded graphics  Support for easy authoring  Low threshold both for authoring and viewing 12

Object Oriented Programming

 Success of OO owes much to UI software field  Popularized by Smalltalk  GUI elements (widgets)

seem

like objects  Have state, accept events (messages)  Rise parallels GUIs  C++ with Windows 3.1

  Java for behaviors in WWW 2009: Flash, etc.

13

Constraints

   Declare a relationship and system maintains it Sketchpad (1963), ThingLab (1979), Higgens (85), Garnet (1990), Amulet (1997), SubArctic (1996) 1999: hadn’t caught on  We thought would be mostly used for graphics  Now: Flash data bindings   Connect data to graphics Address the useful & important aspects of UIs  Predictability   Constraint networks can be hard to debug Especially in multi-way constraints  High threshold  Programmer must specify (or deduce) solving order  Constraints require thinking differently 14

What Hasn’t Caught On

 User Interface Management Systems  Formal Language-Based Tools  Model-Based and Automatic Techniques 15

User Interface Management Systems

 Original goal: like databases, provide high level language that abstracts details of input and output devices  This separation has not worked in practice  Good user interfaces must take into account the pragmatics and detailed behavior of all objects  Standardization of GUI input and output devices has made goal somewhat moot  Doesn’t address the useful & important aspects of UIs 16

Formal Language Based Tools

 Early UIMSs used grammars and state transition diagrams  Focus on

dialog management

 Moving Targets  Direct manipulation made dialog management less important  Path of Least Resistance  State diagrams afford

worse

user interfaces  High threshold  Formal languages are often hard to learn 17

Model-Based and Automatic



Techniques

Automatic techniques for generating UIs from a

model

or declarative specification of contents  Cousin (1985), Mike (1986), UIDE (1993), MasterMind (1993)  Try to separate specification of UI from content  May provide automatic reformating, retargeting, customization to users, etc.

 Result is often unpredictable  Often can be

worse

UI than hand-drawn  Sometimes model is

larger

than the code it would replace 18

Discussion of Themes

 Address the useful & important aspects of UIs   Narrower tools have been more successful than ones that try to do “everything” Do one thing well  Threshold / Ceiling    Research systems often aim for high ceiling Successful systems seem to instead aim for a low threshold Impossible to have both?

19

Discussion of Themes, cont.

 Path of Least Resistance   Tools

should

guide implementers into better user interfaces Goal for the future: do this more?

 Predictability   Programmers do not seem willing to release control Especially when system may do sub-optimal things  Moving Targets  Long stability of Macintosh Desktop paradigm has enabled maturing of tools  1999: We predicted a change soon  2009?

20

Future Prospects and Visions

 Important Trends    Ubiquitous Computing Move to recognition-based interfaces 3-D interfaces   End-user customization and scripting Violate assumptions of today’s tools  Assumptions limit what designers can do   Often unrecognized Implications for future tools 21

Ubiquitous Computing

 Computation embedded in many kinds of devices  Digital pagers and cell phones, Palm Pilots, CrossPads, laptops, wall size displays, “smart” rooms  Next wave: easy communication with radio  E.g., BlueTooth: www.bluetooth.com

 Significant Implications for tools   Tools for coordinating multiple, distributed, communicating devices  “Multi-computer” user interfaces Moving target problem 22

Varying Input and Output

  Today’s Desktop screens vary by a factor of 2.5 in size and a factor of 4 in pixels Tomorrow’s screen will vary by factors of 100 and a factor of 625 in pixels  Cell phone to Stanford’s wall (3796 x 1436 pixels) in size 23

Need New Interaction Techniques

 Interaction techniques for desktop will not work    No room on small devices Can’t reach menubar on wall-size devices 2009: iPhone doesn’t use desktop metaphors  Want to run same application on different devices 24

Need for Prototyping Devices

 User interface will be in hardware  Rapid design and prototyping needed for hardware  Pragmatics and usability cannot be evaluated from a simulation on a screen 25

Multiple, Distributed, Communicating

 Computers more for

communication

, not for

computation

 Already true for WWW, email, digital pagers, cell-phones   Computers as intermediaries between people   CSCW But can’t assume have similar systems Single person with multiple devices   Room-area networks like BlueTooth or HomeRF People communicating with

themselves

 Tools will need to help with data sharing and synchronization 26

Limitations of Today’s Tools for UbiComp

 Tools assume a Pointing Device  Hidden reliance on specific characteristics of common devices    Size of display Many tools cannot handle a different number of mouse buttons Change to a stylus on a touchpad requires different techniques  Assumptions about the setting   Assume user is sitting and looking at UI Assume has user’s full attention 27

Move to Recognition-Based Interfaces

 Speech, gestures, camera-based vision  Multimodal interaction   User will pick which modality to use Use multiple modalities at same time  Today, programming these requires knowing about Hidden-Markov Models, grammars, feature vectors, etc.

 Need tools to hide these complexities 28

Fundamental Differences of Recognition-based UIs

 Input is uncertain   Recognition can make errors Requires monitoring, feedback, correction  Interpreting input requires deep knowledge of data  

Context

of the application “Move the red truck to here” 29

Implications of Recognition-based UIs

 GUI event model no longer works  Do not produce discrete events  Separation of UI from application no longer works  Need a architecture based on accessible application data structures  “Reflection”, “Open Data Model” 30

3-D Interfaces

 Difficult to design the right abstractions for tools  Demise of VRML for Web  Need to settle on the 3-D widgets and interaction techniques that will be standard  Requirement for near-real-time interactivity  Need to hide the mathematics  2009: but what useful for?

31

End-user Customization and Scripting

  Spreadsheet enables end users to specify their own computation Visual Basic, other “scripting” languages  Needed in

all

applications   Threshold for programming is too high Need “gentle slope systems” 32

Gentle Slope Systems

Difficulty of Use Programming in C Visual Basic Flash HyperCard

MFC C Programming ActionScript HyperTalk Basic xCmds C Programming

Goal

Sophistication of what can be created 33

More Assumptions of Today’s Tools

 Skill and Dexterity of users   Older users Makes single, fixed library of widgets untenable  Non-overlapping and opaque components  Preclude translucency, magic lens interactions  Fixed libraries of components (widgets)   Creating new widgets is very difficult New devices will require new interaction techniques  Interactive tools provide freedom of design  Aim for “Mechanism not Policy” 34

Operating Systems Considerations

 What is in the OS?

 Window Manager? Toolkit? Communication? Scripting facilities?

 Need ever increasing services for applications  Need more access to low-level information  E.g., hardware buttons, whether on network  Ideally, API to support competition and research into these components 35

Some Design Guidelines for Future Tools

 Many things require further research  Organize around providing rich context     Of application and device state To inquire about data & methods; “reflection” Enables EUP, Recognition-Based UIs, data sharing for UbiComp Rather than event-based 36

More Design Guidelines

 Replaceable User Interfaces    Ability to have multiple UIs Enabled by procedural interface to everything in UI Enables UbiComp devices, EUP  Aim for low threshold, rather than high ceiling  But cover the

right

parts of the interface  Predictable for programmers rather than “smart” or automatic  Need for support for

evaluation

37

Conclusions

 Research in tools necessarily trails innovation in UI design  Due to consolidation on desktop metaphor, significant progress in tools  UI design poised for radical changes  New opportunities and challenges for tools 38