CS 898: Advanced Interaction Design

Instructor

Professor Michael Terry (mterry@cs.uwaterloo.ca)
Office: DC 2118
Office hours: TBA

Meeting Time and Location

MC 2036
MW 1-2:30PM

Introduction

Today, we are blessed with a wealth of technological capability: Processors run at gigahertz speeds, RAM is measured in gigabytes, and long-term storage is quickly moving from the gigabyte range to the terabyte range. At the same time, the fruits of research in computer science and other domains have created plug-and-play libraries that require minimal theoretical knowledge for developers to be able to effectively use. Physics, machine learning, 3D graphics, and computer vision are all now highly accessible via well-documented, robust libraries, lessening the need for domain experts from these specialized fields when one wishes to incorporate this technology in new user interfaces.

Despite all of these advances, human-computer interfaces have remained relatively stable for the past 25 years. WIMP interfaces are still the dominant form of interaction. Web applications have developed slightly different conventions for interaction, but they still assume a keyboard and mouse. Pen input and modern graphics cards have introduced some new interaction possibilities, though pen interaction is still largely a thin layer on WIMP interfaces, while graphics cards tend to add more aesthetic value than functional value in user interfaces.

There are numerous reasons for the slow adoption of new interaction techniques. It can be costly: People need to be trained in the proper use of the technology, the technology itself may be costly (either in terms of hardware or licensing costs), or it simply might not provide enough of an improvement over existing methods to justify its adoption. Or, the technology may simply not be fully developed, tested, or packaged in a form that enables its easy integration into an application. This course deals with these latter three issues.

Course Goals

In this course, you will gain expertise in the design of novel interaction techniques. You will consider interaction techniques from the following, interrelated views:

The user need you are addressing (i.e., the problem you are solving),
The interaction technique’s functionality (what work it can perform),
Its interaction design (how it is manipulated by the user, including how users can detect and recover from error conditions),
How well it solves the specified problem (which includes comparisons to existing techniques), and
Its architectural design (i.e., how it is implemented).

You will gain this expertise by doing the following:

Identifying a real-world need
Creating a literature review that documents this real-world need and the existing approaches to solving this problem
Designing a novel interaction technique to address the need
Implementing and testing the interaction technique with real people
Packaging your interaction technique as a library, schematic, and/or CAD drawing (or its equivalent), depending on its realization. This package will include documentation and everything necessary for a third party to be able to use without any communication with you. The package will not only be “plug-and-play,” it will also be extensible by third parties via well-defined hooks
Writing a paper in UIST conference paper format describing the technique. A video of your technique must accompany your paper

By the end of the course, you will have designed, implemented, tested, and packaged an interaction technique that end users will find useful and that developers will be able to download and integrate in their applications. This course is your opportunity to apply expertise in fields such as machine learning, computer vision, optimization, visual design, and other specialities, to the design of new interaction tools for end-users.

Because of the workload, it is highly recommended you combine the course project with your thesis work or other course projects, pending approval by all parties. There will be a final demo day (time and place TBA) in which others will be able to interact with you technique.

CS 489 / 698 is highly recommended as a precursor to this course. If you have not taken this course, the following resources will be useful to you:

Contextual Design. Hugh Beyer and Karen Holtzblatt. Morgan Kaufmann Publishers.
Millen, D. R. 2000. Rapid ethnography: time deepening strategies for HCI field research. In Proceedings of the 3rd Conference on Designing interactive Systems: Processes, Practices, Methods, and Techniques (New York City, New York, United States, August 17 - 19, 2000). D. Boyarski and W. A. Kellogg, Eds. DIS '00. ACM, New York, NY, 280-286. DOI= http://doi.acm.org/10.1145/347642.347763
Learning From Strangers: The Art and Method of Qualitative Interview Studies. Robert S. Weiss. Free Press.

Course Format

The course will be a mixture of lectures, student presentations, and design critiques. All lectures and presentations will be given in a round-table format to facilitate interaction.

Course Marking

Topic proposal, assignments 5%
Literature review 15%
Design 15%
Implementation 15%
Final package and paper 40%
Presentations and class participation 10%

Course Readings

Most classes have assigned readings. You must complete the readings prior to the class and submit a short summary for each reading. The summary should contain the following:

A synopsis of the major contributions of the paper. This should not be a rehash of the abstract.
What you personally learned from the paper. What did this paper teach you? How has it clarified your thinking or changed your view of a particular thing/phenomenon?
What could improve the paper. How could it have been better?

The summaries should be no more than one page long per reading. At the end of the course, they will serve as a summary of the readings for your own future research.

After the first week of lecture, students will be leading discussions.

Course Project

The easiest technology to create is technology that is needs-driven – technology that addresses real-world, human needs. Accordingly, in this course, you must identify a real-world problem that your technology will address. This problem must be approved by the course instructor.

You will be addressing your identified problem via the creation of an interaction technique. Interaction techniques are tools that allow people to get work done more efficiently, without severely impacting the range of problems they can solve. Drag-and-drop, pie menus, pen-based gestures, tool-tips, and voice-based input are all examples of interaction techniques. An interaction technique should be a standalone, identifiable entity that can be applied across domains and contexts. Machine learning itself is not an interaction technique, but the recognition of incorrect grammar via machine learning techniques (along with the presentation of those errors) is an interaction technique. Standalone sticky notes are not an interaction technique (they’re an application), but comments embedded within a document are an interaction technique.

To get you thinking about real-world problems and interaction techniques, here are some ideas:

Multiple previews of sound manipulations for sound designers and/or musicians
More efficient ways of exploring a multidimensional parameter space
Two-handed interaction in applications like GIMP or Illustrator
Creativity support tools that support parallel exploration of alternatives
Additional ideas (typing tutor, CD gain...)

Syllabus

The following is a tentative syllabus.

Date	Topic	Assignments / Readings
January 7	Introduction + Overview
January 9	Interaction design	Beaudouin-Lafon, M. 2004. Designing interaction, not interfaces. In Proceedings of the Working Conference on Advanced Visual interfaces (Gallipoli, Italy, May 25 - 28, 2004). AVI '04. ACM, New York, NY, 15-22. DOI= http://doi.acm.org/10.1145/989863.989865 Beaudouin-Lafon, M. and Mackay, W. E. 2000. Reification, polymorphism and reuse: three principles for designing visual interfaces. In Proceedings of the Working Conference on Advanced Visual interfaces (Palermo, Italy). AVI '00. ACM, New York, NY, 102-109. DOI= http://doi.acm.org/10.1145/345513.345267 Beaudouin-Lafon, M. 2000. Instrumental interaction: an interaction model for designing post-WIMP user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (The Hague, The Netherlands, April 01 - 06, 2000). CHI '00. ACM, New York, NY, 446-453. DOI= http://doi.acm.org/10.1145/332040.332473

January 14	Tool design	Resnick, Berg, and Eisenberg. Beyond Black Boxes: Bringing Transparency and Aesthetics Back to Scientific Investigation. Published in Journal of Learning Sciences (vol. 9, no. 1, 2000) Myers, B., Hudson, S. E., and Pausch, R. 2000. Past, present, and future of user interface software tools. ACM Trans. Comput.-Hum. Interact. 7, 1 (Mar. 2000), 3-28. DOI= http://doi.acm.org/10.1145/344949.344959
January 16	Input devices	Card, S. K., Mackinlay, J. D., and Robertson, G. G. 1991. A morphological analysis of the design space of input devices. ACM Trans. Inf. Syst. 9, 2 (Apr. 1991), 99-122. DOI= http://doi.acm.org/10.1145/123078.128726 John et al. Bringing Usability Concerns to the Design of Software Architecture.

January 21	Beyond WIMP interaction	Project proposal due Kurtenbach, G., Fitzmaurice, G., Baudel, T., and Buxton, B. 1997. The design of a GUI paradigm based on tablets, two-hands, and transparency. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, United States, March 22 - 27, 1997). S. Pemberton, Ed. CHI '97. ACM, New York, NY, 35-42. DOI= http://doi.acm.org/10.1145/258549.258574 Bederson, B. B. and Hollan, J. D. 1994. Pad++: a zooming graphical interface for exploring alternate interface physics. In Proceedings of the 7th Annual ACM Symposium on User interface Software and Technology (Marina del Rey, California, United States, November 02 - 04, 1994). UIST '94. ACM, New York, NY, 17-26. DOI= http://doi.acm.org/10.1145/192426.192435
January 23	Alternative menus	Guimbretiére, F. and Winograd, T. 2000. FlowMenu: combining command, text, and data entry. In Proceedings of the 13th Annual ACM Symposium on User interface Software and Technology (San Diego, California, United States, November 06 - 08, 2000). UIST '00. ACM, New York, NY, 213-216. DOI= http://doi.acm.org/10.1145/354401.354778 Kurtenbach, G. and Buxton, W. 1994. User learning and performance with marking menus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating interdependence (Boston, Massachusetts, United States, April 24 - 28, 1994). B. Adelson, S. Dumais, and J. Olson, Eds. CHI '94. ACM, New York, NY, 258-264. DOI= http://doi.acm.org/10.1145/191666.191759

January 28	Physical construction of interfaces	Hudson, S. E. and Mankoff, J. 2006. Rapid construction of functioning physical interfaces from cardboard, thumbtacks, tin foil and masking tape. In Proceedings of the 19th Annual ACM Symposium on User interface Software and Technology (Montreux, Switzerland, October 15 - 18, 2006). UIST '06. ACM, New York, NY, 289-298. DOI= http://doi.acm.org/10.1145/1166253.1166299 Greenberg, S. and Fitchett, C. 2001. Phidgets: easy development of physical interfaces through physical widgets. In Proceedings of the 14th Annual ACM Symposium on User interface Software and Technology (Orlando, Florida, November 11 - 14, 2001). UIST '01. ACM, New York, NY, 209-218. DOI= http://doi.acm.org/10.1145/502348.502388
January 30	Machine learning and interfaces	Fails, J. A. and Olsen, D. R. 2003. Interactive machine learning. In Proceedings of the 8th international Conference on intelligent User interfaces (Miami, Florida, USA, January 12 - 15, 2003). IUI '03. ACM, New York, NY, 39-45. DOI= http://doi.acm.org/10.1145/604045.604056 Horvitz, E. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: the CHI Is the Limit (Pittsburgh, Pennsylvania, United States, May 15 - 20, 1999). CHI '99. ACM, New York, NY, 159-166. DOI= http://doi.acm.org/10.1145/302979.303030

February 4	Gesture recognition	Saund, E. and Lank, E. 2003. Stylus input and editing without prior selection of mode. In Proceedings of the 16th Annual ACM Symposium on User interface Software and Technology (Vancouver, Canada, November 02 - 05, 2003). UIST '03. ACM, New York, NY, 213-216. DOI= http://doi.acm.org/10.1145/964696.964720 Wobbrock, J. O., Wilson, A. D., and Li, Y. 2007. Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In Proceedings of the 20th Annual ACM Symposium on User interface Software and Technology (Newport, Rhode Island, USA, October 07 - 10, 2007). UIST '07. ACM, New York, NY, 159-168. DOI= http://doi.acm.org/10.1145/1294211.1294238
February 6	Sensors	Fogarty, J., Hudson, S. E., Atkeson, C. G., Avrahami, D., Forlizzi, J., Kiesler, S., Lee, J. C., and Yang, J. 2005. Predicting human interruptibility with sensors. ACM Trans. Comput.-Hum. Interact. 12, 1 (Mar. 2005), 119-146. DOI= http://doi.acm.org/10.1145/1057237.1057243 Fogarty, J. and Hudson, S. E. 2007. Toolkit support for developing and deploying sensor-based statistical models of human situations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA, April 28 - May 03, 2007). CHI '07. ACM, New York, NY, 135-144. DOI= http://doi.acm.org/10.1145/1240624.1240645

February 11	Computer vision	Project lit review due Maynes-Aminzade, D., Winograd, T., and Igarashi, T. 2007. Eyepatch: prototyping camera-based interaction through examples. In Proceedings of the 20th Annual ACM Symposium on User interface Software and Technology (Newport, Rhode Island, USA, October 07 - 10, 2007). UIST '07. ACM, New York, NY, 33-42. DOI= http://doi.acm.org/10.1145/1294211.1294219 AR Toolkit OpenCV Wilson, A., and N. Oliver. GWindows: Robust Stereo Vision for Gesture-Based Control of Windows, International Conference on Multimodal Interfaces, 2003. pdf Wilson, A., and N. Oliver. Multimodal Sensing for Explicit and Implicit Interaction, HCI International, 2005 pdf
February 13	Optimization techniques	Fogarty, J. and Hudson, S. E. 2003. GADGET: a toolkit for optimization-based approaches to interface and display generation. In Proceedings of the 16th Annual ACM Symposium on User interface Software and Technology (Vancouver, Canada, November 02 - 05, 2003). UIST '03. ACM, New York, NY, 125-134. DOI= http://doi.acm.org/10.1145/964696.964710 Gajos, K. and Weld, D. S. 2004. SUPPLE: automatically generating user interfaces. In Proceedings of the 9th international Conference on intelligent User interfaces (Funchal, Madeira, Portugal, January 13 - 16, 2004). IUI '04. ACM, New York, NY, 93-100. DOI= http://doi.acm.org/10.1145/964442.964461

February 18	Reading Week

February 25	Tangible user interfaces	Fitzmaurice, G. W., Ishii, H., and Buxton, W. A. 1995. Bricks: laying the foundations for graspable user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Denver, Colorado, United States, May 07 - 11, 1995). I. R. Katz, R. Mack, L. Marks, M. B. Rosson, and J. Nielsen, Eds. Conference on Human Factors in Computing Systems. ACM Press/Addison-Wesley Publishing Co., New York, NY, 442-449. DOI= http://doi.acm.org/10.1145/223904.223964 Ishii, H. and Ullmer, B. 1997. Tangible bits: towards seamless interfaces between people, bits and atoms. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, United States, March 22 - 27, 1997). S. Pemberton, Ed. CHI '97. ACM, New York, NY, 234-241. DOI= http://doi.acm.org/10.1145/258549.258715
February 27	Haptic feedback	MacLean, K. E., Snibbe, S. S., and Levin, G. 2000. Tagged handles: merging discrete and continuous manual control. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (The Hague, The Netherlands, April 01 - 06, 2000). CHI '00. ACM, New York, NY, 225-232. DOI= http://doi.acm.org/10.1145/332040.332435 Snibbe, S. S., MacLean, K. E., Shaw, R., Roderick, J., Verplank, W. L., and Scheeff, M. 2001. Haptic techniques for media control. In Proceedings of the 14th Annual ACM Symposium on User interface Software and Technology (Orlando, Florida, November 11 - 14, 2001). UIST '01. ACM, New York, NY, 199-208. DOI= http://doi.acm.org/10.1145/502348.502387
February 29		Low fidelity project design due

March 3	Lenses et al	Bier, E. A., Stone, M. C., Fishkin, K., Buxton, W., and Baudel, T. 1994. A taxonomy of see-through tools. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating interdependence (Boston, Massachusetts, United States, April 24 - 28, 1994). B. Adelson, S. Dumais, and J. Olson, Eds. CHI '94. ACM, New York, NY, 358-364. DOI= http://doi.acm.org/10.1145/191666.191786 Bier, E. A., Stone, M. C., Pier, K., Buxton, W., and DeRose, T. D. 1993. Toolglass and magic lenses: the see-through interface. In Proceedings of the 20th Annual Conference on Computer Graphics and interactive Techniques SIGGRAPH '93. ACM, New York, NY, 73-80. DOI= http://doi.acm.org/10.1145/166117.166126 Terry, M. and Mynatt, E. D. 2002. Side views: persistent, on-demand previews for open-ended tasks. In Proceedings of the 15th Annual ACM Symposium on User interface Software and Technology (Paris, France, October 27 - 30, 2002). UIST '02. ACM, New York, NY, 71-80. DOI= http://doi.acm.org/10.1145/571985.571996
March 5		Topics from here to the end of class will be related to projects

March 10		Initial functioning technique due
March 12

March 17		*Evaluation results due*
March 19

March 24
March 26

March 31		Final, packaged technique due
April 2

April 7		Final paper due