Remember that readings are due at 11:59 PM the day before lecture.
1/15: Ubiquitous Computing
(No reading responses)
1/20: Input & Output
The Computer for the 21st Century. Mark Weiser. Scientific American. September 1991, pp. 94-104.
This is one of the most influential, highly-cited papers in all of human-computer interaction. Weiser’s vision of ubiquitous computing articulated a vision in which computing recedes into the background rather than stands as a focal point of our attention. Researchers and practitioners have chased this vision since 1991. We read this article because it is the foundation upon which much of the Interaction component of the class sits. What do you think about Weiser’s differentiation between calm computation that recedes into infrastructure, and simply spreading computers around everywhere in the environment? What are the implications of such a vision?
Tangible Bits: Towards Seamless Interfaces Between People, Bits and Atoms. Hiroshi Ishii, Brygg Ulmer. CHI 1997.
The Tangible Bits paper sits alongside Weiser’s Ubiquitous Computing paper as one of the foundational works of modern HCI. It came out several years after Weiser’s article, but in many ways, was the innovation that made Weiser’s vision actionable. Tangible bits takes a strong stance that input and output need to be aligned in concrete physical artifacts, whereas Weiser allowed digital outputs such as screens and projectors. This paper has spawned many, many projects and even entire conferences. What tradeoffs does Tangible Bits make through its strict adherence to input/output co-location?
1/22: Activity Sensing
BeWell: A Smartphone Application to Monitor, Model and Promote Wellbeing. Nicholas D. Lane, Mashfiqui Mohammod, Mu Lin, Xiaochao Yang, Hong Lu, Shahid Ali, Afsaneh Doryab, Ethan Berke, Tanzeem Choudhury, Andrew T. Campbell. Pervasive computing technologies for healthcare 2012.
A smartphone contains a variety of sensors that can be used to track all kinds of detailed user activity. This work shows how such sensor data can be combined together to estimate the overall wellbeing of the user. They suggest visualizations to communicate both the activities and wellbeing scores to users. Which visualizations would be most useful to you if you used this system? Which do you think would be most useful to others? Why?
Privacy as Contextual Integrity. Helen Nissenbaum. Washington Law Review 2004.
How do we manage the privacy tradeoffs inherent in ubicomp technology? Nissembaum’s theory of contextual integrity is a central idea in ubiquitous computing, and one that helps us articulate how and when we might view our privacy as been violated in IoT or ubicomp contexts. What do you think of the theory? How might you extend it?
1/27: Design Cognition (Quiz 1)
For those who want more information about design, check out the 4120 Lectures.
The Design of Everyday Things (Chapter 2). Don Norman. 2013.
This book is one of the best known treatises on design and HCI. Norman describes interfaces and the problems with them from the perspective of a cognitive psychologist. Chapter Two is about the psychology of everyday interactions with systems. It describes how people work with systems to get things done and more importantly what people think about as they are working with the systems. The dhttps://penn-hci.github.io/cis7000-sp26/schedule/escriptions of the gulf of execution and evaluation, the seven stages of action and the corresponding seven principles of design will recur throughout this course. What other interface design problems have you encountered where they might have been relevant?
1/29: Design Process (Quiz 1)
Deconstructing Community-Based Collaborative Design: Towards More Equitable Participatory Design Engagements. Christina Harrington, Sheena Erete, Anne Marie Piper. CHI 2019.
As we will discuss in class, participatory design is a method for bringing community stakeholders in directly to the full design process, rather than only allowing the designer to engage in the concept generation and selection stages. But, in this work, Harrington and her colleagues demonstrate through a series of co-design (participatory) workshops that bringing people into the room does not mean that they have, or view themselves as having, full status and authority. What do you think we ought to do about this problem? Does it suggest deeper issues with participatory design and its goals? Does it suggest alternative design processes that we might consider?
The Promise of Empathy: Design, Disability, and Knowing the ‘Other’. Cynthia L. Bennett, Daniela K. Rosner. CHI 2019.
Needfinding, or developing empathy, are key processes in the design process. In this article, Bennett and Rosner step back and point out how the processes that designers use to build empathy may be causing harm rather than helping. They use the case of empathy activities around disability, and how designers walk away with false confidence that they understand and have empathized with disabled individuals’ experiences. Just like with the Harrington et al. paper, we ask: what ought we to do about this problem? Does it suggest deeper issues with the needfinding and empathy-building processes we use in design? Can you think of alternative processes that might mitigate the risk of this occurring?
2/3: Design Tools
The Reflective Practitioner (Chapter 3). Donald Schön. 1983.
Optional: read Chapter 2, which sets up Schön’s critique of rationality inproblem solving and lays out his theory of reflection-in-action.
This book provides foundational theory that we draw on in HCI design: what is core to the activity of design, and what does expertise in design really mean? Schön’s answer — reflection in action — is a frame that has proven useful, by articulating that the designer iterates by taking an action, reflecting on the result of that action, and using that reflection to gain insight into the problem in order to plan the next action. Schön’s argument that design cannot be planned and is instead a process of iteration and reflection is foundational reasoning in HCI as to why our field works the way it does, and why we teach the way we do. Do you agree with his position? What are the theory’s consequences for the goals of the tools and processes that we develop for design?
Learning Visual Importance for Graphic Designs and Data Visualizations. Zoya Bylinskii. UIST 2017.
Work in design tools has drawn on AI and data-driven models to support the reflection element of reflection-in-action. In particular, designers often have to imagine how people will react to their designs. This paper is one of the first to demonstrate that, within certain constraints, those reactions could be simulated effectively by an AI. Think about the other forms of feedback, and other kinds of designs, that could be supported by this kind of approach.
2/5: Social Media (Quiz 2)
Beyond Being There. Jim Hollan, Scott Stornetta. CHI 1992.
A classic of social computing, which was originally called CSCW (Computer-Supported Cooperative Work). It essentially argues both that remote collaboration will never be as good, and that it could be even better. The question of “does this satisfy the Beyond Being There criteria?” is a good gut check for any social computing system. What implications do you see of this idea, or what alternatives might you pitch? What are the implications of the claim, if we take it in its strongest instantiation that we should never even try to mirror offline interactions?
Groupware and social dynamics: eight challenges for developers. Jonathan Grudin. Communications of the ACM 1992.
A second classic of social computing (Grudin calls it `groupware’). In this article in the Communications of the ACM, Jonathan Grudin lays out all the reasons why social applications fail to get over the cold start problem. These problems remain resiliently, frustratingly challenging today. Which do you feel are the most core of the problems that he articulates, and why are they the most central? Are there any core problems that he overlooked?
2/10: Collaboration (Quiz 2)
Distance Matters. Gary Olson, Judith Olson. Human-Computer Interaction 2000.
Rounding out a week of classics. CSCW (“computer-supported cooperative work”) is a vibrant field of HCI focused on collaboration, and is the field that gave rise to social computing. (Social computing broadened CSCW out from office-based collaboration environments to user-contributed content online.) The Distance Matters paper is part of the CSCW canon. This paper raised an important question as to how effective collaboration software could really ever be, and why. Do you think the Distance Matters limit is fundamental? If so, why? If not, what could change the situation?
2/12: Society
Feminist HCI: taking stock and outlining an agenda for design. Shaowen Bardzell. CHI 2010.
Do we err by assigning genderless constructs such as ’the user’ in HCI? Do our visions of HCI make implicit assumptions about what kinds of sensing and tracking are desirable, what kinds of social interactions ought to be designed for, or what kinds of problems are interesting, based on this notion of the prototypical user? As you read this article about refocusing attention from default users to the marginalized, ask: where else do these gaps arise, and what might we do about them?
Embedding democratic values into social media AIs via societal objective functions. Chenyan Jia, Michelle S. Lam, Minh Chau Mai, Jeffrey T. Hancock, and Michael S. Bernstein. Volume 8, Issue 1, April 2024, CSCW.
What would it mean to explicitly design our sociotechnical systems with awareness and knowledge of the societal goals they are meant to facilitate? What other societal objective functions would you envision? What challenges in governance and decisions about ‘whose values?’ need to be answered here?
2/17: Human-Centered AI
Direct Manipulation Vs. Interface Agents. Ben Shneiderman and Pattie Maes. Interactions 1997.
In a pair of debates in 1997 researchers Pattie Maes and Ben Shneiderman argued about the way in which future interfaces would work. Would users delegate tasks to proactive software agents (Maes’ position) or would users always have full control even as automation increases (Shneiderman position). One could argue that until recently the direct manipulation paradigm has been dominant. But with the advent of LLMs and generative AI are we moving to a world of autonomous software agents?
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. J.D. Zamfirescu-Pereira, Richmond Wong, Bjoern Hartmann, Qian Yang. CHI 2023.
While prompting is often hailed as an accessible interface for non-AI-experts to work with AI, crafting effective prompts is challenging. This paper examines why prompting is challenging for non-experts and suggests opportunities for designing more accessible tools that can aid prompt design. Are the techniques proposed here enough to fully mitigate the challenges of prompt engineering? Are there other approaches we might use?
2/19: Programming and Toolkits
Tea: A high-level language and runtime system for automating statistical analysis. Eunice Jun, Maureen Daum, Jared Roesch, Sarah Chasins, Emery Berger, Rene Just, and Katharine Reinecke. Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 2019.
Could tools help us make sure we are asking the right questions when we run statistical analyses? Pay attention to how Tea systemetizes how we describe our experimental data—toolkit work is all about the representation that we expose to the author. Is this representation effective? How else might we represent this kind of thinking effectively?
Jury learning: Integrating dissenting voices into machine learning models. Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, Michael S. Bernstein. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 2022.
How might we develop AI systems that manage inherent disagreement in what ought to be the ‘correct’ label? Jury learning offers one response: articulate whose voices should determine the AI’s behavior. What works about this representation, and what alternatives might improve upon it?
2/24: Content Creation (Quiz 3)
Design principles for visual communication. Maneesh Agrawala, Wilmot Lit, and Floraine Berthouzoz. Communications of the ACM 2011.
Maneesh Agrawala and his collaborators developed this approach to crafting diagrams, visualizations, and explanations. This article is a review of several of his group’s projects, and an articulation of the general approach they take to their work. What role is computation playing, compared to what it would take to create these graphics manually?
2/26: Visualization (Guest Speaker: Jeff Tao)
A Bayesian Cognition Approach to Improve Data Visualization. Yea-Seul Kim, Logan A Walls, Peter Krafft, Jessica Hullman. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 2019.
This paper was released in a historical moment (arguably still ongoing) where the question of whether people can change their beliefs in respond to new information was top-of-mind. It poses and evaluates a framework through which we can view all data visualizations. It also illustrates one of the core challenges of HCI research: we have few tools for getting access to the inner workings of people’s minds beyond asking them to speak or write about it. Pick a visualization you’ve seen recently (news, social media, work dashboards). What prior beliefs might different audiences bring to it? How might the same visualization lead to different posterior beliefs depending on those priors? Does this framework suggest any design strategies for visualization creators?
Four Experiments on the Perception of Bar Charts. Justin Talbot, Vidya Setlur, Anushka Anand. IEEE Transactions on Visualization and Computer Graphics. 2014.
This paper uses controlled experiments to isolate specific perceptual mechanisms in bar chart reading. Findings like these illustrate that choosing an appropriate visualization isn’t just about data types and appropriate encoding conventions: the data itself can cause very divergent and biased understandings. Pick one of their experiments: what specific question were they trying to answer? What did they learn that we couldn’t have known from “common sense” or design intuition? How would you extend or follow up on that experiment? Or alternatively, propose your own visualization perception experiment.
3/3: Cognitive Models
Please note that these slides are a bit out-of-date, due to a file syncing issue. My apologies, I’m looking into this so it won’t happen again. It is still appropriate to use these slides as a reference for the main ideas. If you want to see video from lecture with the updated slides, you can watch it here. -AH
The Humane Interface (Chapter 4). Jef Raskin. 2000.
Jef Raskin was an HCI expert who led the Macintosh project at Apple. In this chapter of his book he describes a set of methods developed by HCI researchers in the 1980s to quantify human performance based on models of human information processing. The keystroke-level model attempts to compute how long it might take the average person to perform a task using a GUI. The exact numbers produced by these methods are very rough. But they can be useful to obtain ballpark figures for how long it might take to perform a particular task. And they can help designers identify areas of the interface that might require too many clicks or keystrokes. But developing such low-level models is tedious and since the late 1990s these methods have fallen out of favor. Today they are little used in practice. Why do you think this might be? Are there issues today that could be addressed by building such models.
Generative Agents: Interactive Simulacra of Human Behavior. Joon Sung Park, Joseph O’Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, Michael S. Bernstein. UIST 2023.
How might we craft simulations of human individuals and societies that reflect our lives? Many of the core design challenges in human-computer interaction that we discussed in this class, from building computational tools to designing computer-mediated social interactions, must reckon with the complex nature of our world and the way individuals behave and interact with technology. Generative agents point to a future in which the power to simulate hypothetical worlds, where we can ask ‘what if’ counterfactual questions and paint concrete pictures of how a multiverse of different possibilities might unfold, promises an opportunity to navigate this complexity. This week, reflect on how generative agents extend the original visions of the foundational literature on cognitive models that we covered in this class (e.g., GOMS, KLM) in the era of generative AI models. How might these simulacra of human behaviors inform the design of our interactions, and what new interaction paradigms will these agents enable?
3/5: Methodology (Quiz 4)
The Sciences of the Artificial (Chapter 5—The Science of Design: Creating the Artificial). Herbert A. Simon. MIT Press 1970.
In The Sciences of the Artificial, Herb Simon argues for the status of design as a rigorous method in and of itself. In doing so, similar to Schön, he contrasts design with traditional engineering subjects and the ways in which seeking to cast all goals in terms of formalization of optimization objectives narrows our view. Does the argument convince? How else might we argue for design as a rigorous method?
Ways of Knowing in HCI. Judith S. Olson and Wendy A. Kellogg. New York: Springer, 2014.
Pick a chapter representing a method that you’re interested in learning more about. What surprised you about what you learned?
3/17: Mental Health Tech (Guest Speaker: Talie Massachi)
Download Experimental evidence of massive-scale emotional contagion through social networks. Adam D. I. Kramer, Jamie E. Guillory, and Jeffrey T. Hancock. Proceedings of the National Academy of Sciences of the United States of America. 2014.
The 2014 Facebook emotional contagion study is a highly controversial paper in modern HCI. The paper received backlash from both the HCI community and the general public at the time of its publication. While reading the paper, take a close look at the methodology, the claims made by the authors, and the results. If you were on a review committee evaluating this study, what specific questions or concerns would you raise? What implication of this paper do you think had such a strong impact on the general public?
StudentLife: Assessing Mental Health, Academic Performance and Behavioral Trends of College Students using Smartphones. Rui Wang, Fanglin Chen, Zhenyu Chen, Gabriella M. Harari, Stefanie Tignor, Xia Zhou, Dror Ben-Zeev, and Andrew T. Campbell. Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 2014.
In this paper, researchers used continuous, background smartphone sensing (and user surveys) to monitor students’ well-being and stress levels. What do you think are the potential pros and cons of this type of research? Consider the validity of the research, the claims the researchers can make, and the potential future outcomes as a result of this work.
3/19: Accessibility
Ability-Based Design: Concept, Principles and Examples. Jacob O. Wobbrock, Shaun K. Kane, Krzysztof Z. Gajos, Susumu Harada, Jon Froehlich. ACM Transactions on Accessible Computing 2011.
Wobbrock and colleagues critique the typical approach to accessibility design, which is to focus on the dis in disability—the deficit. Instead, they argue for ability-based design, which places the designer’s attention on what the individual can do rather than what they cannot, and provide examples from their own research. What was your reaction to this inversion of the focus in accessibility? What kind of impact can, and can’t, it have on the designs we have around us?
GenAssist: Making image generation accessible. Mina Huh, Yi-Hao Peng, and Amy Pavel. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 2023.
How can blind and low vision users make better use of generative AI tools such as stable diffusion? GenAssist proposes a suite of automated checks to ensure that the generated image respects the desired caption. Obviously, this kind of approach cannot be 100% accurate in its checks. Given our discussions of AI overreliance, does this make it dangerous or useless? If not, why not? What design principles might we distill for these kinds of tools?
3/24: ICT4D (Quiz 4)
Designing mobile interfaces for novice and low-literacy users. Indrani Medhi, Somani Patnaik, Emma Brunskill, S.N. Nagasena Gautama, William Thies, and Kentaro Toyama. TOCHI 2011.
ICT4D focuses on interactive technology support for global populations, and in particular those who have less access to technology than your typical HCI practitioner. This paper is an example of how drastically we need to reconsider our understandings of usability, and its processes, when designing interfaces for users who may not be literate. Instead of focusing on the details of the design, use this paper to think broadly about what elements of HCI design should remain constant and which might need to be reconfigured like this.
‘Yours is better!’: participant response bias in HCI. Nicola Dell, Vidya Vaidyanathan, Indrani Medhi, Edward Cutrell, and William Thies. CHI 2012.
As discussed earlier in the class, this work empirically measures some of the ways that unintended biases can impact your conclusions about your design. What else ought we to look out for? How should we go about doing design that is globally inclusive?
3/26: Algorithm Auditing
3/31: Ed Tech
4/2: Critical Computing
In observance of Passover, commentaries will be due at 12:00 PM for the Critical Computing readings. However, there will be no penalty for lateness if commentaries are submitted by 11:59 PM.
4/7: Tools for Thought (Quiz 5)
4/9: Tech & Policy (Guest Speaker: Ro Encarnación) (Quiz 5)
4/14: Spatial Computing (Guest Speaker: Violet Whitney)
A Brief Rant on the Future of Interaction Design, Bret Victor
What is Spatial UX?, Violet Whitney
4/16: Usable Security (Guest Speaker: Mike Hicks)
Computer systems are used and built by people, and those people are partially responsible for the systems’ security. Whenever you choose a password, or think twice about clicking a link in a suspicious email, or write some code for a system that stores sensitive data, you are participating in the security story. You might wonder: How well do people perform these critical functions? If not well enough, what are the blockers and how can they be effectively addressed? These are the kinds of questions HCI researchers regularly ask and answer, and many HCI researchers have got together to form the usable privacy and security community (with flagship conference SOUPS) to study them. In this talk, I’ll give a brief overview of interesting research around passwords and phishing, giving a flavor of the UPS methods used to produce quantitative and qualitative results. In addition, I’ll talk about my own previous work on studying humans as developers, rather than end users, as part of the Build it, Break it, Fix it security-oriented programming contest. The contest served as a quasi-controlled study of methods human participants used for building hopefully-secure software, and how well different methods fared, both qualitatively and quantitatively. I will conclude my talk by highlighting opportunities for UPS work, especially as GenAI ramps up to become a partner in traditionally human-executed security tasks.