THE POLARA PROJECT
EMOTION AND THE COGNITIVE PROCESSING OF MEDIA
AVC EMOTIONAL MODEL
METHODOLOGY AND DATA COLLECTION
Our understanding of the emotional dimensions of human-computer interaction hasexpanded significantly over the course of the last decade. Human perception is criticallyinformed by emotion. This emotional dimension, a given in the study of the social dynamicsof human-human interaction, has been investigated in the study of human-computerinteraction yielding much the same results; humans tend to interact with computers in verymuch the same fashion as they interact with other humans -- socially (Nass, Steuer &Tauber 1994,). Even command line and text interfaces are, to a certain extent, infusedwith social significance (Reeves & Nass, 1996).
Lately there has been an explosion in the use of embodied autonomous characters andagent mediation in all types of computer software including entertainment (e.g. Grand& Cliff 1998; P. F. Magic), desktop applications (e.g. Microsoft, 1996) andpedagogical software interfaces (Lester et al 1997; Shaw, Johnson & Ganeshan, 1999).In an agent-mediated interface, embodied agents are presented as anthropomorphiccharacters that can be endowed to a greater or lesser extent with various elements oflife, engaging users in life-like interaction with the agent. Many of these agentcharacters are constructed to sense, synthesize and express emotions, while others have nointernal emotional state and rely on explicit scripting of expressive features to conveyemotionally appropriate cues to the user.
We believe endowing interface agents with emotional autonomy can be profitableto users and to designers as well. In this paper, we discuss the significance of emotionin the cognitive processing of mediated content and examine dimensional and discrete (orbasic) models of emotion. We then outline our adaptation of a dimensional model of emotionto an animated agent and the subsequent mapping of the agent's expressive features underthis model. Finally, we report the results of an exploratory test of the affective agent'seffects on user evaluation of the agent, performance on a task, and emotional responses.But first, we introduce the agent.
The Polara Project (Dietz, et al. 1998) is an effort to create an interactivemultimedia story-world that takes the user on an interactive journey to the stars. Themain character in the story is a quirky alien robot named "b'Nex," (pronouncedbin'hex). b'Nex leads users through various levels and activities in the cartoonstoryworld as a persistent character in the story and within the user interface.
The entire Polara Project world and the agent b'Nex were created in MacromediaDirector. The body of the agent is modular, consisting of very simple 2-D vector shapeswhich can manipulated and reshaped based on internal or external control. There areseveral benefits to this approach to animation. Primary among them is the ability toscript the animation, behavior and expressions of the character without having to draweach frame of animation manually. In addition behaviors and expressions can be combined innew ways based on pre-existing ones. The Petz series of interactive titles use a similarapproach (Frank & Stern, 1998).
The Polara environment was not constructed as a test bed for exploring the dynamics ofagent-human interaction. Rather, the agent, which started as a relatively inert interfaceguide through the storyworld, took on a life of its own as an integral character in thestory, a fellow traveler and companion. We wanted the character to be compelling andlikeable in the context of its role in the story and also extend what goodwill accruedfrom its story-based persona to its dual role as interface gatekeeper. We decided that inorder for the agent to be a compelling and likeable character in the story and exist as apersistent component of the interface we needed to endow the character with depth andexpressivity it would otherwise not have. b'Nex needed feelings.
Emotion plays a primary role in the processing of mediated stimuli. The presence ofemotion in mediated content improves attention, evaluation, and memory for mediatedpresentations. In addition, emotional media content elicits emotional responses in users.While intense emotion (both positive and negative) can interfere with cognitiveprocessing, moderate levels of emotion have been shown to improve performance on manyaspects of processing including attention, liking, enjoyment, memory, and vigilance.
Research examining the role of emotion in the processing of mediated messages hasprimarily focused on how different types of emotion in a mediated stimulus alter how userspay attention to, evaluate, and remember the mediated message. This research has shownthat emotional mediated content elicits increases in users self-reported emotion(Lang, Newhagen, & Reeves, 1996), physiological arousal (Lang, Bolls, Potter,Kawahara, in press), cognitive effort (measured using secondary task reaction times)(Lang, Dhillon, & Dong, 1995), and attention (physiological and self-reported) (Lang,Dhillon, & Dong, 1995; Lang et al., in press). As a result, memory for emotionalmessages, and in particular for arousing messages, is greater than memory fornon-emotional or less emotional messages (Lang, 1990; Lang, Dhillon, & Dong, 1995;Lang, it Newhagen, & Reeves, 1996; Lang et al., in press).
By and large, the research discussed above has looked at emotional responses totelevision messages. This research suggests that emotion improves cognition in mediatedsituations. Can the same be said of computer messages? Can the simple presence of emotionin the computer content improve attention, emotional response, liking, and performanceduring computer mediated tasks? Before addressing this we need to lay a foundation for ourinquiry into emotion.
There are two major theoretical approaches to the study of emotion, dimensional andcategorical. Theorists who use the categorical approach to emotion attempt to definespecific categories or types of emotion (Izard, 1977; Plutchik, 1980; Ortony & Turner,1990). Research in this area suggests that there are some number (estimates range fromthree to more than 20) of basic emotions which combine to produce all the emotional stateswhich people experience. Among the emotions most often designated as basic emotions aredisgust, anger, happiness, grief, and fear. In addition, intermediate emotional states areaccounted for through the "blending" of various basic emotions.
The dimensional approach (Bradley, 1994; Lang, Dhillon, & Dong, 1995)conceptualizes emotion as having two or perhaps three basic underlying dimensions alongwhich the entire range of human emotions can be arranged. The most common dimensions arevalence (which ranges from happy to sad) and arousal (which ranges from calm to excited).The third less often mentioned dimension is dominance (ranging from in control to out ofcontrol). Research using the dimensional approach to emotion has shown that emotionselicited by pictures, television, radio, computers, and sounds can be mapped onto anemotional space created by the arousal and valence axes, where the viewers' levels ofarousal and valence consistently predict emotional, cognitive, and physiological responsesto emotional stimuli.
Although evolving from distinctly dimensional theories of human emotional psychology,Our model is not meant to be descriptive of the actual workings of the human mind. Nor isit intended to be a rigorous and exhaustive model for implementation in interactiveagents. It is however intended as a simple and reasonably powerful method to invest agentcharacters with a sufficient range of emotional depth to be convincing and enhance userexperience.
The model implemented in the embodied software agent described in this study is basedon dimensional theories of emotion and is primarily inspired by the InternationalAffective Picture System (IAPS: Lang et al., 1988) and the International AffectiveDigitized Sounds system (IADS: Bradley and Lang, 1991). The IAPS is a standard set ofimages of emotional scenes that have been used in numerous studies of human emotionalresponse to media. The resultant scatter plot of IAPS and IADS responses along the axes ofarousal and valence reveal roughly the form of a parabola, an interaction between arousaland valence. There are relatively few if any stimuli that elicit responses categorized ashigh arousal and neutral valence, likewise for high valence (positive or negative) and lowarousal responses.
In our model of emotion, three dimensions -- arousal, valence and control/dominance define the emotional space. Any given point in this three-dimensional spacedescribes a unique emotional state. Few studies concern themselves with thecontrol/dominance dimension as it accounts for little variance, however it is very usefulin terms of distinguishing between two negative valence, high arousal emotional statessuch as grief and rage. The control dimension is necessary to achieve the range ofexpression needed for a fully expressive character. As in the IAPS and IADS, we find thatonly certain areas of this 3-dimensional emotional space are meaningful.
In the AVC model, there are three components -- temperament, mood and emotion, each ofwhich occupy a point in the emotional space. The temperament is a fixed coordinate whichdefines the at rest state of the system, defining a rudimentary personality for the agent.
The emotion, mood and temperament coordinates define the agent's persona and currentemotional state at any given time. Although temperament is static and constrained to thesurface described in Figure 3., mood and emotion coordinates can move freely throughoutthe space. The position of the emotion coordinate in AVC space gives rise to theexpressions of the agent; it is the emotional state of the agent. Emotion can existanywhere in the AVC space, but gravitates toward the position of the mood. Mood acts muchlike emotion, only it plods through the AVC space slowly, pulled by emotional events, butcontinually gravitating toward the temperament value. The dynamics of this system are suchthat the mood is influenced both by the current emotional state as well as the agent'stemperament. The position of the emotion coordinate can be set by environmental factors ordirect scripting and is influenced by the mood. Temperament is static and set when theagent is first run.
The expression of the agent character across its feature set is by and large a linearmapping of one or two emotional dimensions to a body feature. For instance, the amount thecharacter's eyebrows raise is directly proportion to its arousal and changes its shapebased on the valence of its current emotional state.
Table 1. below presents the mappings of the agent's expressive features from theunderlying AVC model. A wide range of meaningful emotional expressions can be derived fromthe combination of the AVC model and the feature mapping.
Table 1: Mapping emotional dimensions to agent features.
|Arousal and Valence
Valence and Control
Valance and Control
What will happen when agents display "self"- generated emotions whileinteracting with computer users? This study was designed to explore that question. Usersinteracted either with the agent without emotion or the agent with emotion. Usersperformed two tasks, the first was an emotional interview task and the second amemorization task. Users evaluated their emotional responses and the agent following theinteraction. In addition, performance on the memorization task was measured.
Research suggests that users will prefer the agent when it displays appropriateemotion. In addition it is expected that users will report feeling more emotion when theagent is emotional, and that they were perform better on the task working with anemotional agent. Some research has been done to empirically test these sorts of questions.In one study (Lester, Converse, Kahler, Barlow, Stone, & Bhoga, 1997), the affectiveimpact of animated pedagogical agents was tested. In this study the agent offered varioustypes of feedback during a learning task. Results showed that users preferred the agent inwhat was called the fully expressive condition (when all types of feedback were present)to all other conditions (which offered limited types of feedback). In addition,performance on the learning task was best during the fully expressive condition.
Gustafson, Schafer, & Konstan (1998) did not test the presence of affect in agentsbut did examine how workers alter their behavior in response to the presence of an agent.They find that workers accept agents and may accomplish their tasks more rapidly andwithout more errors.
Lester and Stone (1997) and Koda & Maes (1996) suggest that agents which exhibitlife-like expressions have greater motivational impact. Clearly the expression of emotionis a life-like behavior, thus the presence of emotion and in particular appropriateemotion in an animated agent should increase the believability and therefore the impact ofthe agent on users.
Finally Bates (1994) suggests that emotion plays a critical role in the creation ofbelievable agents. The appropriately timed expression of emotion plays a significant rolein whether characters are believable.
Based on this agent-related research and on the research about how emotion altersmediated processing the following hypotheses are made:
Hypothesis 1: Users will report feeling more emotion when interacting with theaffective agent than the non-affective agent.
Hypothesis 2: Users will evaluate the affective agent as more human, better to workwith, and more likable than the non-affective agent.
Hypothesis 3: Users will perform a memorization task better working with the affectiveagent compared to the nonaffective agent.
In this study we tested the effects of a life-like emotional agent whos facialexpressions and activity level are driven by a dimensional emotion model which receivesinputs from user actions and responds in emotionally appropriate ways to user actions. Thedesign of the experiment was a 2 (Task) X 2 (Emotional) between-groups experiment. TheTask factor represents the two tasks the subjects performed, one is an emotionalinterviewing task and the other is a learning (fact memorization) task. The EmotionalFactor had two levels, emotional and non-emotional.
The procedure for the experiment was as follows. First, subjects interacted with theagent in a conversational interview. The agent interviewed the subject, inquiring intohis/her own mood and feelings. In the emotionally convergent condition, the agent overtime matched its mood to that reported by the subject. Thus, as the interview continued,the agent became emotionally congruent with the subject. In the non-emotional conditionthe agent did not change in its emotional state or display emotion. Following thisinterview the subjects rated the agents personality, likeability, and functionalityon several indices.
In the second stage of the experiment, the subject participated in a memorization taskwith the aid of the agent. The agent taught the subject a list of facts, over which thesubject was later tested in the presence of the agent. In the emotional condition, theagent provided appropriate expressive emotional feedback in addition to cognitive feedback(i.e. correct/incorrect information). Specifically, when the subject performed well theagent became more positive and excited. When the subject performs poorly the agent becamemore negative and subdued. In the non-emotional condition, the agent provided onlycognitive feedback. Following completion of the task, subjects rated the agent on variouspersonality and functional indices. Meanwhile, the responses of subjects during theinterview task and accuracy on the memorization exercise were recorded through thecomputer interface and saved to a text file.
This hypothesis predicted that users would report feeling more emotion when interactingwith the affective agent compared to the non-affective agent. This hypothesis was testedby looking at the main effect of Emotion on users self-reported arousal, valence,and control following both the interview and a memorization task. Users reported feelingmore emotional after interacting with the emotional agent compared to the non-emotionalagent for both the interview task (F (1,39) = 4.97, p <.032) and the memorization task(F (1,39) assigned 5.46, p <.025). The means are shown in Table 2.
Table 2: Emotional response following the interview and memorization tasks.
This hypothesis predicted that users would evaluate the affective agent more favorablythan the non-affective agent. To test this hypothesis users rated the agent in terms ofhow helpful, attentive, smart, interested, and emotional the agent was. They also ratedhow much they liked the agent. Users rated the agent following both the interview and thememorization task. The main effect of Emotion on the ratings did not reach significancefor either the interview task (F (1,39) = 1.15, p <.291) or the memorization task (F(1,39) = 1.42, p < .240). The means are given in Table 2. For the memorization task,the affective agent was consistently rated more highly than the non-affective agent.However, for the interview task the non-emotional agent was rated more highly than theemotional agent. Figure 3. shows the results from the memory task.
Table 3: Evaluation of the Affective Agent
This hypothesis predicted that users would perform a memorization task better workingwith the affective agent compared to the nonaffective agent. The main effect ofEmotion on the memorization data was not significant (F (1,38) =.946, p <.337) thoughthe means were in the correct direction. The score for subjects working with thenon-emotional agent memory task was 66.67 percent of facts correct. The score for subjectsworking with the emotional agent memory is 70.37 percent of facts correct.
We have found that the AVC model and expression mapping implemented in the characterb'Nex endows the agent with a substantial range of emotional expressivity. The agent meetsthe goals set out in its design for the Polara Project interactive storyworld.
The research reported here begins to explore how affective agents might improve humancomputer interaction. The results of this study are somewhat encouraging. Although many ofthe results did not achieve statistical significance, they were by and large in thedirection predicted. Because the manipulation in this study was small, the tasks wereshort, and sample somewhat small, it would be unwise to conclude from these results thatadding affect to agents has little effect. Rather, given the above limitations on thisstudy, there is sufficient evidence to warrant further exploration of how affective agentscan be used to improve human computer interaction.
Indeed, future work will address these shortcomings and probe these areas. In addition,further development of the agent character will be to adapt a dimensional personalityspace with the AVC space and refining its expressions. We also intend to examine newquestions as well. What is the role of a sympathetic agent? How emotionally autonomousshould an interface agent be? Does human-agent interaction differ in any quantifiable wayfrom human-human interaction physiologically? From these examinations we hope to drawconclusions that will aid users and designers of agent-mediated interfaces alike as wellas contribute to a greater understanding of the role of emotion in the cognitiveprocessing of information be it mediated or otherwise.
Bates, J. (1994) The role of emotion inbelievable agents. Communications of the ACM 37(7):122-125
Bradley, M. (1994). Emotional Memory: A dimensional analysis. In Emotions: Essays onemotion theory. Hillsdale, NJ: LEA.
Bradley, M. M. and Lang, P. J. (1991) International affective digitized sounds(IADS): technical manual and affective ratings. Gainesville, University of Florida,Center for Research in Psychophysiology.
Grand, S. & Cliff, D. (1998) Creatures: Entertainment software agents withartificial life Autonomous Agents and Multi-Agent Systems, 1(1):39-57, 1998.
Dietz, R. B., Everson, B., Byers, W. H., Hunt, T., Koruth, M. & Kowalewski, K.(1998) The Polara Project. URL: http://www.polara.org (May 1999)
Ekman, P. (1992) An argument for basic emotions. In: Stein, N. L. and Collins,A. The Cognitive Structure of Emotions. Cambridge University Press, 1988.
Frank, A. and Stern, A., (1998) Multiple character interaction between believablecharacters. Proceedings of the Computer Game Developers Conference, Long Beach, CA.
Gustafson, T., Schafer, J. B., Konstan, J. (1998) Agents in their midst: evaluatinguser adaptation to agent-assisted interfaces. In Proceedings- IUI 98, 163-170, ACM
Koda, T., Maes, P. (1996) Agents with face: the effects of personification of agents.Proceedings HCI 96, London, UK.
Lang, A., Bolls, P., Potter, R., and Kawahara, K. (in press). The Effects of ProductionPacing and Arousing Content on the Information Processing of Television Messages. Journalof Broadcasting and Electronic Media.
Lang, A. & Newhagen, J., & Reeves, B. (1996). Negative Video as Structure:Emotion, attention, capacity, and memory. Journal of Broadcasting and Electronic Media,40, 460-477.
Lang, A. Dhillon, P., and Dong, Q. (1995). Arousal, Emotion, and Memory for televisionmessages. Journal of Broadcasting and Electronic Media, 38,1-15.
Lang, A. (1990). Involuntary attention and physiological arousal evoked by structuralfeatures and mild emotion in TV commercials. Communication Research, 17 (3),275-299.
Lang, P. J. (1979). A bio-information theory of emotional imagery. Psychophysiology,16, 495-512.
Lang, P. J. (1985). The cognitive psychophysiology of emotion: Fear and anxiety. In A.Tuma & J. Maser (Eds.), Anxiety and the anxiety disorders (pp. 131-170). Hillsdale,NJ: Lawrence Erlbaum associates.
Lang, P. J. and Greenwald,M. K. (1985). The international affective picture system slides and technical report.Gainesville, University of Florida, Center for Research in Psychophysiology.
Lang, P. J., Greenwald, M., Bradley, M. M., & Hamm, A. O. (1993). Looking atpictures: evaluative, facial, visceral, and the bureau responses. Psychophysiology, 30,261-273.
Lester, J. C., Converse, S. A., Kahler, S. E., Barlow, S. T., Stone, B. A. & Bhoga,R. S. (1997). The persona effect: affective impact of animated pedagogical agents.Proceedings of CHI'97 Human Factors in Computing Systems.
Microsoft (1996). IntelliSense in Microsoft Office 97. Microsoft Office 97Whitepaper.
Nass, C., Moon, Y., Fogg, B. J., Reeves, B., and Dryer, D. C. (1998) Can computerpersonalities be human personalities? International Journal of Human-Computer Studies,49.
Nass, C., Steuer, J. and Tauber, E. R. (1994) Computers Are Social Actors, inProceedings of CHI '94 Human Factors in Computing Systems. ACM Press, 72-78.
Ortony and Turner (1990). Whats Basic about basic emotions? Psychological Review,vol. 97, no. 3, 315-331.
P. F. Magic. Dogz, Catz, Oddballz and Babyz. URL: http://www.pfmagic.com (May 1999)
Plutchik, R. (1980). Emotions: a psychoevolutionary synthesis. New York: Harper andRow.
Reeves, B. and Nass, C. The Media Equation. (1996) Center for the Study ofLanguage and Information. 101-108
Shaw, E., Johnson, W. L. & Ganeshan, R. (1999) Pedagogical agents on the web.Proceedings of Agents '99.