Interactive systems and affective science

Contents

1 Introduction
1.1 Generative narrative
1.2 Tale-Spin
1.3 Computer games
1.4 Interactive drama

2 A sample story in interactive drama
2.1 IDtension
2.2 The mutiny
2.3 Identifying the problems

3 Affective sciences and narrative studies
3.1 Theories of emotions and affect
3.2 Dramatic narrative and affect elicitation
3.3 Interest

4 Immersion in a virtual world
4.1 Façade
4.2 Aesthetics and emotions
4.2.1 Appraisal theory of aesthetic emotions . .
4.2.2 The make-believe theory

5 Uses of player mood modelling in interactive drama pre-sentation
5.1 Cognitive-affective behaviour regulation
5.2 Interactive drama in pedagogical and therapeutic use
5.2.1 Carmen’s Bright IDEAS
5.2.2 Prom Week
5.2.3 FearNot!

6 Integrating the ideas

7 Concluding considerations and outlook

Appendices

Abstract

Zusammenfassung

CV

Acknowledgements

The Austrian Research Institute for Artificial Intelligence is supported by the Austrian Federal Ministry for Transport, Innovation, and Tech- nology.

The research leading to these results was supported in part by the European FP7 project IRIS (contract no. IST FP7 231824).

IDtension is a research project on interactive drama carried out by Dr. Nicolas Szilas. The project is hosted by the TECFA Lab at the University of Geneva, Switzerland.

Introduction

"There are countless forms of narrative in the world", Roland Barthes explains in his Introduction to the Structural Analysis of Narrative. [4, p.237] "Moreover," he continues, "in this infinite variety of forms, it is present at all times, in all places, in all societies; indeed narrative starts with the very history of mankind; there is not, there has never been any- where, any people without narrative; all classes, all human groups, have their stories [...]." [4, p.237] If Barthes is right, we have to regard narra- tive as both transcultural and transmedial. Similarly Marie-Laure Ryan describes narratology, the study of narrative and narrative structure, as a project that transcends disciplines and media. [41, p.289] Moreover, one of the main attractions of narratives, independent of the medium in which they are presented, is that they elicit emotional response in their audiences. They have an affective impact. [46, p.136] This thesis inves- tigates a recent form of digital narrative, known as interactive drama. Interactive drama shares many similarities with two other forms of nar- rative, namely filmic narrative and computer games, but there are also significant differences. While this thesis emphasises the identification of these similarities and differences the main focus is on arguing that an interactive drama, based on a model of the user’s affective state, should be able to provide a truly immersive user experience.

1.1 Generative narrative

Writing any form of narrative is a creative process. Some works of narrative are regarded as more creative than others but as long as the final product is, at least to some degree, novel, we have to attribute cre- ativity to the author. The notion of novelity is important for generative systems, systems that basically create new story content "on the fly". As Pablo Gervás proposes, "whatever is generated must be somewhat unexpected or different from what others might have produced." [22, p.49] Another important aspect is the notion of a "creator", an agent which acts in a creative way to create results that "can be perceived or evaluated." [22, p.50] A fascinating pattern of clouds, even if new and unexpected, is therefore not considered as a result of creativity because there is no creator. With the needs of being perceivable and evaluable another factor comes to play: the audience. Gervás simply defines the audience as "the particular person or persons for which the creative ac- tion is intended" [ibid.] and henceforth I will stick to this definition for the audience of all kinds of narrative. In interactive systems the audi- ence is not a passive one, and this is exactly what fuels my position that interactive narrative provides exciting new possibilities, not available to non-interactive forms.

In Computational Approaches to Storytelling and Creativity 22 Pablo Gervás provides an overview about the history of generative storytelling. The first system Gervás introduces was developed in 1973 and was sim- ply called Novel Writer 25 in22. It creates murder stories in a story- world represented as a semantic network. The states of the storyworld constantly change according to probability rules, yet the sequence of

scenes is hardwired. As Gervás explains, the set of rules is highly con- straining and allows for the construction of only one very specific type of story, the happening of a murder during a weekend party. While the interplay between the characters is generative, the plot itself is not. Au- thor 16 in22, developed in 1981 tries a different approach. It is based on the claim that storyworlds "are developed by authors as a post hoc justification for events that the author has already decided have to be part of the system." [22, p.53] Author tries to model the kind of knowl- edge a real author needs to create a story and also the way this knowl- edge is organised and accessed. According to Gervás the model has two metagoals, namely to achieve the current narrative goal and to find bet- ter narrative goals to follow. "It is this second metagoal [that allows] for changes in direction when unforeseen opportunities arise." [22, p.54] Instead of modelling the author’s mind, Universe 27 in22, devel- oped in 1983, models the generation of scripts for soap opera episodes. According to Gervás it is the first storytelling system to devote special attention to the creation of characters. [22, p.54] However, Universe cannot be considered as an autonomous system but rather as a tool to aid human authors. Complex data structures are presented to represent the characters but only a small amount is filled in automatically. Most of the actual characterisation has to be done by the user. While Uni- verse was the first storytelling system to devote special attention to the creation of characters, Gervás explains, Minstrel 56 in22, developed in 1993, was the first storytelling system to address specifically issues of creativity. [22, p.54] Minstrel constructs short King Arthur stories in two stages, a planning stage and a problem-solving stage, the first trying to break down author-level goals into smaller author-level goals, the second trying to solve the goals by adding the required ingridients to the story. According to Gervás there are two operations of particular interest:

"One involves solving author-level goals, which takes the form of instantiating a set of partially complete char- acter schemas. This is done by querying episodic memory with partially complete schemas and using the results to in- stantiate the corresponding author-level goal. Another is the way opportunistic goals are triggered. Each time a new scene is created, Minstrel revises it to check whether it pro- vides an opportunity to apply one of the author-level goals that ensure consistency or introduces one of the desired lit- erary motifs." [22, p.55]

While these systems are certainly interesting for their pioneering work in generative narrative, they are lacking one important aspect: interac- tivity. Their only purpose is to generate "new stories" at runtime but the user, or in this case rather the reader, has no influence on this process. While also still lacking the immersive experience we want to achieve in interactive drama, the next system I will present brings us one step closer to the idea of having the user engage in the story generation pro- cess.

1.2 Tale-Spin

Tale-Spin was a landmark project that has been carried out at Yale University in the mid 1970s as part of James Meehan’s dissertation32. The idea behind Tale-Spin was to create a system that brings us closer to what Meehan calls a "Metanovel". The idea behind such a metanovel, as presented in Meehan’s dissertation goes like this:

"A metanovel is a computer program that tells stories that only a computer can tell, stories of such complexity of detail that only a computer could handle, stories with more flexibility—even reversibility—of events and charac- ters than a human could manage. A metanovel time-sharing system tells a story to many people at once, no two of whom read the same thing, because they have each expressed dif- ferent interests in the events and characters they want to hear about, and because they may each desire a different style of storytelling. And yet, among all these readers, there is but one story—the metanovel itself—and each reader is only following those threads of the story that interest him." [32, p.ii]

There have been attempts in digital narrative where users could decide between different paths to continue, similar to "Pick your path"-stories in children books (Figure 1.1); but in these books, and systems, the user just makes a local choice that leads to the next pre-written chunk of narrative instead of having real impact on the narrative stucture itself.

Tale-Spin is quite different to these approaches and in a way brings us closer to what we want to achieve in interactive drama. The difference, Wardrip-Fruin explains, is due to the granularity of story processes and data: The system models story as "a relatively fine-grained set of pro- cesses and data that are used to generate story events. In the case of Tale-Spin, this is accomplished by creating a simulated world, and char- acters and objects that populate the world." [57, p.116] Therefore the slightest changes can lead to very different stories. The architecture is based on the AI concepts of plans and goals, which are, along with ob- stacles, also essential elements of narrative structure but it is important to understand that stories produced by Tale-Spin are far from the literary

illustration not visible in this excerpt

Figure 1.1: Meanwhile: Pick Any Path47

quality of stories a capable human author can produce. Furthermore, we should not mistake Tale-Spin for something it was not meant to be: an interactive drama system like the ones I will discuss later on. Nevertheless, understanding the way Tale-Spin generates stories can teach us a lot about generative narrative.

The storyworlds in Tale-Spin are rather simple, containing only some animals (like a bear and a bird) and some objects (like honey, berries or a worm) and dependencies between them. A basic story would then for example be about a hungry bear, named Arthur, who wants to get some honey, however, only the bird, named George, knows where honey can be found and he only tells Arthur about the location in exchange for a worm. The user who interacts with Tale-Spin does not take part in the storyworld by controlling a character but rather by answering questions the system asks. For the story above, the system would have asked for example who the main character is (Arthur) and what his problem is (he is hungry). The possible answers are provided by the system and the user has to pick one. In this simple world, possible answers to the first question would have been either George or Arthur and to the second hungry, tired, thirsty or horny, respectively.

Tale-Spin stories, like most narrative, are about a main character who has to overcome some obstacles to reach a desired goal and to overcome these obstacles they need a plan. The plan always depends on the given circumstances. If Arthur knows where to find honey he only has to go to that location but if he does not know he has to try something else, like asking George for help. The control structure of Tale-Spin is shown in Figure 1.2.

illustration not visible in this excerpt

Figure 1.2: Control structure of Tale-Spin ’ s

simulator [57, p.128, In the example scenario presented, the user selected "hungry" as Arthur’s Figure 5.1]

problem. From this selection Tale-Spin creates a so-called dependency expression "Arthur knows that he is hungry". Without going into too much detail on the control structure, the process resulting from Arthur realising that he is hungry can be summed up like this:

1. The assertion mechanism (top right) adds the information that Arthur is hungry to the memory (bottom left) and is indexed as something Arthur knows.
2. The information is passed on to the inference mechanism (bottom right) which leads to an expression like "Arthur knows that Arthur intends not to be hungry". This information is again asserted.
3. The problem solver (top left) is invoked and the planning process is started. It checks the memory if Arthur knows where food can be found.
4. If he knows where honey can be found the control to get it is invoked, if he does not the memory is checked if Arthur knows who knows where honey can be found.
5. These processes are continued until the problem is solved or until there is nothing left to check and reaching the goal fails. Failed goals are kept in memory.

Whether Arthur asks George about the location of the honey depends on whether Arthur thinks that George is his friend. Tale-Spin asks the au- dience if this is the case: "Does Arthur think George likes him?" Again the audience can select an answer from four possibilities (a lot, a little, not much, not at all). Actually Tale-Spin keeps four different states like that in memory, namely whether Arthur thinks that George likes him, whether Arthur thinks he likes George, whether George thinks Arthur likes him and whether George thinks he likes Arthur. The same hap- pens for the relationship variables trust, dominate and feeling indebted. These relationships are used for developing different plans and creating somewhat believable character behaviour. The user does not interact with the storyworld as a character but rather defines how they want the story to evolve. Later on I show that there already exist interac- tive drama systems, especially for therapeutic use, that pursue a similar strategy.

While Tale-Spin creates interesting behaviours, a problem of the sys- tem is that a lot of the interesting behaviours happen at system level but never reach the audience. This problem has become known as the Tale-Spin effect [57, p.115] which means that, for a layperson, a sys-tem appears to be very simple but is actually based on very complex

technology. In contrast, there are systems that seem to be very sophisticated but are based on rather simple mechanism.This phenomenon has become known as the Eliza Effect [57, p.23] after Joseph Weizenbaum’s famous computer system.58

1.3 Computer games

When talking about interactive drama one question inevitably arises: "Isn’t that just like a computer game?" The idea behind interactive drama is to provide the player, who interacts with the system, with a feeling of freedom, by providing a high number of different choices and dif- ferent outcomes and still maintaining a cohesive story structure. Those familiar with computer games, when reading this description, will most likely think of roleplaying games, or RPGs. After all, as Noah Wardrip- Fruin explains in his book Expressive Processing, this is the most story-ambitious genre of computer games. [57, p.44] As we will see, the high number of possible choices in interactive drama as well as in computer games has its share of problems. While it is true that they provide the feeling of freedom, the problem is that the impact of most of these choices on the structure of the narrative is minimal. Therefore, sooner or later, the player will realise that this feeling of freedom is just an illusion. The ultimate goal of interactive drama systems has to be providing choices that "really matter". There is another downside of providing a large amount of user choices, namely the possibility that the user, who is usually not an experienced author, destroys the struc- ture of the narrative. This is a problem far more relevant for interactive drama than for computer games, simply because in games the player does not really interact with the structure of the narrative itself. In an RPG the player character rather interacts with objects and characters in a fictional world while the execution of the game rules, the presentation of the game world and the actions of the none-player characters (NPCs) are controlled by the computer system via quest flags and dialogue trees, which will be discussed in a moment.

In modern RPGs the premises are often similar. The player character starts in a fictional world, may that be a medieval fantasy world, the post-apocalyptic future or a far away galaxy, without much knowledge of the characters’ past (amnesia is quite common) and the future ahead. Then it is up to the player to discover the storyworld, solve some early quests and usually find out that the fate of the whole land or galaxy depends on them. The player engages in many different activities like exploration, dialog, puzzle solving, and, in most cases armed combat. For example the official synopsis of Fallout: New Vegas, one of the last years’ most successful RPGs, goes like this:

Welcome to Vegas. New Vegas.

It’s the kind of town where you dig your own grave prior to being shot in the head and left for dead... and that’s before things really get ugly. It’s a town of dreamers and desperados being torn apart by warring factions vying for complete control of this desert oasis. It’s a place where the right kind of person with the right kind of weaponry can really make a name for themselves, and make more than an enemy or two along the way.

As you battle your way across the heat-blasted Mo- jave Wasteland, the colossal Hoover Dam, and the neon drenched Vegas Strip, you’ll be introduced to a colorful cast of characters, power-hungry factions, special weapons, mu-tated creatures and much more. Choose sides in the upcom-ing war or declare "winner takes all" and crown yourself the King of New Vegas.7

What is especially interesting for our approach is that "many RPGs give the sense that the story itself is playable by offering the player freedom to roam across a world infused by quests that operate on many scales, can sometimes be completed in different ways, and are often optional or available for partial completion. As each player chooses which quests to accept—as well as how, whether, and when to complete them—this creates a different story structure for each playing." [57, p.47] As we will see this is quite similar to what we want to achieve in interactive drama: to truly interact not only with objects in a storyworld but with the structure of the narrative itself. But is this really the case in RPGs?

Despite the variety of experiences that players can have with quests, Wardrip-Fruin explains, it is commonly observed by players and authors that there are a limited number of types of quests. In RPGs what you do when solving a quest is basically going through a checklist that represents states of the game world. (see Figure 1.3)

This checklist is normally directly available to the player via a journal or quest log. In Fallout: New Vegas the quest log is accessible through a computer devise the player character wears on their arm. Figure 1.4 shows this log for one example quest called The Advance Scouts. As you can see the main task is to deal with the White Legs camp. The White Legs are a hostile tribe that is about to attack a friendly tribe to which the player character has offered help. Along with the task itself the log presents two different ways of solving the problem. The player could either simply kill the enemy tribe or they could choose to sneak into the camp and steal the tribe’s war totem which causes them to retreat.

illustration not visible in this excerpt

Figure 1.3: Part of the storyworld in Fallout: New

Vegas7

In my opinion, this has nothing to do with interaction with the narra- tive. Granted, between starting the quest and finishing it the player can choose between two different paths to reach the goal and get some re- ward (money and experience points), but both paths are basically a sim- ple interaction with objects in the game world. The player can choose to interact with the members of the enemy tribe (by shooting them) or with the totem (by stealing it) but this is not where the narrative unfolds or makes any progress at all. Progress of the narrative happens during scripted scenes before and after the quest - during the non-interactive parts of the game. Wardrip-Fruin comes to a similar conclusion:

"Yet given the goal of the audience feeling able to play within the fictional world - not just in areas such as com- bat but also in terms of the stories being told - the quest flag structure quickly brings game authors to a limit point of complexity, holding back the future development of the form." [57, p.51]

illustration not visible in this excerpt

Figure 1.4: Quest log in Fallout: New Vegas7

Computers are not particularly good in understanding free form dialog. There are systems that try to implement free form dialog, one of which I will discuss in Chapter 4.1, but the results are very limited to say the least. A more common way of dialog in RPGs (and some other genres) is the so-called dialogue tree. The user can for example choose one out of five different dialogue options (see Figure 1.5 for an example in Fallout: New Vegas) and the system selects an appropriate response (see Figure 1.6) by traversing a hierarchically organised data structure.

Wardrip-Fruin cites Chris Bateman who is rather skeptical about this approach:

"Despite the name, dialogue trees are seldom true trees but rather converging and diverging chains of conversation. They can be a nightmare to work with, and the benefits they provide are somewhat minimal. Nonetheless, some players greatly appreciate the illusion that they have control over what their character can say, with the consequence that di-

illustration not visible in this excerpt

Figure 1.5: A question from the dialogue tree in

Fallout: New Vegas7

alogue trees remain important, especially in cRPG games." [5, p.277] in [57, p.55]

However Wardrip-Fruin disagrees with Bateman on the notion of "illu- sion of control". While a system like that might be an illusion of real conversation, it is still an important method of decision making. Even though we are dealing basically with simple menu selection the choices matter in the game world and the different outcomes can be vast. This is also be important aspect of interactive drama.

These quests and dialogue trees described above are essential for an en- gaging user experience. As Wardrip-Fruin writes, an RPG uses them "to reward and sustain engagement with its fictional world; establish patterns that, when altered, produce small moments of surprise and pleasure; and direct the audience’s attention to a series of things that must be accomplished through play." [57, p.59] So what makes a game compelling enough to keep players continuing for endless hours and even replay the game multiple times after they have finished it? The narrative itself is most likely an important aspect. The player wants to

illustration not visible in this excerpt

Figure 1.6: An answer given by an NPC in Fallout: New Vegas7

know how the story unfolds and how it ends. However, most players of RPGs do not just follow the main storyline but rather engage in nu- merous optional quests which do not really add much to the narrative itself. So there have to be other reasons for the player to keep exploring the storyworld. The player has to be rewarded for their engagement. In most RPGs these rewards are similar: money, more powerful weapons and items and experience. The latter can be used to "level-up" the player character which means they become more powerful, which again allows them to solve harder tasks which provide even more money, items and experience. Wardrip-Fruin calls this the "just one more thing" mindset that keeps players going for hours upon hours.

1.4 Interactive drama

Interactive drama is a recent form of experiencing narrative. One of the central goals for every interactive drama system is to provide the user with the feeling of agency. Agency is the power to take meaningful actions and to see the results of the decisions and choices made. An interactive drama offers a world in which the participant can have a real effect on the drama which they is experiencing. It is important to un- derstand that a linear or multi-linear story as found in computer games is clearly not an interactive drama. As we have seen, game genres like roleplaying games provide a large amount of interactivity and story vari- ance but the point here is that in these games the player interacts with objects and characters while following a predetermined story. Even if these games provide different story branches and different endings, each of these branches and ending is pre-written by the story authors. The player does not interact with the structure of the narrative itself. Each time the user participates in an interactive drama they should identify the story as being essentially a new story. To achieve this, the main sto- ryline will need to significantly differ every time the user participates. Insignificant changes, such as only the ending differing or characters having a different trivial conversation will not be sufficient.

The Oz Project, led by Joseph Bates at Carnegie Mellon, started un- der a rather unusual premise. As Wardrip-Fruin puts it, "the Oz Project may be the only computer science research project most famous for an experiment that did not require computers." [57, p.317] What they ini- tially did was creating interactive drama, but quite different from the others I will discuss in this thesis. Instead of using computer simula- tions, they decided to equip real actors with headsets and guide their actions, when required, by instruction by a human director. Brenda Laurel, one of the pioneers in the research field of interactive drama, defines interactive drama as "[...] a first-person experience within a fan- tasy world, in which the user may create, enact, and observe a character whose choices and actions affect the course of events just as they might in play." [26, p.6] in [57, p.318] The first Oz experiment matched this definition. The first-person interactor was a test-person who was asked to enter a simple theatre set, representing a bus station, to buy a ticket.

On this set there were also three actors, the clerk from whom the test- person should buy a ticket, another passenger and a young punk, and they all had the goal to make the test-persons task as hard as possi- ble. They acted autonomously, like in an improvisational theatre, but as mentioned above they had headsets to get directions from a director to intervene if things got out of hand. Along with the experimenters, there was also additional audience to watch the play. It turned out, and this is essential for interactive drama, that while the test-person reported an in- credibly engaging experience, the additional audience were less excited and sometimes even bored. So obviously the experience in interactive media is quite different to the experience in other forms of media like novels or film. The basic design philosophy of interactive drama was born:

"[P]lacing the audience member as an interacting char- acter within the drama (as an interactor), creating expres- sive and relatively independent non-player characters (NPCs) within the same environment, and guiding the higher level actions of the NPCs through the interventions of a drama manager tasked with adjusting pacing for the interactor and guiding the story to a successful conclusion." [57, p.319]

Following these guidelines, Joseph Bates led the first effort in applying agent technology to entertainment software in the early 1990s. They developed technology for dramatic virtual worlds and the first of these worlds was called "Edge of intention". (see Figure 1.7)

The creatures inhabiting this world were called "Woggles". The Wog- gles’ behaviour was guided by an agent architecture that integrated re- activity, goal-directed behaviour, and emotions. Three of the four Wog- gles were controlled by this architecture and one was controlled by the

illustration not visible in this excerpt

Figure 1.7: The Edge of

Intention6

user who interacted with the storyworld. As Wardrip-Fruin explains, "these characters needed to have both long-term and short term behav- iors, with some behaviors supporting others, multiple behaviors able to take place simultaneously, and some reasonable method of what to do at any given time." [57, p.320] In their graphical representation the Woggles were ball-like creatures with eyes that could move around their environment by bouncing. The user, who controlled their Woggle with a mouse, could engage in simple activities like for example a game of catch. Goal success, failure, prospective failure, and decisions that other Woggles made gave rise to happiness, sadness, fear, gratitude, and anger to varying degrees. The authoring language for the agent’s be- haviour was called Hap. "Instead of visual appearance, it was focused on characters that could act appropriately and autonomously in a fic- tional world, and do so based on goals and behaviors crafted by authors for that character." [57, p.322] Goals and sub-goals are organized in a tree-like structure and each goal has an author-defined set of possible behaviours that would satisfy it. The problem here is the same as the one I mentioned when discussing Tale-Spin: most of the interesting pro-cesses happen on the inside and never reach the audience. The audience sees only the successful outcomes of the internal "reasoning" processes but all the failures that happened before would be—at least—equally interesting.

Phoebe Sengers, who at that time did her PhD as member of the Oz Project, saw this problem. She built a set of extensions, inspired by narrative psychology, which she called the Expressivator. Michele L. Crossley describes narrative psychology as an approach that was formu- lated "as an alternative to dominant quantitative approaches which, in their attempt to numerically categorize experience through quantifica- tion and statistical procedures, failed radically to incorporate or address these hermeneutic dimensions of experience and thus lost any sense of the ’lived’ nature of human reality and identity." [15, p.361] The cen- tral premise of narrative psychology is, as Crossley explains, the exis- tence of an essential and fundamental link between experience of self, temporality, relationships with others, and morality which are "[...] the kinds of stories and narratives through which we make sense of our lives [...]." [15, p.361] If these links are not understood, simply because they are not communicated, the audience fails to interpret why behaviour changes occur, and the behaviour of the Woggles often seems—in Sen- gers term—schizophrenic. Therefore the idea of the Expressivator is to communicate why changes are being made. Communicating why things happen is an essential step in creating an engaging interactive drama.

[...]