Dave Chan for The New York Times
NOT QUITE REALITY Stéphane Bouchard, of University of Quebec in Ottawa, reacting with his very similar avatar.
Published: November 22, 2010
OTTAWA — His talk was going just fine until some members of the audience became noticeably restless. A ripple of impatience passed through the several dozen seated listeners, and a few seemed suddenly annoyed; then two men started to talk to each other, ignoring him altogether.
“When I saw that, I slowed down and then stopped what I was saying,” said the speaker, a 47-year-old public servant named Gary, who last year took part in an unusual study of social anxiety treatment at the University of Quebec.
The anxiety rose in his throat — What if I’m not making sense? What if I’m asked questions I can’t answer? — but subsided as his therapist, observing in the background, reminded him that the audience’s reaction might have nothing to do with him. And if a question stumped him, he could just say so: no one knows everything.
Michal Czerwonka for The New York Times
Arno Hartholt interacts with characters in a simulated Old-West saloon at U.S.C.
He relaxed and finished the talk, and the audience seemed to settle down. Then he removed a headset that had helped create an illusion that the audience was actually there, not just figures on a screen. “I just think it’s a fantastic idea to be able to experience situations where you know that the worst cannot happen,” he said. “You know that it’s controlled and gradual and yet feels somehow real.”
For more than a decade, a handful of therapists have been using virtual environments to help people to work through phobias, like a fear of heights or of public spaces. But now advances in artificial intelligence and computer modeling are allowing them to take on a wider array of complex social challenges and to gain insight into how people are affected by interactions with virtual humans — or by inhabiting avatars of themselves. Researchers are populating digital worlds with autonomous, virtual humans that can evoke the same tensions as in real-life encounters. People with social anxiety are struck dumb when asked questions by a virtual stranger. Heavy drinkers feel strong urges to order something from a virtual bartender, while gamblers are drawn to sit down and join a group playing on virtual slot machines. And therapists can advise patients at the very moment those sensations are felt.
In a series of experiments, researchers have shown that people internalize these virtual experiences and their responses to them — with effects that carry over into real life. The emerging field, called cybertherapy, now has annual conferences and a growing international following of therapists, researchers and others interested improving behavior through the use of simulations. The Canadian military has invested heavily in virtual-reality research; so has the United States Army, which has been spending about $4 million annually on programs with computer-generated agents, for training officers and treating post-traumatic stress reactions. The trend has already generated a few critics, who see a possible downside along with benefits.
“Even if this approach works, there will be side effects that we can’t anticipate,” said Jaron Lanier, a computer scientist and author of “You Are Not a Gadget: A Manifesto” (Knopf, 2010). “And in some scenarios I would worry about defining humans down: defining what’s normal based on what we can model in virtual environments.”
But most researchers say that virtual therapy is, and will remain, no more than a therapist’s tool, to be used only when it appears effective. “There’s a real and understandable distrust of technology as a shortcut for good clinical skills,” said Albert Rizzo, a psychologist at the University of Southern California, “but I think, deep down, most therapists will want any tool that can help them do their work, and they’ll be open to using virtual approaches.” Virtual Humans, Real Therapy
“My abilities are somewhat limited,” says a female voice. “For example, I can speak and listen to what you say, but I can’t do any physical activity.” In an office at the Institute for Creative Technologies at the University of Southern California, a virtual woman named Angelina is addressing a college student from a computer screen. Angelina looks to be about 30 or so, a pretty, athletic figure with an open, intelligent face framed by short black hair. Her eyes and expression, guided by video cameras and microphones, stay in sync with the student’s, as an empathetic therapist’s would. “What are some of the things you hate about yourself?” asks the voice.
The student stalls for a moment. “Well,” she says, in a video of the exchange, “I don’t like that I can be really quiet in social situations. Sometimes people take that as me being rude, but it’s just me being quiet.” Angelina nods sympathetically and then asks another question, about what the student fears most.
Interacting with a virtual human programmed to be socially sensitive in this way is oddly liberating. The figures are clearly not human; some are balky with language, others mute. Many have a two-dimensional graphic-arts quality.
But the faces are mobile, blinking, alive, the body language and gestures seemingly natural; in some cases, the voice recognition and choice of replies are good enough to conduct a stiff but convincing conversation. The result is a living presence that is responsive but not judgmental.
In a recent study using this virtual confidant, researchers at U.S.C. have found that Angelina elicits from people the crucial first element in any therapy: self-disclosure. People with social anxiety confessed more of their personal flaws, fears and fantasies to virtual figures than to live therapists conducting video interviews, the study found.
Michal Czerwonka for The New York Times
SIMULATING WAR Josh Williams, left, and Andrew Smith, demonstrate a program at U.S.C. designed to help veterans.
The researchers are incorporating the techniques learned from Angelina into a virtual agent being developed for theArmy, called SimCoach. Guided by language-recognition software, SimCoach — there are several versions, male and female, young and older, white and black — appears on a computer screen and can conduct a rudimentary interview, gently probing for possible mental troubles. Using SimCoach on a laptop, veterans and family members would anonymously ask about difficulties they’re having, whether due to post-traumatic stress or other strains of service.
Michal Czerwonka for The New York Times
NO COUCH REQUIRED Sin-hwa Kang demonstrates visual therapy at U.S.C.
“It does not give a diagnosis,” said Jonathan Gratch, a co-author of the Angelina study with Sin-Hwa Kang, also of U.S.C. “But the idea is that the SimCoach would ask people if they would like to see a therapist; and if so, could then guide them to someone in their area, depending on what it has learned.”
Once people are in treatment, therapists can use virtual technology to simulate threatening situations — and guide patients through them, gradually and incrementally, calibrating the intensity of the experience.
In person-to-person sessions to address anxieties or phobias, for instance, therapists may have patients do this in their imaginations. Revisit a dreaded experience — say, a rooftop party, for a person afraid of heights — while defusing the physical reactions to the memory in the office. Out in the world, patients then practice the same techniques, gradually increasing their exposure, beginning with modest heights, for instance, and working up.
Using virtual environments, therapists can run this entire drill in their offices. At theVirtual Reality Medical Center in San Diego, psychologists have treated hundreds of patients using gradual virtual exposure, for post-traumatic stress and agoraphobia, among other anxieties. At U.S.C., Dr. Rizzo has designed a program specifically for veterans of the Iraq war. In one scenario, wearing a headset, the patient is in a virtual Humvee, motoring along a desert road toward a small Iraqi village. To the right is a passenger, another soldier; behind and above rides a gunner; in front is another Humvee. As the motorcade approaches the village, engines rumbling, there is a flurry of gunfire, and more. A roadside bomb goes off, bullets pierce the window — your fellow soldier on the right is wounded badly, now dying — all of it under control of the therapist.
“We can control the intensity of the experience, and then work on the patient’s response,” Dr. Rizzo said.
When it works, the therapy breaks the association between reminders of an upsetting experience and the racing heart, the flushing, the panic that the person has been struggling with.
Adding autonomous virtual humans to the landscape allows therapists to begin addressing some of the most complex problems of them all — social ones. In one continuing study at the University of California, Davis, for instance, researchers are trying to improve high-functioning autistic children’s ability to think and talk about themselves while paying attention to multiple peers. The hope is similar for people with social anxiety: that practice interacting with a virtual boss, suspicious strangers or virtual partygoers who are staring as one enters the room will also lead to increased comfort, with the help of a therapist. “The figures themselves don’t even have to be especially realistic to evoke reactions,” said a psychologist, Stéphane Bouchard, who directs the cybertherapy program at the University of Quebec in Outaouais. “People with social anxiety, for example, will feel they are being judged by virtual humans who are simply watching them.”
In the pilot study that included Gary, the University of Quebec researchers tracked two groups of patients: one that received an hour of talk therapy once a week for 14 weeks and another that got talk therapy with a virtual component, practicing virtual interactions. Both groups showed improvement, faring much better than a comparison group put on a waiting list, preliminary results suggest. But those who got virtual therapy achieved the same gains without having to practice interactions in the real world, deliberately putting themselves in embarrassing situations or dreaded encounters. The researchers are now working to identify which people benefit most, and whether combining virtual and real-world experiences accelerates recovery.
My Avatar, Myself
The face in the mirror does not look familiar; it has a generic, computer-generated look. Yet it does appear to be staring out from a mirror. Lift a hand and up goes its hand. Nod, wave, smile, and it does the same, simultaneously. Now, look down at your own body: and there, through the virtual reality headset, are a torso, legs, clothes identical to those in the mirror.
In a matter of minutes, people placed in front of this virtual mirror identify strongly with their “body” and psychologically inhabit it, researchers at Stanford University have found. And by subtly altering elements of that embodied figure, the scientists have established a principle that is fundamental to therapy — that an experience in a virtual world can alter behavior in the real one. “The remarkable thing is how little a virtual human has to do to produce fairly large effects on behavior,” said Jeremy Bailenson, director of the Virtual Human Interaction Lab at Stanford and the author, with James Blascovich, of the coming book “Infinite Reality” (HarperCollins 2011). In one recent experiment , Dr. Bailenson and Nick Yee, now at the Palo Alto Research Center, had 50 college students enter a virtual environment and acquire a virtual body, an avatar. Each student then participated in a negotiation game with a member of the experimental team, who was introduced as another student. But all the avatars were not created equal. Some were four inches taller than their human counterparts, and others were four inches shorter. The participants didn’t notice this alteration, but those made taller negotiated in the virtual game much more aggressively than those made shorter. A later study led by Dr. Yee found that this effect carried over into face-to-face negotiations after the virtual headsets were removed.
The researchers have demonstrated a similar effect in the case of attractiveness. In another experiment, they created generic avatars for some participants that were about 25 percent “more attractive” than average, based on features that the group had rated as attractive. Compared with study participants whose avatars were made 25 percent “less attractive,” the virtual beauties were more socially confident, standing closer in virtual conversation, revealing more about themselves — an effect that also seeped into social interactions after the headsets came off.
Again, no one noticed the manipulation; its effects were entirely subconscious.
The authors argue that the participants, in effect, psychologically internalized their virtual experience. “What we learn in one body is shared with other bodies we inhabit, whether virtual or physical,” they concluded.
It seems people will psychologically inhabit almost any virtual body if the cues are strong.In recent research a team led by Mel Slater, a computer scientist at the University of Barcelona, induced what it calls body-transfer illusion — showing that men will mentally take on the body of a woman, for instance, if that’s the body it appears they’re walking around in virtually. The experience is especially powerful, Dr. Slater said, when the men feel a touch (on a shoulder, in a recent study) at the same time the avatar is touched. “You can see the possibilities already,” said Dr. Slater. “For example, you can put someone with a racial bias in the body of a person of another race.”
These kinds of findings have inspired a variety of simple experiments. Dropping a young man or woman into the virtual body of an elderly person does in fact increase sympathy for the other’s perspective, research suggests. “This is to me the most exciting thing about using virtual environments for behavior change,” Dr. Bailenson said. “It’s not only that you can create these versions of reality; it’s that you can cross boundaries — that you can take risks, break things, do things you could not or would not do in real life.”
Mini-Me in Action
In the virtual studio at the University of Quebec, patients wearing a headset can have a short conversation with a diminutive, attentive virtual therapist. Except for slight stature, it is a ringer for Dr. Bouchard: the same open face, the same smile, the same pelt of dark hair around a bald pate.
“Mini-Me, we call it,” Dr. Bouchard said.
The hologramlike figure seems at first to be minding its own business, looking around, biding time. Then it approaches slowly, introduces itself and kindly asks a question, like some digital-age Socrates: “What is the best experience you’ve ever had?”
For now, Mini-Me cannot do much more than cock its head at the answer and nod, before programmers begin to guide the conversation; the scientists are adding more language-recognition software, to extend interactions. Yet Mini-Me offers a glimpse of where virtual humans are headed: three-dimensional forms that can be designed to resemble people in the real world.
“You could scan in a picture of your mother or your boss or someone else significant and, with some voice recording samples, use a system that would automatically and quickly recreate a virtual facsimile of that person,” said Dr. Rizzo of U.S.C., where programmers have set up an Old West bar scene, complete with a life-size, autonomous virtual bartender, a waitress and a bad guy. “Then, perhaps, we’d be able to stage interactions that might closely resemble those in a patient’s life to help work through challenging issues.”
Anyone could rehearse the dance of social interaction, tripping without consequence, until the steps feel just about right.
“The great thing about it,” said Gary, the civil servant, referring to his own virtual therapy, “is that you can do anything you want and just see what happens. You get to practice.”
