Army Avatars Push the Envelope on Conversational Technologies

Defense Systems published an article last week about conversational avatars used by the U.S. Army. The avatars are the result of a partnership between the Army Research Laboratory (ARL) and the University of Southern California’s Institute for Creative Technologies (ICT). In the fall of 2013 I did an interview with Arno Hartholt of ICT and published the results in a post called Virtual Humans to the Rescue.

ICT SimSenseiThe Defense Systems article describes several ARL/ICT-built avatars and their specific use cases. Ellie is a virtual human that can pick up on facial and other cues to detect the emotion of the person she’s speaking with. ICT calls the mood-sensing technology used by Ellie SimSensei and you can see a demo in this video. Ellie has been effectively employed to converse with war veterans and detect markers that could signal Post Traumatic Stress Syndrome (PTSD) or depression.

The Army’s continued partnership with ICT points to a goal of expanding the use of immersive training. Conversational embodied virtual agents seem to be a significant part of that work. Two other use cases covered in the article are the Virtual Standard Patient and the Emergent Leader Immersive Training Environment. The virtual patient helps medical students hone their interviewing and diagnostic skills, while the Emergent Leader avatar works with junior leaders to improve their communication skills using role-playing exercises.

The Army has been employing and developing these avatars for several years now and continues to partner with ICT, so the technology must be having some positive results. The Defense Systems article notes that DARPA is also exploring ways to improve a computer’s ability to carry on human-like conversation. DARPA announced the Communicating with Computers project back in February of this year. As yet, I haven’t seen any news about the winners of DARPA’s competitive announcement.

Conversational technologies will undoubtedly be a key area of intelligent assistants going forward, and the ARL’s partnership with ICT is helping to forge new ground in that arena.

Virtual Humans to the Rescue

Ada and GraceForget about the Singularity for a moment. While we may be able to map our brains into software one day or benefit from machines that outsmart us, those are dreams for the future. While we’re at it, forget about humanoid robots or clever C-3PO clones. In the here and now, a technology already exists that can give us human-like beings capable of helping us in amazing ways. Virtual humans are digitally rendered human actors that have already proven their potential. The only thing holding virtual humans back is our ability to create them and learn how to leverage their possibilities.

That’s where Arno Hartholt and the team at the University of Southern California’s Institute for Creative Technologies (ICT) come in. After speaking with Hartholt recently on the phone, I sensed that he and the ICT team are on a mission to get virtual humans into the mainstream where they can work their magic. The ICT has been working on virtual humans for a decade and they’ve had great success. Sergeant Star is a virtual human that’s used by the Army to answer questions from prospective recruits. Ada and Grace are engaging twins who interact with children at Boston’s Museum of Science. Other ICT virtual humans help soldiers prepare for situations in battle, train health workers to interact effectively with patients, and teach battlefield arbitrators the skills of negotiation. But these first generations of virtual humans barely scratch the surface of the possibilities.

Hartholt indicated that a big hurdle to the progress of virtual humans is their inherent complexity. Not many people have the expertise or resources to build a complete, well-rounded virtual human. An effective virtual human has to be able to understand language and meaning, speak intelligently, perceive and react to other people, and exhibit realistic emotions and movements. To create all those capabilities from scratch would require a huge investment in time and resources.

So what did the ICT decide to do? Over the past decade, they’ve been working on a Virtual Human Toolkit. The toolkit contains all the components that a builder of virtual humans might need to construct their own fully capable character. Since users of the toolkit are able to leverage these pre-built pieces, they can focus on the personality and functions of the virtual human, instead of sweating the technical details. In our conversation, Hartholt stated the importance of including people from a broad range of disciplines and interests in the development of virtual humans. The toolkit can enable psychologists, educators, medical professionals and others to develop and experiment with virtual humans. The more people and perspectives applied to discovering the potential benefits of virtual humans, the better.

Virtual Human ToolkitOne question I pondered with Hartholt was: what makes a virtual human work? The current generation of intelligent digital assistants like Apple’s Siri, Google Now and others offer a lot of useful features. Yet while they can provide helpful answers, they aren’t truly engaging personalities. Few people develop a real connection with these assistants. So why would someone be more drawn in by a virtual human? Hartholt postulates that part of the deficiency of current mobile personal assistants is simply their lack of a physical presence. They are disembodied voices. These assistants also don’t have any real sense of who we are or how we might be feeling when we push the button that activates their digital ears. They can’t express empathy.

A convincing virtual human needs a general awareness of the person it’s interacting with. You’re not going to spend a lot of time with a pretend human if it doesn’t acknowledge any cues you give it. The ICT has developed technology that it calls SimSensei that can detect facial expressions and body language in real time. When the technology is paired with a virtual interviewer, it results in a virtual character who can engage effectively with a real person. The SimSensei character can respond convincingly to the interviewee’s smiles, body language, and voice tones, for example. The virtual interviewer can even be trained to detect stress in the subject and adjust its responses accordingly.

Now that we live in a connected world, we could surround ourselves with smart, helpful virtual humans. Imagine a world where children had a set of virtual teachers or smart virtual playmates available to interact with them whenever they were bored or had a question. They could ask as many questions as they wanted and their virtual human would never get frustrated or tell them to go watch TV. They could even turn to their virtual human for empathy and encouragement, in the sad event that they weren’t able to find these critical acts of reinforcement from the real humans in their lives. Imagine a world where an ill person could have his personal virtual human contact his doctor and send in all his relevant vital statistics and either get a prescription or make an appointment. What about a companion virtual human for a senior living alone, or a virtual human coach for someone learning a new language, or a virtual human tutor for a child struggling with math?

The possibilities for virtual humans are as limitless as their potential benefits. Now all we need to do is get busy building them.