Virtual Humans to the Rescue

Ada and GraceForget about the Singularity for a moment. While we may be able to map our brains into software one day or benefit from machines that outsmart us, those are dreams for the future. While we’re at it, forget about humanoid robots or clever C-3PO clones. In the here and now, a technology already exists that can give us human-like beings capable of helping us in amazing ways. Virtual humans are digitally rendered human actors that have already proven their potential. The only thing holding virtual humans back is our ability to create them and learn how to leverage their possibilities.

That’s where Arno Hartholt and the team at the University of Southern California’s Institute for Creative Technologies (ICT) come in. After speaking with Hartholt recently on the phone, I sensed that he and the ICT team are on a mission to get virtual humans into the mainstream where they can work their magic. The ICT has been working on virtual humans for a decade and they’ve had great success. Sergeant Star is a virtual human that’s used by the Army to answer questions from prospective recruits. Ada and Grace are engaging twins who interact with children at Boston’s Museum of Science. Other ICT virtual humans help soldiers prepare for situations in battle, train health workers to interact effectively with patients, and teach battlefield arbitrators the skills of negotiation. But these first generations of virtual humans barely scratch the surface of the possibilities.

Hartholt indicated that a big hurdle to the progress of virtual humans is their inherent complexity. Not many people have the expertise or resources to build a complete, well-rounded virtual human. An effective virtual human has to be able to understand language and meaning, speak intelligently, perceive and react to other people, and exhibit realistic emotions and movements. To create all those capabilities from scratch would require a huge investment in time and resources.

So what did the ICT decide to do? Over the past decade, they’ve been working on a Virtual Human Toolkit. The toolkit contains all the components that a builder of virtual humans might need to construct their own fully capable character. Since users of the toolkit are able to leverage these pre-built pieces, they can focus on the personality and functions of the virtual human, instead of sweating the technical details. In our conversation, Hartholt stated the importance of including people from a broad range of disciplines and interests in the development of virtual humans. The toolkit can enable psychologists, educators, medical professionals and others to develop and experiment with virtual humans. The more people and perspectives applied to discovering the potential benefits of virtual humans, the better.

Virtual Human ToolkitOne question I pondered with Hartholt was: what makes a virtual human work? The current generation of intelligent digital assistants like Apple’s Siri, Google Now and others offer a lot of useful features. Yet while they can provide helpful answers, they aren’t truly engaging personalities. Few people develop a real connection with these assistants. So why would someone be more drawn in by a virtual human? Hartholt postulates that part of the deficiency of current mobile personal assistants is simply their lack of a physical presence. They are disembodied voices. These assistants also don’t have any real sense of who we are or how we might be feeling when we push the button that activates their digital ears. They can’t express empathy.

A convincing virtual human needs a general awareness of the person it’s interacting with. You’re not going to spend a lot of time with a pretend human if it doesn’t acknowledge any cues you give it. The ICT has developed technology that it calls SimSensei that can detect facial expressions and body language in real time. When the technology is paired with a virtual interviewer, it results in a virtual character who can engage effectively with a real person. The SimSensei character can respond convincingly to the interviewee’s smiles, body language, and voice tones, for example. The virtual interviewer can even be trained to detect stress in the subject and adjust its responses accordingly.

Now that we live in a connected world, we could surround ourselves with smart, helpful virtual humans. Imagine a world where children had a set of virtual teachers or smart virtual playmates available to interact with them whenever they were bored or had a question. They could ask as many questions as they wanted and their virtual human would never get frustrated or tell them to go watch TV. They could even turn to their virtual human for empathy and encouragement, in the sad event that they weren’t able to find these critical acts of reinforcement from the real humans in their lives. Imagine a world where an ill person could have his personal virtual human contact his doctor and send in all his relevant vital statistics and either get a prescription or make an appointment. What about a companion virtual human for a senior living alone, or a virtual human coach for someone learning a new language, or a virtual human tutor for a child struggling with math?

The possibilities for virtual humans are as limitless as their potential benefits. Now all we need to do is get busy building them.

Virtual Human Toolkit – A Treasure Trove for Virtual Agent Developers

Virtual Human ToolkitThe University of Southern California’s Institute for Creative Technologies offers a Virtual Human Toolkit for constructing animated conversational characters. I ran across the Virtual Human Toolkit while browsing through the official proceedings from the 13th International Conference, IVA 2013, Edinburgh, UK, August 29-31, 2013. A team from the USC Institute for Creative Technologies wrote a paper titled “All Together Now: Introducing the Virtual Human Toolkit” that was presented at  IVA 2013.

The goal of the Virtual Human Toolkit is to provide a suite of ready-made components that developers can use to more quickly build well-rounded virtual agents. Virtual agent characters can have many benefits, but they are comprised of numerous complex technical components. Most teams don’t have access to all the knowledge and skills needed to build virtual characters with a broad range of capabilities. A versatile virtual human would ideally be able to simulate human behavior, perceive and adequately react to the behavior of others, and respond appropriately to questions or statements. Virtual humans are costly to develop, so the toolkit from USC’s Institute for Creative Technologies should be a great help to small teams looking to experiment with the technology.

Based on the documentation available, the virtual human toolkit currently consists of the following components:

  • Speech Recognition
  • Natural Language Understanding
  • Nonverbal Behavior Understanding
  • Natural Language Generation
  • Nonverbal Behavior Generation

These capabilities are embedded within individual modules that are all connected via an underlying messaging platform. A core module is called Multisense. This module enables the virtual human to track and interpret the non-verbal behavior of its human conversational partner. The virtual human can track facial expressions and body language using various input devices and then analyze the input to make generalizations about the human conversational partner’s emotional state.

The NPCEditor module understands incoming dialog and then determines an appropriate response. Currently the Virtual Human Toolkit uses chatbot-like pattern matching technology to engage in dialog. The editor does appear to have the ability to use statistical models to find the best perceived response if it encounters an utterance that doesn’t match, so this capability would put it ahead of basic pattern matching scripts.

The NonVerbal Behavior Generator helps the Virtual Human plan out its nonverbal responses, which can consist of things like nodding, gesturing with arms and hands, and so on. Other components work to synchronize behaviors associated with conversational speech, which include speech, gaze, gesturing and head movements.

In their IVA 2013 article, the Institute for Creative Technologies team suggests a number of practical applications for the Virtual Human Toolkit. Among the types of uses for the technology are: Question-Answering Characters, Virtual Listeners, Virtual Interviewers,  and Virtual Role-Players.

The toolkit is available free of charge for the academic research community and for U.S. Government uses. There’s an email address to use if you’d like to contact the team about using the toolkit for commercial purposes.