Architecture

Virtual Human Toolkit is a research tool developed in the Unity game engine.

VH Toolkit is developed with the Rapid Integration & Development Environment (RIDE). RIDE is a research and development (R&D) middleware platform that combines government, industry, and academic technologies to provide a rapid prototyping sandbox for real-time applications. RIDE is developed at USC ICT, in collaboration with its partners, sponsored by the U.S. Army and Navy.

RIDE follows a three-layered architecture, see Figure 1. The foundation is the Engine Layer, with a current focus on the Unity game engine. This provides core capabilities, including rendering, user interface (UI), and audio. The Middleware Layer adds common prototyping functionalities (e.g., networking and cloud interfaces) and VH-specific technologies, including speech recognition, nonverbal behavior generation, and text-to-speech generation. The VH technologies are based on the Virtual Human Toolkit, a collection of modules and tools that supports creating VHs. Individual projects are developed in the Project Layer, leveraging the RIDE API. This layer can act as an incubation area where proven technology can be incorporated back into the Middleware Layer, benefiting all RIDE users.

(TODO - insert architecture image here)

VH Toolkit allows interactions between human and virtual characters using audio-visual sensing, automated speech recognition (ASR), natural language processing (NLP), nonverbal behavior generation (NVB), and text-to-speech generation (TTS). Most of these use 3rd party cloud services, called through RIDE’s Common Web Services System (CWSS) that streamlines integrating and using RESTful services. The CWSS is implemented in C# and handles common authorization requirements, constructing calls, awaiting the response, and relegating the results to a callback function to be processed further. Having these core functionalities in a dedicated system provides several advantages, including 1) once a technology vendor is supported (e.g., AWS) integrating new services from the same vendor require less effort, 2) similarly, vendor support can be more easily shared among researchers and developers, 3) once a particular type of service has been implemented (e.g., natural language processing) alternative services from different vendors can be integrated with less effort, and 4) it simplifies creating service calls and processing the results.

To learn more about RIDE, see https://ride.ict.usc.edu/getting-started/quickstart/