Interactive Content Using Gesture Capture System

Let's Nyanbo!

This digital interactive content lets visitors recreate iconic poses from Space Brothers. When you mimic a character's pose from a memorable scene, the matching character and speech bubble appear, allowing you to take a commemorative photo. It uses a custom-developed “Gesture Capture System” powered by bone-tracking technology. The experience was featured at the Space Brothers Exhibition #3 held at TeNQ from November 2021 to February 2022.

Space Brothers Exhibition #3 at TeNQ

Venue: Space Museum TeNQ

Produced by: Toei Agency

© Chuya Koyama / Kodansha

Issue

An immersive experience driven by active user engagement.

Interactive content that's easy to try invites users to become the characters themselves, offering a deeper sense of immersion and a fresh perspective.

Creation

Ten signature poses

The selection includes quirky poses like “ Nyanbo ,” “Pow!,” and “It’s an alien!!,” as well as memorable moments such as “That pose,” “ Hibito, Safe Return ,” and “ Yeeaaahhh !!” Each pose is easy to mimic, allowing visitors to fully step into the character's role.

Technology

Gesture Capture System
The Gesture Capture System is a gesture-based experience that uses motion-tracking cameras to display visuals in sync with the user’s body movements, enabling commemorative photos with digital content. It employs the Azure Kinect DK—a 3D camera with depth-sensing and spatial recognition capabilities—to accurately track body poses and motion. When a user’s pose matches a pre-defined gesture, the system overlays illustrations and visual elements on screen, allowing for immersive photo opportunities. Captured images are uploaded to the cloud in real time, and a QR code is instantly generated, making it easy to download or share photos on social media.

Future

Flexible customization options

Beyond driving event attendance and media exposure, this system is highly adaptable—by changing the paired content or poses, it can be tailored to suit a wide range of themes and scenarios. Its ease of use makes it accessible to all age groups, making it ideal for deployment in event venues, tourist attractions, educational institutions, or even reception areas. It offers an interactive experience that actively involves visitors.
A notable example is “Capture Dinosaur,” a commemorative photo experience featured at the *DINOVENTURE – Forest of Light and Mysterious Eggs* event. In collaboration with German dinosaur figure brand Schleich, the system displayed animated dinosaur visuals that responded to visitors' poses inspired by the figures.