Tetra AR Projector

Machine Learning (YOLO) | AR | Unity Development | Experience Design

As part of a team, we custom-designed an Augmented Reality (AR) projector as well as a framework for non-programmers that allow them to build AR experiences. As a proof of concept, we built an educational interactive magazine and projection mapping using our framework.

Tetra is a user-friendly tool for building AR experiences. It features intuitive interaction using gesture recognition and the AR Creator ARc. ARc is an interface that allows you to develop your own AR and projection mapping software with visual scripting and object identification.

Motivation:

Interactions with students in the design department revealed a strong resistance from my colleagues to adopt new technologies to complement their projects. One of the main barriers to incorporate technology in design is the need for coding skills. Through research, we found that projection mapping as well as creating custom AR applications is expensive, difficult and non-standardised. A tool with modular components that would allow an interactive and standardised way of developing applications is lacking, and thus would be a new addition to the local and global markets.

HARDWARE

CAMERA: INTEL REALSENSE D415

PROJECTOR: WOWOTO H9

COMPUTER: ANY LINUUX COMPUTER

Screen Shot 2021-10-26 at 1.18.01 PM.png

Object Identification:

The object identification system is using ARToolkit SDK with Unity to extract the image features allowing for accurate object detection

Screen Shot 2021-10-26 at 1.18.08 PM.png

Gesture Recognition:

The gesture system is responsible for the detection of the user’s hand gestures and interaction with the camera. The system relies on the state-of-the-art machine learning algorithm YOLO (You Only Look Once) and computer vision components for efficient recognition of multiple hand gestures.

Screen Shot 2021-10-26 at 1.18.26 PM.png

AR PROJECTIONS CONTROL:

Projections can be controlled by simple hand gestures captured by the camera. Here is an example of an application: By swiping [left or right] the user can rotate the 3D projected model 360 degrees.

VISUAL PROGRAMMING:

By connecting simple blocks, users can create a complete AR application. As an example, this one displays a 3D model

Allows users to create dynamic Augmented Reality objects using Unity such as images, 3D models, text, videos, etc... The system interacts with the gesture recognition module and allows the user to create or edit a certain AR object in a scene using specific gestures. The AR creator also interacts with the object identification system to augment some of recognized objects.

Projection Mapping:

Users can create projection mapping masks and edit these masks via anchor points or nodes.