Live Driver (Face Tracking)

From Unity3D

Jump to: navigation, search

Contents

What is the Live Driver?

Please refer to Image Metrics for more information.

Image Metrics Live Driver™ Software Development Kit is the real-time facial analysis, expression tracking and animation technology, which is now also available for mobile app developers building on iOS. Developers can now easily integrate completely natural, intuitive and automated expression for the end user into everything from character animation, video chat, and user-created content to augmented reality applications, games and social networks.

Live Driver uses a webcam or the onboard camera found on iPads and iPhones to instantly capture, analyze and replicate the user’s facial expressions, gestures and emotions with instant calibration. The proprietary technology creates compelling new channels for companies looking to leverage the ubiquity of today’s laptops, smartphones and tablet cameras, along with the end user’s growing desire for personal expression.

Whether used as a Human computer Interface (HCI) or to control a virtual 2D or 3D character, Live Driver immerses the user in the environment allowing them to express themselves in new and innovative ways.

All taking place in real-time, supported by any consumer-grade camera, the experience is simple, efficient, and user-friendly.


Installing the Live Driver

All Live Driver need is a webcam. You don't need any extra setup unless you want to change stuff in LiveDriverNativeWrapper project. If so you need install the Live Driver SDK and setup correct library path for the Visual Studio project.

Integrating with Unity

The Face Tracking Wrapper contains three levels.

  • FaceTrackingWrapper.cs and other C# scripts. These scripts are what you need to use for using Live Driver in Unity.
  • LiveDriverNativeWrapper.dll. It is a C++ DLL that wraps up all Live Driver function in a way that can be accessed by C# scripts. You need to put this file in Plugins folder under your Assets folder.
  • LiveDriverSDK.dll and QuadProg.dll. They are DLLs provided by Live Driver. You need to put them inside your project folder (under the same path as you Assets folder).


IMPORTANT:

  • When you build your game, LiveDriverSDK.dll and QuadProg.dll should go with your exe file. Put them under the same path as your exe file. You don't need to worry about LiveDriverNativeWrapper.dll since Unity will do the work for you.
  • To run Live Driver you need a license file .lic and tracking model file .imbin. Put these files under StreamingAssets folder under your Assets folder. Unity will aumatically copy these files when you build your game.
  • LiveDriverSDK.dll, QuadProg.dll, and the .imbin file should be copied from the SDK installation folder. The .lic file will be provided by Image Metrics. See section "Obtaining a License to the SDK".


Bringing the Move.me SDK in to Unity

Install the Unity Package

The pre-built Unity package includes all of the scripts required to start running your world. To install, simply import the package, and you all of the required assets will appear in your project.

The following items are included within the package:

Scenes

  • FaceDemo - This scene gives you an example of how to start up Live Driver and access its data. It will show webcam image and tracking points for your face.

Scripts

  • FaceTrackingWrapper - This is the script acting as a main bridge between Unity and Live Driver, grabing webcam image from Unity, feeding to Live Driver and taking data back. It covers most of the common functions and data. It is written in singleton pattern.
  • FaceTrackingDemo - the script showing you how to connect to Live Driver and access to data. You can start your development based on this script.
  • WebcamAccessor - the script that actually gets webcam image.
  • DynamicTexture - the script to show webcam image.
  • LiveDriver folder - it contains a bunch of scripts used for Live Driver data structure. You may also need to access to LiveDriver.cs if you want more functions/data other than those got wrapped by FaceTrackingWrapper. NOTE: all scripts in this folder are under LiveDriverWrapper namespace.

Plugins

  • LiveDriverNativeWrapper - the native wrapper dll.

StreamingAssets

  • It should contain the license file and tracking model files.

Prefabs

  • TrackingPoint - just a small sphere represents tracking points.


Using the Scripts

FaceTrackingWrapper

This scripts wraps up both Live Driver and webcam. Basically this is the only script you need to talk to for Live Driver. All properties are read only.

Static Properties

  • Wrapper - a FaceTrackingWrapper. Use this property to get FaceTrackingWrapper. You CANNOT make FaceTrackingWrapper instance otherwise.

Properties

  • Driver - a LiveDriver. It gives you the access to LiveDriver script that actually do all the work. Use this property when you want to use functions/data not included in this wrapper.
  • WebcamImage - a Color32[]. Use the this property for assigning webcam image to texture, take shots, etc. null if the wrapper is not started.
  • WebcamImageWidth - a int. 0 if the wrapper is not started.
  • WebcamImageHeight - a int. 0 if the wrapper is not started.
  • TrackingPoints - a LiveDriverWrapper.Point[]. An array of all tracking points used by Live Driver. null if the wrapper is not started.
  • AnimationControls - a LiveDriverWrapper.AnimationControl[]. An array of all animation control values output by Live Driver. null if the wrapper is not started.
  • HeadPosition - a Vector3. (0,0,0) if the wrapper is not started.
  • HeadRotation - a Vector3. (0,0,0) if the wrapper is not started.
  • CharacterHeadPosition - a Vector3. (0,0,0) if the wrapper is not started.
  • CharacterHeadRotation - a Vector3. (0,0,0) if the wrapper is not started.
  • CameraPosition - a Vector3. (0,0,0) if the wrapper is not started.
  • CameraRotation - a Vector3. (0,0,0) if the wrapper is not started.
  • IsStarted - a bool represents whether the wrapper successfully started. Please check this value before you call other functions or access data.
  • IsCalibrating - a bool represents whether Live Driver is calibrating. Do NOT call Calibrate function when it is true. false if the wrapper is not started.
  • IsCalibrated - a bool represents whether Live Driver successfully finished calibration. false if the wrapper is not started.

Methods

  • bool StartFaceTracking(string licensePath, string modelPath) - licensePath the path of lisence file, modelPath the path of the tracking model file. Start Live Driver and webcam. Return true if success, false otherwise (Live Driver fail/cannot find webcam).
  • void void Calibrate() - Calibrate Live Driver. You need to successfully calibrate at least once to be able to use Live Driver data. Use IsCalibrated and IsCalibrating to determine whether to call calibrate.
  • void UpdateFaceTracking() - Update one frame for Live Driver. It will convert webcam image and feed to Live Driver. Normally you want to call it in Update() function. NOTE: since Live Driver is asynchronous, the data you get this frame may not be the data analyzed from last frame's image.
  • void StopFaceTracking() - Stop Live Driver and webcam. Normally you call it before your application ends, in OnApplicationQuit() for example.


FaceTrackingDemo

Here shows you how to run the demo correctly.

Fields

  • pointPrefab - a GameObject. The prefab to generate spheres to mark tracking points. Drag in TrackingPoint prefab in Prefabs folder.
  • licensePath - a string. Input the name/path of the license file. The file should be in StreamingAssets folder.
  • modelPath - a string. Input the name/path of the tracking model file. The file should be in StreamingAssets folder.
  • failThreshold - a float from 0 - 1. Set the Live Driver fail threshold. Higher value will have higher quality data, but less smooth. The default value for Live Driver is 0.5, for FaceTrackingDemo is 0.4.
  • canvasTex - a DynamicTexture. Used for display webcam image. Drag in a game object with renderer and DynamicTexture script.


WebcamAccessor

This script shows how to access webcam in Unity. You may use this script for any other webcam application.

Properties

  • WebcamPixels32 - a Color32[]. null if the webcam is not started.
  • Width - a int. Default value is 640.
  • Height - a int. Default value is 480.

Methods

  • bool StartWebcam() - the same as StartWebcam(0).
  • bool StartWebcam(int num) - the same as StartWebcam(num, 640, 480, 30).
  • bool StartWebcam(int num, int width, int height, int fps)' - num, the number of camera you want to start, 0 is the first camera found by WebCamTexture.devices. width, image width. height, image height. fps, requested fps.
  • void StopWebcam() - Stop webcam.


LiveDriver scripts

Please refer to LiveDriver SDK documentations. Most of the functions/properties can be considered as a C# version of LiveDriver functions. Some functions may be missing in C# scripts (Callback functions for exmaple), that is because Unity does not support everything. But it doesn't not affect normal use of Live Driver.


Further Discussion

Compromise and TODOs

  • Unity does not support C++/CLI library, so we did a C style native wrapper dll to wrap up LiveDriver.
  • Unity is not thread safe, it can only accept callbacks from its own thread. Callback from LiveDriver(another thread) will cause Unity crash. So we have to drop callback functions.
  • The structure is like: C# script -> wrapper DLL -> SDK DLL. Unity can only pick up the first level DLL(wrapper DLL in this case) in its Plugins folder. That's why we need to put SDK DLLs in project folder/with exe file.
  • TODO: Live Driver gives a tutorial about how to make appropriate rigging for 3D models, so it can be animated by AnimationControls data. Since I'm not modeler and don't have a model to test on. I'll leave this for now, please fill in this part if you find out anything.

Contact

If you have any question or if you are interested in further development, please contact [Xun Zhang].


Obtaining a License to the SDK

Please go to http://www.image-metrics.com/livedriver/evaluation/ for an evalution or academic version of the software!
Please be aware that the evaluation license does not allow you to share any part of the SDK with other people so you will need to acquire the SDK and license (.lic file) from Image Metrics to use Live Driver with Unity3D.


Package and Example Project

Note: The following files do not contain Live Driver SDK, please follow session Obtaining a License to the SDK to get required files.
Face Tracking Package for Unity3D
Live Driver Native Wrapper Project for Visual Studio

Personal tools