Hand tracking in HoloLens2 is awesome but detecting gestures is not. Currently, HoloLens allows the following in-built hand gestures,

HoloLens 1

HoloLens 2

Hand Tracking

If no controllers are present then the default controllers are the user's hands.

• The position of each joint on the user's hands can be obtained by the following one-line statement. This example will give the pose of the right index tip joint. More info about this here.

MixedRealityPose pose;
if (HandJointUtils.TryGetJointPose(TrackedHandJoint.IndexTip, Handedness.Right, out pose))
{
	// Use pose object to access the position and orientation of IndexTip
}

Gesture Detection

HoloLens uses *Pointers* to focus on different GameObjects.

• By default, the air tap is registered only when a user has focused the pointer at a GameObject. Again, there are many ways to create pointers but there is a pointer present by default for each of the user's hands. Here is an example,

• If you want to register (detect) Pointer events irrespective of whether the user has focused on a GameObject or not, you need to register Pointer events as Global Events. This can be easily done using a single statement like so (see the Awake function only),

public class FiringProjectiles: Singleton<FiringProjectiles>, IMixedRealityPointerHandler
{
	void Awake()
  {
      CoreServices.InputSystem?.RegisterHandler<IMixedRealityPointerHandler>(this);
  }
	
	void IMixedRealityPointerHandler.OnPointerUp(MixedRealityPointerEventData eventData)
  {
      // Requirement for implementing the interface
  }

  void IMixedRealityPointerHandler.OnPointerDown(
       MixedRealityPointerEventData eventData)
  {
      // Requirement for implementing the interface
  }

  void IMixedRealityPointerHandler.OnPointerDragged(
       MixedRealityPointerEventData eventData)
  {
      // Requirement for implementing the interface
  }

  // Detecting the air tap gesture
  void IMixedRealityPointerHandler.OnPointerClicked(MixedRealityPointerEventData eventData)
	{
			if (eventData.InputSource.SourceName == 'Right Hand' || eventData.InputSource.SourceName == 'Mixed Reality Controller Right')
			{
					// Do something when the user does an air tap using their right hand only
			}
	}
}

• We can access the controller that led to that specific event using the MixedRealityPointerEventData object as shown above in the OnPointerClicked function. Note that by default the Air Tap gesture is a pointer gesture and hence this mapping is possible. Unity automatically maps the detected controller controls to the in-built gestures (unless specified otherwise).

• We can detect the controller type using the SourceName attribute of the InputSource object as shown above. The reason we check the source name twice is the following:

  1. Name of right hand controller in Unity: Right Hand
  2. Name of right hand controller in HoloLens: Mixed Reality Controller Right
  3. Name of left hand controller in Unity: Left Hand
  4. Name of left hand controller in HoloLens: Mixed Reality Controller Left
  • So, if we don't check the name twice the gestures might get detected in Unity but not in HoloLens (or vice-versa).