A-Frame hand-tracking on the HoloLens

On the HoloLens you only have the UWP version of Edge out of the box. This doesn't have the newer WebXR device API implemented. The WebXR device API can even handle hand tracking. But, Edge UWP doesn't, this only has the old WebVR API. I already noticed you could use the Gamepad API to get an Air tap event and handle that as a click, but I wanted needed to get the position of the hands.

It took me a while to get this figured out this one. It turned out to be a bit easier than I thought. I thought I had to track the controls myself through the gamepad API, but it turned out A-Frame has us covered.

There's a component that is the basis for a lot of other components that handle tracking on various devices, for example, the Oculus touch controls or the Windows motions controls. If you know what ID is used in a specific situation you can just handle the rest yourself.

I created a very small test application and used Vorlon as I described in a previous post to get the information. I was implementing the whole thing myself at first, but later realized I could use the Tracked-Controls component that's in A-Frame. Below is a small snippet of HTML from that test app that shows how to use the left and right hand as a controller. The controller numbers 4 and 5 are actually the indices of the gamepads returned by the GamePad API. For the idPrefix I used Hand, which is short for 'Hand (Spatial Interaction Source)', the full id. I used the default low poly hand models used by A-Frame, but had to tweak the rotation and position a little bit to make them overlay the real hands a bit better.

<a-entity tracked-controls="controller: 4;hand:left; idPrefix: Hand">
    <a-entity id="hand" 
            rotation="-60 0 60" 
            position ="0 -.03 -.02 " 
            gltf-model="src:url(https://cdn.aframe.io/controllers/hands/leftHandLow.glb)">
        </a-entity>
</a-entity>
<a-entity tracked-controls="controller: 5;hand:right; idPrefix: Hand">
    <a-entity id="hand" 
            rotation="-60 0 -60" 
            position ="0 -.03 -.02 " 
            gltf-model="src:url(https://cdn.aframe.io/controllers/hands/rightHandLow.glb)">
        </a-entity>
</a-entity>

The only thing that this does not do is track the individual fingers. That's something you need the newer WebXR device API for, but this is better than nothing. I hope, one day, Edge on the HoloLens will support the full WebXR device API.

Next, I need to figure out a way of creating interaction between hands and objects in 3D space to be able to press buttons or touch objects. More on that in a future post.