Building a Video Chat App, Part 2 - Accessing Cameras

On my Twitch channel we’re continuing to build our video chat application on Azure Communication Services (ACS).

For today’s post, we’re going to look at the next major milestone, accessing your camera and microphone.

How Browsers Access Devices

We're going to use the ACS SDK to do this, but before we get there let's first understand how we access cameras and microphones in the browser. Browsers have had this functionality for a while now, it came about as a need for the WebRTC specification, since that allows you to do what we're doing, run a video stream through the browser, and it works using the navigator.mediaDevices API which replaced navigator.getUserMedia.

This API is promised based, so it works nicely with async/await, and will return us the MediaStream available to the browser.

There is a catch though, the user has to consent to providing access to the devices, which makes sense as you don't want any random website to be able to access your camera and mic without you knowing about it, do you? The user will see a prompt like so:

In "raw JavaScript" we'd write something like this:

navigator.mediaDevices
    .getUserMedia({ audio: true, video: true })
    .then(function(stream) {
        /* use the stream */
    })
    .catch(function(err) {
        /* handle the error */
    });

If the user denies the request then the catch of the promise is triggered (or if they've previously denied it), otherwise you'll end up in the MediaStream for the camera/mic they have selected. The MediaStream can be provided to a <video> element and you can look at yourself.

Accessing Devices with ACS

Now that we understand the fundamentals, let's look at how we use this in the ACS SDK to get one step closer to establishing out video call.

We'll need to add some npm packages to our UI:

npm install --save @azure/communication-calling @azure/communication-common

With these packages, we're going to need four APIs, AzureCommunicationUserCredential, CallClient, CallAgent and DeviceManager.

To make the important parts of this available throughout our application, we're going to create a React Context to hold it, so let's get started with that.

Defining Our Context

Let's create a file called useCallingContext.tsx since we'll have the context in there as well as a hook to access context, and define our context:

import {
    AudioDeviceInfo,
    CallAgent,
    CallClient,
    DeviceManager,
    VideoDeviceInfo
} from "@azure/communication-calling";
import { AzureCommunicationUserCredential } from "@azure/communication-common";
import React, { useState, useEffect, useContext } from "react";
import useToken from "./useToken";

export type CallingProps = {
    micList?: AudioDeviceInfo[];
    cameraList?: VideoDeviceInfo[];
    callAgent?: CallAgent;
    deviceManager?: DeviceManager;
};

const CallingContext = React.createContext<CallingProps>({});

The context will have available on it the list of cameras and mics, along with the CallAgent and DeviceManager instances since they will be useful later.

Since the logic to setup all the data available on the context only happens once, we'll implement the context provider within this file to, so let's do that.

export const CallingContextProvider = (props: {
    children: React.ReactNode;
}) => {
    return (
        <CallingContext.Provider value={/* todo */}>
            {props.children}
        </CallingContext.Provider>
    );
};

Lastly, we'll expose a hook to make it easy to access the context elsewhere in the application:

export const useCallingContext = () => useContext(CallingContext);

Great, we're now ready to implement the context provider.

Implementing the Context Provider

The context provider here is key, as it's the thing that'll be responsible for getting the devices and making them available elsewhere in our application, and for that we're going to need some local state.

export const CallingContextProvider = (props: {
  children: React.ReactNode;
}) => {
  const token = useToken();
  const [, setClient] = useState<CallClient>();
  const [callAgent, setCallAgent] = useState<CallAgent>();
  const [deviceManager, setDeviceManager] = useState<DeviceManager>();
  const [cameraList, setCameraList] = useState<VideoDeviceInfo[]>();
  const [micList, setMicList] = useState<AudioDeviceInfo[]>();

We're going to need the token that is generated for the user in }}">Part 1, and we're doing that through a custom hook:

import { useState, useEffect } from "react";

export type TokenResponse = {
    token: string;
    expiresOn: Date;
    communicationUserId: string;
};

const useToken = () => {
    const [token, setToken] = useState("");

    useEffect(() => {
        const run = async () => {
            const res = await fetch("/api/issueToken");
            const tokenResponse: TokenResponse = await res.json();

            setToken(tokenResponse.token);
        };

        run();
    }, []);

    return token;
};

export default useToken;

Then we've got some more state for the different parts of the ACS SDK that we're going to expose, except for the CallClient which we only need to establish the other parts of the API.

We'll use an effect hook to set this up, that'll be triggered when the token is available to us:

useEffect(() => {
    const run = async (callClient: CallClient, token: string) => {
        const tokenCredential = new AzureCommunicationUserCredential(token);
        let callAgent: CallAgent | undefined = undefined;
        try {
            callAgent = await callClient.createCallAgent(tokenCredential);
            const deviceManager = await callClient.getDeviceManager();
            const result = await deviceManager.askDevicePermission(true, true);

            if (result.audio) {
                setMicList(deviceManager.getMicrophoneList());
            }

            if (result.video) {
                setCameraList(deviceManager.getCameraList());
            }

            setCallAgent(callAgent);
            setDeviceManager(deviceManager);
        } catch {
            if (callAgent) {
                callAgent.dispose();
            }
        }
    };

    if (token) {
        const callClient = new CallClient();
        setClient(callClient);
        run(callClient, token);
    }
}, [token]);

Ok, that's a lot of code, let's break it down piece by piece, starting at the bottom:

if (token) {
    const callClient = new CallClient();
    setClient(callClient);
    run(callClient, token);
}

This is a check to make sure that the user token has been issued, and once it has been we're going to call an async function (run), because an effect hook can't take an async function directly, and the run function is really where things happen.

First off, this function is going to create the credentials for ACS from the token provided:

const tokenCredential = new AzureCommunicationUserCredential(token);

Next, we'll setup a try/catch block to access the devices, and remember that the reason we'd do it this way is so that if the user declines the request to access devices, we can gracefully handle the error (the async/await unwraps a promises catch into the catch of the try/catch block).

We'll create the callAgent using the credentials:

callAgent = await callClient.createCallAgent(tokenCredential);

We're not actually using the callAgent yet, it's what we use to connect to calls, but we need to create an instance of it before we access the DeviceManager. I'm unclear as to why it's this way, and it's something I'm going to raise with the ACS team.

With our callAgent created, it's now time to access the DeviceManager, which will give us all the devices:

const deviceManager = await callClient.getDeviceManager();
const result = await deviceManager.askDevicePermission(true, true);

if (result.audio) {
    setMicList(deviceManager.getMicrophoneList());
}

if (result.video) {
    setCameraList(deviceManager.getCameraList());
}

From the deviceManager, which we get from callClient.getDeviceManager, we need to request permissions from the user to access their device list using askDevicePermissions. This method takes two arguments, whether you want audio and video access, and for our case we do. Assuming the user grants permissions, we can then use deviceManager.getMicrophoneList and deviceManager.getCameraList to get arrays of AudioDeviceInfo and VideoDeviceInfo that we can present to the user for their selection.

This is the same as if you were to call the enumerateDevices method from MediaDevices, but the SDK takes the liberty of splitting the enumerated devices into their appropriate types. What's important to know about this is that you must call askDevicePermissions first, otherwise you'll get an array with a single unknown device. That's because enumerateDevices, which is what's used internally by the SDK, accesses the available devices without prompting for consent and if consent hasn't been provided, you can't get the devices.

Conclusion

Our React context is all ready for integration into the application. We've learnt how to get started using the ACS SDK and its DeviceManager to request permission for the devices and then display the full list of them.

If you want to catch up on the whole episode, as well as look at how we integrate this into the overall React application, you can catch the recording on YouTube, along with the full playlist