Deepgram x Twitch Hackathon Submission

Twitch is being used more and more for live streaming. Whether it's gamers playing through video games, developer live coding, music streams, or someone cooking, everyone is using Twitch for live streams.

Something that was released a little while ago, is closed captions on Twitch live streams. Unfortunately the capability of this service is very limited. Closed captions must be enabled by the streamer, and isn't necessarily a straightforward process. What would be awesome is something built into the browser, ie. a browser extension or similar where users can enable closed captions, using Deepgram for any and all videos!

Introduction

According to the Census, in Australia alone, almost 12,000 people use Auslan to connect and communicate online on a daily basis. According to the World Health Organisation, about 430 million people have hearing loss. These people require translations or closed captions in order to communicate.

Many of these people (approx. 432 million) are adults who use the internet on a daily basis. As developers, we need to ensure they are able to use the tools we create effectively. There are hundreds of companies around the world helping people with various disabilities to use the internet and other technologies. This type of inclusivity is important if products and services are to cater to everyone around the world.

Thus, allowing for users to enable closed captions on any video content on any website would be a blessing to millions of people.

My Deepgram Use-Case

Many websites with video already have closed captions enabled. YouTube, TikTok, Twitter, and many others. For simplicity, and MVP, I'm focusing on Twitch. As a streamer myself I use Twitch on a daily basis. I would love for my audience to be able to participate in my streams no matter what disability they have. Not other this, but I want to make it available for all viewers.

Currently, there are several ways to already use closed captions. These include Twitch stream Closed Captioner extension, Closed Captions for Streams extension, as well as other third party applications like built in OBS plugins, Web Captioner, Pixel Chat, or PubNub.

Subtitles being shown off by developer Talk2megooseman. These captions are great because they support multiple languages, and can be moved around anywhere on the screen. Watch the full demo.

There's one key problem however with all the above products. All these methods need to be enabled by the broadcaster, and not the viewer. Thus the broadcaster must enable the capability to have closed captions on their stream as they go live. The viewer on the other hand has very limited capability to view subtitles. If a streamer doesn't have subtitles enabled, then unfortunately people won't be able to toggle subtitles. Thus, accessibility becomes a real problem for many people.

This is where Deepgram comes in. Deepgram allows for translation from live streaming audio in real time. By using this SDK, it shouldn't be too difficult to create a browser extension that allows users to enable subtitles on any audio coming from the browser.

Dive into Details

A browser extension that is able to be toggled on by viewers would go a long way to accessibility for the 432 million people with hearing impairments. It would also be useful for people who have multiple windows and tabs open and are working on something else. Many users like to have audio muted while they read subtitles. This is especially useful when users are in busy or noisy environments.

The browser extension removes the need for streamers and broadcasters to have closed captions on their streams. It opens up their content to more viewers, and increases the accessibility and diversity of their streams.

Deepgram already supports multiple languages too and thus allowing translation into various languages shouldn't be a problem.

I use Twitch as the example here, but it this was a browser extension or similar, closed captions could be enabled on anything with voice; Twitter Spaces (which is built in already but has issues), Twitch VODs, Facebook streams, TikTok, and more. YouTube already has it's own built in caption manager so this wouldn't really be on the roadmap.

Roadmap

The research phrase has already been complete and we could move onto building the extension!

Conclusion

I believe this extension would go a long way to helping millions of people. It would open the doors for people who are hearing impaired, and those users who prefer to read content. Streamers and content creators who be able to reach more people and would be reminded that their content is being viewed by people from all walks of life.