AudioSign: Converting audio to animated sign language (P2)

In the last post, we talked about the idea of taking deepgram and using it to implement a speech-to-animated sign language system.

💭 My Deepgram usage

Deepgram is a Speech-To-Text platform that leverages deep learning to transcribe text from voice data.

They make use of end to end deep learning, which entails the system(model) trying to figure out the transcript from the audio without having to break it down into parts like phonemes, words and such. I believe the main reason for this is to make learning new tasks easier and allows for more robustness in the system.

Deepgram has very low latency and high accuracy when using the live/recorded audio transcription services and as such is a reliable choice. So, what I had in mind was to go a step further with the transcriptions and use that to animate a character signing that data.

Now the question is how to transfer the transcription system into one for the signing of languages.

Signing differs from the spoken word as it is not a direct translation, it deals more with concepts than words. It uses other details like tone, visual cues, facial expressions to make good translations. Developing a system to do all this would be a worthwhile undertaking as it would help the 70+ million people that sign have more impactful interactions.

The applications of such a system include the following:

• Live signing of videos on platforms for streaming, news and other forms of media to make them more inclusive.
• Signing of pre-existing media, archives, history and other forms of information that was previously only accessible to those who weren't hearing challenged.
• An advancement in holographic technology could also be used in conjunction with the system such that when an individual is speaking, a real-time 3d rendering of the signing is taking place.

🛠 Possible ways we can achieve this

In the next and final post on the innovation idea track, I'll be writing about how we could achieve this with current deep learning techniques and technologies.

Stay tuned for more and have a great day 👋