Overview of My Submission

Anime OAV (Anime or any other video) is a chrome extension that automatically translate, transcribes, and generates overlaying subtitles for videos.

const translator = 
TranslatorFactory.createTranslator(Translators.AZURE);
...
const translator = 
TranslatorFactory.createTranslator(Translators.GOOGLE);

Tech Stack

• TailwindCSS - Ui style
• Chrome extension
• React.js - Front-end library
• Deepgram - Transcriber
• Azure / Google - Translator
• Webpack - Code bundler

Submission Category:

Accessibility Advocates

Why

Sometimes my favorite or interesting shows are produced outside my native language and my only options are to wait for an official or fan-made translation.

I wondered what if I had a way to automatically translate videos outside my native language so I can enjoy them.

Translation isn't the only problem I've had. Sometimes audio can be hard to understand, and I think, if only I had a way to transcribe what everyone is saying.

My extension solves both of the issues above and increases awareness of issues that come with accessibility.

How does it work

Terminology

Below is a short description on how a chrome extension works.

Chrome extension diagram by Aryclenio Barros

• Content Scripts - Code that runs on the page the user is currently viewing. Commonly used for operating on the DOM and interacting with the current page. Because of security concerns, some actions can't be preformed on content scripts and must be done in a background script.

• Background Scripts - Code that runs in a service worker in the background and can't interact with the current page or DOM. Commonly used for long lived tasks or managing state across the extension. It can be thought of as a back-end of sorts for the extension. To communicate, the background and content scripts can register listeners to pass messages between each other. Since background scripts can preform actions content scripts can't, certain actions are executed there then passed to the content script.

Translating / Transcribing

The background script captures the current tabs audio then passes the audio to Deepgram for transcription.

const transcriber = 
SpeechRecognizerFactory
.createSpeechRecognizer(SpeechRecognizers.Deepgram);

After transcribing the audio, the extension uses one of its multiple translation engines to translate the text. For now, only Azure and Google are implemented.

const translator = 
TranslatorFactory.createTranslator(Translators.AZURE);
...
const translator = 
TranslatorFactory.createTranslator(Translators.GOOGLE);

The translated text is then sent to the client as a message.

const clientMessage = {
  command: "draw_text",
  data: translated,
}

chrome.tabs.sendMessage(activeTabId, clientMessage, (response) => {
  console.log("Sending text to client: " + translated);
});

Subtitles

Because of needed configurations, certain domains are white-listed for translations / transcriptions. When the client encounter a white-listed domain, the content script pulls the needed configuration to find the video player container and its dimensions and position, then draws a canvas overlay for displaying subtitles.

const videoContainerSelector = {
  crunchyroll: "#showmedia_video_box",
}

const videoContainer = queryElement(crunchyroll);
const containerPos = getPos(videoContainer);
const containerDim = getDim(videoContainer);

renderSubtitleCanvas(containerPos, containerDim);

The content script then listens for subtitles from the background script so it can render them on the canvas.

chrome.runtime.onMessage.addListener(
  (request, sender, sendResponse) => {
    switch (request.command) {
      case 'draw_text':
        console.log("Received command to draw text: " + request.data);
        drawTextToCanvas(request.data);
        sendResponse(true);
        return true;

      default:
        console.log("Unknown client command: " + request.command);
    }

    return true;
  },
);

Features

• Audio transcribing
• Text translation
• Render subtitles onto videos
• Multiple text translation engines: Azure Cognitive Translator, Google Cloud Translation

Roadmap

• Ability to change complex options without modifying code
• Increase transcribing accuracy
• Add more transcribing engines
• Ability to customize subtitle font
• Clean up code!!

Screenshots

Conclusion

Not too bad for three days of work huh? This was an incredible project to work on. Sadly, I learned about the hackathon too late, but I had fun regardless.

I want to continue updating the extension if there is a community interest to keep it alive. Open source contributions are extremely encouraged. Thanks for reading!