Detexify in your browser

What I built

Detexify is a well-known service to recognize hand drawn math symbols and translate them to LaTeX expressions, so one who works with LaTeX not need to dive into symbols-a4.pdf that you just can't memorize.

I have built a replicate of this service using ML and deploy it as a browser-side application, i.e., it does not have a backend. The recognition and search purely runs in your browser.

Category Submission:

Program for the People

App Link

detexify.yuhuishi.me

Screenshots

Description

Users can draw a math symbol they cannot recall what the corresponding LaTeX expression is. The model will try to recognize the most likely symbols given the pixels, and show the LaTeX expressions together with their probabilities.

Link to Source Code

https://github.com/Yuhuishishishi/detexify-replicate

Permissive License

MIT License

Background

When I was doing my graduate school and writing in LaTeX, finding the LaTeX expression for a math symbol is the most time consuming and unpleasant thing I had experienced.
One can dive into the doc symbols-a4.pdf and search among hundreds of math symbols to find out the correct expression for the one in your mind.

Or one can go directly to detexify.yuhuishi.me and draw it and let the ML model help you find the expression for it.

Hope this application can help people who need to work with LaTeX to make their life bit easier.

How I built it

The application is a pure front-end app (there is no backend).

Model training

For the ML part, I trained a recognition model that takes in a user drawn symbol and outputs the LaTeX expression.

The model is trained using the Kaggle notebook.
Under the hood, the model is a simple CNN learner.

The data the model is trained on consists of a bunch of labeled math symbol images. You can find more details here HASYv2 Dataset.

Overall, the model can achieve 60% accuracy. I haven't spent much time on tuning and finding a better architecture.

Deployment

I built the front-end using React. This is the first-ever React app I have built since I have no previous experience in it.

To serve the model, I used tensorflow.js to convert it into a javascript-friendly format and load it into the app. The inference runs purely in the browser.

I used DO app platform to deploy the whole application. Since there is no backend involved, the deployment is quite straightforward (deployed as a static site).