This is a submission for the Twilio Challenge
What I Built
The goal of this project is to demonstrate the integration of Generative AI (GenAI) in embedded systems and to highlight the ease of accessing AI capabilities via Twilio's API.
I have integrated a continuous motion recognition system with generative AI that recognizes and tracks motion and uses Twilio's WhatsApp API to alert users and respond to their questions.
How I Built
Continuous Motion Recognition
Unlike rule-based programming, which struggles with the variability in human gestures, machine learning easily adapts to these differences.
The Continuous Motion Recognition project using Edge Impulse leverages advanced ML techniques to enable real-time, on-device motion detection and classification. Edge Impulse, a leading platform for embedded machine learning, provides a comprehensive toolset for collecting data, designing, and deploying custom models directly to edge devices. This project typically involves gathering motion data from sensors such as accelerometers and gyroscopes, then using Edge Impulse to preprocess the data, train models, and optimize them for efficient, low-power inference on microcontrollers or other edge hardware.
You can follow the Continuous motion recognition | Edge Impulse Documentation tutorial to get started with it. You can view and clone the project here.
So, if you have gone through it successfully and deployed it with your firmware and when you run the inference, you will be able to see the real-time recognition of the motion.
Storing the result
For this project, I used Xata as the database. I have filtered out the result with the maximum value from each prediction and stored it in the predictions table. If you are new to Xata, checkout Getting started with Xata guide.
Twilio and AI
Twilio's WhatsApp Messaging Service
Twilio provides access to the WhatsApp Business Platform through its APIs. This allows developers to integrate WhatsApp messaging into their applications. We can leverage familiar Twilio APIs like Programmable Messaging to send and receive messages with text, media, and even certain pre-built templates.
I have used Python-Flask for creating the server and responding to the incoming messages. You can check out the Twilio documentation on Receive and Reply to Incoming Messages - Python for a better understanding of how it needs to be implemented.
Delivering the message
By using the Twilio sandbox, we are able to send and receive messages, create automated responses, and integrate interactive messaging features within a controlled setting.
When someone replies to one of our messages, we will receive a webhook request from Twilio. So, we need to configure the webhooks by connecting to our server/app. I ran my server using Replit. Search for the Flask template and integrate the Twilio API.
Google Gemini AI
I used Google's 1.5 Flash Model to generate responses based on predicted data. When a user replies on WhatsApp, the endpoint URL is configured to our server on Replit, which listens for these messages. We then retrieve all records from the database and pass them, along with the user's message, to the Gemini AI for generating a response. The generated response is then sent back to the user on WhatsApp.
Demo
References
The primary objective of this project is to demonstrate the use of GenAI in EdgeAI/ML devices. By integrating Twilio's AI, users can now easily analyze ML-predicted data simply by asking questions through WhatsApp. While this project may not directly solve specific problems, it showcases the potential of applying GenAI to predicted data on a large scale.
Feedback is welcome :)
Thank you for reading and have a great day!
Top comments (0)