How To Build an AI-Powered Voice Assistant With Twilio, Laravel, and OpenAI

Voice assistants, such as Amazon Alexa and Apple's Siri have become integral to people’s lives, as they're so helpful with mundane tasks, such as setting reminders and turning on smart home devices. However, most voice assistants struggle with complex questions and queries, leaving users disappointed.

In this tutorial, you will learn how to build an AI-powered voice assistant that can understand and respond to complex questions using Twilio Programmable Voice and OpenAI.

Prerequisites

To complete this tutorial, you will need the following:

• A Twilio account, free or paid. If you don't have one, create a Twilio account for free.
• A phone number you own. Add it to your Twilio account.
• PHP 8.2 (or higher) installed
• Composer installed globally
• An OpenAI account. Create a free account here.
• ngrok and a free ngrok account
• Basic knowledge of the Laravel framework would be nice, although it’s not required

Build the AI-powered voice assistant

Create a new Laravel project

To create a new Laravel project using Composer, you need to run the command below in your terminal.

composer create-project laravel/laravel voice-assistant

Next, navigate to the project’s working directory and start the application development server by running the commands below in the terminal.

cd voice-assistant
php artisan serve

Once the application server is up, open http://localhost:8000/ in your browser to access the default Laravel welcome page, as shown in the image below.

The next step is to open the project in your preferred IDE or text editor.

Install the Twilio PHP Helper Library

The Twilio PHP Helper Library provides functionality for interacting with Twilio's APIs. With this library, you can easily interact with Twilio Programmable Voice in the application. Twilio Programmable Voice uses Twilio Markup language(TwiML), to specify the desired behaviour when receiving an incoming call or SMS.
In your project’s working directory, run the command below in a new terminal window or tab to install the library.

composer require twilio/sdk

Retrieve your Twilio API credentials

You will need your Account SID and Auth Token to interact with Twilio Programmable Voice using the Twilio PHP Helper Library. You can find them in the Account Info panel on your Twilio Console dashboard, as shown in the image below. Copy the respective values.

Store the Twilio access token in your .env file

After retrieving your Twilio access tokens, you need to ensure that they are stored securely in your project using environment variables.
In your project’s root directory, locate the .env file. This file is used to securely store sensitive data and other configuration data. Update it with the following:

TWILIO_SID=<your_twilio_account_sid>
TWILIO_AUTH_TOKEN=<your_twilio_account_auth_token>
TWILIO_PHONE_NUMBER=<your twilio_account_phone_number>

Then, replace the placeholders with your corresponding Twilio access token values which you just copied.

Install the OpenAI Laravel Helper Library

OpenAI is a leading AI research lab known for creating advanced language models trained on extensive amounts of data. The OpenAI Laravel package allows you to easily use OpenAI models in your application. Install the package by running the command below.

composer require openai-php/laravel

Finally, set up the package configurations by executing the below command:

php artisan openai:install

This command will create a new file called openai.php in the config folder. The file contains the configurations required to connect to the OpenAI API.
To use the OpenAI API, you'll need an API key for authentication. Log in to your OpenAI account and navigate to the API Keys section. Click Create new secret key to generate your key.

After retrieving the key, store it and add the below line to your .env file

OPENAI_API_KEY=<your_openAI_api_key>

Remember also to replace the placeholder with your corresponding API Key.

Update the routes

Run the command below in your terminal.

php artisan install:api

This command will create a new file named api.php in the routes folder. Add the below import statement at the beginning of the new file.

use App\Http\Controllers\VoiceController;

Then, update the file contents to include the following routes.

Route::post('/voice', [VoiceController::class, 'voiceInput']);
Route::post('/chat', [VoiceController::class, 'speechtoText']);

Here you are defining the routes for the voice and chat endpoints.

Note: In Laravel, API routes defined in the api.php file have a prefix of api/. This means the route defined as /voice will be accessible at /api/voice. The same applies to /chat.

Create the controller

The next step is to create a controller class that will house all the application logic. To create the controller, run the command below in your terminal.

php artisan make:controller VoiceController

This command will create a new file called VoiceController.php in the app/Http/Controllers folder.
Open the VoiceController.php file and replace the content with the following.

<?php

namespace App\Http\Controllers;

use Illuminate\Http\Request;
use Twilio\TwiML\VoiceResponse;
use OpenAI\Laravel\Facades\OpenAI;

class VoiceController extends Controller
{
    private $response;

    public function __construct()
    {
        $this->response = new VoiceResponse();
    } 

    public function voiceInput(Request $request)
    {
        $gather = $this->response->gather(['input' => 'speech', 'action' => '/api/chat']);
        $gather->say('Hello, I am your personal assistant. How can I help you today?');

        return $this->response;
    }

    public function speechtoText(Request $request)
    {
        $result = OpenAI::chat()->create([
            'model' => 'gpt-3.5-turbo',
            'messages' => [
                ['role' => 'user', 'content' => $request->SpeechResult],
            ],
        ]);
        $this->response->say($result->choices[0]->message->content);
        $this->response->gather(['input' => 'speech', 'action' => '/api/chat']);

        return $this->response;
    }
}

In the code above, the necessary imports required for the voice assistant are added. In the voiceInput() function, the TwiML (Twilio Markup Language) gather() method listens for the caller's input, and specifies the action to take on receiving it. Next, the TwiML say() method is used to prompt the caller. The gather() method converts the audio received into text and adds the transcribed audio to the $request object using the parameter speechResult.
After gathering the user's input, the $request object is sent as a POST request to the /api/chat endpoint, corresponding to the speechToText() function. The speechToText() function retrieves the transcribed audio from the request using the speechResult parameter and then forwards this transcribed audio to OpenAI Chat API for processing.
OpenAI's Chat API requires a role parameter containing a message with a system role and content for the intended actions. The model must also be configured, it is set to gpt-3.5-turbo in this tutorial. Finally, TwiML is used to return the response from OpenAI to the user in audio format.

Set up the Ngrok server

To make your voice assistant accessible from the internet, you can use a tool called Ngrok. Ngrok allows you to expose your local Laravel server to the internet.
Run the command below in your project's root directory.

ngrok http 8000 

This command creates a secure tunnel that allows you to access your local Laravel server running on port 8000 from a public URL provided by Ngrok.
After executing the command, you should see the following in your terminal.

Copy the Forwarding URL displayed here.

Configure TwiML

Create a New TwiML App

Next, you need to create a new TwiML app. This is to ensure reusability across multiple phone numbers. You would be able to easily add the created TwiML app to any number in your Twilio account.
In your Twilio Console, navigate to Explore Products > Voice > Manage > TwiML apps, and select Create new TwiML App.

Then, fill out the form, as in the screenshot above, and replace <your_ngrok_address> with the Ngrok Forwarding URL you copied earlier, append "/api/voice" to the end of it, and click Create.

Add TwiML to your number

You must also add the created TwiML app to your phone number to ensure that Twilio executes your code when a call is made to/from the phone number.
In the Twilio Console, navigate to Phone Numbers > Manage > Active numbers, and click on your Twilio number.

Then, in the Voice Configuration section for your phone number, set Configure with to TwiML App, set TwiML App to your TwiML app, and then click Save.

Test the application

It’s finally time to see your AI-powered voice assistant in action! You can test it by calling your Twilio phone number from your verified phone number.
When you successfully set up the system, a call will be made to your Twilio phone number. A voice prompt will ask for your request and the system will intelligently respond to it.

What’s next for building an AI-powered voice assistant with Twilio, Laravel, and OpenAI?

Congratulations on successfully creating your very own AI-powered voice assistant using Twilio Programmable Voice and OpenAI!

This comprehensive tutorial guided you through setting up the necessary tools, integrating Twilio and OpenAI, and building a voice assistant using the Laravel framework. Now, your voice assistant is ready to respond intelligently to any questions you might have.

So, what's next on your journey? Here are a few suggestions:

• Integration with External APIs: You can improve your voice assistant by integrating with external APIs. For example, you could integrate with e-commerce platforms to enable voice-based shopping experiences.
• Interaction History: You can also implement a feature that allows your voice assistant to remember previous interactions, and extract context from it. This would allow your voice assistant to provide more tailored and relevant responses.

Cheers to building and learning!