In this example, we use Langchain, but perhaps the original API alone can be used.
Welcome to LangChain | 🦜️🔗 Langchain
First, the server side.
import { ChatOpenAI } from "langchain/chat_models"
import { HumanChatMessage } from "langchain/schema"
import { CallbackManager } from "langchain/callbacks"
const api = (req, res, ctx) => {
if (req.method !== "POST") return
const input = req.body.input
const chat = new ChatOpenAI({
openAIApiKey: OPENAI_API_KEY,
streaming: true
callbackManager: CallbackManager.fromHandlers({
async handleLLMNewToken(token: string) {
res.write(token)
},
}),
})
await chat.call([new HumanChatMessage(input)])
res.end()
})
The point is to enable streaming with streaming: true
and write()
. This is called Writable streams in Node.js and is enabled for files and responses.
When all the Langchain processing is done, we terminate it with end()
.
Next is the client side.
const reply = async (input) => {
const decoder = new TextDecoder()
const res = await fetch(ENDPOINT, {
method: "POST",
headers: {
"Content-Type": "application/json",
},
body: JSON.stringify({
input,
}),
})
const reader = res?.body?.getReader()
let output = ""
while (true) {
if (reader) {
const { value, done } = await reader.read()
if (done) break
output += decoder.decode(value)
}
}
}
The point is getReader()
. Also, we need to decode it, so we put that process in.
Top comments (0)