How to migrate from Sendbird to TalkJS

In this tutorial, we're going to show you how to export data from Sendbird and import it to TalkJS. This is really helpful if you are trying to switch platforms and onboard with TalkJS. In case you’re using some other chat API the steps would be very similar to this one. The basic flow is to get all the data using Sendbird’s API, make some data transformations so that TalkJS can identify it, and then finally invoke the TalkJS APIs with the transformed data. You can find the entire source code in our Github repository.

Exporting from Sendbird

Shown above is a screenshot of our group chat on Sendbird. They have three main entities that you must import to TalkJS. They are Users, Channels, and Messages. The table below shows the mapping between entities in Sendbird and TalkJS.

There are two ways to export data from Sendbird. One is to use the export option from the dashboard, and the other is to use their Export APIs. We are going to use the dashboard here as it is quick and easy.

On your dashboard, navigate to the Data exports page and click on Request. Once there, select the entity you want to export (Channels, Messages, Users), its format, and time frame. Sendbird only allows you to select a date range of 7 days, so if you have older data, you need to perform multiple exports. Click on Request to submit the export request. On the Data exports page, you can see a new entry for the export. It goes through different statuses before becoming available to download. Click on Download and save it to your local storage.

Using the Sendbird Export API

You can also export data from Sendbird using their export REST API. The base URL for the export REST API is:

https://api-APPLICATION_ID.sendbird.com/v3/export/ENTITY_NAME

Replace APPLICATION_ID with your application ID from the Sendbird dashboard and ENTITY_NAME with users, messages or channels based on what you want to export. You also have to provide a custom header called Api-Token and provide it with the value from your Sendbird dashboard as well.

The request must be an HTTP POST and the body requires just two parameters, the start_ts and end_ts. Both of these are in UNIX milliseconds. Sendbird allows you to fetch messages within a 7-day timeframe only even when using the API, so if you have more data, you must submit multiple requests.

{
  "start_ts": "1673352000",
  "end_ts": "1673907200"
}

The default format is JSON, but if you want it in CSV just add the format attribute in the request body and set it to CSV. Don’t forget to add the Api-Token header when you submit each request. After submitting, you should get a response as shown below:

{
  "request_id": "x00000xx0000000",
  "status": "scheduled",
  "data_type": "users",
  "start_ts": 1673352000000,
  "end_ts": 1673907200000,
  "user_ids": [],
  "format": "json",
  "created_at": 1675207070000,
  "timezone": "UTC"
}

The status is scheduled and you can retrieve the file once it is complete by executing a GET request with the request_id as a query parameter. For this example, the request is as shown below:

https://api-APPLICATION_ID.sendbird.com/v3/export/users/x00000xx0000000

Once the status is done, you can download the data using the URL in the response. It has an expiry date of 7 days.

Importing to TalkJS

We have separated the import process into three different programs to make it easier to explain. The code in all three is similar to each other and you can combine them while using it in production. This is just to make the explanation a bit easier for the sake of this tutorial. We have three NodeJS applications that import users, conversations, and messages.

Importing Users

"use strict";
const fs = require("fs");
const axios = require("axios");

const instance = axios.create({
  baseURL: "https://api.talkjs.com",
  headers: {
    Authorization: "Bearer YOUR_SECRET_KEY_FROM_TALKJS_DASHBOARD",
    "Content-Type": "application/json",
  },
});

function importUsers() {
  const userDataRaw = fs.readFileSync("exported-data/active_users.json");
  const usersJson = JSON.parse(userDataRaw);
  const users = usersJson.active_users;
  for(let user of users){
    instance.put(`/v1/YOUR_APP_ID/users/${user.user_id}`, {
      name: user.nickname,
      email: [user.metadata.emailId],
      photoUrl: user.profile_url,
      id: user.user_id,
    })
  }
}
importUsers();

The code above shows how you can import users to TalkJS. We have used two modules in the program. The file system module (fs) reads the exported data and the axios module to make requests. After importing the modules, we create an instance of the axios module passing in the baseURL of the TalkJS API and the necessary headers.

You must obtain the secret key from your TalkJS dashboard to authenticate requests. Be sure to set the Content-Type as application/json, since TalkJS only supports that currently.

The next function is the actual import process. We read the JSON data and loop through each user. Then we call the endpoint with the HTTP PUT method passing in the parameters necessary to create a user. The id and email are the minimum parameters required. Sendbird doesn’t have an email address field for their users, but they do support the addition of custom fields. The metadata field contains the custom fields and we can map the email address from that to the email field for a TalkJS User.

Importing Conversations

function importConversations() {
  const channelsRaw = fs.readFileSync("exported-data/group_channels.json");
  const channelsJson = JSON.parse(channelsRaw);
  const channels = channelsJson.group_channels;
  for(let channel of channels){
    const channelParticipants = [];
    for (let member of channel.members) {
      channelParticipants.push(member.user_id);
    }
    instance.put(`/v1/YOUR_APP_ID/conversations/${channel.channel_url}`, {
      subject: channel.name,
      participants: channelParticipants,
    })
  }
}

importConversations();

The next program is to import conversations. Channels in Sendbird are analogous to Conversations in TalkJS. We read the JSON file just like before and store the data in a variable called channels. The participants of a conversation are stored in a variable called members in Sendbird. We retrieve this and store it in an array before invoking the API to create conversations in TalkJS. The channel’s URL from Sendbird is used as the unique Conversation ID in TalkJS.

Importing Messages

const messagesRaw = fs.readFileSync("exported-data/messages.json");
const messagesJson = JSON.parse(messagesRaw);
const messages = messagesJson.messages;

const uniqueConversations = [ ...new Set(messages.map((item) => item.channel_url))];

let conversationJsonArray = [];

for (let conversation of uniqueConversations) {
  let conversationJsonObj = {};
  conversationJsonObj[conversation] = [];
  conversationJsonArray.push(conversationJsonObj);
}

for(let conversation of conversationJsonArray) {
  for(let message of messages){
    if(Object.keys(conversation)[0] === message.channel_url){
      let messageObj = {};
      messageObj.text = message.message,
      messageObj.sender = message.user.user_id,
      messageObj.type = message.type === "MESG" ? "UserMessage" : "SystemMessage",
      messageObj.timestamp = message.created_at,
      messageObj.readBy = []
      conversation[message.channel_url].push(messageObj);
    }
  }
}

This part is a little different from the other two. You can use the import endpoint in TalkJS to bulk import messages from conversations. But before that, you must prepare your data. The structure of the data is as shown below.

[
  {
    "converstationId": [
      { message_1 }, { message_2 }, { message_3 }
    ]
  },
  {
    "converstationId": [
      { message_1 }, { message_2 }, { message_3 }
  }
]

The first three lines of code are similar to the previous programs where we read the exported data and store it in a constant. Then, we create a constant called uniqueConversations to store the unique IDs of the Channels from Sendbird. We use a Set because it only stores unique values.

In the next for loop, we create a JSON array containing the conversation IDs as keys. The JSON array currently looks as shown below:

[
  {
    "converstationId": []
  },
  {
    "converstationId": []
  }
]

The last part consists of two for loops. One that iterates over the JSON array shown above and the other that iterates over our messages. Then, we check if each message’s conversation ID is equal to the key of the object we are currently in. If they match, we populate it with the required values for TalkJS. The timestamp field helps in keeping the order of the messages intact when importing it from Sendbird to TalkJS. We have used the message.type MESG from Sendbird to map it to the UserMessage type in TalkJS.

function importMessages(conversationJsonArray) {
  for (let conversation of conversationJsonArray) {
    instance.post(`/v1/tAU5JKLC/import/conversations/${Object.keys(conversation)[0]}/messages`, [...conversation[Object.keys(conversation)[0]]])
  }
}
importMessages(conversationJsonArray);

Finally, we invoke the importMessages function passing in the constructed JSON array. We iterate through each of the conversation and pass the entire messages array inside it to the API.

Opening the application in TalkJS

Now, we can create an application in TalkJS and then view the imported data. You can also go to your TalkJS dashboard and navigate to Activity → Conversation History and then select the newly created conversation. It shows you the users and chats in a TalkJS-style UI.

In our sample application, we are going to add their user IDs and names to set them up quickly. When using it in production, you can use the TalkJS APIs to retrieve users, conversations, messages, etc. There are three available UI modes for TalkJS. One is the inbox, the second is the chatbox and the last one is the popup widget. You can read more about them here. We are using the chatbox widget in this example. You can read the Getting Started section to quickly set up your app in TalkJS. Then copy the code below in your script file to retrieve the conversation we exported from Sendbird.

Talk.ready.then(function () {
  var me = new Talk.User({
    id: "523915",
    name: "Mathew Jacob"
  });
  var other1 = new Talk.User({
    id: "523916",
    name: "Morgan Stanley"
  });
  var other2 = new Talk.User({
    id: "523917",
    name: "Shane Riley"
  });
  var other3 = new Talk.User({
    id: "523918",
    name: "Clyde Howell"
  });
  window.talkSession = new Talk.Session({
    appId: "YOUR_APP_ID",
    me: me
  }); 
var conversation = window.talkSession.getOrCreateConversation(
    "SENDBIRD_CONVERSATION_ID"
  );
  conversation.setParticipant(me);
  conversation.setParticipant(other1);
  conversation.setParticipant(other2);
  conversation.setParticipant(other3); 
var chatbox = window.talkSession.createChatbox(conversation);
  chatbox.mount(document.getElementById("talkjs-container"));
});

As mentioned earlier, you may want to use an API to retrieve the names and IDs of users. But in this example, we have hardcoded them based on the values we imported. Once all this is set up, you should have a chatbox UI on your browser with the exported chats from Sendbird. The entire source code for this example is available in our Github repository.