Fine-Tune MistralAI on Koyeb GPUs

git clone https://github.com/mistralai/mistral-finetune.git

{
  "messages": [
    {
      "role": "user",
      "content": "User interaction n°1 contained in document n°1"
    },
    {
      "role": "assistant",
      "content": "Bot interaction n°1 contained in document n°1"
    },
    {
      "role": "user",
      "content": "User interaction n°2 contained in document n°1"
    },
    {
      "role": "assistant",
      "content": "Bot interaction n°2 contained in document n°1"
    }
  ]
}
{
  "messages": [
    {
      "role": "user",
      "content": "User interaction n°1 contained in document n°2"
    },
    {
      "role": "assistant",
      "content": "Bot interaction n°1 contained in document n°2"
    },
    {
      "role": "user",
      "content": "User interaction n°2 contained in document n°2"
    },
    {
      "role": "assistant",
      "content": "Bot interaction n°2 contained in document n°2",
      "weight": 0,  # don't train on n°2
    },
    {
      "role": "user",
      "content": "User interaction n°3 contained in document n°2"
    },
    {
      "role": "assistant",
      "content": "Bot interaction n°3 contained in document n°2"
    }
  ]
}

import json
import requests
import pandas as pd
import os
from decouple import config
from mistralai import Mistral

NEWS_API_KEY = config("NEWS_API_KEY")
MISTRAL_API_KEY = config("MISTRAL_API_KEY")

# Function to fetch financial news data related to a specific topic
def fetch_financial_news(api_key, query="financial market", page_size=100):
    url = f"https://newsapi.org/v2/everything?q={query}&pageSize={page_size}&apiKey={api_key}"
    response = requests.get(url)
    news_data = response.json()
    articles = news_data.get('articles', [])
    return articles

# Function to process and save the data to a CSV file
def save_news_to_csv(articles, topic):
    # Create a DataFrame from the articles
    data = {
        "source": [article['source']['name'] for article in articles],
        "author": [article['author'] for article in articles],
        "title": [article['title'] for article in articles],
        "description": [article['description'] for article in articles],
        "url": [article['url'] for article in articles],
        "publishedAt": [article['publishedAt'] for article in articles],
        "content": [article['content'] for article in articles],
    }
    df = pd.DataFrame(data)

    # Save the DataFrame to a CSV file
    output_file = os.path.join(f"{topic.replace(' ', '_')}_news.csv")
    df.to_csv(output_file, index=False)
    print(f"Data saved to {output_file}")

def process_csv_to_jsonl(input_file):
    try:
        # Read the CSV file
        df = pd.read_csv(input_file)
        # Process each row and append each content to a list
        json_list = []
        for index, row in df.iterrows():
            client = Mistral(api_key=MISTRAL_API_KEY)
            result = client.chat.complete(model="mistral-small-latest", messages=[
                {
                    "content": """You will receive news article. Analyze it and generated user and assistant
                                interactions in a chat like format. The output should be a JSON with the
                                following format:
                               '{
                                  "messages": [
                                    {
                                      "role": "user",
                                      "content": "User interaction n°1 contained in document n°1"
                                    },
                                    {
                                      "role": "assistant",
                                      "content": "Bot interaction n°1 contained in document n°1"
                                    },
                                    {
                                      "role": "user",
                                      "content": "User interaction n°2 contained in document n°1"
                                    },
                                    {
                                      "role": "assistant",
                                      "content": "Bot interaction n°2 contained in document n°1"
                                    }
                                  ]
                                }'
                               Return only the JSON and nothing else, and no JSON tags.
                               """
                    ,
                    "role": "system",
                },
                {
                    "content": f"Here is the news content: {row['content']}",
                    "role": "user",
                },
            ])
            result_txt = result.choices[0].message.content
            result_txt = result_txt.replace("```

json", "").replace("

```", "")
            json_content = json.loads(result_txt)
            print(json_content)
            json_list.append(json_content)
        # Save the list to a JSONL file
        output_file = input_file.replace(".csv", ".jsonl")
        with open(output_file, "w") as f:
            for item in json_list:
                f.write(json.dumps(item) + "\n")
    except Exception as e:
        print(e)
        return "", ""

def separate_jsonl_train_eval(input_file):
    try:
        # Read the JSONL file and separate into training and evaluation sets using pandas
        df = pd.read_json(input_file, lines=True)
        df_train = df.sample(frac=0.90, random_state=200)
        df_eval = df.drop(df_train.index)

        # Save the training and evaluation sets to JSONL files keeping the same file name
        train_output_file = input_file.replace(".jsonl", "_train.jsonl")
        eval_output_file = input_file.replace(".jsonl", "_eval.jsonl")
        df_train.to_json(train_output_file, orient="records", lines=True)
        df_eval.to_json(eval_output_file, orient="records", lines=True)
    except Exception as e:
        print(e)
        return "", ""

# Main function to run the data processing pipeline
def main():
    # Specify the topic you want to search for
    topic = "financial market"

    # Fetch data
    articles = fetch_financial_news(NEWS_API_KEY, query=topic)

    # Save the data to a CSV file
    save_news_to_csv(articles, topic)

    # Process the CSV file to generate a JSONL file
    process_csv_to_jsonl("financial_market_news.csv")

    # Separate the JSONL file into training and evaluation sets
    separate_jsonl_train_eval("financial_market_news.jsonl")

# Entry point of the script
if __name__ == "__main__":
    main()

```
{% endraw %}
{% raw %}`

This Python script automates the process of collecting financial news data, formatting it for fine-tuning a MistralAI language model, and preparing it into training and evaluation datasets:

- **`fetch_financial_news`**: This function fetches financial news articles from NewsAPI based on a specified topic.
- **`save_news_to_csv`**: This function saves the fetched news data into a CSV file.
- **`process_csv_to_jsonl`**: This function converts the CSV data into a JSONL format, generating chat-based interactions using MistralAI.
- **`separate_jsonl_train_eval`**: This function splits the JSONL data into training and evaluation datasets.

The main function orchestrates the entire workflow:

- Fetching news data.
- Saving the data to a CSV file.
- Converting the CSV data to JSONL format.
- Splitting the JSONL data into training and evaluation datasets.

This script will generate the following dataset files, which will be used to train the model (you will run this script later on the remote machine):

- **`financial_market_news_train.jsonl`**: Contains the training data, in this case, 90 records of questions and answers related to news source data.
- **`financial_market_news_eval.jsonl`**: Contains the evaluation data, in this case, 10 records of questions and answers related to news source data.

## Configure the Training Script

Before we deploy to the Koyeb CPU to validate the dataset and train the model, you can start preparing the training configuration file. This file, which is a YAML file, will include all the necessary settings for the training process, as mentioned earlier.

So, go ahead and create a `7B.yaml` file:

```yaml
# data
data:
  instruct_data: '/mistral-finetune/financial_market_news_train.jsonl' # Fill
  data: '' # Optionally fill with pretraining data
  eval_instruct_data: '/mistral-finetune/financial_market_news_eval.jsonl' # Optionally fill

# model
model_id_or_path: '/mistral-finetune/mistral_models/' # Change to downloaded path
lora:
  rank: 64

# optim
seq_len: 32768
batch_size: 1
max_steps: 300
optim:
  lr: 6.e-5
  weight_decay: 0.1
  pct_start: 0.05

# other
seed: 0
log_freq: 1
eval_freq: 100
no_eval: False
ckpt_freq: 100

save_adapters: True # save only trained LoRA adapters. Set to `False` to merge LoRA adapter into the base model and save full fine-tuned model

run_dir: '/mistral-finetune/chat_test' # Fill
``{% endraw %}{% raw %}`

This is the important information that you need to fill in:

- **`instruct_data`**: This is the path to the training dataset. This dataset will be generated when you run the **`dataset_processing.py`** script on the remote machine.
- **`eval_instruct_data`**: This is the path to the evaluation dataset. This dataset will also be generated when you run the **`dataset_processing.py`** script on the remote machine.
- **`model_id_or_path`**: This is the identifier or path of the model you will be training. You will download this model later on the remote machine.
- **`batch_size`**: You can adjust this if needed, but a batch size of 1 will work well for this case.
- **`max_steps`**: This is the number of steps to train the model with. The default of 300 steps provides a good balance between speed and training capabilities. You can reduce to 100 steps for faster processing at a cost of less accuracy.
- **`run_dir`**: This is the directory where the trained model will be saved.

After deployment, you will need to download the model to train, execute the dataset script, and then train the model. These settings are prepared for the commands you will execute later on.

## Deploying to Koyeb GPU

To deploy the fine-tuning process to Koyeb, you will need to create a Dockerfile that sets up the environment for training the model, a repository to store the code, and finally deploy the app to Koyeb via git and built using the Dockerfile.

### Create a Dockerfile

We'll start by preparing a Dockerfile to ensure we have all the necessary dependencies installed, especially for GPU support. Create a `Dockerfile` with the following contents:

```docker
# Use the official Python base image
FROM python:3.11

# Clone the repository
RUN git clone https://github.com/mistralai/mistral-finetune.git

# Set the working directory
WORKDIR /mistral-finetune

# Update pip, install torch and other dependencies
RUN pip install --upgrade pip && pip install -r requirements.txt

# Copy the 7B.yaml file
COPY 7B.yaml /mistral-finetune/example/7B.yaml

# Script to prepare the training data
COPY dataset_processing.py /mistral-finetune/dataset_processing.py
```

This Dockerfile is designed to set up an environment for fine-tuning the MistralAI language model. It automates the process of cloning the necessary repository, installing dependencies, and copying both the training configuration and the training dataset script.

### Create the repository

The final step is to create a new repository on GitHub to store the project files.

Once you're ready, run the following commands in your terminal to commit and push your code to the repository:

```docker
echo "# MistralFineTuning" >> README.md
git init
git add .
git commit -m "First Commit"
git branch -M main
git remote add origin [Your GitHub repository URL]
git push -u origin main
``{% endraw %}{% raw %}`

You should now have all your local code in your remote repository. Now it is time to deploy the Dockerfile.

### Deploy to Koyeb

In the [Koyeb control panel](https://app.koyeb.com/), while on the **Overview** tab, initiate the app creation and deployment process by clicking **Create App.** You can select a Worker application.

On the App deployment page:

1. Select **GitHub** as your deployment method.
2. Choose the repository where your code resides. For example, `MistralFineTuning`.
3. Select the GPU you wish to use, for example, `A100`. The training might work on other GPUs, but for performance and training accuracy, this is the recommended.
4. In the **Builder** section, choose **Dockerfile**.
5. In the **Service name** section, choose an appropriate name.
6. Finally, click **Deploy**.

## Running the Fine-Tuning process

Once the deployment is complete, you can start preparing and running the fine-tuning of the model.

The Dockerfile deployment has set up the base system needed to train the model, but it didn’t download a model, so that will be one of the first steps.

Since the next commands need interaction with the remote machine, you'll use the Koyeb CLI to access the remote machine through the terminal.

First, make sure you have the Koyeb CLI installed. You can find the installation instructions [**here**](https://www.koyeb.com/docs/build-and-deploy/cli/installation). Then, generate an API Token, which you can do [**here**](https://app.koyeb.com/user/settings/api/).

Now you are ready to log in with the Koyeb CLI:

```bash
koyeb login
```

First, input your API token key when asked for it.

To see a list of running instances, use the following command:

```bash
koyeb instances list
```

Note the instance ID you want to connect to. Then, create a remote terminal session to the remote machine:

```bash
koyeb instances exec <instance_id> /bin/bash
```

You now have an active remote session to the remote machine. All commands executed from now on will be on the remote machine.

As mentioned, the first step is to download the model to train, in this case the Mistral 7B Instruct:

```bash
mkdir mistral_models
wget https://models.mistralcdn.com/mistral-7b-v0-3/mistral-7B-Instruct-v0.3.tar && tar -xf mistral-7B-Instruct-v0.3.tar -C mistral_models
```

It might take a couple of minutes for the model to be downloaded and extracted.

Next, to ensure proper compatibility, make sure that the Numpy package installed is at version 1.26.4:

```bash
pip install numpy==1.26.4
```

Now you can install the necessary libraries for executing the dataset script:

```bash
pip install requests pandas mistralai python-decouple
```

You can then copy the necessary information for the **`.env`** file:

```bash
echo "NEWS_API_KEY=<YOUR_NEWS_API_KEY>" > .env
echo "MISTRAL_API_KEY=<YOUR_MISTRAL_API_KEY>" >> .env

```

Make sure to replace the values with your own API keys.

And then you can execute the **`dataset_processing.py`** script:

```bash
python3 dataset_processing.py
```

It might take a couple of minutes to prepare the dataset. After it finishes, you should have two JSONL files corresponding to the training and evaluation datasets.

You can now validate those datasets with:

```bash
python3 -m utils.validate_data --train_yaml example/7B.yaml
```

You should get an estimate on the ETA for the training and there should not be any validation errors. If there are errors, you can fix them with:

```bash
python3 -m utils.reformat_data cryptocurrency_market_news_train.jsonl
python3 -m utils.reformat_data cryptocurrency_market_news_eval.jsonl
```

Validate the dataset again (if needed) and now there should be no errors:

```bash
python3 -m utils.validate_data --train_yaml example/7B.yaml
```

Everything is now ready to train the model, which you can do with:

```bash
CUDA_VISIBLE_DEVICES=0 python3 -m torch.distributed.run --nproc-per-node 1 --master_port 65123 -m train example/7B.yaml
```

The **`CUDA_VISIBLE_DEVICES=0`** is necessary to make sure the training script recognizes the GPU on the remote machine.

This process will take several minutes, possibly even hours. It will show an estimate of the remaining processing time.

After it is finished, you will be able to evaluate the trained model against the standard model, which we will see how to do in the next section.

## Evaluating the Fine-Tuned Model

To evaluate the fine-tuned model, we first need to establish a baseline with the default model.

First, you need to install the necessary package on the remote machine:

```bash
pip install mistral_inference
```

Now you can test the default model by running:

```bash
mistral-chat /mistral-finetune/mistral_models/ --max_tokens 1024 --temperature 0.7 --instruct
```

It will ask you for a prompt. Let’s try this one:

```bash
Prompt: how is x adverstising doing?

To provide an accurate answer, I would need specific details about X advertising. However, I can share some general insights about advertising in today's digital landscape:

1. Digital Advertising: With the rise of the internet and mobile devices, digital advertising has become a significant part of the marketing mix. Platforms like Google, Facebook, Instagram, and YouTube offer targeted advertising options based on user demographics, interests, and online behavior.

2. Social Media Advertising: Social media platforms are popular advertising channels due to their vast user base and the ability to target ads based on user profiles. Brands use social media ads to increase brand awareness, drive traffic to their websites, and generate leads.

3. Influencer Marketing: Influencer marketing has grown in popularity, as brands partner with social media influencers to promote their products or services. This strategy allows brands to tap into the influencer's audience and leverage their credibility.

4. Traditional Advertising: While digital advertising is on the rise, traditional advertising methods like television, radio, print, and outdoor advertising still have a place in the marketing mix, especially for reaching broader audiences or those who are less tech-savvy.

5. Measuring Success: Advertising success is often measured by metrics such as reach, impressions, click-through rates, conversion rates, and return on investment (ROI). Brands use these metrics to evaluate the effectiveness of their advertising campaigns and make data-driven decisions.

Without more specific information about X advertising, it's difficult to say how they are doing. It's essential for brands to regularly analyze their advertising performance and adjust their strategies as needed to maximize their return on investment.

```

As you can see, the default model gave a very generic answer.

Now let’s run the fine-tuned model with:

```bash
mistral-chat /mistral-finetune/mistral_models/ --max_tokens 1024 --temperature 0.7 --instruct --lora_path /mistral-finetune/chat_test/checkpoints/checkpoint_000300/consolidated/lora.safetensors
```

And we use the same prompt:

```bash
Prompt: how is x adverstising doing?

X advertising is currently seeing a slight increase in performance.
```

As you can see, the fine-tuned model gave a much more accurate and precise answer.

### Impact on Domain Knowledge

Fine-tuning MistralAI on financial data significantly improves the model's ability to understand and operate within the financial domain. This process transforms the model into a specialized tool that has a deep understanding of the financial domain.

Here we have just exposed the model to a subset of recent news, but by exposing the model to more domain-specific data, such as financial reports, market analysis, and regulatory documents, it learns the precise meanings and nuances of financial terminology.

Fine-tuning also helps the model stay current with ongoing trends in the financial industry. This includes understanding the implications of market movements, economic indicators, and geopolitical events on financial markets.

This enhanced understanding and specialization enable the model to perform a wide range of finance-related tasks with greater accuracy, relevance, and compliance. This makes it an invaluable asset for financial professionals and organizations, helping them to make more informed decisions and improve their overall performance in the financial domain.

## Conclusion

You've just completed this tutorial on fine-tuning MistralAI on Koyeb Serverless GPUs.

You can check out the example repository for this tutorial on Koyeb's GitHub for [fine-tuning MistralAI](https://github.com/koyeb/example-finetune-mistralai) with serverless GPUs.

While this guide focused on fine-tuning MistralAI for finance, the approach and techniques covered here are the same across various domains. Whether you're working with healthcare data, legal documents, technical manuals, or customer service interactions, fine-tuning can significantly improve the relevance and accuracy of AI models.

Have fun experimenting with your own datasets and seeing how fine-tuning can add value and improve performance in your specific area of interest!