IBM released Granite 3.0, and it includes General purpose, Language , Guardrails & Safety, Mixture-of-Experts(MoE), Accelerated inference and Granite Time Series Tiny Time Mixer models. All models are available under the Apache 2.0 License making them accessible to everyone.
IBM’s Granite 3.0 models are specifically designed to meet the evolving needs of today’s enterprises, offering a blend of high performance, tailored customization, and a commitment to ethical AI practices. The company emphasizes addressing key concerns often associated with advanced models from providers like OpenAI, Anthropic, Google, and Meta. This includes clear disclosure of training data and development processes, legal protection through indemnification, and granting full commercial usage rights. These efforts make it easier and safer for enterprises to adopt and integrate these models into their operations with confidence.
In IBM's early testing, the text-only dense LLMs known as the IBM Granite 2B and 8B models—trained on more than 12 trillion tokens of data—showed notable speed and performance gains over its predecessors. Llama 3.1 8B-Instruct is now competitive with Granite-8B-Instruct on both OpenLLM Leaderboard v1 and OpenLLM Leaderboard v2 benchmarks.
Capabilities
✅ Summarization
✅ Text classification
✅ Text extraction
✅ Question-answering
✅ Retrieval Augmented Generation (RAG)
✅ Code related
✅ Function-calling
✅ Multilingual dialog use cases
Prerequisites for Your System:
Make sure you have the following:
✅ GPUs: RTXA6000 (for smooth execution).
✅ Disk Space: 70 GB free.
✅ RAM: At least 40 GB.
Step-by-Step Process to Deploy Granite Dense 2B and 8B on a Virtual Machine in the Cloud
For the purpose of this tutorial, we will use a GPU-powered Virtual Machine offered by NodeShift; however, you can replicate the same steps with any other cloud provider of your choice. NodeShift provides the most affordable Virtual Machines at a scale that meets GDPR, SOC2, and ISO27001 requirements.
Step 1: Sign Up and Set Up a NodeShift Cloud Account
Visit the NodeShift Platform and create an account. Once you've signed up, log into your account.
Follow the account setup process and provide the necessary details and information.
Step 2: Create a GPU Node (Virtual Machine)
GPU Nodes are NodeShift's GPU Virtual Machines, on-demand resources equipped with diverse GPUs ranging from H100s to A100s. These GPU-powered VMs provide enhanced environmental control, allowing configuration adjustments for GPUs, CPUs, RAM, and Storage based on specific requirements.
Navigate to the menu on the left side. Select the GPU Nodes option, create a GPU Node in the Dashboard, click the Create GPU Node button, and create your first Virtual Machine deployment.
Step 3: Select a Model, Region, and Storage
In the "GPU Nodes" tab, select a GPU Model and Storage according to your needs and the geographical region where you want to launch your model.
We will use 1x RTX A6000 GPU for this tutorial to achieve the fastest performance. However, you can choose a more affordable GPU with less VRAM if that better suits your requirements.
Step 4: Select Authentication Method
There are two authentication methods available: Password and SSH Key. SSH keys are a more secure option. To create them, please refer to our official documentation.
Step 5: Choose an Image
Next, you will need to choose an image for your Virtual Machine. We will deploy Granite Dense 2B and 8B on an NVIDIA Cuda Virtual Machine. This proprietary, closed-source parallel computing platform will allow you to install Granite Dense 2B and 8B on your GPU Node.
After choosing the image, click the 'Create' button, and your Virtual Machine will be deployed.
Step 6: Virtual Machine Successfully Deployed
You will get visual confirmation that your node is up and running.
Step 7: Connect to GPUs using SSH
NodeShift GPUs can be connected to and controlled through a terminal using the SSH key provided during GPU creation.
Once your GPU Node deployment is successfully created and has reached the 'RUNNING' status, you can navigate to the page of your GPU Deployment Instance. Then, click the 'Connect' button in the top right corner.
Now open your terminal and paste the proxy SSH IP or direct SSH IP.
Next, if you want to check the GPU details, run the command below:
nvidia-smi
Step 8: Install Granite Dense 2B and 8B
After completing the steps above, it's time to download Granite Dense 2B and 8B from the Ollama website.
Website Link: https://ollama.com/library/granite3-dense
Then run the following command to install the Ollama:
curl -fsSL https://ollama.com/install.sh | sh
After the installation process is complete, run the following command to see a list of available commands:
ollama
Next, run the following command to host the Granite Dense 2B and 8B model so that it can be accessed and utilized efficiently.
ollama serve
Step 9: Select 2B Size of the Model
On website select the 2B size of the model, First we will Run 2B size:
Website Link: https://ollama.com/library/granite3-dense:2b
Step 10: Pull Granite Dense 2B Model
To pull the Granite Dense 2B Model, run the following command:
ollama pull granite3-dense:2b
Step 11: Run Granite Dense 2B Model
Now, you can run the model in the terminal using the following command and interact with your model:
ollama run granite3-dense:2b
Step 12: Select 8B Size of the Model
On website select the 8B size of the model, Next we will Run 8B size:
Website Link: https://ollama.com/library/granite3-dense:8b
Step 13: Pull Granite Dense 8B Model
To pull the Granite Dense 8B Model, run the following command:
ollama pull granite3-dense:8b
Step 14: Run Granite Dense 8B Model
Now, you can run the model in the terminal using the following command and interact with your model:
ollama run granite3-dense:8b
Conclusion
Granite Dense 2B and 8B is a groundbreaking open-source model from IBM that brings state-of-the-art AI capabilities to developers and researchers. Following this step-by-step guide, you can quickly deploy Granite Dense 2B and 8B on a GPU-powered Virtual Machine with NodeShift, harnessing its full potential. NodeShift provides an accessible, secure, affordable platform to run your AI models efficiently. It is an excellent choice for those experimenting with Granite Dense 2B & 8B and other cutting-edge AI models.
For more information about NodeShift:
Top comments (0)