Introduction to BERT Language Model

BERT, an acronym for Bidirectional Encoder Representations, is a language model architecture that was created by Google in 2018.

This architecture was trained using a massive dataset consisting of approximately 2.5 billion words from the entire Wikipedia library and around 800 million words from the Google Books Corpus.

Training a model using such a large dataset would typically require a very long period of time, but thanks to the newly introduced Transformer architecture and the use of high-speed TPU's (Tensor Processing Units), Google was able to complete the training process in just four days

So as I said in the paragraph above , it is built on the transformer architecture a novel approach to NLP modelling which uses techniques like self-attention to identify the context of words.

Transformers usually consist of encoder and decoder blocks , but the Bert architecture only used encoders stacked onto one another.

Google initially released 2 versions:

1. Bert Base: with 12 encoders
2. Bert large: with 24 encoders

Bert was such a hit because it could also be used for many NLP problems like sentiment analysis, text summarization, and question answering.

We would be building a simple sentiment analysis classifier using a Bert model and the transformers library

You would want to use a Jupyter notebook for this tutorial(a Google Colab Environment will be preferable)

Firstly , we install these 2 libraries

!pip install transformers torch

Then we proceed to use these few lines of code for importing and loading our model

from transformers import pipeline
pipe = pipeline(task= "sentiment-analysis")
# the pipeline object defaults to using a lightweight version of BERT

Then we can simply check the semantic score of a piece of text by doing

pipe("This book was amazing , great read")

#[{'label': 'POSITIVE', 'score': 0.9998821020126343}]

pipe("The pig smelled very terribly")

#[{'label': 'NEGATIVE', 'score': 0.9984949827194214}]

The transformers library implements a lot of large language models very easily in python with only a few lines of code

You can check out a lot of other usecases of BERT here on HuggingFace

Thanks for the read, See ya👋