DEV Community

Cover image for Distributed SQL Change Management with Liquibase and YugabyteDB on GKE
YugabyteDB Distributed PostgreSQL Database profile image Jimmy Guerrero
Jimmy Guerrero for YugabyteDB Distributed PostgreSQL Database

Posted on • Edited on

Distributed SQL Change Management with Liquibase and YugabyteDB on GKE

#yugabyte #postgres #kubernetes #distributedsystems

Liquibase is an open source and extensible change management project that supports a variety of databases including Snowflake, MySQL, and PostgreSQL via JDBC. Liquibase allows users to easily define changes in SQL, XML, JSON, and YAML. These changes are then managed in a version control system so the changes can be documented, ordered, and standardized. For more information on the features and benefits of Liquibase, check out their documentation site.

In this blog post we’ll show you how to:

  • Install a 3 node YugabyteDB cluster on Google Kubernetes Engine
  • Build the sample Northwind database
  • Install and configure Liquibase
  • Create a simple changeset and verify the results
  • Explore how changes are documented and managed in Liquibase

New to distributed SQL or YugabyteDB? Read on.

What is Distributed SQL?

Distributed SQL databases are becoming popular with organizations interested in moving data infrastructure to the cloud or cloud native environments. This is often motivated by the desire to reduce TCO or move away from the horizontal scaling limitations of monolithic RDBMS like Oracle, PostgreSQL, MySQL, and SQL Server. The basic characteristics of Distributed SQL are:

  • They must have a SQL API for querying and modeling data, with support for traditional RDBMS features like foreign keys, partial indexes, stored procedures, and triggers.
  • Automatic distributed query execution so that no single node becomes a bottleneck.
  • Should support automatic and transparent distributed data storage. This includes indexes, which should be sharded across multiple nodes of the cluster so that no single node becomes a bottleneck. Data distribution ensures high performance and high availability.
  • Distributed SQL systems should also provide for strongly consistent replication and distributed ACID transactions.

For a deeper discussion about what Distributed SQL is, check out, “What is Distributed SQL?”

What is YugabyteDB?

YugabyteDB is an open source, high-performance distributed SQL database built on a scalable and fault-tolerant design inspired by Google Spanner. YugabyteDB is PostgreSQL wire compatible, cloud native, offers deep integration with GraphQL projects, plus supports advanced RDBMS features like stored procedures, triggers and UDFs.

Got questions? Make sure to ask them in our YugabyteDB Slack channel. Ok, let’s dive in…

Step 1: Install YugabyteDB on a GKE Cluster using Helm 3

In this section we are going to install YugabyteDB on the cluster. The complete steps are documented here. We’ll assume you already have a GKE cluster up and running as a starting point.

The first thing to do is to add the charts repository.

$ helm repo add yugabytedb https://charts.yugabyte.com
Enter fullscreen mode Exit fullscreen mode

Now, fetch the updates.

$ helm repo update
Enter fullscreen mode Exit fullscreen mode

Create a namespace. In this case we’ll call it yb-demo.

$ kubectl create namespace yb-demo
Enter fullscreen mode Exit fullscreen mode

Expected output:

namespace/yb-demo created
Enter fullscreen mode Exit fullscreen mode

We are now ready to install YugabyteDB. In the command below we’ll be specifying values for a resource constrained environment.

$ helm install yb-demo yugabytedb/yugabyte \
--set resource.master.requests.cpu=1,resource.master.requests.memory=1Gi,\
resource.tserver.requests.cpu=1,resource.tserver.requests.memory=1Gi,\
enableLoadBalancer=True --namespace yb-demo --wait
Enter fullscreen mode Exit fullscreen mode

To check the status of the cluster, execute the below command:

$ kubectl get services --namespace yb-demo
Enter fullscreen mode Exit fullscreen mode

check the status of the YugabyteDB cluster Liquibase GKE tutorial

Note the external-IP for yb-tserver-service which we are going to use to establish a connection between YugabyteDB and Liquibase. From the screenshot above we can see that the IP is 34.72.XX.XX and the YSQL port is 5433.

Step 2: Creating the Northwind sample database

The next step is to download a sample schema and data. You can find a variety of sample databases that are compatible with YugabyteDB in our Docs. For the purposes of this tutorial we are going to use the Northwind sample database. The Northwind database contains the sales data for a fictitious company called “Northwind Traders,” which imports and exports specialty foods from around the world. The Northwind database is an excellent tutorial schema for a small-business ERP, with customers, orders, inventory, purchasing, suppliers, shipping, employees, and single-entry accounting.

Connect to the yb-tserver-pod by running the following command:

$ kubectl exec -n yb-demo -it yb-tserver-0 /bin/bash
Enter fullscreen mode Exit fullscreen mode

To download the schema and data files, run the following commands:

$ wget https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/sample/northwind_ddl.sql
$ wget https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/sample/northwind_data.sql
Enter fullscreen mode Exit fullscreen mode

Note: If the Google Cloud Shell tells you that the wget command does not exist, you can execute:

$ yum install wget -y
Enter fullscreen mode Exit fullscreen mode

To connect to the YSQL service exit out of the pod shell and run the following command:

$ exit 
$ kubectl exec -n yb-demo -it yb-tserver-0 -- ysqlsh -h yb-tserver-0.yb-tservers.yb-demo
Enter fullscreen mode Exit fullscreen mode

Create a database and connect to it using the following commands:

yugabyte=# CREATE DATABASE northwind;
northwind=# \c northwind;
Enter fullscreen mode Exit fullscreen mode

We can now create the database objects and load them with data using the files we downloaded to yb-tserver-pod using the following commands:

northwind=# \i 'northwind_ddl.sql';
northwind=# \i 'northwind_data.sql';
Enter fullscreen mode Exit fullscreen mode

Verify that the tables are created by running the following command:

northwind-# \d

                 List of relations
 Schema |          Name          | Type  |  Owner
--------+------------------------+-------+----------
 public | categories             | table | yugabyte
 public | customer_customer_demo | table | yugabyte
 public | customer_demographics  | table | yugabyte
 public | customers              | table | yugabyte
 public | employee_territories   | table | yugabyte
 public | employees              | table | yugabyte
 public | order_details          | table | yugabyte
 public | orders                 | table | yugabyte
 public | products               | table | yugabyte
 public | region                 | table | yugabyte
 public | shippers               | table | yugabyte
 public | suppliers              | table | yugabyte
 public | territories            | table | yugabyte
 public | us_states              | table | yugabyte
(14 rows)
Enter fullscreen mode Exit fullscreen mode

Verify we have data by issuing a simple SELECT:

northwind=# SELECT count(*) FROM products;

 count
-------
    77
(1 row)
Enter fullscreen mode Exit fullscreen mode

By default, a YugabyteDB installation doesn’t have a password setup for the default yugabyte user. Specifying one is done the same way you’d do it in PostgreSQL.

northwind=# ALTER ROLE yugabyte WITH PASSWORD 'password';
Enter fullscreen mode Exit fullscreen mode

Step 3: Install and configure Liquibase

Locate the appropriate version of Liquibase for your platform from their downloads page.

https://www.liquibase.org/download

For the purposes of this demo, we’ll be installing Liquibase locally on a Mac.

Create a Liquibase project

On my Mac I created a new folder called LiquibaseYugabyteDB.

Download the appropriate PostgreSQL driver

Visit https://jdbc.postgresql.org/download.html and download the appropriate JDBC driver for your environment. I downloaded JDBC 4.2 (42.2.14) and placed it in the LiquidbaseYugabyteDB folder.

Create a changelog file

Next, in this same folder I created a file called dbchangelog.xml file. This changelog contains the sequence of changesets, each one of which makes small changes to the structure of the database. Add the following boilerplate to this file to get started.

<?xml version="1.0" encoding="UTF-8"?>  
  <databaseChangeLog  
    xmlns="http://www.liquibase.org/xml/ns/dbchangelog"  
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
  xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
    http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd">    
  </databaseChangeLog>
Enter fullscreen mode Exit fullscreen mode

Create a properties file

Now in the same directory, create a file called liquibase.properties and add the following content to it that is reflective of your environment.

changeLogFile:  ../LiquibaseYugabyteDB/dbchangelog.xml  
    url:  jdbc:postgresql://34.72.XX.XX:5433/northwind
    username:  yugabyte  
    password:  password
    driver:  org.postgresql.Driver  
    classpath:  ../LiquibaseYugabyteDB/postgresql-42.2.14.jar
Enter fullscreen mode Exit fullscreen mode

Step 4: Create a changeset and verify the results

Create a changeset

To create a changeset, return to the dbchangelog.xml file and add the following copy. In the changeset we are going to create a promotions table in the northwind database with columns for the name of the promotion and the discounted amount.

<?xml version="1.0" encoding="UTF-8"?>  
  <databaseChangeLog  
    xmlns="http://www.liquibase.org/xml/ns/dbchangelog"  
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
    xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
    http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd">  

    <changeSet  id="1"  author="jguerrero">  
        <createTable  tableName="promotions">  
            <column  name="id"  type="serial">  
                <constraints  primaryKey="true"  nullable="false"/>  
            </column>  
            <column  name="name"  type="varchar(50)">  
                <constraints  nullable="false"/>  
            </column>  
            <column  name="discount"  type="numeric(3,2)"/>                  
        </createTable>  
   </changeSet>
  </databaseChangeLog>
Enter fullscreen mode Exit fullscreen mode

The SQL equivalent of the changeset above would be:

CREATE  TABLE  promotions  
  (id serial PRIMARY KEY,  
   name  VARCHAR(50) UNIQUE NULL,  
   discount  NUMERIC (3,2) 
  );
Enter fullscreen mode Exit fullscreen mode

Execute the changeset

Execute the update command locally in the directory you created to send the changeset to the database.

$ liquibase update
Liquibase Community 4.0.0 by Datical
Starting Liquibase at 11:00:28 (version 4.0.0 #19 built at 2020-07-13 19:45+0000)
Liquibase: Update has been successful.
Enter fullscreen mode Exit fullscreen mode

Verify the results

To verify that the promotions table was created we can open up a YSQL shell and insert two records into the table and select the data out.

INSERT INTO promotions 
(id, name, discount)
VALUES 
(DEFAULT, 'Memorial Day Promotion', 3.50),
(DEFAULT, 'International Coffee Day Promotion', 5.35);

SELECT * FROM promotions;
Enter fullscreen mode Exit fullscreen mode

verify that the promotions table was created YugabyteDB Liquibase GKE tutorial

Step 5: Managing changes

Along with the promotions table, you should now see two additional tables that Liquibase has created in the northwind database.

Northwind database YugabyteDB Liquibase GKE tutorial

The databasechangelog table

This table keeps a record of all the changesets that were deployed. This means that the next time you deploy again, the changesets in the changelog will be compared with the databasechangelog tracking table and only the new changesets that were not found in the databasechangelog will be deployed. You will notice that a new row was created in that table with the changeset information we have just deployed. For this example:

new row was created in that table with the changeset information YugabyteDB Liquibase GKE tutorial

The databasechangelock table

This table is used internally by Liquibase to manage access to the changelog table during deployment. So, nothing to see here!

Step 6: Issue a second changeset

Create a second changeset

Let’s create an additional changeset. In this case let’s add two more columns to the promotions table. A start_date column and a stop_date column so we know when a promotion starts and ends. Instead of XML, let’s use SQL to initiate the changes. To do this, create a file in your Liquibase project directory called changelog.sql.

Add the following commands to this file:

--liquibase formatted sql  
--changeset jimmy:1
ALTER TABLE promotions
ADD COLUMN start_date TIMESTAMP,
ADD COLUMN stop_date TIMESTAMP;
Enter fullscreen mode Exit fullscreen mode

Issue the following command to send the change to the database.

$ liquibase --changeLogFile=changelog.sql update
Enter fullscreen mode Exit fullscreen mode

Verify the change

We should now be able to see a second record in the databasechangelog tracking table.

see a second record in the databasechangelog tracking table YugabyteDB Liquibase GKE tutorial

We should also be able to see the new start_time and stop_date columns in the promotions table.

see the new start_time and stop_date columns in the promotions table YugabyteDB Liquibase GKE tutorial

Conclusion

That’s it! You now have a PostgreSQL-compatible, 3 node YugabyteDB cluster running on GKE whose changes are now being managed and tracked by Liquibase. To learn more about supported third-party PostgreSQL tools that work with YugabyteDB, check out the YugabyteDB Documentation or join our community Slack and hit us up there.

Top comments (0)

Read next

programmerraja profile image

Sherlock Holmes: The Case Of Missing Cookies

Boopathi -

utsav_sonkar profile image

Get Started with KUBERNETES(Including HELM): Basic to Intermediate

Utsav Sonkar -

soumya14041987 profile image

Kubectl Top command:-Secrets behind scenes

Soumya -

endykaufman profile image

Access to the site on NestJS and Angular by domain name with SSL certificate in Kubernetes via Ingress

ILshat Khamitov -