Deploy Postgres with replications Database in a minutes, with Terraform (Infrastructure as Code)

#aws #terraform #postgres #cloudskills

Hi everyone !

So many thanks for your supports in my last blog post:
https://dev.to/simerca/use-native-gps-with-vuejs-for-ios-and-android-1kk7

Today ! It's a great time for you !
I'will show you how to deploy Postgres database in a minutes with Terraform and AWS, with Primary / Replica and Hourly backup !

So first, whats is Terraform ?
Terraform it's a tool for Infrastructure as Code, it's very simple to use it, for exemple, you just write the configuration of you need and execute it with one command like :
terraform apply

First install Terraform:
https://learn.hashicorp.com/tutorials/terraform/install-cli

Next, start the configuration files

please use my public repo in gitlab:
git clone https://gitlab.com/Simerca/terraform-postgres

Next, see the first file and edit with your own configuration
variables.tf

// Default AWS Access Credentials
variable "access_key" {
    default = "****"
}
variable "secret_key" {
    default = "*****"
}
variable "region" {
    default = "eu-west-3"
}
variable "bucket" {
    default = "bucket"
}
variable "aws_ami" {
    default = "ami-00798d7180f25aac2"
}
variable "aws_type" {
    default = "t2.micro"
}
// Other Configuration
variable "workspace" {
    default = "user"
}
variable "password" {
    default = "*****"
}

So next see the main file
app-instances.tf

/* Setup our aws provider */
provider "aws" {
  access_key  = "${var.access_key}"
  secret_key  = "${var.secret_key}"
  region      = "${var.region}"
}

resource "aws_instance" "primary_1" {
  ami           = "${var.aws_ami}"
  instance_type = "${var.aws_type}"
  security_groups = ["${aws_security_group.swarm.name}"]
  key_name = "${aws_key_pair.deployer.key_name}"
  connection {
    host = self.public_ip
    user = "ec2-user"
    private_key = "${file("${path.module}/id_rsa.pem")}"
  }
  provisioner "remote-exec" {
    inline = [
      "sudo yum update -y",
      "sudo yum install git -y",
      "sudo amazon-linux-extras install docker -y",
      "sudo service docker start",
      "sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose",
      "sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose",
      "sudo chmod +x /usr/local/bin/docker-compose;",
      "sudo docker network create web",
      "sudo docker run --restart=unless-stopped --name=postgres -d -p 5432:5432 -e POSTGRES_DB=${var.workspace} -e POSTGRES_USER=${var.workspace} -e POSTGRES_PASSWORD=${var.password} -v $(pwd)/data:/var/lib/postgresql/data postgres",
      "sudo docker run -d --restart=unless-stopped -p 3000:3000 -e PW2_ADHOC_CONN_STR=\"postgresql://${var.workspace}:${var.password}@${self.public_ip}:5432/${var.workspace}\" -e PW2_GRAFANAUSER=admin -e PW2_GRAFANAPASSWORD=admin -e PW2_ADHOC_CONFIG=exhaustive -e PW2_ADHOC_CREATE_HELPERS=true --name pw2 cybertec/pgwatch2-postgres",
      "sudo docker run -d --restart=unless-stopped --name=postgres_backup -e SCHEDULE='@hourly' -e S3_REGION=${var.region} -e S3_ACCESS_KEY_ID=${var.access_key} -e S3_SECRET_ACCESS_KEY=${var.secret_key} -e S3_BUCKET=${var.bucket} -e POSTGRES_DATABASE=${var.workspace} -e POSTGRES_USER=${var.workspace} -e POSTGRES_HOST=${self.public_ip} -e POSTGRES_PASSWORD=${var.password} -e S3_PREFIX=${var.workspace} -e POSTGRES_EXTRA_OPTS='--schema=public --blobs' schickling/postgres-backup-s3"
    ]
  }
  tags = { 
    Name = "${var.workspace}-primary"
  }
}
resource "aws_instance" "replica_1" {
  ami           = "ami-00798d7180f25aac2"
  instance_type = "${var.aws_type}"
  security_groups = ["${aws_security_group.swarm.name}"]
  key_name = "${aws_key_pair.deployer.key_name}"
  connection {
    host = self.public_ip
    user = "ec2-user"
    private_key = "${file("${path.module}/id_rsa.pem")}"
  }
  provisioner "remote-exec" {
    inline = [
      "sudo yum update -y",
      "sudo yum install git -y",
      "sudo amazon-linux-extras install docker -y",
      "sudo service docker start",
      "sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose",
      "sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose",
      "sudo chmod +x /usr/local/bin/docker-compose;",
      "sudo docker network create web",
      "sudo docker run --restart=unless-stopped --name=postgres -d -p 5432:5432 -e POSTGRES_DB=${var.workspace} -e POSTGRES_USER=${var.workspace} -e POSTGRES_PASSWORD=${var.password} -v $(pwd)/data:/var/lib/postgresql/data postgres",
      "sudo docker run -d --restart=unless-stopped -p 3000:3000 -e PW2_ADHOC_CONN_STR=\"postgresql://${var.workspace}:${var.password}@${self.public_ip}:5432/${var.workspace}\" -e PW2_GRAFANAUSER=admin -e PW2_GRAFANAPASSWORD=admin -e PW2_ADHOC_CONFIG=exhaustive -e PW2_ADHOC_CREATE_HELPERS=true --name pw2 cybertec/pgwatch2-postgres",
      "sudo docker run -d --restart=unless-stopped --name=postgres_backup -e SCHEDULE='@hourly' -e S3_REGION=${var.region} -e S3_ACCESS_KEY_ID=${var.access_key} -e S3_SECRET_ACCESS_KEY=${var.secret_key} -e S3_BUCKET=${var.bucket} -e POSTGRES_DATABASE=${var.workspace} -e POSTGRES_USER=${var.workspace} -e POSTGRES_HOST=${self.public_ip} -e POSTGRES_PASSWORD=${var.password} -e S3_PREFIX=${var.workspace} -e POSTGRES_EXTRA_OPTS='--schema=public --blobs' schickling/postgres-restore-s3"
    ]
  }
  tags = { 
    Name = "${var.workspace}-replica"
  }
}

So ! Try to understand how it works

Step 1 :

Use the variable to set the configuration, very simple to do this.

provider "aws" {
  access_key  = "${var.access_key}"
  secret_key  = "${var.secret_key}"
  region      = "${var.region}"
}

You can use environment variable with the schema :

TF_VAR_access_key=XXXXXX terraform apply

Step 2:

Launch instance and basic configuration with set of command lines

resource "aws_instance" "primary_1" {
  ami           = "${var.aws_ami}"
  instance_type = "${var.aws_type}"
  security_groups = ["${aws_security_group.swarm.name}"]
  key_name = "${aws_key_pair.deployer.key_name}"
  connection {
    host = self.public_ip
    user = "ec2-user"
    private_key = "${file("${path.module}/id_rsa.pem")}"
  }
  provisioner "remote-exec" {
    inline = [
      "sudo yum update -y",
      "sudo yum install git -y",
      "sudo amazon-linux-extras install docker -y",
      "sudo service docker start",
      "sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose",
      "sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose",
      "sudo chmod +x /usr/local/bin/docker-compose;",
      "sudo docker network create web",
      "sudo docker run --restart=unless-stopped --name=postgres -d -p 5432:5432 -e POSTGRES_DB=${var.workspace} -e POSTGRES_USER=${var.workspace} -e POSTGRES_PASSWORD=${var.password} -v $(pwd)/data:/var/lib/postgresql/data postgres",
      "sudo docker run -d --restart=unless-stopped -p 3000:3000 -e PW2_ADHOC_CONN_STR=\"postgresql://${var.workspace}:${var.password}@${self.public_ip}:5432/${var.workspace}\" -e PW2_GRAFANAUSER=admin -e PW2_GRAFANAPASSWORD=admin -e PW2_ADHOC_CONFIG=exhaustive -e PW2_ADHOC_CREATE_HELPERS=true --name pw2 cybertec/pgwatch2-postgres",
      "sudo docker run -d --restart=unless-stopped --name=postgres_backup -e SCHEDULE='@hourly' -e S3_REGION=${var.region} -e S3_ACCESS_KEY_ID=${var.access_key} -e S3_SECRET_ACCESS_KEY=${var.secret_key} -e S3_BUCKET=${var.bucket} -e POSTGRES_DATABASE=${var.workspace} -e POSTGRES_USER=${var.workspace} -e POSTGRES_HOST=${self.public_ip} -e POSTGRES_PASSWORD=${var.password} -e S3_PREFIX=${var.workspace} -e POSTGRES_EXTRA_OPTS='--schema=public --blobs' schickling/postgres-backup-s3"
    ]
  }
  tags = { 
    Name = "${var.workspace}-primary"
  }
}

Be carreful to this line

private_key = "${file("${path.module}/id_rsa.pem")}"

Yoo need to create new SSL key for access to your instances

ssh-keygen -t rsa -N "" -b 2048 -C "assignment"
ssh-keygen -y -f ~/.ssh/id_rsa.pem > ~/.ssh/id_rsa.pub

After,
The command line execute command in order.

Install dependencies
Install docker
Docker run some containers

We run this set twice, 1 primary, 1 replica (master/slave)

Step 3

Open security group

In other file we have :
security-group.tf

/* Default security group */
resource "aws_security_group" "swarm" {
  name = "swarm-group-${var.workspace}"
  description = "Default security group that allows inbound and outbound traffic from all instances in the VPC"

  ingress {
    from_port   = "0"
    to_port     = "0"
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
    self        = true
  }

  ingress {
    from_port = 22
    to_port   = 22
    protocol  = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port = 5432
    to_port   = 5432
    protocol  = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  ....

Here we open the needed port to access on our Postgres

Lets Try !

First try your configuration (it's just a test)
terraform plan

Next, you can use this exemple command line to launch Postgres instance inside your AWS account :

TF_VAR_access_key=XXXX TF_VAR_secret_key=XXXX TF_VAR_user=DBUser TF_VAR_password=SomePassword terraform apply

This command launch:

2 Ec2 instances (Primary /Replica)
2 PostgresSQL Database on port 5432 and with the user : DBUser and password : SomePassword
2 Monitoring container with Grafana
2 Hourly backup on S3 Bucket(Primary to Replica)

and when you want to destroy all instances

terraform destroy

very simple no ?