Day 18 of 100 Days of Cloud: Migrating Data from a PostgreSQL Instance to a Docker Container

Welcome to Day 18 of our 100 Days of Cloud series! Today’s focus is on migrating PostgreSQL data from a remote server to a Docker container running PostgreSQL. This guide will walk you through system updates, PostgreSQL installation, data backup, Docker setup, and finally, restoring the database inside a Docker container.

Table of Contents

1. Update and Upgrade Your System
2. Install PostgreSQL
3. Backup PostgreSQL Database
4. Prepare for Docker Installation
5. Install Docker
6. Clone Git Repository and Set Up Docker Containers
7. Move Environment Configuration
8. Copy Backup File to Docker Container
9. Restore Database in Docker Container
10. Access Docker Container

1. Update and Upgrade Your System

Begin by updating your system to ensure that all existing packages are current. This helps avoid conflicts and ensures compatibility.

sudo apt update
sudo apt upgrade

• sudo apt update: Updates the list of available packages and their versions.
• sudo apt upgrade: Upgrades all installed packages to their latest versions.

2. Install PostgreSQL

Install PostgreSQL 16 to manage your databases.

sudo apt-get install -y postgresql-16

• sudo apt-get install -y postgresql-16: Installs PostgreSQL 16, automatically confirming any prompts.

To verify available PostgreSQL packages, use:

apt search postgres

• apt search postgres: Lists PostgreSQL packages and versions available in the repository.

3. Backup PostgreSQL Database

Create a backup of your PostgreSQL database from the remote server. Adjust the command according to your database details.

pg_dump -h verisafe.xxxxxxxxxxxxpx.us-east-1.rds.amazonaws.com -U verisafe -F c -b -v -f /home/ec2-user/verisafe/backup_file.dump postgres

• -h: Hostname of the PostgreSQL server.(The RDS endpoint in my case)
• -U: Username for database access.
• -F c: Specifies the custom format for the backup.
• -b: Includes large objects (blobs).
• -v: Enables verbose output.
• -f: Path where the backup file will be saved.

4. Prepare for Docker Installation

Before installing Docker, ensure you have the necessary tools and dependencies.

sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common

• apt-transport-https: Allows apt to handle HTTPS.
• ca-certificates: Manages SSL certificates.
• curl: Command-line tool for transferring data from or to a server.
• software-properties-common: Provides the add-apt-repository command for managing PPAs.

5. Install Docker

Add Docker’s official GPG key and repository, then install Docker Community Edition (CE).

First, add Docker’s GPG key:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

• curl -fsSL ... | sudo apt-key add -: Adds Docker’s GPG key to verify package integrity.

Add Docker’s repository:

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"

• sudo add-apt-repository ...: Adds Docker’s repository to your package sources.

Update the package list and install Docker CE:

sudo apt update
sudo apt install docker-ce

• sudo apt install docker-ce: Installs Docker Community Edition.

Verify the Docker installation:

apt-cache policy docker-ce

• apt-cache policy docker-ce: Displays the Docker CE version and installation status.

6. Clone Git Repository and Set Up Docker Containers

Clone the repository that contains your Docker setup and bring up the Docker containers.

Check your Git installation:

git --version

• git --version: Displays the installed Git version.

Clone the repository:

git clone https://github.com/dita-daystaruni/verisafe.git

• git clone ...: Downloads the repository to your local machine.

Navigate to the repository:

cd verisafe

• cd verisafe: Changes directory to the cloned repository.

Start the Docker containers:

sudo docker compose up

• sudo docker compose up: Launches the Docker containers defined in docker-compose.yml.

7. Move Environment Configuration

If your project includes an example.env file, rename it to .env to configure your environment.

Edit the example.env file if necessary:

nano example.env

• nano example.env: Opens the file in the nano text editor.

Rename the file:

mv example.env .env

• mv example.env .env: Renames example.env to .env to be used by Docker Compose.

Restart Docker containers to apply the configuration changes:

sudo docker compose up

8. Copy Backup File to Docker Container

Transfer the backup file into the Docker container where PostgreSQL is running.

docker cp /home/ubuntu/verisafe/backup_file.dump verisafe-postgres-1:/backup_file.dump

• docker cp ...: Copies the backup file to the Docker container.

If you encounter permissions issues, use sudo:

sudo docker cp /home/ubuntu/verisafe/backup_file.dump verisafe-postgres-1:/backup_file.dump

9. Restore Database in Docker Container

Access the Docker container and restore the database using pg_restore.

First, access the container:

docker exec -it verisafe-postgres-1 bash

• docker exec -it ... bash: Opens an interactive bash shell inside the container.

Restore the database:

pg_restore -U verisafe -d verisafe /backup_file.dump

• -U postgres: PostgreSQL user to perform the restoration.
• -d your_database_name: The target database where the data will be restored.
• /backup_file.dump: Path to the backup file inside the container.

10. Access Docker Container

To perform operations within the Docker container, you might need to access its shell.

docker exec -it verisafe-postgres-1 bash

• docker exec -it ... bash: Opens a bash shell in the specified container.

If necessary, use sudo:

sudo docker exec -it verisafe-postgres-1 bash

Conclusion

In today’s tutorial, we covered the entire process of migrating PostgreSQL data from a remote server to a Docker container. This included updating your system, installing PostgreSQL, creating and restoring backups, setting up Docker, and managing Docker containers.

Feel free to revisit any steps as needed and ensure everything is correctly configured. Stay tuned for more cloud management tips and techniques in our ongoing series. Happy migrating!