AWS makes regular automated snapshots of your hosted Elasticsearch services. But if you for whatever reason wanna create a manual snapshot, you can only do this scripted. At the time of writing this there's no way of doing this from the AWS console.
This Ruby script will help you creating manual snapshots of an AWS hosted Elasticsearch service using signed requests.
I had to create a manual snapshot of my ES service to migrate to a new VPC. You cannot change the VPC for an existing service, so I had no other option than creating a new ES service, manually snapshotting my old service and restoring this snapshot on the new service. And btw this is the documented way to do it by AWS.
Before you begin, start by reading the guide from AWS to familiarize yourself with the process:
Creating Amazon ES Index Snapshots
These are the steps you need to take to prepare yourself to create the manual snapshots and restoring them.
- Start by creating an AWS S3 bucket for your snapshots.
- Next go to the AWS IAM console and create a new policy. The content of the policy is also listed in the above link:
{
"Version": "2012-10-17",
"Statement": [{
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::s3-bucket-name"
]
},
{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::s3-bucket-name/*"
]
}
]
}
Now create an AWS IAM Role - this will be used in the script to permit the snapshot.
- The console will ask for a use case. Select
DMSfor now - we'll change this later as Elasticsearch is not listed. - Click next to set the role permissions. Here you can search for the policy you just created.
- Name your role something meaningful and create it. You'll have to reference it in the script.
- Once created find your role and open it in the console.
- Click on the
Trust Relationsshipstab and clickEdit trust relationship - Replace the existing policy with the following:
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "es.amazonaws.com"
},
"Action": "sts:AssumeRole"
}]
}
The script
Now we are ready to configure the script.
I'm using the aws-sdk-elasticsearchservice gem - also listed in the script. Start by installing it:
gem install aws-sdk-elasticsearchservice
You need to do some small configuration in the script. The AwsEsSnapshot class defines a number of constants that you should replace with your own values:
- AWS bucket name and region (notice you can only migrate Elasticsearch to a new service in same region)
- Role arn for your freshly created role. It's listed in the top of the summary screen in the AWS console.
- Finally you'll have to name a repository and snapshot. For a one time migration these values are not that important, but if you wanna continue to create manual snapshots you might wanna set some meaningful names.
Finally I'm fetching access key id and secret from environment variables. You can run your script like this:
AWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=xxx ruby miration.rb
gem 'aws-sdk-elasticsearchservice'
require 'aws-sdk-elasticsearchservice'
class AwsEsSnapshot
attr_accessor :logger
REPO = 'my-repository-name'
SNAPSHOT_NAME = 'my-snapshot-name'
ROLE_ARN = 'arn:aws:iam::123456789012:role/TheSnapshotRole'
BUCKET = 's3-bucket-name'
REGION = 'eu-west-1'
ACCESS_KEY_ID = ENV['AWS_ACCESS_KEY_ID']
SECRET_ACCESS_KEY = ENV['AWS_SECRET_ACCESS_KEY']
def initialize(logger: Logger.new(STDOUT))
@logger = logger
@logger.info('AwsEsSnapshot') { <<~HEREDOC
Initialized with:
Repository: #{REPO}
Snapshot name: #{SNAPSHOT_NAME}
Role ARN: #{ROLE_ARN}
S3 bucket: #{BUCKET}
AWS Region: #{REGION}
HEREDOC
}
end
# Creating a signed request
def signer
@signer ||= Aws::Sigv4::Signer.new(
service: 'es',
region: REGION,
access_key_id: ACCESS_KEY_ID,
secret_access_key: SECRET_ACCESS_KEY
)
end
def signature(method:, url:, payload: nil)
signer.sign_request(
http_method: method.to_s.upcase,
url: url,
body: payload ? payload.to_json : nil
)
end
def request(method: :put, host: nil, path:, payload: nil)
url = host + '/' + path
signature = signature(method: method, url: url, payload: payload)
uri = URI(url)
logger.debug("AwsEsSnapshot#request") { "#{method.to_s.upcase}ing #{payload} to #{url}"}
Net::HTTP.start(uri.host, uri.port, :use_ssl => true) do |http|
request = send(method, uri)
request.body = payload.to_json if payload
request['Host'] = signature.headers['host']
request['X-Amz-Date'] = signature.headers['x-amz-date']
request['X-Amz-Security-Token'] = signature.headers['x-amz-security-token']
request['X-Amz-Content-Sha256']= signature.headers['x-amz-content-sha256']
request['Authorization'] = signature.headers['authorization']
request['Content-Type'] = 'application/json'
response = http.request request
if response.code.to_i == 200
logger.debug("AwsEsSnapshot#request") { response.body }
else
logger.fatal("AwsEsSnapshot#request") { response.body }
raise "Received unexpected status code #{response.code} in response."
end
response
end
end
def post(uri)
Net::HTTP::Post.new uri
end
def put(uri)
Net::HTTP::Put.new uri
end
def get(uri)
Net::HTTP::Get.new uri
end
def register(host:)
payload = {
type: 's3',
settings: {
bucket: BUCKET,
region: REGION,
role_arn: ROLE_ARN
}
}
request host: host, path: "_snapshot/#{REPO}", payload: payload
logger.debug("AwsEsSnapshot#request") { "Registered S3 bucket on #{host}" }
end
def create_snapshot(host:)
# First we'll ensure our S3 bucket is registered
register(host: host)
request host: host, path: "_snapshot/#{REPO}/#{SNAPSHOT_NAME}"
logger.debug("AwsEsSnapshot#request") { "Completed snapshot for #{SNAPSHOT_NAME}" }
end
def restore(host:)
# First we'll ensure our S3 bucket is registered
register(host: host)
payload = {
indices: "-.kibana*,-.opendistro_security",
include_global_state: false,
ignore_unavailable: true,
}
request method: :post, host: host, path: "_snapshot/#{REPO}/#{SNAPSHOT_NAME}/_restore", payload: payload
logger.debug("AwsEsSnapshot#request") { "Restored snapshot for #{SNAPSHOT_NAME}" }
end
def status(host:)
request method: :get, host: host, path: "_snapshot/_status"
end
end
Now you are ready to use the script.
By default it's logging to STDOUT, but you can initialize with a custom logger:
es = AwsEsMigration.new(logger: MyCustomLogger.new)
There are tree methods of interest.
AwsEsMigration#create_snapshot
Expects a host: argument. This is your existing Elasticsearch service endpoint URL from where you wanna create a snapshot.
es.create_snapshot(host: 'https://my-legacy-endpoint-url.eu-west-1.es.amazonaws.com')
AwsEsMigration#status
will tell you the current snapshot progress. It's my experience the snapshot is completed fairly quickly.
Like litereally in minutes for a 10gb ES storage.
puts es.status(host: 'https://my-legacy-endpoint-url.eu-west-1.es.amazonaws.com').body
AwsEsMigration#restore
Finally if you like me need to restore a snapshot:
es.restore(host: 'https://my-new-endpoint-url.eu-west-1.es.amazonaws.com')
I hope this will help somebody out there :)
Top comments (0)