Using acme.sh & Let’s Encrypt on IBM i

I recently deployed a Node.js application on IBM i and wanted to use Let’s Encrypt for our certificates. We ran into a few bumps along the way. This post is going to go over the process of installing acme.sh and actually generating certificates.

What is acme.sh

Acme.sh is an open-source shell script to automatically call out to Let’s Encrypt to generate a certificate for you to use in your application. The quote on the GitHub repository is “It's probably the easiest & smartest shell script to automatically issue & renew the free certificates from Let's Encrypt.”. The repo can be found at https://github.com/Neilpang/acme.sh/.

First things first, you should install acme.sh with an SSH shell. As a reminder:

1. You can start the SSH daemon with STRTCPSVR SERVER(*SSHD) on the 5250 command line.
2. You can then log into the IBM i using ssh user@youribmi from your machine terminal.
3. You will want to run these commands in the bash shell.

To install:

1. Clone the repo: git clone https://github.com/Neilpang/acme.sh.git
2. Run the install script:

cd acme.sh
./acme.sh --install --force
alias acme.sh=~/.acme.sh/acme.sh

To use acme.sh you also need to have your domain DNS settings set up to point to your IBM i.

Can not init api (error code: 77)

I was following some old slides Aaron Bartell had written when doing this. When I first tried to install a certificate, I was getting a libcurl error.

bash-4.2$ export CURL_CA_BUNDLE=~/certs/cacert.pem
bash-4.2$ acme.sh --issue -d website.com -w /mywebsite/public
Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 77
Can not init api.

I imagine I was getting this error because the file I am referencing (cacert.pem) did not exist. After some Googling, I found this GitHub issue on IbmiToolkit repository. There are two important notes on this thread.

1. “The certificate bundle in question is typically provided on Linux distributions by default, but this is naturally not the case for IBM i.” - great! /s
2. phpdave sharing this pseudocode about finding the local CA.

phpdave’s script actually contains an important link on line 19: http://curl.haxx.se/ca/cacert.pem. I thought: if I download this certificate and make curl use it then I wonder if it will solve my issue.

bash-4.2$ wget http://curl.haxx.se/ca/cacert.pem
bash-4.2$ export CURL_CA_BUNDLE=/home/USER/cacert.pem
bash-4.2$ acme.sh --issue -d website.com -w /mywebsite/public
[Thu Jan 24 15:04:02 EST 2019] Creating domain key
....

How Let’s Encrypt authenticates against your server

Notice in the ache.sh command, we include a URL to our website and a path to a public folder. We do this because acme.sh will

1. Create a directory in the public folder: /mywebsite/public/.well-known/acme-challenge
2. Create a random file in the acme-challenge folder with a secret key in it
3. Tell Let’s Encrypt to compare your key at website.com/.well-known/acme-challange/randomkey with the key from the acme.sh script
4. If it’s successful, then your keys will be generated.

This does mean that the “public” folder should be accessible on your web server. For example, if you were building a Node.js app with Express and it had this folder structure:

mywebsite/
    index.js
    node_modules/
    public/
        index.html

Your index.js would tell the web server that the public directory is for static files: app.use(express.static('public'))

So when acme.sh generates /mywebsite/public/.well-known/acme-challenge/randomkey, it will be accessible via website.com/.well-known/acme-challange/randomkey

Using the generated keys

If all is successful when you run acme.sh it should generate a key and certificate for you:

bash-4.2$ acme.sh --issue -d website.com -w /mywebsite/public/
…
Your cert is in  /home/USER/.acme.sh/website.com/website.com.cer
Your cert key is in  /home/USER/.acme.sh/website.com/website.com.key
The intermediate CA cert is in  /home/USER/.acme.sh/website.com/ca.cer
And the full chain certs is there:  /home/USER/.acme.sh/website.com/fullchain.cer

You can then copy these keys to a location where they’re useful to you: cp /home/USER/.acme.sh/website.com/ /mywebsite/letsencrypt. In your Node.js Express app, you can then reference the key and certificate when creating your HTTPS server:

var express = require('express');
var https = require('https');
var http = require('http');
var fs = require('fs');

// This line is from the Node.js HTTPS documentation.
var options = {
  key: fs.readFileSync(path.join('letsencrypt', 'website.com.key')),
  cert: fs.readFileSync(path.join('letsencrypt', 'website.com.cer'))
};

// Create a service (the app object is just a callback).
var app = express();

// Create an HTTP service.
http.createServer(app).listen(80);
// Create an HTTPS service identical to the HTTP service.
https.createServer(options, app).listen(443);

ERR_SSL_VERSION_INTERFERENCE in Chrome

I noticed when we got the certificates working in our app, the website wouldn’t load in Chrome with the ERR_SSL_VERSION_INTERFERENCE error. After some research (trusty Google!) it looked like Node.js 8 doesn’t support TLS 1.3 (as noted in the second paragraph here).

Luckily, we also have Node.js 10 on IBM i - so a simple uninstall of Node.js 8 and install of Node.js 10 (with yum of course!) we can get around this problem. You may also have to update some of your packages if you upgrade to Node.js 10.