A few tips for Perl on Docker and Kubernetes

@davorg started a thread on the FB Perl Community and Perl Programmers about docker-perl, and as its maintainer, I'm overdue sharing some tips about it so might as well write them here now:

Use a custom name/tag for Perl images you use

As documented in the Docker Hub, one can use the perl image with something like

docker pull perl
docker run -it perl perl -E 'say "hi there from Docker!"'

While doing this is enough for simple or exploratory cases, it is inadequate for development or deployment scenarios, as perl alone will pull the latest Perl image build of the latest supported Perl version (which is 5.32.0 now but could be different later on.) Fortunately, docker-perl provides tags to indicate specific versions (as well as options like :threaded) or size variants (like :slim) so one can also do

docker pull perl:5.32-buster
docker run -it perl:5.32-buster perl -E 'say qq{hi again!}'

There is another concern though: while Docker images have tags, these tags are floating in the sense that they don't always point to the same underlying image layer at creation time, as tags can be updated to point to another layer. This is best seen in the :latest tag which Docker uses by default when calling images with their bare names, but this behavior is also present in any other tags like :5 which is pointing currently to :5.32.0, but might later be updated to point to :5.32.1 or even :5.34.0, when the perl manifest on docker-library/official-images is updated.

These tags also get updated indirectly by way of updates on their base images; as docker-perl uses the buildpack-deps or debian:slim base images, when these get updated for security patches, these patches will eventually make their way into the perl images as well through the official-images build tooling.

For cases that require finer control of images, it might be helpful then to use a custom image name, tag, or both, when using docker-perl:

docker pull perl:5.32-slim-threaded-buster
docker tag perl:5.32-slim-threaded-buster myorg/perl:5.32
docker run -it myorg/perl:5.32 perl -e 'printf "hello from myorg/perl v%vd!\n", $^V'

This allows for some independence for updates and development between the perl published on Docker Hub and myorg's custom perl image:

# docker build -t myorg/myapp:dev .
FROM myorg/perl:5.32
ADD . /app
WORKDIR /app
RUN cpanm --installdeps .
CMD ["perl", "myapp.pl"]
...

Consider keeping a local/private registry to host Perl images

Using the perl images (especially like in the previous section) means that these will be copied into the local Docker host image storage, usually in /var/lib/docker/images but could vary depending on what storage driver your host is using. While this is usually enough in simple cases, consider that the base perl image is usually big (around 700-800MB unpacked,) so pulling this image afresh over multiple container hosts on a network will probably be wasteful.

Hence it is usually recommended to have some kind of local or private registry in the host or on a network if frequently working with containers (and more so if internet connectivity is slower/expensive.) There are several approaches to do this, and I've written about these before in my old blog:

• Composing a Docker Hub mirror and private registry
• Mirroring the Docker Hub

Nowadays though, there are more options for using local/private registries: for example, if you have a Kubernetes cluster running in your network (or even as a small setup like microk8s, or an internal Docker container cluster using KinD or k3d,) there's usually an addon
that enables a local registry on the cluster. Here's an example for microk8s, which I extended for my own local network to be served under a TLS ingress:

microk8s enable registry
microk8s kubectl apply -f - <<REGISTRY_SVC
apiVersion: v1
kind: Service
metadata:
  name: registry
  namespace: default
spec:
  # microk8s deploys registry in its own namespace, so reach out
  externalName: registry.container-registry.svc.cluster.local
  ports:
  - name: registry
    port: 5000
    protocol: TCP
    targetPort: 5000
  type: ExternalName
REGISTRY_SVC
microk8s kubectl apply -f - <<REGISTRY_INGRESS
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: registry
  namespace: default
spec:
  rules:
  # I don't own home.example btw, just for illustration;
  # I use globally-accessible subdomains pointing to private IP
  - host: registry.home.example
    http:
      paths:
      - backend:
          serviceName: registry
          servicePort: 5000
  tls:
  - hosts:
    - registry.home.example
    secretName: wildcard-home-example
REGISTRY_INGRESS

I have a LetsEncrypt wildcard SSL cert installed and maintained on my microk8s via cert-manager, so this lets me push to this registry anywhere on my local network as

docker push myorg/perl:5.32 registry.home.example/myorg/perl:5.32
docker push myorg/myapp:dev registry.home.example/myorg/myapp:dev

and on somewhere else which might be running Podman instead:

podman run -it registry.home.example/myorg/perl:5.32 perl -V
podman run -d registry.home.example/myorg/myapp:dev

It also works on authoring Dockerfiles, especially useful when implementing GitOps workflows:

# docker build -t myorg/mojo:8.65
# docker push myorg/mojo:8.65 registry.home.example/myorg/mojo:8.65
FROM registry.home.example/myorg/perl:5.32
RUN cpanm Mojolicious@8.65
EXPOSE 3000
CMD ["/usr/local/bin/mojo", "daemon"]

For more sophisticated/production deployments, consider using Harbor or vendors such as Amazon ECR, Google Container Registry or Red Hat Quay.

Use container entrypoint for proper signals handling in Perl

This is already documented in the official-images docs for perl, but bears repeating here: containers don't provide a parent init process by default, which means that when starting a new container, any ENTRYPOINT or CMD set in the image the container will boot from will usually become the parent process within this container. This is important to note here for Perl (and also for other languages) as these tend to fall back to their parent init process for handling signals, so without this, it can appear to become unresponsive:

docker run perl:5.32 perl -E 'sleep 300'
^C
[refuses to die, even if sent SIGINT like above]

This is particularly important for Kubernetes, as its container pod lifecycle can use signals to probe for liveness or readiness of containers, and a running pod/container can seem to "hang" around indefinitely if these signals aren't handled correctly.

For perl, one needs to install a %SIG handler if they want/need to run perl directly as PID 1:

docker run perl:5.32 perl -E '$SIG{TERM} = sub { $sig++; say "recv TERM" }; sleep 300; say "waking up" if $sig'
^C
waking up
recv TERM

For most normal deployments however, it is easier to use a tool like tini, catatonit, or dumb-init for the container's ENTRYPOINT, alongside your perl script/executable in CMD, for example this small Mojolicious demo:

# docker build -t mojo-with-tini:dev .
FROM perl:5.32-buster
RUN apt-get update && \
    apt-get install -y --no-install-recommends tini && \
    cpanm Mojolicious
ENTRYPOINT ["/usr/bin/tini", "--"]
EXPOSE 3000
CMD ["/usr/local/bin/mojo", "daemon"]

For Kubernetes, one can also set the command and args for a pod:

kubectl apply -f - <<MOJO_POD
apiVersion: v1
kind: Pod
metadata:
  name: mojo-with-tini-demo
  labels:
    purpose: demonstrate-perl-signal-handling
spec:
  containers:
  - name: mojo-with-tini-demo-container
    image: mojo-with-tini:dev
    # redundant as we already set these in the Dockerfile above,
    # but showing here for illustration
    command: ["/usr/bin/tini", "--"]
    args: ["/usr/local/bin/mojo", "daemon"]
MOJO_POD

Outro

There's a few more tips on the queue that I'd write more here but this post is getting rather long now, so maybe later.

I'd love to hear more about how people use docker-perl, and especially more about how make developing Perl projects on container environments easier!