One of the most common deployments of services in k8s has always been a simple mysql-service. You google and you find a thousand examples out there.
But recently I started experimenting with debezium, and it requires a mysql with a custom my.cnf file in order to enable full binary logs.
First I deployed this service on my minikube with no issues at all:
apiVersion: v1
kind: Service
metadata:
name: mysql
spec:
ports:
- port: 3306
clusterIP: None
selector:
app: mysql
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pvc
labels:
app: mysql
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: mysql
spec:
selector:
matchLabels:
app: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: mysql
spec:
containers:
- image: debezium/example-mysql:0.10
name: mysql
env:
# Use secret in real usage
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass
key: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pvc
But to my surprise, it would not start on GKE. Instead I would get this:
> [ERROR] --initialize specified but the data directory has files in it. Aborting.
What? Isn't Kubernetes meant to be the fabric on which we could easily port our workloads across different providers with no issues?
After a good amount of googling, it turns out that GKE Persistence Volume provider mounts the /var/lib/mysql volume with a lost+found directory that is not empty. That does not happen on minikube. And mysql does not like that (not when you are not running with default my.cnf at least).
So to fix this, you need to pass --ignore-db-dir=lost+found to the command line arguments of the container as in this snippet:
spec:
containers:
- image: debezium/example-mysql:0.10
args:
- "--ignore-db-dir=lost+found"
name: mysql
Since this took me a while to find, I thought it would be worth sharing for others that may bump into this.
Happy coding
Top comments (1)
Dude! Thank you! This was such a pain in the backend! Of course, it makes perfect sense that a Linux filesystem would have a lost+found folder. But I sometimes forget we're dealing with Linux thanks to all the K8s abstraction and distraction!