CockroachDB and persistent volumes
When deployed on Kubernetes or OpenShift, CockroachDB uses persistent volumes (PVs) to store DB data, metadata, state-data, user-data, log files, configuration files. These volumes are typically file-system mounts that are mapped to disks/SSDs where the data is physically saved in a distributed fashion. When you operate CockroachDB and run queries, data must be read or written where these operations translate to frequent or continuous disk reads & writes.
Managing the disk: IOPS & throughput
On cloud-managed orchestrators, when you read or write data to disk (PVs), this consumes IOPS and utilizes some of the available IO throughput. These are limiting factors that can result bandwidth saturation, or worse, throttling by the cloud provider under heavier workloads. This condition can be identified by the combination of low CPU usage and high disk latencies, visualized through the CockroachDB UI console hardware dashboard metrics and charts.
Divide & conquer
To overcome these limitations, CockroachDB lets you take advantage of multiple, independent PVs to separate the destination of the cockroach runtime data. CockroachDB Logging is a good candidate to move out of the critical path by dedicating its own volume/storage. This will help with performance tuning since your SQL/schemas live on their own dedicated volume. In fact it's the production readiness recommendation to split the data from the logs into separate PVs.
Typical CockroachDB deployments
Most CockroachDB clusters implement a single PVC that is assigned to each node in a stateful set. Default configurations in both HELM and Operator managed environments create this 1:1 mapping as follows:
Our planned deployment with multiple PVs
By introducing a second PV dedicated for logs, we split the workload and effectively double the IO channels and allows for each to be independently configured. Storage for logs can be significantly reduced when compared to the cockroach-data PV since logs can be rotated/truncated while your business data can grow over time. This illustration highlights the logical infrastructure layout between nodes and PVs. 
…to the implementation
We need to make additions to the StatefulSet template along with custom log-configuration settings to direct CockroachDB logs into the new destination PV.
The logging “secret” configuration
This resource is the one-stop-shop for all your customized logging properties, including log sinks (output logs to different locations, including over the network), logging channels that are mapped to each sink, the format used by the log messages, any redaction-flags of log messages, the buffering and max sizes of log messages.
The following log configuration is the smallest/simplest configuration that we will use as a starting point. Here we keep most defaults, only adjusting the file-defaults destination path for the actual files, where this path will be mounted to a separate PV defined in the StatefulSet template.
file-defaults:
dir: /cockroach/cockroach-logs
sinks:
file-groups:
default:
channels:
- ALL
For a comprehensive explanation of this fragments, along with working examples and code-fragments, please refer to the Cockroach log configuration documentation so you can tailor the actual logging to your needs.
The StatefulSet template configuration
This statefulset fragment only highlights the added template properties to define the PVC and specific mount points to both the log config secret and the new logs folder. A full, complete StatefulSet example follows this fragment to show the entirety of an actual solution I deployed.
kind: StatefulSet
apiVersion: apps/v1
spec:
volumeClaimTemplates:
# ...
# ...
# Fragment 1
# New volumeClaimTemplate to generate Log PVC & PV
- kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: logsdir
labels:
app.kubernetes.io/instance: zlamal
app.kubernetes.io/name: cockroachdb
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
volumeMode: Filesystem
template:
spec:
containers:
- # ...
# ...
volumeMounts:
# ...
# ...
# Fragment 2
# Additional mount-points for path to logs and log-config
- name: logsdir
mountPath: /cockroach/cockroach-logs/
- name: log-config
readOnly: true
mountPath: /cockroach/log-config
# Fragment 3
# Addition of a new “cockroach start” parameter --log-config-file=...
# This parameter points CRDB to the mounted log-config secret
args:
- shell
- '-ecx'
- |-
exec /cockroach/cockroach start --log-config-file=/cockroach/log-config/log-config.yaml --join=... --advertise-host=... --certs-dir=/cockroach/cockroach-certs/ --http-port=8081 --port=26257 --cache=11% --max-sql-memory=10%
volumes:
- name: datadir
persistentVolumeClaim:
claimName: datadir
# Fragment 4
# Establish the logical YAML reference to the logging directory
- name: logsdir
persistentVolumeClaim:
claimName: logsdir
# Fragment 5
# Establish logical YAML reference to the log-config secret resource
- name: log-config
secret:
secretName: zlamal-cockroachdb-log-config
defaultMode: 420
# ...
# ...
Here is the complete StatefulSet of these changes,including tags/labels specific to my cluster as a reference example that you can copy and edit to make your own (eg sizes, storage classes, IOPS, tags/labels. etc):
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: zlamal-cockroachdb
labels:
app.kubernetes.io/component: cockroachdb
app.kubernetes.io/instance: zlamal
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: cockroachdb
helm.sh/chart: cockroachdb-14.0.4
spec:
serviceName: zlamal-cockroachdb
volumeClaimTemplates:
- kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: datadir
labels:
app.kubernetes.io/instance: zlamal
app.kubernetes.io/name: cockroachdb
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
volumeMode: Filesystem
- kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: logsdir
labels:
app.kubernetes.io/instance: zlamal
app.kubernetes.io/name: cockroachdb
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
volumeMode: Filesystem
template:
metadata:
labels:
app.kubernetes.io/component: cockroachdb
app.kubernetes.io/instance: zlamal
app.kubernetes.io/name: cockroachdb
spec:
restartPolicy: Always
initContainers:
- resources: {}
terminationMessagePath: /dev/termination-log
name: copy-certs
command:
- /bin/sh
- '-c'
- cp -f /certs/* /cockroach-certs/; chmod 0400 /cockroach-certs/*.key
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
imagePullPolicy: IfNotPresent
volumeMounts:
- name: certs
mountPath: /cockroach-certs/
- name: certs-secret
mountPath: /certs/
terminationMessagePolicy: File
image: busybox
serviceAccountName: zlamal-cockroachdb
schedulerName: default-scheduler
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/component: cockroachdb
app.kubernetes.io/instance: zlamal
app.kubernetes.io/name: cockroachdb
topologyKey: kubernetes.io/hostname
terminationGracePeriodSeconds: 300
securityContext: {}
containers:
- resources: {}
readinessProbe:
httpGet:
path: /health?ready=1
port: http
scheme: HTTPS
initialDelaySeconds: 10
timeoutSeconds: 1
periodSeconds: 5
successThreshold: 1
failureThreshold: 2
terminationMessagePath: /dev/termination-log
name: db
livenessProbe:
httpGet:
path: /health
port: http
scheme: HTTPS
initialDelaySeconds: 30
timeoutSeconds: 1
periodSeconds: 5
successThreshold: 1
failureThreshold: 3
env:
- name: STATEFULSET_NAME
value: zlamal-cockroachdb
- name: STATEFULSET_FQDN
value: zlamal-cockroachdb.mz-helm-v11.svc.cluster.local
- name: COCKROACH_CHANNEL
value: kubernetes-helm
ports:
- name: grpc
containerPort: 26257
protocol: TCP
- name: http
containerPort: 8081
protocol: TCP
imagePullPolicy: IfNotPresent
volumeMounts:
- name: datadir
mountPath: /cockroach/cockroach-data/
- name: logsdir
mountPath: /cockroach/cockroach-logs/
- name: log-config
readOnly: true
mountPath: /cockroach/log-config
- name: certs
mountPath: /cockroach/cockroach-certs/
terminationMessagePolicy: File
image: 'cockroachdb/cockroach:v23.2.1'
args:
- shell
- '-ecx'
- |-
exec /cockroach/cockroach start --log-config-file=/cockroach/log-config/log-config.yaml --join=${STATEFULSET_NAME}-0.${STATEFULSET_FQDN}:26257,${STATEFULSET_NAME}-1.${STATEFULSET_FQDN}:26257,${STATEFULSET_NAME}-2.${STATEFULSET_FQDN}:26257 --advertise-host=$(hostname).${STATEFULSET_FQDN} --certs-dir=/cockroach/cockroach-certs/ --http-port=8081 --port=26257 --cache=11% --max-sql-memory=10%
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app.kubernetes.io/component: cockroachdb
app.kubernetes.io/instance: zlamal
app.kubernetes.io/name: cockroachdb
serviceAccount: zlamal-cockroachdb
volumes:
- name: datadir
persistentVolumeClaim:
claimName: datadir
- name: logsdir
persistentVolumeClaim:
claimName: logsdir
- name: log-config
secret:
secretName: zlamal-cockroachdb-log-config
defaultMode: 420
- name: certs
emptyDir: {}
- name: certs-secret
projected:
sources:
- secret:
name: zlamal-cockroachdb-node-secret
items:
- key: ca.crt
path: ca.crt
mode: 256
- key: tls.crt
path: node.crt
mode: 256
- key: tls.key
path: node.key
mode: 256
defaultMode: 420
dnsPolicy: ClusterFirst
podManagementPolicy: Parallel
replicas: 3
updateStrategy:
type: RollingUpdate
selector:
matchLabels:
app.kubernetes.io/component: cockroachdb
app.kubernetes.io/instance: zlamal
app.kubernetes.io/name: cockroachdb
Conclusion & References
This is a versatile addition to the standard statefulset because the IOPS can be managed between the PVs, and the plumbing is in-place for log customization. DB admins can easily make changes the to logging channels in a running environment by editing a single log-config file that saved as a secrets object.
Cockroach Logging Overview
Cockroach log configuration
Cockroach start: logging
Production recommendations
Top comments (2)
Mark, I'm guessing you could take a similar approach to having multiple data store devices?
Yes indeed! Adding additional data-stores is an ideal solution to address several use-cases:
CRDB on high-vCPU worker-nodes: From our production readiness guidelines, we do not recommend workers with > 32 vCPUs. If you're bound to servers with 32 or more vCPUs, the additional store will benefit from the extra compute/processing power by creating additional processes per-store. These include splitting the GC workload, compactions, replica management, WAL, monitoring, etc. In the end you will leverage the additional CPU and will experience less waiting times on I/O operations.
You can create custom stores dedicated for specialized activities such as encryption at-rest for a subset of your data. In most cases there is a performance cost to encrypt/decrypt data, and you may not want to do this for the entirety of your data, maybe just a few tables managing PII. This is nicely written up with tangible examples in this blog: cockroachlabs.com/blog/selective-e...