The Rancher Logging Operator supports a number of backends, but a very common one is logging to a syslog endpoint. However, rather than maintaining separate logging infrastructure if this is a new piece of your environment, you can deploy it to your Kubernetes clusters directly, either using the method in the UI above, or via the Helm chart.
If you plan to deploy charts via Fleet (which I've written about before, or some other GitOps flow, you would pull the following charts from the Rancher chart repo:
helm repo add rancher-charts https://charts.rancher.io
helm repo update
helm fetch rancher-charts/rancher-logging-crd
helm fetch rancher-charts/rancher-logging
but before we apply them, we want to setup the syslog endpoint in the cluster for the logging operator to log to:
apiVersion: apps/v1
kind: Deployment
metadata:
name: rsyslog-deployment
labels:
app: rsyslog
spec:
replicas: 1
selector:
matchLabels:
app: rsyslog
template:
metadata:
labels:
app: rsyslog
spec:
containers:
- name: rsyslog
image: sudheerc1190/rsyslog:latest
ports:
- containerPort: 514
resources:
requests:
cpu: 250m
memory: 524Mi
restartPolicy: Always
terminationGracePeriodSeconds: 30
here we're just creating a syslog Deployment
; for production, I would recommend creating a PersistentVolume
and mount it to /var/log/remote
so in subsequent recreations of this Pod
, you can restore past logging data for this or other clusters.
Because we want this just available internal to the cluster in this example, we'll just create the following Service
objects to create cluster DNS entires for this endpoint:
---
apiVersion: v1
kind: Service
metadata:
name: "syslog-service-tcp"
spec:
ports:
- port: 514
targetPort: 514
protocol: TCP
type: LoadBalancer
selector:
app: "rsyslog"
---
apiVersion: v1
kind: Service
metadata:
name: "syslog-service-udp"
spec:
ports:
- port: 514
targetPort: 514
protocol: UDP
type: LoadBalancer
selector:
app: "rsyslog"
so depending upon which protocol you'd like to use for logging, you can use a hostname like syslog-service-udp.default.svc.cluster.local
.
Now, you can create the CRDs and Logging Operator:
helm install rancher-logging-crd ./rancher-logging-crd-100.1.2+up3.17.4.tgz -n cattle-logging-system --create-namespace
helm install rancher-logging ./rancher-logging-100.1.2+up3.17.4.tgz -n cattle-logging-system
Note that the version on these charts should match, and that they should be deployed to the cattle-logging-system
namespace.
In the logging operator, there are two pieces Flow
(and cluster-wide ClusterFlow
) resources and Output
(and ClusterOutput
) resources. This example will just use cluster-scoped examples that span namespaces, but the matching rules and output formatting will work similarly in both cases.
Our ClusterOutput
will be what connects to our Syslog service, using host: syslog-service-udp.default.svc.cluster.local
on port 514, in this case using transport: udp
:
apiVersion: v1
items:
- apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
name: testing
namespace: cattle-logging-system
spec:
syslog:
buffer:
total_limit_size: 2GB
flush_thread_count: 8
timekey: 10m
timekey_use_utc: true
timekey_wait: 1m
format:
app_name_field: kubernetes.pod_name
hostname_field: custom-cluster-name
log_field: message
rfc6587_message_size: false
host: syslog-service-udp.default.svc.cluster.local
insecure: true
port: 514
transport: udp
kind: List
and you can verify the status of the output (deployment status, problems, etc.) with kubectl get clusteroutput/testing -n cattle-logging-system
. The ClusterOutput
should be namespaces with the operator deployment. In the above spec, under format
, you can refer to the RFC format for other options for logging fields.
In our ClusterFlow
, we're going to use just a basic rule, ask to exclude logs from the kube-system
namespace, and then select
everything from the applications
namespace:
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
name: global-ns-clusterflow
spec:
match:
- exclude:
namespaces:
- kube-system
- select:
namespaces:
- applications
globalOutputRefs:
- testing
and you'll see we use the testing
output as the global endpoint for sending logs. You can check the status of the Flow using kubectl get clusterflow/global-ns-clusterflow
.
I usually test after this point using a deployment like this:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
type: LoadBalancer
and then deploy it once to an excluded namespace and then an included one, then make a few requests (curl -s http://{LB_IP}
, if the output and flow report no issues, you'll see the logs start to populate after a few minutes). You can then check the logs manually (if nothing is scraping syslog externally) with something formatted like kubectl exec -it {rsyslog_pod} -- tail -f /var/log/remote/{date hierarchy}/{hour}.log
.
More about the BanzaiCloud Operator can be found here, and more on the Logging features in Rancher from this operator can be found here.
Top comments (0)