hayao-k for AWS Community Builders

Posted on Jan 6, 2021 • Originally published at hayao-k.dev

Integration of AWS Security Hub and kube-bench

#aws #kubernetes #security

Introduction

kube-bench has been added to the 3rd party integrations in AWS Security Hub.

🚀 AWS Security Hub adds open source tool integrations with Kube-bench and Cloud Custodian

https://aws.amazon.com/jp/about-aws/whats-new/2020/12/aws-security-hub-adds-open-source-tool-integration-with-kube-bench-and-cloud-custodian/

With this integration, the results of the CIS Kubernetes Benchmark and CIS Amazon EKS Benchmark checks run by kube-bench can now be centrally managed in AWS Security Hub.

What is CIS Benchmark?

CIS Benchmark is a set of guidelines published by the Center for Internet Security (CIS), a non-profit organization in the United States, for strengthening various operating systems, servers, and cloud environments.

CIS Benchmark is referenced in compliance requirements such as the PCI DSS when it states "industry-accepted system hardening standards".

You can download over 140 CIS Benchmarks in PDF format from the CIS website.
https://www.cisecurity.org/cis-benchmarks/

What is kube-bench?

kube-bench is a Go application that allows you to check if your environment complies with the recommendations listed in the CIS Kubernetes Benchmark.

aquasecurity / kube-bench

Checks whether Kubernetes is deployed according to security best practices as defined in the CIS Kubernetes Benchmark

kube-bench is a tool that checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark.

Tests are configured with YAML files, making this tool easy to update as test specifications evolve.

CIS Scanning as part of Trivy and the Trivy Operator

Trivy, the all in one cloud native security scanner, can be deployed as a Kubernetes Operator inside a cluster Both, the Trivy CLI, and the Trivy Operator support CIS Kubernetes Benchmark scanning among several other features.

Quick start

There are multiple ways to run kube-bench. You can run kube-bench inside a pod, but it will need access to the host's PID namespace in order to check the running processes, as well as access to some directories on the host where config files and other files are stored.

The supplied job.yaml file can be applied to run the tests as a…

View on GitHub

kube-bench supports checking not only the CIS Kubernetes Benchmark, but also the CIS Amazon Elastic Kubernetes Service (EKS) Benchmark and CIS Google Kubernetes Engine (GKE) Benchmark.

What is AWS Security Hub?

AWS Security Hub is a service for aggregating and centrally managing various security data for the entire AWS environment.

Security Hub integrates with many 3rd party security products, as well as AWS services such as Amazon GuardDuty, Inspector, and Macie.
Data can be sent from Security Hub-enabled products to the Security Hub and vice versa.

Internally, the result information is managed in a JSON type format called AWS Security Finding Format (ASFF), so you can import your own data as long as it conforms to this format.

Prerequisites

I have confirmed that the following versions work.

Amazon EKS: 1.17
eksctl: 0.30.0
kube-bench: 0.40.0

Ran the CIS Amazon EKS Benchmark v1.0 check in an EKS cluster.

Enable integration

Search for kube-bench from the Security Hub console integration and click "Accept findings" to see information about the IAM policies required to send the findings to Security Hub.

Select "Accept findings" again on the confirmation screen, and the status will be activated.

Configue IAM Roles

In order for kube-bench to run in an EKS cluster, the pod must have permissions to send check results to Security Hub.

There are two ways to assign access to AWS resources to a Pod.

Use IAM Roles for Service Accounts (IRSA)
Use IAM roles configured for node groups

Using IRSA, you can associate an IAM role with a Kubernetes service account.
This allows you to provide Security Hub access only to pods launched by kube-bench.

Note that if you use an IAM role that is configured for a node group, all Pods running under that group will be assigned access to the Security Hub.

This time we will use IRSA.
If you haven't created an IAM OIDC Provider to use IRSA, do so first!

$ eksctl utils associate-iam-oidc-provider \
--cluster {CLUSTER_NAME} --approve --region ap-northeast-1

Next, create an IAM role with the following policies attached.
Replace the regions as needed.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "securityhub:BatchImportFindings",
            "Resource": [
                "arn:aws:securityhub:ap-northeast-1::product/aqua-security/kube-bench"
            ]
        }
    ]
}

Add the following policy to the trust relationship of the IAM role.
Set <ACCOUNT_ID> and <OIDR_PROVIDER_ID> to your own.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.ap-northeast-1.amazonaws.com/id/<OIDC_PROVIDER_ID>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringLike": {
          "oidc.eks.ap-northeast-1.amazonaws.com/id/<OIDC_PROVIDER_ID>:sub": "system:serviceaccount:kube-bench:*",
          "oidc.eks.ap-northeast-1.amazonaws.com/id/<OIDC_PROVIDER_ID>:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

This is written assuming that kube-bech is used for Namespace.

Create container image

We need to build a container image of kube-bench and push it to ECR.
First, let's create an ECR repository to store the image.

$ aws ecr create-repository --repository-name k8s/kube-bench --image-tag-mutability MUTABLE
{
    "repository": {
        "repositoryArn": "arn:aws:ecr:ap-northeast-1:123456789012:repository/k8s/kube-bench",
        "registryId": "123456789012",
        "repositoryName": "k8s/kube-bench",
        "repositoryUri": "123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/k8s/kube-bench",
        "createdAt": 1607747704.0,
        "imageTagMutability": "MUTABLE",
        "imageScanningConfiguration": {
            "scanOnPush": false
        },
        "encryptionConfiguration": {
            "encryptionType": "AES256"
        }
    }
}

Clone the source code of kube-bench from GitHub.

$ git clone https://github.com/aquasecurity/kube-bench.git
Cloning into 'kube-bench'...
remote: Enumerating objects: 4115, done.
remote: Total 4115 (delta 0), reused 0 (delta 0), pack-reused 4115
Receiving objects: 100% (4115/4115), 7.69 MiB | 5.28 MiB/s, done.
Resolving deltas: 100% (2644/2644), done.

$ cd kube-bench

If you want to send the execution result to Security Hub, you need to edit cfg/eks-1.0/config.yaml and rewrite the AWS account, region and cluster name to be executed before building the image.

---
AWS_ACCOUNT: "123456789012"
AWS_REGION: "ap-northeast-1"
CLUSTER_ARN: "arn:aws:eks:ap-northeast-1:123456789012:cluster/{YOUR_CLUSTER_NAME}"

Build the container image and push it to ECR.

$ aws ecr get-login-password | docker login --username AWS --password-stdin https://123456789012.dkr.ecr.ap-northeast-1.amazonaws.com
Login Succeeded

$ docker build -t k8s/kube-bench .
...
Successfully built 6ad073f96455
Successfully tagged k8s/kube-bench:latest

$ docker tag k8s/kube-bench:latest 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/k8s/kube-bench:latest
$ docker push 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/k8s/kube-bench:latest
The push refers to repository [123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/k8s/kube-bench]
bd6c279efeaa: Pushed 
3032a8c3bf7a: Pushed 
b0efa7564210: Pushed 
fe4cf80d4f2c: Pushed 
31c3d3db74eb: Pushed 
d2e36eff2b5d: Pushed 
f4666769fca7: Pushed 
latest: digest: sha256:da95de0edccad7adb6fd6c80137a65b5458142efe14efa94ac44cc5c6ce6b2ef size: 1782

Run kube-bench

Creating a Service Account

Create a service account for kube-bench using a manifest file like the following. Note that the annotations specifies the IAM role that we just created.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-bench-sa
  namespace: kube-bench
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/<IAM_ROLE_NAME>

Create a namespace kube-bench and apply it.

$ kubectl create ns kube-bench
kubectl create ns kube-bench

$ kubectl apply -f sa.yaml
serviceaccount/kube-bench-sa created

$ kubectl get sa -n kube-bench
NAME            SECRETS   AGE
default         1         6m32s
kube-bench-sa   1         11s

Run a job

Open jobs-eks.yaml cloned from GitHub, and edit image and command. Adding the --asff flag to the command will send the results to Security Hub.

Add the service account name that you just created.

The edited file should look like the following.

---
apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench
spec:
  template:
    spec:
      hostPID: true
      serviceAccountName: kube-bench-sa
      containers:
        - name: kube-bench
          image: 123456789012.dkr.ecr.region.amazonaws.com/k8s/kube-bench:latest
          command: ["kube-bench", "node", "--benchmark", "eks-1.0", "--asff"]
          volumeMounts:
            - name: var-lib-kubelet
              mountPath: /var/lib/kubelet
              readOnly: true
            - name: etc-systemd
              mountPath: /etc/systemd
              readOnly: true
            - name: etc-kubernetes
              mountPath: /etc/kubernetes
              readOnly: true
      restartPolicy: Never
      volumes:
        - name: var-lib-kubelet
          hostPath:
            path: "/var/lib/kubelet"
        - name: etc-systemd
          hostPath:
            path: "/etc/systemd"
        - name: etc-kubernetes
          hostPath:
            path: "/etc/kubernetes"

Run kube-bench as a Kubernetes job and make sure it exits successfully.

$ kubectl apply -f job-eks.yaml -n kube-bench
job.batch/kube-bench created

$ kubectl get all -n kube-bench                                                                                                                                                    
NAME                   READY   STATUS      RESTARTS   AGE
pod/kube-bench-q4sbd   0/1     Completed   0          55s

NAME                   COMPLETIONS   DURATION   AGE
job.batch/kube-bench   1/1           5s         55s

If the following message is output to the Pod's log and the job fails, please check whether the contents of cfg/eks-1.0/config.yaml are correct.

failed to output to ASFF: finding publish failed: MissingEndpoint: 'Endpoint' configuration is required for this service

Check the findings of Security Hub

Normally, the results of running kube-bench will be output to the Pod's log.

$ kubectl logs -f pod/kube-bench-2j2ss -n kube-bench
[INFO] 3 Worker Node Security Configuration
[INFO] 3.1 Worker Node Configuration Files
[PASS] 3.1.1 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored)
[PASS] 3.1.2 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored)
[PASS] 3.1.3 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored)
[PASS] 3.1.4 Ensure that the kubelet configuration file ownership is set to root:root (Scored)
[INFO] 3.2 Kubelet
[PASS] 3.2.1 Ensure that the --anonymous-auth argument is set to false (Scored)
[PASS] 3.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Scored)
[PASS] 3.2.3 Ensure that the --client-ca-file argument is set as appropriate (Scored)
[PASS] 3.2.4 Ensure that the --read-only-port argument is set to 0 (Scored)
[PASS] 3.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Scored)
[PASS] 3.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Scored)
[PASS] 3.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Scored) 
[PASS] 3.2.8 Ensure that the --hostname-override argument is not set (Scored)
[WARN] 3.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Scored)
[PASS] 3.2.10 Ensure that the --rotate-certificates argument is not set to false (Scored)
[PASS] 3.2.11 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)

== Remediations node ==
3.2.9 If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level.
If using command line arguments, edit the kubelet service file
/etc/systemd/system/kubelet.service on each worker node and
set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable.
Based on your system, restart the kubelet service. For example:
systemctl daemon-reload
systemctl restart kubelet.service


== Summary node ==
14 checks PASS
0 checks FAIL
1 checks WARN
0 checks INFO

== Summary total ==
14 checks PASS
0 checks FAIL
1 checks WARN
0 checks INFO

If you specify the --asff flag and send the results to Security Hub, only the number of cases imported to Security Hub will be listed in the Pod's log.

$ kubectl logs -f pod/kube-bench-q4sbd -n kube-bench
2020/12/12 14:51:07 Number of findings that were successfully imported:1

In the case of this environment, the result was one.

Note that the results will be sent to Security Hub only for items whose check result is FAIL or WARN. If all checks are PASSed, no results will be generated to Security Hub.

If you check the findings in Security Hub console, you will see that the one result sent was successfully captured.

Using Custom Insights

Security Hub has a feature called Custom Insights that allows you to aggregate data by resource or environment using specific grouping condition and filters.

Custom Insights can be created in the Insights section of the Security Hub console by simply setting any grouping condition, filters

For example, if you are using Security Hub in a multi-account configuration, you can use the management account to aggregate the detection results for each member account.

Let's check out an example of a custom insight with the grouping condition set to AWS account ID and the product name Kube-bench set as a filter.
By clicking on the account ID, you will be able to instantly see the kube-bench results for the target account.

Even if you are operating with a single account, you can use it to list the detection results for each EKS cluster by setting the resource ID as a group by condition.