When you think about making your microservice architecture more secure, would you know where to start? There are so many ways to begin securing a system that it quickly becomes overwhelming.
This blog post introduces you to two approaches to minimize security exposure.
It starts with documentation
What does documentation have to do with security? Paired with the right tools, it can quickly help us discover shadow APIs and mismatches.
In this context, documentation refers to machine-readable documents instead of bulky Word documents.
Let's take a look at OpenAPI specifications.
An OpenAPI spec describes endpoints, accepted parameters, formats of returned payloads, etc.
OpenAPI documentation has several advantages, such as auto-generating client code. In this context, we can, with the right tools, also leverage it to:
- Find unused API endpoints
- Discover undocumented API endpoints
Let's examine how this plays out with a Node.js/Express project.
Create a new project
To get started, we create a small Node.js API with Express:
mkdir api
cd api
npm init -y
npm i express cors body-parser helmet morgan express-openapi
Once the installation has finished, open package.json
and configure the following:
"type": "module",
"scripts": {
"start": "node src/index.js"
}
We'll keep our source code in a src/
folder. Run the following command in the project root to create a src
folder:
mkdir src
The following source code creates a new Express application and adds a few middlewares:
//src/index.js
import express from 'express';
import bodyParser from 'body-parser';
import cors from 'cors';
import helmet from 'helmet';
import morgan from 'morgan';
const app = express();
// adding Helmet to enhance your API's security
app.use(helmet());
// using bodyParser to parse JSON bodies into JS objects
app.use(bodyParser.json());
// enabling CORS for all requests
app.use(cors());
// adding morgan to log HTTP requests
app.use(morgan('combined'));
app.get('/', (_req, res) => {
res.send("OK");
});
app.listen(3001, () => {
console.log('listening on port 3001');
});
This code serves as our scaffold. We'll have a default route that replies "OK" if invoked.
Add OpenAPI documentation
Now, let's add some routes and documentation.
We'll store our routes and docs in a separate folder:
mkdir src/api-v1
In src/api-v1
, create api-doc.yaml
. This file will contain all documentation needed for our endpoints:
swagger: '2.0'
basePath: '/'
info:
title: 'A getting started API.'
version: '1.0.0'
definitions:
Student:
type: 'object'
properties:
name:
description: 'The name of this student.'
type: 'string'
required: ['name']
Grade:
type: 'object'
properties:
subject:
description: 'The subject the grade is for'
type: 'string'
grade:
description: 'The actual grade'
type: 'string'
required: ['subject', 'grade']
paths:
/students:
get:
summary: 'Returns all Students'
operationId: 'getStudents'
parameters: []
responses:
200:
description: 'A list of students'
schema:
type: 'array'
items:
$ref: '#/definitions/Student'
default:
description: 'An error occurred'
schema:
additionalProperties: true
/grades:
get:
summary: 'Returns all Grades'
operationId: 'getGrades'
parameters: []
responses:
200:
description: 'A list of grades'
schema:
type: 'array'
items:
$ref: '#/definitions/Grade'
default:
description: 'An error occurred'
schema:
additionalProperties: true
post:
summary: 'creates a new grade for a student'
operationId: 'createGrade'
parameters:
- name: 'grade'
in: 'body'
schema:
$ref: '#/definitions/Grade'
responses:
201:
description: 'Created'
default:
description: 'An error occurred'
schema:
additionalProperties: true
The file defines a few different things:
- Data types in use
Student
Grade
- Routes and their supported HTTP methods
Next, we need to create the actual handlers for each endpoint. All endpoints will be defined in src/api-v1/paths
. Run the following command to create the folder:
mkdir src/api-v1/paths
Inside paths
, we will create two files. First, grades.js
:
// src/api-v1/paths/grades.js
export default function(grades) {
let operations = {
get,
post
};
function get(_req, res, _next) {
res.status(200).json(grades);
}
function post(req, res, _next) {
console.log(req.body);
grades.push(req.body);
res.send({status: 'OK'});
}
return operations;
}
Next, create the students
endpoint:
// src/api-v1/paths/students.js
export default function(students) {
let operations = {
get
};
function get(req, res, next) {
res.status(200).json(students);
}
return operations;
}
Unlike grades.js
, this endpoint only supports GET
. Finally, we need to wire everything together in src/index.js
.
First, add the following additional imports at the top of the file:
// src/index.js
//...
import { initialize } from 'express-openapi';
import { readFileSync } from 'node:fs';
After the last import, we will load static data the endpoint can return. First, create src/data.json
:
{
"students": [
{ "name": "alice", "email": "alice@example.com" },
{ "name": "bob", "email": "bob@example.com" },
{ "name": "alex", "email": "alex@example.com" }
],
"grades": [
{"name": "Alice", "grade": "A+"},
{"name": "Bob", "grade": "C"},
{"name": "Alex", "grade": "F"}
]
}
In src/index.js
, we now load the data:
// src/index.js
import { initialize } from 'express-openapi';
import { readFileSync } from 'node:fs';
//add this line
const { students, grades } = JSON.parse(readFileSync('./src/data.json').toString('utf-8'));
const app = express();
//...
Lastly, we need to initialize express-openapi
:
// src/index.js
//...
app.get('/', (_req, res) => {
res.send("OK");
});
//add this line
initialize({
app,
apiDoc: readFileSync('./src/api-v1/api-doc.yaml', 'utf8'),
dependencies: {
students: students,
grades: grades,
},
paths: 'src/api-v1/paths'
});
app.listen(3001, () => {
console.log('listening on port 3001');
});
With this, we're good to dockerize the application:
# Dockerfile
FROM node:18-alpine
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3001
CMD [ "node", "src/index.js" ]
docker build -t apisecuritybackend .
Deploy APIClarity
We'll install APIClarity into a Kubernetes cluster to test our API documentation. We're using a Kind cluster for demonstration purposes.
Of course, if you have another Kubernetes cluster up and running elsewhere, all steps also work there.
We start by creating a new cluster (this step might take up to 2 minutes):
kind create cluster
Expected output:
Creating cluster "kind" ...
✓ Ensuring node image (kindest/node:v1.25.3) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:
kubectl cluster-info --context kind-kind
Have a nice day! 👋
Next, we will install APIClarity using Helm.
Create a values.yaml
with the following contents:
#######################################################################################
## Global Values
global:
## Database password
##
databasePassword: apiclarity
## Docker image
##
docker:
## Configure registry
##
registry: "ghcr.io/openclarity"
## Whether or not persistence is enabled
##
persistentVolume:
## Persistent Volume size
##
size: 100Mi
## Persistent Volume Storage Class
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
## set, choosing the default provisioner. (gp2 on AWS, standard on
## GKE, AWS & OpenStack)
# storageClass: ""
traceSampling:
enable: false
bflaAutomaticLearningAndDetection:
enable: false
## End of Global Values
#######################################################################################
#######################################################################################
## APIClarity Values
apiclarity:
# Send APIClarity notifications to this backend
# notificationURLPrefix: example.local:8888/
tls:
enabled: false
# Secret that contains server tls key and cert
tlsServerCertsSecretName: "apiclarity-tls"
# TLS secret (tlsServerCertsSecretName) cert field name
tlsServerCertFileName: "server.crt"
# TLS secret (tlsServerCertsSecretName) key field name
tlsServerKeyFileName: "server.key"
# ConfigMap that contains the root cert
rootCACertConfigMapName: "apiclarity-root-ca.crt"
# Root cert ConfigMap (rootCACertConfigMapName) cert field name
rootCertFileName: "ca.crt"
# Defines env variables for apiclarity pod
env:
plugins:
## ENV_1: example123
FUZZER_DEPLOYMENT_TYPE: "configmap"
## Docker Image values.
docker:
imageTag: "v0.14.5"
imagePullPolicy: Always
## Logging level (debug, info, warning, error, fatal, panic).
logLevel: warning
## Enable/disable rbac resource creation (i.e. ClusterRole, ClusterRoleBinding)
rbac:
create: true
## ServiceAccount settings
serviceAccount:
## Enable/disable ServiceAccount creation. Set false to use a pre-existing account
create: true
## Override name of ServiceAccount
# name:
## Resource limits for APIClarity deployment
resources:
requests:
memory: "200Mi"
cpu: "100m"
limits:
memory: "1000Mi"
cpu: "1000m"
## Resource limits for APIClarity init container deployment
initResources:
requests:
memory: "200Mi"
cpu: "100m"
limits:
memory: "1000Mi"
cpu: "200m"
## End of APIClarity Values
#######################################################################################
#######################################################################################
## APIClarity Postgres Values
apiclarity-postgresql:
enabled: true
## Specify posgtresql image
# image:
# registry: docker.io
# repository: bitnami/postgresql
# tag: 14.4.0-debian-11-r4
## initdb parameters
# initdb:
## ConfigMap with scripts to be run at first boot
## NOTE: This will override initdb.scripts
# scriptsConfigMap
## Secret with scripts to be run at first boot (in case it contains sensitive information)
## NOTE: This can work along initdbScripts or initdbScriptsConfigMap
# scriptsSecret:
## Specify the PostgreSQL username and password to execute the initdb scripts
# user:
# password:
## Setup database name and password
auth:
existingSecret: apiclarity-postgresql-secret
database: apiclarity
## Enable security context
containerSecurityContext:
enabled: true
runAsUser: 1001
runAsNonRoot: true
# End of APIClarity Postgres Values
#######################################################################################
#######################################################################################
## APIClarity Traffic Source Values
trafficSource:
global:
## Proxy configuration for the traffic source post install jobs
httpsProxy: ""
httpProxy: ""
envoyWasm:
## Enable Envoy wasm traffic source
##
enabled: false
## Enable Istio verification in a Pre-Install Job
##
enableIstioVerify: true
## Enable APIClarity WASM filter in the following namespaces
##
namespaces:
- default
tap:
## Enable Tap traffic source
##
enabled: true
## Enable APIClarity Tap in the following namespaces
##
namespaces:
- default
- students
## APIClarity Tap logging level (debug, info, warning, error, fatal, panic)
##
logLevel: "warning"
## Docker Image values.
docker:
imageTag: "v0.14.5"
imagePullPolicy: Always
kong:
## Enable Kong traffic source
##
enabled: false
## Carry out post-install patching of kong container to install plugin
patch: true
## Specify the name of the proxy container in Kong gateway to patch
##
containerName: "proxy"
## Specify the name of the Kong gateway deployment to patch
##
deploymentName: ""
## Specify the namespace of the Kong gateway deployment to patch
##
deploymentNamespace: ""
## Specify the name of the ingress resource to patch
##
ingressName: ""
## Specify the namespace of the ingress resource to patch
##
ingressNamespace: ""
tyk:
## Enable Tyk traffic source
##
enabled: false
## Enable Tyk verification in a Pre-Install Job
##
enableTykVerify: true
## Specify the name of the proxy container in Tyk gateway to patch
##
containerName: "proxy"
## Specify the name of the Tyk gateway deployment to patch
##
deploymentName: ""
## Specify the namespace of the Tyk gateway deployment to patch
##
deploymentNamespace: ""
# End of APIClarity Traffic Source Values
#######################################################################################
#######################################################################################
## APIClarity Runtime Fuzzing Values
APIClarityRuntimeFuzzing:
## Fuzzer jobs and pods labels.
##
labels:
app: apiclarity-fuzzer
docker:
## Fuzzer docker image to load
##
image: gcr.io/eticloud/k8sec/scn-dast:f425b0aefe272d6707649c0a9845eabceade7f91
## Fuzzing methods (scn-fuzzer, restler, crud). It is a comma separated list.
##
methods: "scn-fuzzer,restler,crud"
## Internal use only, do not change
##
restlerRootPath: "/tmp"
## Internal use only, do not change
##
restlerTokenInjPath: "/app/"
debug: false
## Resource limits for Fuzzer deployment
##
resources:
requests:
memory: "200Mi"
cpu: "100m"
limits:
memory: "1000Mi"
cpu: "200m"
# End of APIClarity Runtime Fuzzing Values
#######################################################################################
#######################################################################################
## APIClarity External Trace Source Values
supportExternalTraceSource:
enabled: false
# End of APIClarity External Trace Source Values
#######################################################################################
Next, run the following commands:
helm repo add apiclarity https://openclarity.github.io/apiclarity
helm upgrade --values values.yaml --create-namespace apiclarity apiclarity/apiclarity -n apiclarity --install
After running these commands, wait a few seconds to give the pods enough time to start up. You can verify if all pods are present by running:
kubectl get pods -n apiclarity
NAME READY STATUS RESTARTS AGE
apiclarity-apiclarity-7f8f74699b-wcjj8 1/1 Running 0 104s
apiclarity-apiclarity-postgresql-0 1/1 Running 0 104s
apiclarity-apiclarity-taper-mft65 1/1 Running 0 104s
Once all pods are up and running, port-forward the dashboard port:
kubectl port-forward --namespace apiclarity svc/apiclarity-apiclarity 9999:8080
Schedule workload
Next, we will load the API Docker image we built earlier into Kind (if you're not using Kind, make sure to push this into a container registry of your choice):
kind load docker-image apisecuritybackend:latest
Once finished, we can schedule an API workload:
kubectl create ns students
cat << EOF | kubectl apply -n students -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: students-deployment
labels:
app: students
spec:
replicas: 1
selector:
matchLabels:
app: students
template:
metadata:
labels:
app: students
spec:
containers:
- name: students
image: apisecuritybackend:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3001
---
apiVersion: v1
kind: Service
metadata:
name: students-service
spec:
selector:
app: students
ports:
- protocol: TCP
port: 3005
targetPort: 3001
EOF
This YAML snippet provides a basic deployment and a service that makes the pod accessible. Let's spin up another workload to run a few HTTP requests:
$ kubectl run -i --tty --rm debug --image=ubuntu --restart=Never -- bash
# apt-get update && apt-get install -y curl
We use another container to send HTTP requests so APIClarity recognizes the traffic. Since the standard Ubuntu container does not have curl
preinstalled, we're installing it first.
Once the installation is finished, run a few queries:
curl -XGET http://localhost:3005
OK%
curl -XGET http://localhost:3005/students
[{"name":"alice","email":"alice@example.com"},{"name":"bob","email":"bob@example.com"},{"name":"alex","email":"alex@example.com"}]
curl -XGET http://localhost:3005/students
[{"name":"alice","email":"alice@example.com"},{"name":"bob","email":"bob@example.com"},{"name":"alex","email":"alex@example.com"}]
These HTTP requests should generate enough data in the APIClarity dashboard.
Explore APIClarity findings
Open a new browser tab and visit http://localhost:9999
.
The dashboard shows students
as the currently most used API.
Access http://localhost:9999/inventory/INTERNAL/1
:
On this screen, we upload our API documentation file (src/api-v1/api-doc.yaml
). The uploaded documentation will serve as the reference for all further analysis. Once uploaded, click on default tag to see all configured endpoints:
In the terminal tab with the Ubuntu container, run:
curl -XGET http://students-service.students.svc.cluster.local:3005
Then, come back to the dashboard. You'll notice under Latest spec diffs, one or more entries showed up:
We just discovered a so-called "shadow API". A shadow API is an endpoint that's in use but does not have any documentation.
It's crucial to detect these early because a shadow API might expose functionality or data that has yet to be reviewed and approved.
Once found, we can investigate further in the UI and the respective code base to understand where this endpoint comes from and update our microservice architecture.
Advanced Security - ensuring correct authorizations
While API documentation helps to detect undocumented endpoints in our infrastructure, we can't determine who accesses these endpoints.
For instance, in the case of our demo application, usually, a student would only access /students
. Under normal circumstances, a student wouldn't create new grades or create new student records. These operations are only performed by teachers or IT staff.
With APIClarity, we can detect so-called BFLA (Broken Function Level Authorization). A broken function level authorization could be a student accessing endpoints to manage grades.
With this capability, we get a step closer to better understanding how production services interact.
Learn normal patterns
It all starts by teaching APIClarity what standard traffic patterns look like.
Open the APIClarity dashboard and click on students-service.students in the Most used APIs box.
On the next screen, select BFLA in the tab list.
Click on START. APIClarity now records all API traffic to our API.
We use another Ubuntu container to generate some traffic. In your terminal, run:
kubectl run -i --tty --rm teacher --image=ubuntu --restart=Never -- bash
apt-get update && apt-get install -y curl
curl -XGET http://students-service.students.svc.cluster.local:3005/grades
curl -XGET http://students-service.students.svc.cluster.local:3005/grades
curl -XGET http://students-service.students.svc.cluster.local:3005/grades
These requests simulate legitimate API traffic, for instance, a teacher accessing student grades.
Next, come back to your browser tab and stop learning:
Click on Stop BFLA model learning. On the following screen, click Start BFLA detection.
Detect illegitimate access
In a separate terminal window, run the following:
kubectl run -i --tty --rm student --image=ubuntu --restart=Never -- bash
curl -XGET http://students-service.students.svc.cluster.local:3005/grades
curl -XGET http://students-service.students.svc.cluster.local:3005/grades
curl -XGET http://students-service.students.svc.cluster.local:3005/grades
These requests simulate illegitimate API access, such as a student who tries to access grades.
Return to your browser window and click on Stop BFLA model detecting.
Next, click on default-tag and GET /grades:
APIClarity correctly identified the student trying to access grades as illegitimate.
This provides us with some information to investigate further to find out why these API calls happen in the first place.
Make sure to star APIClarity on GitHub and comment below how you're using APIClarity to strengthen security.
Also check out the video version of this post here!
Top comments (6)
Nice and thorough, helpful post here!
I would argue that cors makes your app less secure though...
I'd be curious to hear how CORS makes an application less secure.
Enabling CORS makes it so hackers who make a fake domain can fetch data from your server and display it to the user, without the browser warning the user.
Shade light on how you mean
Enabling CORS makes it so hackers who make a fake domain can fetch data from your server and display it to the user, without the browser warning the user.