As i am an Ibmcloud (Bluemix)user, i share with you a ready Devops toolchain that i use for my Kubernetes Deployment .
In the interest of efficiency and continuous deployment, the implementation of a ToolChain of continuous deployment remains a must. The latter is divided into 3 Steps :
Step 1
- Validate docker file
- Build Docker image
- Push docker images to Registry
Step2 : Non blocking Step
- Vulnerability scanning for the docker image
- Get the recommendations to fix them
Step3
- validate deployment files
- deploy changes to Kubernetes Cluster
🛠Toolchain
Step 1 : Build
This step is responsible to verify docker file with a linter then build the image and push it to registry .
Bellow we describe the shell scripte that do the stuff :
Verify Docker file
#!/bin/bash
echo "REGISTRY_URL=${REGISTRY_URL}"
echo "REGISTRY_NAMESPACE=${REGISTRY_NAMESPACE}"
echo "IMAGE_NAME=${IMAGE_NAME}"
echo "ARCHIVE_DIR=${ARCHIVE_DIR}"
echo "DOCKER_ROOT=${DOCKER_ROOT}"
echo "DOCKER_FILE=${DOCKER_FILE}"
# View build properties
if [ -f build.properties ]; then
echo "build.properties:"
cat build.properties
else
echo "build.properties : not found"
fi
echo "=========================================================="
echo "Checking for Dockerfile at the repository root"
if [ -z "${DOCKER_ROOT}" ]; then DOCKER_ROOT=. ; fi
if [ -z "${DOCKER_FILE}" ]; then DOCKER_FILE=Dockerfile ; fi
if [ -f ${DOCKER_ROOT}/${DOCKER_FILE} ]; then
echo -e "Dockerfile found at: ${DOCKER_FILE}"
else
echo "Dockerfile not found at: ${DOCKER_FILE}"
exit 1
fi
echo "Linting Dockerfile"
npm install -g dockerlint
dockerlint -f ${DOCKER_ROOT}/${DOCKER_FILE}
echo "=========================================================="
echo "Checking registry current plan and quota"
bx cr plan
bx cr quota
echo "If needed, discard older images using: bx cr image-rm"
echo "Checking registry namespace: ${REGISTRY_NAMESPACE}"
NS=$( bx cr namespaces | grep ${REGISTRY_NAMESPACE} ||: )
if [ -z "${NS}" ]; then
echo "Registry namespace ${REGISTRY_NAMESPACE} not found, creating it."
bx cr namespace-add ${REGISTRY_NAMESPACE}
echo "Registry namespace ${REGISTRY_NAMESPACE} created."
else
echo "Registry namespace ${REGISTRY_NAMESPACE} found."
fi
echo -e "Existing images in registry"
bx cr images --restrict ${REGISTRY_NAMESPACE}
Build Docker Image
#!/bin/bash
echo "REGISTRY_URL=${REGISTRY_URL}"
echo "REGISTRY_NAMESPACE=${REGISTRY_NAMESPACE}"
echo "IMAGE_NAME=${IMAGE_NAME}"
echo "BUILD_NUMBER=${BUILD_NUMBER}"
echo "ARCHIVE_DIR=${ARCHIVE_DIR}"
echo "GIT_BRANCH=${GIT_BRANCH}"
echo "GIT_COMMIT=${GIT_COMMIT}"
echo "DOCKER_ROOT=${DOCKER_ROOT}"
echo "DOCKER_FILE=${DOCKER_FILE}"
# View build properties
if [ -f build.properties ]; then
echo "build.properties:"
cat build.properties
else
echo "build.properties : not found"
fi
# To review or change build options use:
echo -e "build --help"
bx cr build --help
echo -e "Existing images in registry"
bx cr images
# Minting image tag using format: BUILD_NUMBER--BRANCH-COMMIT_ID-TIMESTAMP
# e.g. 3-master-50da6912-20181123114435
# (use build number as first segment to allow image tag as a patch release name according to semantic versioning)
TIMESTAMP=$( date -u "+%Y%m%d%H%M%S")
IMAGE_TAG=${TIMESTAMP}
if [ ! -z "${GIT_COMMIT}" ]; then
GIT_COMMIT_SHORT=$( echo ${GIT_COMMIT} | head -c 8 )
IMAGE_TAG=${GIT_COMMIT_SHORT}-${IMAGE_TAG}
fi
if [ ! -z "${GIT_BRANCH}" ]; then IMAGE_TAG=${GIT_BRANCH}-${IMAGE_TAG} ; fi
IMAGE_TAG=${BUILD_NUMBER}-${IMAGE_TAG}
echo "=========================================================="
echo -e "BUILDING CONTAINER IMAGE: ${IMAGE_NAME}:${IMAGE_TAG}"
if [ -z "${DOCKER_ROOT}" ]; then DOCKER_ROOT=. ; fi
if [ -z "${DOCKER_FILE}" ]; then DOCKER_FILE=${DOCKER_ROOT}/Dockerfile ; fi
#set -x
#bx cr build -t ${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG} ${DOCKER_ROOT} -f ${DOCKER_FILE}
#set +x
bx cr build -t ${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${TIMESTAMP} ${DOCKER_ROOT} -f ${DOCKER_FILE}
bx cr image-inspect ${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${TIMESTAMP}
# Set PIPELINE_IMAGE_URL for subsequent jobs in stage (e.g. Vulnerability Advisor)
export PIPELINE_IMAGE_URL="$REGISTRY_URL/$REGISTRY_NAMESPACE/$IMAGE_NAME:$IMAGE_TAG"
bx cr images --restrict ${REGISTRY_NAMESPACE}/${IMAGE_NAME}
echo "=========================================================="
echo "COPYING ARTIFACTS needed for deployment and testing (in particular build.properties)"
echo "Checking archive dir presence"
if [ -z "${ARCHIVE_DIR}" ]; then
echo -e "Build archive directory contains entire working directory."
else
echo -e "Copying working dir into build archive directory: ${ARCHIVE_DIR} "
mkdir -p ${ARCHIVE_DIR}
find . -mindepth 1 -maxdepth 1 -not -path "./$ARCHIVE_DIR" -exec cp -R '{}' "${ARCHIVE_DIR}/" ';'
fi
# If already defined build.properties from prior build job, append to it.
cp build.properties $ARCHIVE_DIR/ || :
# IMAGE information from build.properties is used in Helm Chart deployment to set the release name
echo "IMAGE_NAME=${IMAGE_NAME}" >> $ARCHIVE_DIR/build.properties
echo "IMAGE_TAG=${IMAGE_TAG}" >> $ARCHIVE_DIR/build.properties
# REGISTRY information from build.properties is used in Helm Chart deployment to generate cluster secret
echo "REGISTRY_URL=${REGISTRY_URL}" >> $ARCHIVE_DIR/build.properties
echo "REGISTRY_NAMESPACE=${REGISTRY_NAMESPACE}" >> $ARCHIVE_DIR/build.properties
echo "GIT_BRANCH=${GIT_BRANCH}" >> $ARCHIVE_DIR/build.properties
echo "TIMESTAMP=${TIMESTAMP}" >> $ARCHIVE_DIR/build.properties
echo "File 'build.properties' created for passing env variables to subsequent pipeline jobs:"
cat $ARCHIVE_DIR/build.properties
Step 2 : Validate
Security Check
#!/bin/bash
# uncomment to debug the script
#set -x
# View build properties
if [ -f build.properties ]; then
echo "build.properties:"
cat build.properties
else
echo "build.properties : not found"
fi
PIPELINE_IMAGE_URL=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${TIMESTAMP}
echo "PIPELINE_IMAGE_URL=${PIPELINE_IMAGE_URL}"
echo "REGISTRY_URL=${REGISTRY_URL}"
echo "REGISTRY_NAMESPACE=${REGISTRY_NAMESPACE}"
echo "IMAGE_NAME=${IMAGE_NAME}"
echo "IMAGE_TAG=${IMAGE_TAG}"
echo "TIMESTAMP=${TIMESTAMP}"
bx cr images --restrict ${REGISTRY_NAMESPACE}/${IMAGE_NAME}
echo -e "Checking vulnerabilities in image: ${PIPELINE_IMAGE_URL}"
for ITER in {1..30}
do
set +e
STATUS=$( bx cr va -e -o json ${PIPELINE_IMAGE_URL} | jq -r '.[0].status' )
set -e
# Possible status from Vulnerability Advisor: OK, UNSUPPORTED, INCOMPLETE, UNSCANNED, FAIL, WARN
if [[ ${STATUS} != "INCOMPLETE" && ${STATUS} != "UNSCANNED" ]]; then
break
fi
echo -e "${ITER} STATUS ${STATUS} : A vulnerability report was not found for the specified image."
echo "Either the image doesn't exist or the scan hasn't completed yet. "
echo "Waiting for scan to complete..."
sleep 10
done
set +e
bx cr va -e ${PIPELINE_IMAGE_URL}
set -e
STATUS=$( bx cr va -e -o json ${PIPELINE_IMAGE_URL} | jq -r '.[0].status' )
[[ ${STATUS} == "OK" ]] || [[ ${STATUS} == "UNSUPPORTED" ]] || [[ ${STATUS} == "WARN" ]] || { echo "ERROR: The vulnerability scan was not successful, check the OUTPUT of the command and try again."; exit 1; }
Step 3 : Deploy
Verify K8S deployments file
#!/bin/bash
# uncomment to debug the script
#set -x
echo "IMAGE_NAME=${IMAGE_NAME}"
echo "IMAGE_TAG=${IMAGE_TAG}"
echo "REGISTRY_URL=${REGISTRY_URL}"
echo "REGISTRY_NAMESPACE=${REGISTRY_NAMESPACE}"
echo "DEPLOYMENT_FILE=${DEPLOYMENT_FILE}"
# Fix : after getting error with IMAGE_TAG variable
IMAGE_TAG=${TIMESTAMP}
# Input env variables from pipeline job
echo "PIPELINE_KUBERNETES_CLUSTER_NAME=${PIPELINE_KUBERNETES_CLUSTER_NAME}"
if [ -z "${CLUSTER_NAMESPACE}" ]; then CLUSTER_NAMESPACE=default ; fi
echo "CLUSTER_NAMESPACE=${CLUSTER_NAMESPACE}"
echo "=========================================================="
echo "DEPLOYING using manifest"
echo -e "Updating ${DEPLOYMENT_FILE} with image name: ${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG}"
if [ -z "${DEPLOYMENT_FILE}" ]; then DEPLOYMENT_FILE=deployment.yml ; fi
if [ -f ${DEPLOYMENT_FILE} ]; then
sed -i "s~^\([[:blank:]]*\)image:.*$~\1image: ${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG}~" ${DEPLOYMENT_FILE}
cat ${DEPLOYMENT_FILE}
else
echo -e "${red}Kubernetes deployment file '${DEPLOYMENT_FILE}' not found${no_color}"
exit 1
fi
set -x
kubectl apply --namespace ${CLUSTER_NAMESPACE} -f ${DEPLOYMENT_FILE}
set +x
echo ""
echo "=========================================================="
IMAGE_REPOSITORY=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}
echo -e "CHECKING deployment status of ${IMAGE_REPOSITORY}:${IMAGE_TAG}"
echo ""
for ITERATION in {1..30}
do
DATA=$( kubectl get pods --namespace ${CLUSTER_NAMESPACE} -o json )
NOT_READY=$( echo $DATA | jq '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | select(.ready==false) ' )
if [[ -z "$NOT_READY" ]]; then
echo -e "All pods are ready:"
echo $DATA | jq '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | select(.ready==true) '
break # deployment succeeded
fi
REASON=$(echo $DATA | jq '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | .state.waiting.reason')
echo -e "${ITERATION} : Deployment still pending..."
echo -e "NOT_READY:${NOT_READY}"
echo -e "REASON: ${REASON}"
if [[ ${REASON} == *ErrImagePull* ]] || [[ ${REASON} == *ImagePullBackOff* ]]; then
echo "Detected ErrImagePull or ImagePullBackOff failure. "
echo "Please check proper authenticating to from cluster to image registry (e.g. image pull secret)"
break; # no need to wait longer, error is fatal
elif [[ ${REASON} == *CrashLoopBackOff* ]]; then
echo "Detected CrashLoopBackOff failure. "
echo "Application is unable to start, check the application startup logs"
break; # no need to wait longer, error is fatal
fi
sleep 5
done
APP_NAME=$(kubectl get pods --namespace ${CLUSTER_NAMESPACE} -o json | jq -r '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | .name' | head -n 1)
echo -e "APP: ${APP_NAME}"
echo "DEPLOYED PODS:"
kubectl describe pods --selector app=${APP_NAME} --namespace ${CLUSTER_NAMESPACE}
if [ ! -z "${APP_NAME}" ]; then
APP_SERVICE=$(kubectl get services --namespace ${CLUSTER_NAMESPACE} -o json | jq -r ' .items[] | select (.spec.selector.app=="'"${APP_NAME}"'") | .metadata.name ')
echo -e "SERVICE: ${APP_SERVICE}"
echo "DEPLOYED SERVICES:"
kubectl describe services ${APP_SERVICE} --namespace ${CLUSTER_NAMESPACE}
fi
#echo "Application Logs"
#kubectl logs --selector app=${APP_NAME} --namespace ${CLUSTER_NAMESPACE}
echo ""
if [[ ! -z "$NOT_READY" ]]; then
echo ""
echo "=========================================================="
echo "DEPLOYMENT FAILED"
exit 1
fi
echo ""
echo "=========================================================="
echo "DEPLOYMENT SUCCEEDED"
if [ ! -z "${APP_SERVICE}" ]; then
echo ""
IP_ADDR=$(bx cs workers ${PIPELINE_KUBERNETES_CLUSTER_NAME} | grep normal | head -n 1 | awk '{ print $2 }')
PORT=$( kubectl get services --namespace ${CLUSTER_NAMESPACE} | grep ${APP_SERVICE} | sed 's/.*:\([0-9]*\).*/\1/g' )
echo ""
echo -e "VIEW THE APPLICATION AT: http://${IP_ADDR}:${PORT}"
fi
Deploy to K8S
#!/bin/bash
# uncomment to debug the script
#set -x
echo "IMAGE_NAME=${IMAGE_NAME}"
echo "IMAGE_TAG=${IMAGE_TAG}"
echo "REGISTRY_URL=${REGISTRY_URL}"
echo "REGISTRY_NAMESPACE=${REGISTRY_NAMESPACE}"
echo "DEPLOYMENT_FILE=${DEPLOYMENT_FILE}"
IMAGE_TAG=${TIMESTAMP}
# Input env variables from pipeline job
echo "PIPELINE_KUBERNETES_CLUSTER_NAME=${PIPELINE_KUBERNETES_CLUSTER_NAME}"
if [ -z "${CLUSTER_NAMESPACE}" ]; then CLUSTER_NAMESPACE=default ; fi
echo "CLUSTER_NAMESPACE=${CLUSTER_NAMESPACE}"
echo "=========================================================="
echo "DEPLOYING using manifest"
echo -e "Updating ${DEPLOYMENT_FILE} with image name: ${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG}"
if [ -z "${DEPLOYMENT_FILE}" ]; then DEPLOYMENT_FILE=deployment.yml ; fi
if [ -f ${DEPLOYMENT_FILE} ]; then
sed -i "s~^\([[:blank:]]*\)image:.*$~\1image: ${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG}~" ${DEPLOYMENT_FILE}
cat ${DEPLOYMENT_FILE}
else
echo -e "${red}Kubernetes deployment file '${DEPLOYMENT_FILE}' not found${no_color}"
exit 1
fi
set -x
kubectl apply --namespace ${CLUSTER_NAMESPACE} -f ${DEPLOYMENT_FILE}
set +x
echo ""
echo "=========================================================="
IMAGE_REPOSITORY=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}
echo -e "CHECKING deployment status of ${IMAGE_REPOSITORY}:${IMAGE_TAG}"
echo ""
for ITERATION in {1..30}
do
DATA=$( kubectl get pods --namespace ${CLUSTER_NAMESPACE} -o json )
NOT_READY=$( echo $DATA | jq '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | select(.ready==false) ' )
if [[ -z "$NOT_READY" ]]; then
echo -e "All pods are ready:"
echo $DATA | jq '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | select(.ready==true) '
break # deployment succeeded
fi
REASON=$(echo $DATA | jq '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | .state.waiting.reason')
echo -e "${ITERATION} : Deployment still pending..."
echo -e "NOT_READY:${NOT_READY}"
echo -e "REASON: ${REASON}"
if [[ ${REASON} == *ErrImagePull* ]] || [[ ${REASON} == *ImagePullBackOff* ]]; then
echo "Detected ErrImagePull or ImagePullBackOff failure. "
echo "Please check proper authenticating to from cluster to image registry (e.g. image pull secret)"
break; # no need to wait longer, error is fatal
elif [[ ${REASON} == *CrashLoopBackOff* ]]; then
echo "Detected CrashLoopBackOff failure. "
echo "Application is unable to start, check the application startup logs"
break; # no need to wait longer, error is fatal
fi
sleep 5
done
APP_NAME=$(kubectl get pods --namespace ${CLUSTER_NAMESPACE} -o json | jq -r '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | .name' | head -n 1)
echo -e "APP: ${APP_NAME}"
echo "DEPLOYED PODS:"
kubectl describe pods --selector app=${APP_NAME} --namespace ${CLUSTER_NAMESPACE}
if [ ! -z "${APP_NAME}" ]; then
APP_SERVICE=$(kubectl get services --namespace ${CLUSTER_NAMESPACE} -o json | jq -r ' .items[] | select (.spec.selector.app=="'"${APP_NAME}"'") | .metadata.name ')
echo -e "SERVICE: ${APP_SERVICE}"
echo "DEPLOYED SERVICES:"
kubectl describe services ${APP_SERVICE} --namespace ${CLUSTER_NAMESPACE}
fi
#echo "Application Logs"
#kubectl logs --selector app=${APP_NAME} --namespace ${CLUSTER_NAMESPACE}
echo ""
if [[ ! -z "$NOT_READY" ]]; then
echo ""
echo "=========================================================="
echo "DEPLOYMENT FAILED"
exit 1
fi
echo ""
echo "=========================================================="
echo "DEPLOYMENT SUCCEEDED"
if [ ! -z "${APP_SERVICE}" ]; then
echo ""
IP_ADDR=$(bx cs workers ${PIPELINE_KUBERNETES_CLUSTER_NAME} | grep normal | head -n 1 | awk '{ print $2 }')
PORT=$( kubectl get services --namespace ${CLUSTER_NAMESPACE} | grep ${APP_SERVICE} | sed 's/.*:\([0-9]*\).*/\1/g' )
echo ""
echo -e "VIEW THE APPLICATION AT: http://${IP_ADDR}:${PORT}"
fi
Problems
âš Cluster kube always on "Pending"
After creating k8s cluster , the latter is still always in 'Pending' status
✅ Resolution : Update Kubernetes cluster to last proposed version
âš Can't publish docker image on registry
On the BUILD step of the Toolchain FAILED , because of the name of the image
The supplied image name was invalid.
Correct the image name, and try again.
✅ add a TIMESTAMP to docker image name . Replace the ${TAG_NAME} with ${TIMESTAMP}
That's all , thank you for reading and your feedback is highly appreciated
Top comments (0)