The disconnect between Tech and Client
I had a talk with one of my clients which went like this:
Tekton and Kubernetes are nice and all, our devs seem to be quite happy, but where do I see Test results? Jenkins has a feature like this, but now all developers use this tekton stuff and all these fancy console commands.
I just want to look at my reports and get a glimpse of how the softwaretests went. Those terminals don't really do that for me.
Can you get me something like that?
That's a spot-on observation by my client. A lot of Kubernetes tools are heavily focused on CLI-Usage, or in the "early stages of development", when features are higher up on the priority-list than shiny UIs.
One of these examples would be the tekton ecosystem. There are rarely any UIs, and if there are, like the Tekton Dashboard for example, they are very barebones.
Since I am a very CLI-Centric guy and by habit don't really think about using UIs that often, the request of my client ran me over like a truck.
So we dug around a bit and found a very promising tool.
The Allure Framework.
What is allure and why should I use it?
The allure framework does a few things:
- Generating Results in json-Format (thank god no more xml)
- Report creation
- History-Data Collection
All of this is then slapped together in a pretty static-html report, ready to be served on your favorite server.
If you don't have one on hand, the optionally downloadable cli can also spin up a Webserver via the allure serve pathToReportDir command and host the reports itself. Magical!
To give you a little tour of how a report looks, have a gif!
I tried to get a gif to work, but 80% of the time I get errors from dev, and the ones that render turn into potato-quality.
No idea what this cloudinary thing does, but it's not great..
So if this is too small/ugly for you, which I can probably only blame on myself, don't worry! I'll show you how to spin up a demo yourself :)
How does it work?
First off, Allure has tons of connectors for different frameworks and languages, which are documented here.
But let me warn you: The documentation seems to be very minimal right now. There is a lot of trial and error involved to get what you want
To be fair, it seems to be a very powerful tool with many supported languages and Frameworks. Documenting everything to the smallest detail would probably be an enormous task.
My client was especially interested in the Cucumber and JUnit5 support. As of now we roughly finished implementing a continous integration pipeline with Report generation for JUnit5 Testsuites. Cucumber is probably following as soon as I find the time.
Let's test stuff!
To get allure to generate a report for you, all you need is the following additions to your pom.xml:
<properties>
<surefire-plugin.version>2.22.2</surefire-plugin.version>
<allure.version>2.13.3</allure.version>
<aspectj.version>1.9.5</aspectj.version>
</properties>
<dependency>
<groupId>io.qameta.allure</groupId>
<artifactId>allure-junit5</artifactId>
<version>${allure.version}</version>
<scope>test</scope>
</dependency>
<build>
<plugins>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire-plugin.version}</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
<argLine>
-javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
</argLine>
<systemPropertyVariables>
<allure.results.directory>${project.build.directory}/allure-results</allure.results.directory
<java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
</systemPropertyVariables>
</configuration>
<dependencies>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>${aspectj.version}</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>io.qameta.allure</groupId>
<artifactId>allure-maven</artifactId>
<version>2.10.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<reportVersion>${allure.version}</reportVersion>
<allureDownloadUrl>https://repo.maven.apache.org/maven2/io/qameta/allure/allure-commandline/${allure.version}/allure-commandline-${allure.version}.zip</allureDownloadUrl>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
Allure also brings its own set of annotations for Testcases.
Here are a few I like to use:
@Feature("Some Name")
@Epic("Some Name")
@Story("Some Name")
@Severity(SeverityLevel.CRITICAL)
@Description("non-dev readable description")
@Link("link to JIRA-Tickets or some cool resources)
Want to give it a go?
Here is a simple Dockerfile I slapped together, that needs minimal work from your side.
Just create a simple maven project, configure your pom.xml like I've shown above, add some tests, annotate them and give it a run for it's money with a mvn package!
FROM maven:3.6.3-jdk-11 as builder
WORKDIR /opt/service
# This is for caching purposes in local builds
# That way you don't have to download the whole internet every time your source changes
COPY pom.xml .
RUN mvn dependency:go-offline
COPY src/ src/
# Run the Build and generate a report
RUN mvn clean install && mvn allure:report
FROM nginx:1.19.0-alpine as server
# Copy the report and serve it via nginx
COPY --from=builder /opt/service/target/site/allure-maven-plugin/ /usr/share/nginx/html/
The challenge of CIs
Now we have a nice tool, but what now?
I quickly noticed that those reports may be nice, but if I am going to deploy these reports in a container, I need a little bit more than what I showed you earlier.
Allures history-feature can't do anything this way. Allure can't create a history if there is no data of previous runs to be found.
So let's create a solution for this!
I thought I had a nifty idea by designing the report generation process like this:
As you can see, I introduced a new component. MinIO as S3 Storage Provider. Why minIO? Because it's commandline client has a great --newer-than/--older-than Feature. So minio could do the sorting work for me, I don't have to do that myself!
If I wanted to create a report with Data up to 7 Days old, it would be as easy as
mc cp --recursive --newer-than 7d minio/allure/results/ results/
allure generate results/ -o report/
mc cp --recursive report/ minio/allure/reports-history/$(date +%y-%m-%d-%H%M)/
And I'd get a report with Data for 7 Days. It would get stored in minio and would be ready to be put into a container.
Honestly, I loved that idea and was kind of proud of that solution. If you have a better idea, I am all ears!
Let's containerize it!
So how would this look inside a container?
FROM maven:3.6.3-jdk-11 as builder
WORKDIR /opt/service
# This is for caching purposes in local builds
# That way you don't have to download the whole internet every time your source changes
COPY pom.xml .
RUN mvn dependency:go-offline
COPY src/ src/
RUN mvn clean install && mvn allure:report
FROM openjdk:14-alpine as generator
WORKDIR /tmp
COPY --from=builder /opt/service/target/allure-results/* results/
COPY --from=builder /opt/service/target/site/allure-maven-plugin/* reports/
RUN apk add curl tar gzip \
&& curl -L https://bintray.com/qameta/maven/download_file?file_path=io%2Fqameta%2Fallure%2Fallure-commandline%2F2.13.4%2Fallure-commandline-2.13.4.tgz | tar xvz \
&& curl -LO https://dl.min.io/client/mc/release/linux-amd64/mc \
&& chmod +x mc \
&& ln -s $PWD/mc /usr/bin/mc \
&& mc config host add minio http://ipofminio:9000 minioadmin minioadmin \
&& mc mb -p minio/allure \
&& mc cp --recursive results/* minio/allure/results/ \
&& mc cp --recursive reports/* minio/allure/reports/$(date +'%Y-%m-%d-%H%M')/ \
&& mc cp --recursive minio/allure/results/ results/ \
&& allure-2.13.4/bin/allure generate results/ -o reports-with-history-data/ \
&& mc cp --recursive reports-with-history-data/ minio/allure/reports-history/$(date +'%Y-%m-%d-%H%M')/
FROM nginx:1.19.0-alpine as server
COPY --from=generator /tmp/reports-with-history-data/ /usr/share/nginx/html/
What we did is, we introduced another stage with FROM openjdk:14-alpine as generator
in which we do all the upload/fetching/sorting stuff for our history-reports.
All you would need to build this is a minio instance, which is, for local testing, easily deployed via:
docker run -p 9000:9000 --name minio --rm minio/minio server /data
But what about tekton and kubernetes?
You may ask this, rightfully, since I teased it in the beginning.
Making this run in tekton was pretty easy, even though it needed a little bit more prep-work.
First off would be to create an Image, that would be buildable outside of Kubernetes, and still has all the connections to minio we need.
The Dockerfile up there would probably crash the build, since a minio running in kubernetes wouldn't be reachable by your CI, which normally is running outside of kubernetes. So let's modify our Container a little bit to work with those requirements.
What do we have to do?
First off, I want to split up my container into two.
- a reporting container that builds the reports and pushes them to minio
- a nginx container that pulls the report and serves it
Why do I want this?
Having to build an image that contains your report every time the report changes, doesn't seem viable to me. It takes unnecessary space in your docker registry and doesn't bring that much benefit. IMO it would be better to have a dynamic container that can be parameterized and does the whole result-pulling, generating and archiving in it's entrypoint. So let's do just that!
Allure-Reporter Container
Dockerfile.allure
FROM openjdk:14-alpine
WORKDIR /tmp
RUN apk add curl tar gzip \
&& curl -L https://bintray.com/qameta/maven/download_file?file_path=io%2Fqameta%2Fallure%2Fallure-commandline%2F2.13.4%2Fallure-commandline-2.13.4.tgz | tar xvz \
&& curl -LO https://dl.min.io/client/mc/release/linux-amd64/mc \
&& chmod +x mc \
&& ln -s $PWD/mc /usr/bin/mc \
&& ln -s $PWD/allure-2.13.4/bin/allure /usr/bin/allure \
&& allure --version \
&& mc --version
WORKDIR /opt/myproject
VOLUME /opt/myproject/results
COPY entrypoint-allure.sh /usr/local/bin/entrypoint
RUN chmod +x /usr/local/bin/entrypoint
USER nobody
ENTRYPOINT bash -c entrypoint
entrypoint-allure.sh
#!/bin/bash
set -e
# Capture the name of the project, so we can differentiate results in minio
CURRENT_DATE=$(date '+%Y%m%d%H%M')
# Gather all results you can find
# Should be Emulated via Volume Mount locally
mkdir results
for resultDir in $(find . -type d -iname allure-results);do
cp $resultDir/* results
done
# Generate the actual allure report of this testrun and publish it
allure generate results/ -o allure-report/
# Configure Minio and make sure the bucket allure exists
mc config host add minio $MINIO_ENDPOINT $MINIO_ACCESS_KEY $MINIO_SECRET_KEY
mc mb -p minio/allure
# Publish the results
mc cp --recursive results/ minio/allure/$PROJECTNAME/allure-results/
# Publish the report
mc cp --recursive allure-report/ minio/allure/$PROJECTNAME/allure-report-independent/$CURRENT_DATE/
# Get all results, including those from past runs, and generate a history report to publish
mc cp --recursive minio/allure/$PROJECTNAME/allure-results/ results/
allure generate results -o allure-report-history/
mc cp --recursive allure-report-history/ minio/allure/$PROJECTNAME/allure-report-history/$CURRENT_DATE/
Allure-Server Container
Dockerfile.nginx:
FROM nginx:1.19.0-alpine as server
ARG MINIO_ENDPOINT=http://minio.default.svc.cluster.local:9000
ARG MINIO_SECRET_KEY=minioadmin
ARG MINIO_SECRET_PASS=minioadmin
ARG PROJECTNAME=myproject
RUN curl -LO https://dl.min.io/client/mc/release/linux-amd64/mc \
&& chmod +x mc \
&& ln -s $PWD/mc /usr/bin/mc \
&& ln -s $PWD/allure-2.13.4/bin/allure /usr/bin/allure \
&& mc --version
COPY entrypoint-nginx.sh /usr/local/bin/entrypoint
RUN chmod +x /usr/local/bin/entrypoint
USER nobody
ENTRYPOINT bash -c entrypoint
entrypoint-nginx.sh:
#!/bin/bash
set -e
# Capture the name of the project, so we can differentiate results in minio
CURRENT_DATE=$(date '+%Y%m%d%H%M')
# Configure Minio and make sure the bucket allure exists
mc config host add minio $MINIO_ENDPOINT $MINIO_ACCESS_KEY $MINIO_SECRET_KEY
mc mb -p minio/allure
NEWEST_REPORT=$(mc find minio/allure/$PROJECTNAME/allure-report-history/ --maxdepth 1 --newer-than 1d | cut -d'/' -f5- | sort -r | head -n1)
NEWEST_REPORT=${NEWEST_REPORT%/}
mc cp --recursive minio/allure/$PROJECTNAME/allure-report-history/$NEWEST_REPORT report
rm /opt/nginx/html/index.html
cp -r report/$NEWEST_REPORT/* /opt/nginx/html
nginx
With these two containers and four files, you should be pretty much set to go.
We split our all-in-one-container into two separate containers, that are buildable without a connection to minio.
Our reporter and it's nginx-server.
Let's go k8s
I am assuming, you have a running kubernetes setup, if not, you can take a look at my "lazy k8s solution" to running kubernetes, which can take care of that for you.
As final part of the tekton integration, you can use this task to run the report-generator (The first Container up there) and deploy the nginx-server via the file that is referenced in the kubectl step.
allure-reporter-task.yaml:
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: allure-reporter-task
namespace: tekton
spec:
workspaces:
- name: git
params:
- name: minioEndpoint
default: http://minio.default.svc.cluster.local:9000
- name: minioAccessKey
default: minioadmin
- name: minioSecretKey
default: minioadmin
- name: pathToDeploymentYaml
default: k8s/allure-server.yaml
steps:
- name: generate-and-publish-allure-report
image: allure-report-generator:1.0.0
command:
- /bin/bash
args:
- -c
- |
cd /workspace/git/
entrypoint
env:
- name: MINIO_ENDPOINT
value: $(params.minioEndpoint)
- name: MINIO_ACCESS_KEY
value: $(params.minioAccessKey)
- name: MINIO_SECRET_KEY
value: $(params.minioSecretKey)
- name: serve-allure-report
image: lachlanevenson/k8s-kubectl
command: ["kubectl"]
args:
- "apply"
- "-f"
- "$(workspaces.git.path)/$(params.pathToDeploymentYaml)"
To get your allure-server deployed, the following file should be in your apps git-repository, so tekton can find it during a taskrun
Example of k8s/allure-server.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: allure-server
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: allure-server
template:
metadata:
labels:
app: allure-server
spec:
containers:
- name: allure-server
image: your-registry/allure-server:1.0.0
imagePullPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: allure-server
namespace: default
spec:
selector:
app: allure-server
type: ClusterIP
ports:
- port: 80
targetPort: 80
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: allure-server-ingress-route
namespace: default
spec:
entryPoints:
- http
routes:
- match: Host(`localhost`) && PathPrefix(`/allure`)
kind: Rule
services:
- name: allure-server
port: 80
middlewares:
- name: allure-stripprefix
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: allure-stripprefix
namespace: default
spec:
stripPrefix:
prefixes:
- /allure
The difference in this approach, in comparison to the Dockerfile I demonstrated up there, I put all the logic of the Dockerfile into entrypoint scripts.
You know, all the copying, report generation, sorting, everything.
That way I don't have to push the same image over and over into my registry, just because the report was updated. I can just have one reporter image that dynamically pulls the data it needs from storage.
Also, the images you need can be built by a CI, which normally shouldn't have any access to storage solutions like minio. So a build of the Dockerfiles with a minio-connection would break your build. That's exactly why we put all the networking stuff into the entrypoints, as they will not be executed by your CI.
The future
I am already working on a practical demo of allure and tekton with zero-to-k8s. So you don't have to do all the copying and pasting from here.
End of story
But for now, I leave you with this.
A random assortment of stuff, just to get some nice shiny UI for your testresults :D
If you have questions, suggestions, or other feedback, feel free to hammer away at your keyboard :)
Top comments (0)