What we want to implement
What we have (pre-requisite)
- A kubernetes cluster (kind cluster in my case => no external urls possible)
- Knative serving installed (installation script)
- Knative eventing installed (installation script)
- Kafka related broker, source, sink .. (installation script)
- A kafka cluster (internal cluster installation script) - broker url will be
http://my-cluster.kafka-ns.cluster.local:9092
. Note: I am using a cluster hosted on173.39.62.80:9092
in my examples (installation link) - Microservices invoked via POST:
6a. Builder-MS - external (emits CE with "[BUILD] ")
6b. Tester-MS - internal (sequence: emits CE with "..Create TB..", "Run Tests", "..Clean TB..")
6c. Committer-MS - external (emits CE with "[COMMIT]")
6d. Alerter-MS - internal (records and displays the events - for demo; can be event-display service with
kn service create alert-ms --image=gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
)
Intended workflow
GitHubSource
- is registered with GitHub which pushes events (PR related) toGitHubSource
(installation)
Note: Since I am working on localhost/kind, I will skip this part
1a. EveryEventSource
needs aSink
; in this case we usehttp://Builder-MS-URL
as theSink
Note: I will directly invoke thishttp://Builder-MS-URL
via POSTBuilder MS - Microservice that builds and posts a
build.done{}
with aCe-buildstatus=xyz
event to Kafka topicbuild.done
KafkaSource - Subscribes to topic
build.done
and passes it on-toSink=Broker:kb-build
3a. Broker kb-build passes on the event totrigger:kt-build
Trigger(kt-build) - if
buildstatus=pass
pass the event toSink:sequence
, ifbuildstatus=fail
pass it tohttp://Alerter-MS-URL
Sequence - calls 3 services
create-tb-ms
,run-tests-ms
,clean-tb-ms
and passes the results on-toSink=Broker:kb-test
Broker(kb-test)/Trigger(kt-test) - if
teststatus=pass
pass the event toSink:KafkaSink
, ifteststatus=fail
pass it tohttp://Alerter-MS-URL
KafkaSink - posts
test.done{}
withteststatus=pass
to kafka topic:test.done
KafkaSource - Subscribes to topic
test.done
and passes it on-toSink=http://Commit-MS-URL
Commit MS - Does commit process and posts a Cloud-event
Notes
- All the manifests are under github directory
- The demo uses
InMemory Channel
for the brokers when created with dummy.sh, use run.sh for Kafka based broker implementation - Steps #6, #7 and #8 can be avoided by directly calling Commit MS; I use it to add
teststatus
withCeOverrides
which is only available through anEventSource
- Use
SinkBinding
if you directly want to call an EventSink - Tester MS is always returning
teststatus=pass
in my case as it is a dummy sequence - Kafka brokers and triggers need to be in the same namespace to work properly
Test run
# post installation, these are the knative objects
❯ kn sources list -A
NAMESPACE NAME TYPE RESOURCE SINK READY
kafka-ns ks-build-done KafkaSource kafkasources.sources.knative.dev broker:kb-test True
kafka-ns ks-test-done KafkaSource kafkasources.sources.knative.dev broker:kb-commit True
❯ kn service list -A
NAMESPACE NAME URL LATEST AGE CONDITIONS READY REASON
default alert-ms http://alert-ms.default.svc.cluster.local alert-ms-00001 17h 3 OK / 3 True
default clean-tb-ms http://clean-tb-ms.default.svc.cluster.local clean-tb-ms-00001 17h 3 OK / 3 True
default create-tb-ms http://create-tb-ms.default.svc.cluster.local create-tb-ms-00001 17h 3 OK / 3 True
default run-tests-ms http://run-tests-ms.default.svc.cluster.local run-tests-ms-00001 17h 3 OK / 3 True
❯ kubectl get sequences.flows.knative.dev
NAME URL AGE READY REASON
test-ms http://test-ms-kn-sequence-0-kn-channel.default.svc.cluster.local 17h True
❯ kn broker list -A
NAMESPACE NAME URL AGE CONDITIONS READY REASON
kafka-ns kb-build http://broker-ingress.knative-eventing.svc.cluster.local/kafka-ns/kb-build 17h 6 OK / 6 True
kafka-ns kb-commit http://broker-ingress.knative-eventing.svc.cluster.local/kafka-ns/kb-commit 17h 6 OK / 6 True
kafka-ns kb-test http://broker-ingress.knative-eventing.svc.cluster.local/kafka-ns/kb-test 17h 6 OK / 6 True
❯ kn trigger list -A
NAMESPACE NAME BROKER SINK AGE CONDITIONS READY REASON
kafka-ns kt-build-start kb-build http://builer-ms-url 17h 6 OK / 6 True
kafka-ns kt-commit-start kb-commit http://commiter-ms-url 17h 6 OK / 6 True
kafka-ns kt-test-start kb-test sequence:test-ms 17h 6 OK / 6 True
# trigger the workflow by calling `builder-ms-url`
❯ http POST http://builer-ms-url ...
HTTP/1.0 200 OK
Content-Length: 263
Date: Wed, 29 Mar 2023 08:36:44 GMT
Server: Werkzeug/2.0.3 Python/3.6.3
content-type: application/cloudevents+json
{
"buildstatus": "pass",
"data": {
"message": "[BUILD] BUGmn15840",
"completed": true
},
"id": "6110dd6e-f670-4ede-a41b-3f671219ae0d",
"source": "demo.build",
"specversion": "1.0",
"task": "build",
"time": "2023-03-29T08:36:37.353013+00:00",
"type": "build.pass"
}
## output events in the order of arrival
# 1. output from build
# ce-source: /apis/v1/namespaces/kafka-ns/kafkasources/ks-build-done#build.pass
# ce-buildstatus: pass
# ce-type: dev.knative.kafka.event
{
"message": "[BUILD] BUGmn15840",
"completed": true
}
#2. output from test
# ce-source: /apis/v1/namespaces/kafka-ns/kafkasources/ks-build-done#build.pass
# ce-buildstatus: pass
# ce-teststatus: pass
# ce-type: dev.knative.kafka.event
{
"id": 1234,
"message": "[BUILD] BUGmn15840...Create TB......Run tests......Clean TB..."
"completed": true
}
#3. output from commit
# ce-source: demo.commit
# ce-buildstatus: pass
# ce-teststatus: pass
# ce-commitstatus: pass
# ce-type: commit.pass
{
"message": "[BUILD] BUGmn15840...Create TB......Run tests......Clean TB... [COMMIT] ",
"completed": true
}
Top comments (0)