Hello ποΈ !
Following the mechanisms described here to send CSV files into Kafka in JSON using kcat through shell pipelines, we packed π¦ all of that in a new home-made tool named producer-file
. Here are the features :
- π¨ Sending line by line
- πͺ CSV to JSON conversion
- π¦ Support message headers by file and/or by line
- β‘οΈ Delta-changes mode (only send changes occurred after last send)
βοΈ More about the shell way :
β¨οΈ Pipe xlsx files into/from Kafka... From cli with (k)cat π
adriens for opt-nc γ» Oct 10 '22
Motivations
Using jq
+ kcat
with shell pipelines do the job well, but this doesn't fit the company's requirements today about software packaging and logging capabilities π€·π»ββοΈ.
π½οΈ Demo
Short video demonstrating how to use this tool to send a CSV file in JSON into Kafka within a Docker environnement :
Top comments (2)
Here is a related and worhtwhile content :
opt-nc / atelier-spring-kafka
Projet d'exemple d'utilisation de la lib `org.springframework.kafka:spring-kafka`
Ce projet explique comment dΓ©velopper un producer/consumer sans la lib interne historique au profit de celle dΓ©veloppΓ©e et maintenue par Spring.
Projet d'exemple d'utilisation de la lib
org.springframework.kafka:spring-k...
kafka
docker
&docker-compose
(si utilisation dekafka.yml
)Avant de passer Γ la suite, prendre soin de consulter les ressources ci-dessous:
demo.user
Entrer dans le container
kafka
:CrΓ©er le producer :
Copier les lignes :
demo.sms
Initialiserβ¦