DEV Community

Cover image for 🎯 Sending flat files to Kafka
opt-nc profile image Daniel Santos
Daniel Santos for opt-nc

Posted on

🎯 Sending flat files to Kafka

#kafka #docker #productivity #devops

Hello πŸ–οΈ !

Following the mechanisms described here to send CSV files into Kafka in JSON using kcat through shell pipelines, we packed πŸ“¦ all of that in a new home-made tool named producer-file. Here are the features :

  • πŸ’¨ Sending line by line
  • πŸ’ͺ CSV to JSON conversion
  • πŸ”¦ Support message headers by file and/or by line
  • ⚑️ Delta-changes mode (only send changes occurred after last send)

☝️ More about the shell way :

opt-nc

⌨️ Pipe xlsx files into/from Kafka... From cli with (k)cat πŸ™€

adriens for opt-nc ・ Oct 10 '22

#tutorial #productivity #kafka #linux

Motivations

Using jq + kcat with shell pipelines do the job well, but this doesn't fit the company's requirements today about software packaging and logging capabilities πŸ€·πŸ»β€β™‚οΈ.

πŸ“½οΈ Demo

Short video demonstrating how to use this tool to send a CSV file in JSON into Kafka within a Docker environnement :

πŸ“– Ressources

Top comments (2)

Collapse
 
adriens profile image
adriens •

Here is a related and worhtwhile content :

GitHub logo opt-nc / atelier-spring-kafka

Projet d'exemple d'utilisation de la lib `org.springframework.kafka:spring-kafka`

Project Website Confluent tutorial

❔ A propos

Ce projet explique comment dΓ©velopper un producer/consumer sans la lib interne historique au profit de celle dΓ©veloppΓ©e et maintenue par Spring.

βœ‹Pourquoi Kafka

image

πŸ“ Description

Projet d'exemple d'utilisation de la lib org.springframework.kafka:spring-k...

βœ… PrΓ©-requis

  • Instance kafka
  • docker & docker-compose (si utilisation de kafka.yml)

πŸ‘Ά Kafka ?! WTF ❔

Avant de passer Γ  la suite, prendre soin de consulter les ressources ci-dessous:

πŸ§‘β€πŸŽ“ Supports custome

πŸ“œ Scripts

1️⃣ Initialiser le topic demo.user

Entrer dans le container kafka :

docker exec -ti kafka bash

CrΓ©er le producer :

kafka-console-producer.sh --broker-list kafka:9092 --topic demo.user --property "parse.key=true" --property "key.separator=|"

Copier les lignes :

112233|{"phoneNumber":"112233", "firstName":"Hubert", "lastName":"Bonisseur de la Bath"}
998877|{"phoneNumber":"998877", "firstName":"Jean", "lastName":"Soudajman"}
446655|{"phoneNumber":"446655", "firstName":"Henri", "lastName":"Tathan"}

2️⃣ Ecriture automatique dans le topic demo.sms

Initialiser…

View on GitHub
Collapse
 
adriens profile image
adriens •

Read next

adnanlatif profile image

From 41 Minutes to 8 Minutes: How I Made Our CI/CD Pipeline 5x Faster

Adnan Latif -

bobur profile image

From a Content Creator to an AI Tool Maker

Bobur Umurzokov -

wallacefreitas profile image

Docker Networking: A Comprehensive Guide

Wallace Freitas -

muzaffar640 profile image

Why I Can't Stop Talking About Arc Browser: A Developer's Story

Muzaffar Hossain -