At first when I started writing Flink application I thought of it as any other Java console application and started writing the same using Spring Boot Framework. However I soon realized that running the application in Flink is a bit different than executing a Java application on say command line using
java -jar <jar_path>
My application was connecting to Kafka for reading stream from the topic and performing operation on the streams. Our Kafka cluster is secured using SSL and ACL that means I need certificate to read from the topic and publish the processed output on another topic and the certificates will change per environment. In the past I had written a Java consumer app which works on the similar principle but it was a containerized app and passing parameters to Docker container is simple enough using,
-e <env_variable>=<value>
However this was a Flink application and things are quite different since the Flink manager has all the controls to run the application. After breaking my head for several hour(s) I figured out that its like running any other Java app. Now the beauty of Java app is you can pass the properties you are reading from application.properties from the arguments without writing a single line of code as,
java -jar <jar_path> --<property_name>=<value>
So I thought why not try the same thing with Flink application after all its a Java app. So I ran below command,
~/flink/bin/flink run app.jar --brokers=Broker1:9093 --topic=some-topic
And to my surprise this worked and I could verify that by checking the logs on the Flink dashboard. Since this option worked I thought what if I pass the property and their values in a properties file, why not give it a try by running below command,
~/flink/bin/flink run app.jar -Dspring.config.name=<full-path>/application.properties
This also worked since I had implemented the application using Spring boot, I could pass the properties file with all the required values.
You can choose either of the method according to your preference and it should work seamlessly.
Happy Coding :)
Top comments (0)