DEV Community

Connecting to Kafka cluster using SSL with Python

Aditya Kanekar on January 19, 2021

This article specifically talks about how to write producer and consumer for Kafka cluster secured with SSL using Python. I won't be getting into h...
Collapse
 
bcmdevtl profile image
bcmdevtl

Hi Aditya,
Great tutorial. Did you do up the consumer part of this as well? Tried to look but couldn't find it.

Collapse
 
adityakanekar profile image
Aditya Kanekar

Glad you liked it. No I have not added the consumer part yet. But it would be similar to producer. I will try to add that.

Collapse
 
giuseppepegasus profile image
Giuseppe • Edited

Hi Aditya, I'm trying to use your code for my SSL Kafka but whene I use the producer I receive the error on kafka Prodcer command:
self._wrap_ssl() File "/home/kafka/anaconda3/envs/kafka/lib/python3.8/site-packages/kafka/conn.py", line 473, in _wrap_ssl self._ssl_context.load_verify_locations(self.config['ssl_cafile'])

It seems doesn't find the CARoot file..but it is in that folder..
Can you help me? Thank you.

Collapse
 
adityakanekar profile image
Aditya Kanekar

I think I made a typo in the code, instead of CARoot.pem it was CARoote.pem. Can you please check once.

Collapse
 
giuseppepegasus profile image
Giuseppe

Great it works....but self signed certificate are not good for the produce because I receive the error:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)

any suggestion?

Thread Thread
 
sachin_tyagi profile image
Sachin Tyagi

Hello,
I also have same issue, anyone has any solution for it.

Thanks

Collapse
 
stefan_stojanovic profile image
Stefan Stojanovic

Please double check client.keystore.jks file before executing jkstopem.sh
It is possible that the file contains 2 aliases ( one for CARoot and one for Certificate )
In that case, you might have to hardcode Certificate or CA alias in jkstopem script or add an extra arg for it. Hope it helps :)

Collapse
 
__98067b65 profile image
Дмитрий Соловьёв • Edited

These lines look exactly the same

echo "Generating certificate.pem"
keytool -exportcert -alias $alias -keystore $keyStore -rfc -file $outputFolder/certificate.pem -storepass $password

echo "Generating CARoot.pem"
keytool -exportcert -alias $alias -keystore $keyStore -rfc -file $outputFolder/CARoot.pem -storepass $password

The only difference is the *pem file name (certificate.pem vs CARoot.pem)

As a result, after running the script, I got absolutely the same certificate.pem and CARoot.pem files.
Is this normal behavior?

Collapse
 
shubhendumadhukar profile image
Shubhendu Madhukar • Edited

I have a truststore.jks and password. But I don't have a keystore.jks. I am able to use the truststore and password to connect to the cluster using java code. Will this method work for truststore.jks also or is it only for keystore? I tried to generate those files and got certificate.pem but then I got an error stating "java.lang.Exception: Alias does not exist" and other files were not generated.

Collapse
 
adityakanekar profile image
Aditya Kanekar

The method will work for keystore.jks. It appears to me that you are using SSL Enabled Kafka without any ACL enabled. If thats true you might not be providing ssl_keyfile and ssl_certfile while connecting to Kafka through your Java code. In my view you should only provide the CARoot.pem to connect to Kafka other properties might not be required. Please try and let me know if that works.

Collapse
 
shailendra_jain_c0ae8daad profile image
Shailendra Jain • Edited

Hi Aditya,
While running this I get empty key.pem file. What should I do to get the populated key.pem file. Also I have multiple alias in the jks file, should I combine all the CARoot and certificate for different alias and use them while connecting.

Collapse
 
rameshwargupta97 profile image
Rameshwar Gupta

Hi Aditya,

Thanks for the detailed explanation for python version.
I'm also getting empty key.pem file. Can you please suggested something on it?

Collapse
 
code4purpose profile image
code4purpose

hi Aditya, First of all thanks for the detailed steps. It has been very helpful. I followed the exact steps. I am getting zero byte private key. My environment is centos with openssl version OpenSSL 1.0.2k-fips 26 Jan 2017. Any pointers will be highly appreciated. Thank you.