Repository navigation
Closed
Description
Hi,
SSL support it suppose to be working according to issue #581.
However, I've spent two days trying to get it working with Kafka 0.9.0. I can connect to to my kafka cluster with openssl s_client -connect domain.com:9093 -tls1 so I know it is configured correctly.
But when I try connecting with sarama I get errors:
[Sarama] 2016/04/20 11:16:39 client/metadata retrying after 250ms... (1 attempts remaining)
[Sarama] 2016/04/20 11:16:39 client/metadata fetching metadata for all topics from broker domain.com:9093
[Sarama] 2016/04/20 11:16:39 Failed to connect to broker domain.com:9093: tls: first record does not look like a TLS handshake
[Sarama] 2016/04/20 11:16:39 client/metadata got error from broker while fetching metadata: tls: first record does not look like a TLS handshake
[Sarama] 2016/04/20 11:16:39 client/metadata no available broker to send metadata request to
[Sarama] 2016/04/20 11:16:39 client/brokers resurrecting 1 dead seed brokers
[Sarama] 2016/04/20 11:16:39 Closing Client
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
In the kafka logs I see the following errors
[2016-04-20 11:16:39,073] DEBUG Connection with a.b.c.d/a.b.c.d disconnected (org.apache.kafka.common.network.Selector)
javax.net.ssl.SSLHandshakeException: no cipher suites in common
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1348)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:519)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1200)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1172)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:377)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:242)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:68)
at org.apache.kafka.common.network.Selector.poll(Selector.java:281)
at kafka.network.Processor.run(SocketServer.scala:413)
at java.lang.Thread.run(Thread.java:745)
Further investigation using Wireshark indicates that the Kafka server does not respond with a ServerHello in the TLS handshake. Which is all due to Sarama and Kafka couldn't agree on common cipher.
This issue is probably not in Sarama, more correctly in Go TLS or in Java, but I'd like to know how people claims this is working.
Metadata
Metadata
Assignees
Labels
No labels
Activity
eapache commentedon Apr 20, 2016
The ciphers available in go and Java are fairly standard and have quite a bit of overlap by default. If they can't find one in common then I suspect one or the other is misconfigured. Since I don't know of a way to change the available ciphers for go (short of recompiling the go toolchain from scratch) I must assume that something is configured oddly about your Java installation. By default both openjdk and oracle have plenty of common cipher suites available.
tom-pang commentedon Apr 20, 2016
As a note, we've been using the SSL support in production from sarama for months now and it's been flawless, so I too suspect a JVM config issue.
jabbors commentedon Apr 21, 2016
Yeah, I ran this java code http://stackoverflow.com/questions/20000020/enabled-ciphers-on-ubuntu-openjdk-7 to print the available ciphers in the JVM (both openjdk and oracle) and the list was long, with many common ciphers supported by the Go configuration. However, when I ran a sslscan against the Kafka broker the list was much shorter and none of them where common with those in Go.
So I too suspect a JVM config issue. And since I'm not a Java entusiast nor expert, I've chosen the go with default settings when installing anything Java related. So the "working" out-of-the-box Java installation I was hoping for is not a fact. Guess I have to start digging and debugging the JVM.
jabbors commentedon Apr 21, 2016
@tcrayford: Which platform do run Kafka on, which Java and version, and what are the supported ciphers if you run a sslscan against a broker? Also which Go version and platform do you run Sarama on?
I have installed Kafka on a Ubuntu 14.04 cloud machine, tested with openjdk7 and oracle8. sslscan lists the following ciphers for openjdk7 (even less supported ciphers in oracle8):
Tested with Go 1.6 on both OSX and Ubuntu 14.04.
elvarb commentedon May 2, 2016
Might be interested in this ticket https://issues.apache.org/jira/browse/KAFKA-3647
elvarb commentedon May 2, 2016
I have been testing Kafka with lots of different versions of Java but the supported server ciphers never change.
tom-pang commentedon May 2, 2016
@jabbors we use Ubuntu and oracle8 and we've never had to tweak anything.
jabbors commentedon May 3, 2016
@elvarb, thanks for this link, I did install the JCE as suggested but still no go. I'll post to the kafka issue and let them know I also experience the same issue.
@tcrayford, do you mind sharing your kafka configuration? Could be a trivial setup I'm overseeing as I don't get it to work.
This is currently my ssl config.
tom-pang commentedon May 3, 2016
@jabbors we don't do much more than the standard config does, just add an SSL listener and a key/truststore
elvarb commentedon May 4, 2016
@tcrayford it's currently pointing to a problem with the jks / certs that results in only the DSS ciphers being used, do you think it might be something related to how the certs are created?
elvarb commentedon May 4, 2016
When you create the keystore use -keyalg RSA, then you will get the right ciphers.
jabbors commentedon May 9, 2016
Thanks @elvarb, got it working with the keying RSA parameter.
brikis98 commentedon Aug 19, 2017
Thank you @elvarb, I just hit the same issue and spent a lot of time trying to figure out what the cause was, until stumbling across your comment.