Skip to content

SSL support (Kafka 0.9+) revisit #643

Closed

Description

@jabbors

Hi,

SSL support it suppose to be working according to issue #581.

However, I've spent two days trying to get it working with Kafka 0.9.0. I can connect to to my kafka cluster with openssl s_client -connect domain.com:9093 -tls1 so I know it is configured correctly.

But when I try connecting with sarama I get errors:

[Sarama] 2016/04/20 11:16:39 client/metadata retrying after 250ms... (1 attempts remaining)
[Sarama] 2016/04/20 11:16:39 client/metadata fetching metadata for all topics from broker domain.com:9093
[Sarama] 2016/04/20 11:16:39 Failed to connect to broker domain.com:9093: tls: first record does not look like a TLS handshake
[Sarama] 2016/04/20 11:16:39 client/metadata got error from broker while fetching metadata: tls: first record does not look like a TLS handshake
[Sarama] 2016/04/20 11:16:39 client/metadata no available broker to send metadata request to
[Sarama] 2016/04/20 11:16:39 client/brokers resurrecting 1 dead seed brokers
[Sarama] 2016/04/20 11:16:39 Closing Client
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

In the kafka logs I see the following errors

[2016-04-20 11:16:39,073] DEBUG Connection with a.b.c.d/a.b.c.d disconnected (org.apache.kafka.common.network.Selector)
javax.net.ssl.SSLHandshakeException: no cipher suites in common
  at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1348)
  at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:519)
  at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1200)
  at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1172)
  at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
  at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:377)
  at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:242)
  at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:68)
  at org.apache.kafka.common.network.Selector.poll(Selector.java:281)
  at kafka.network.Processor.run(SocketServer.scala:413)
  at java.lang.Thread.run(Thread.java:745)

Further investigation using Wireshark indicates that the Kafka server does not respond with a ServerHello in the TLS handshake. Which is all due to Sarama and Kafka couldn't agree on common cipher.

This issue is probably not in Sarama, more correctly in Go TLS or in Java, but I'd like to know how people claims this is working.

Activity

  1. eapache commented on Apr 20, 2016

    @eapache
    Contributor

    The ciphers available in go and Java are fairly standard and have quite a bit of overlap by default. If they can't find one in common then I suspect one or the other is misconfigured. Since I don't know of a way to change the available ciphers for go (short of recompiling the go toolchain from scratch) I must assume that something is configured oddly about your Java installation. By default both openjdk and oracle have plenty of common cipher suites available.

  2. tom-pang commented on Apr 20, 2016

    @tom-pang
    Contributor

    As a note, we've been using the SSL support in production from sarama for months now and it's been flawless, so I too suspect a JVM config issue.

  3. jabbors commented on Apr 21, 2016

    @jabbors
    Author

    Yeah, I ran this java code http://stackoverflow.com/questions/20000020/enabled-ciphers-on-ubuntu-openjdk-7 to print the available ciphers in the JVM (both openjdk and oracle) and the list was long, with many common ciphers supported by the Go configuration. However, when I ran a sslscan against the Kafka broker the list was much shorter and none of them where common with those in Go.

    So I too suspect a JVM config issue. And since I'm not a Java entusiast nor expert, I've chosen the go with default settings when installing anything Java related. So the "working" out-of-the-box Java installation I was hoping for is not a fact. Guess I have to start digging and debugging the JVM.

  4. jabbors commented on Apr 21, 2016

    @jabbors
    Author

    @tcrayford: Which platform do run Kafka on, which Java and version, and what are the supported ciphers if you run a sslscan against a broker? Also which Go version and platform do you run Sarama on?

    I have installed Kafka on a Ubuntu 14.04 cloud machine, tested with openjdk7 and oracle8. sslscan lists the following ciphers for openjdk7 (even less supported ciphers in oracle8):

      Supported Server Cipher(s):
    Preferred TLSv1.2  256 bits  DHE-DSS-AES256-SHA256         DHE 1024 bits
    Accepted  TLSv1.2  256 bits  DHE-DSS-AES256-SHA            DHE 1024 bits
    Accepted  TLSv1.2  128 bits  DHE-DSS-AES128-SHA256         DHE 1024 bits
    Accepted  TLSv1.2  128 bits  DHE-DSS-AES128-SHA            DHE 1024 bits
    Accepted  TLSv1.2  112 bits  EDH-DSS-DES-CBC3-SHA          DHE 1024 bits
    Preferred TLSv1.1  256 bits  DHE-DSS-AES256-SHA            DHE 1024 bits
    Accepted  TLSv1.1  128 bits  DHE-DSS-AES128-SHA            DHE 1024 bits
    Accepted  TLSv1.1  112 bits  EDH-DSS-DES-CBC3-SHA          DHE 1024 bits
    Preferred TLSv1.0  256 bits  DHE-DSS-AES256-SHA            DHE 1024 bits
    Accepted  TLSv1.0  128 bits  DHE-DSS-AES128-SHA            DHE 1024 bits
    Accepted  TLSv1.0  112 bits  EDH-DSS-DES-CBC3-SHA          DHE 1024 bits
    

    Tested with Go 1.6 on both OSX and Ubuntu 14.04.

  5. elvarb commented on May 2, 2016

    @elvarb

    Might be interested in this ticket https://issues.apache.org/jira/browse/KAFKA-3647

  6. elvarb commented on May 2, 2016

    @elvarb

    I have been testing Kafka with lots of different versions of Java but the supported server ciphers never change.

  7. tom-pang commented on May 2, 2016

    @tom-pang
    Contributor

    @jabbors we use Ubuntu and oracle8 and we've never had to tweak anything.

  8. jabbors commented on May 3, 2016

    @jabbors
    Author

    @elvarb, thanks for this link, I did install the JCE as suggested but still no go. I'll post to the kafka issue and let them know I also experience the same issue.

    @tcrayford, do you mind sharing your kafka configuration? Could be a trivial setup I'm overseeing as I don't get it to work.

    This is currently my ssl config.

    security.protocol=SSL
    security.inter.broker.protocol=SSL
    ssl.keystore.location=/etc/kafka/server.keystore.jks
    ssl.keystore.password=*****
    ssl.key.password=*****
    ssl.truststore.location=/etc/kafka/server.truststore.jks
    ssl.truststore.password=*****
    #ssl.client.auth=none
    #ssl.client.auth=requested
    ssl.client.auth=required
    #ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
    #ssl.cipher.suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
    
  9. tom-pang commented on May 3, 2016

    @tom-pang
    Contributor

    @jabbors we don't do much more than the standard config does, just add an SSL listener and a key/truststore

  10. elvarb commented on May 4, 2016

    @elvarb

    @tcrayford it's currently pointing to a problem with the jks / certs that results in only the DSS ciphers being used, do you think it might be something related to how the certs are created?

  11. elvarb commented on May 4, 2016

    @elvarb

    When you create the keystore use -keyalg RSA, then you will get the right ciphers.

  12. jabbors commented on May 9, 2016

    @jabbors
    Author

    Thanks @elvarb, got it working with the keying RSA parameter.

  13. brikis98 commented on Aug 19, 2017

    @brikis98

    Thank you @elvarb, I just hit the same issue and spent a lot of time trying to figure out what the cause was, until stumbling across your comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址