Recommended configuration for a production server.

I am new to Kafka and I have setup a basic single node cluster using Ambari.

I want to know what is the recommended configuration for a production server.
if I will have 6 topics and each getting traffic in the range of 500,000 to 50,000,000 in a day.

what about setting up a 3-4 node kafka cluster using EC2 r5.xlarge instances.

I know zookeeper needs odd number of nodes and zookeeper is installed on all kafka nodes, then how do we run Kafka with even number of nodes. If this is true it will limit Kafka to odd number of nodes as well.

Is it really needed to install Zookeeper on all Kafka nodes. Can I install Zookeeper on separate nodes and Kafka brokers on separate nodes, how ?

What if i want to run multiple Kafka clusters. Is it possible to manage multiple Kafka clusters through single Zookeeper cluster, how if possible ?

Tagged:

Comments

  • Can I install Zookeeper on separate nodes and Kafka brokers on separate nodes, how ?

    You can, and you should if you have the available resources.

    Run zookeeper-server-start zookeeper.properties on an odd number of servers. (max 5 or 7 for larger Kafka clusters)

    On every other machine that is a Kafka broker, not the same servers as Zookeeper, edit server.properties to point to that set of Zookeeper machine addresses for the zookeeeper.connect property.

    Then do kafka-server-start server.properties for every new Kafka broker.

    From there, you can scale Kafka independently of Zookeeper

    Is it possible to manage multiple Kafka clusters through single Zookeeper cluster

    Look up Zookeeper chroots

    One Kafka cluster would be defined as

    zoo1:2181/kafka1
    And a second

    zoo1:2181/kafka2
    be careful not to mix those up if machines shouldn't be in the same Kafka cluster

    You can find various CloudFormation, Terraform, or Ansible repos for setting up Kafka in a distibuted way in the Cloud on Github, or go for Kubernetes if you are familiar with it.

Sign In or Register to comment.