kafka – OverView

Kafka is a distributed pub/sub broker which can scale horizontally. Kafka broker is designed to work in clusters , where you will run multiple Kafka brokers on different machines. One broker will be selected as controller automatically  from running brokers. The controller is responsible for administrative tasks in the cluster , the controller also assigns partitions to the brokers and monitor the cluster for any failed brokers. Kafka uses zookeeper to maintain metadata of the cluster.

As with any pub/sub system there will be producers (which will write messages to kafka) and consumers (read messages from kafka) . Producers will write messages to topics , each topic can have multiple partitions.A partition will have only one owner known as leader. All the operation performed that partition needs to be done leader node. a partition can be assigned to multiple brokers in which case the leader will take care of replicating the partition to other brokers. Number partitions can only be increased for a topic once it is created. it is necessary to have optimal number of partitions to balance the load across multiple brokers.

Producers

Producers can give key to each message , key will influence the decision on to which partition a message needs to be written.  You can also design a custom partitioner to implement logic arbitrary business logic . Often the messages are written in batches to increase throughput (The kafka producer API will take care of batching of messages based on the given configuration). The producer API can wait until the message has been written to all of the replicas or atleast to the leader of the partition.

Consumers

Often Consumers will be part of a consumer group.There can be more than one consumer group reading from the same topic, there will be no inter dependency between consumer groups reading from the same topic. consumers read from a partition, they can record their offset in kafka or zookeeper. If the consumer is crashed/down for schedules maintenance, it can simply start reading from it’s offset when it comes back. No messages will be lost as kafka persists the messages to disk  (you can retention of the messages based on time and size of the log). Consumers works as part of consumer group, a consumer group will consume a topic and partitions will be assigned to individual consumers , If consumer is down , the consumer group will automatically shares the load among the available consumers (rebalance).