Kafka Overview

Apache Kafka is a distributed publish-subscribe messaging system that is designed to be fast, scalable, and durable. Modern applications are increasingly relying on Kafka for their large-scale message processing needs. Each Kafka cluster is comprised of multiple brokers, ideally running on separate hosts to provide fault-tolerance. Kafka clusters have multiple topics replicated throughout the brokers for organizing messages. These messages are published from producers and are received by consumers.

Since there is a bit more to configuring Kafka than some of our other services, we've broken things out into four different articles with special instructions in each:

OpsClarity not only discovers Kafka brokers automatically but also clusters them together to discover the overall Kafka cluster as well. It does exactly the same for the consumers and producers. Thereafter, it collects performance metrics from each of the brokers, consumers and producers using JMX (if consumers & producers are JVM-based applications). Default monitors are applied on the collected metrics for monitoring the health of the cluster and individual broker at both the service level as well as at the JVM level.
Please review our support article on how to configure service monitors for instructions on how to initially access the Opsclarity configuration screen and to see which settings there are common to every service.  In this article, we'll go over all necessary specific details for managing the OpsClarity - Kafka service integration.


Supported Versions

OpsClarity supports Kafka versions 0.8.2 to 0.10.0.


Opsclarity Published Blogs
We also have some very helpful blogs that you'll want to refer to if you wish to expand your understanding of what Kafka is and how it is used and monitored generally:
Understanding, Operating, and Monitoring Apache Kafka
Evaluating Message Brokers: Kafka vs. Kinesis vs. SQS
Understanding Kafka Consumer Groups and Consumer Lag



If you have any questions or comments about this article, feel free to contact us at support@opsclarity.com.

Have more questions? Submit a request


Please sign in to leave a comment.