Kafka Consumer Lag

Introduction

Here we'll go over the Kafka Consumer Lag Data Source configuration fields, and show how to add a Consumer Lag graph.  Please see our Kafka Overview article for details about other Kafka data sources besides Consumer Lag.

 
 
Requirements
 
The Kafka Lag plugin collects lag metrics for each consumer group talking to Kafka. It can be deployed anywhere such as on the Kafka cluster or on the application cluster reading from Kafka.
 
 
 
Configure Kafka Consumer Lag Data Sources
 
Open up the configuration modal for Kafka, and click on the Configure Data Sources tab.  In the Add a data source drop down menu [1], select Kafka Consumer Group.  Click on Edit [2] to the right to expose the fields that need to be configured [3]:
 
 
 
The plugin requires the following information:
 
Field Name Default Value Description
Collector Host empty

Hostname / IP for the host in this service cluster which should do the work of collecting data.

ZooKeeper host:port empty Hostname(or IP):Port # pair for the Zookeeper service managing the Kafka cluster e.g. Host:2181.
ZooKeeper chroot empty Optional suffix to reinterpret all paths relative to this root.
Consumer Group empty If you specify the group name, it will collect metrics only for that consumer group.  If you leave it blank, it will collect metrics for all the consumer groups.  Metrics will have the group name as a tag.
Run as unix user nobody Leave empty or type "nobody".

 

 
Adding Consumer Lag Graph
 
With the correct service selected, click on Metrics from the upper right navigation panel [1], and then click on Add Graph [2]:
 
 
 
 
In the top filter field, type "lag" [1] and scroll down to find and click on "lag" [2].  Under the curve option, you can select topic for a per topic graph [3]:
 
 
 
 
If you have any questions or comments about this article, feel free to contact us at support@opsclarity.com.
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.