Kafka Consumer

Introduction

Here we'll go over the Kafka Consumer Consumer JMX Data Source configuration fields, and list the Consumer JMX metrics.  Please see our Kafka Overview article for details about other Kafka data sources besides Consumer.

 

Requirements

Before we continue, you will need JMX enabled.  It is turned off by default, so you'll want to enable JMX it as follows:

Add the line "export JMX_PORT=9999" to the JMX settings section inside the $KAFKA_HOME/bin/kafka-run-class.sh file at this location, where "9999" is the port you wish to use, then Restart the Kafka Consumer. 

 
 
Configure Kafka Consumer JMX Data Sources
 
Open up the configuration modal for Kafka, and click on the Configure Data Sources tab.  In the Add a data source drop down menu [1], select Kafka Consumer JMX.  Click on Edit [2] to the right to expose the fields that need to be configured [3]:

 
The plugin requires the following information:
 
Field Name Default Value Description
JMX URL service:jmx:rmi:///jndi/rmi://localhost:PORT/jmxrmi

When JMX is enabled at 9999 (for example), Opsclarity Recommends the following string:

service:jmx:rmi:///jndi/rmi://_HOST:9999_:9999/jmxrmi

When the agent sees this configuration, it will automatically figure out what address it should use to make a connection at the specified port.  This is extremely helpful in cases where listening service binds to local IP address instead of localhost.

JAVAHOME empty In most of the cases, the Opsclarity Agent will pick Up Java Home, but you can provide it here.
User Name empty If authentication is enabled, provide a username here.
Password empty If authentication is enabled, provide a password here. 
Classpath empty Input your Java classpath

 

 

 
Consumer Metrics
 
OpsClarity Metric Name Source Metric Name
assigned_partitions kafka.consumer:type=consumer-coordinator-metrics, name=assigned-partitions, client-id=([-.w]+)
bytes_consumed_rate
kafka.consumer:type=consumer-fetch-manager-metrics, name=bytes_consumed_rate, client-id=([-.w]+)
commit_latency_avg kafka.consumer:type=consumer-coordinator-metrics, name=commit_latency_avg, client-id=([-.w]+) 
commit_latency_max
kafka.consumer:type=consumer-coordinator-metrics, name=commit_latency_max, client-id=([-.w]+)
commit_rate kafka.consumer:type=consumer-coordinator-metrics, name=commit_rate, client-id=([-.w]+)
connection_close_rate kafka.consumer:type=consumer-metrics, name=connection_close_rate, client-id=([-.w]+)
connection_count kafka.consumer:type=consumer-metrics, name=connection_count, client-id=([-.w]+)
connection_creation_rate kafka.consumer:type=consumer-metrics, name=connection_creation_rate, client-id=([-.w]+) 
fetch_latency_avg

kafka.consumer:type=consumer-fetch-manager-metrics, name= fetch_latency_avg, client-id=([-.w]+)

fetch_latency_max kafka.consumer:type=consumer-fetch-manager-metrics, name=fetch_latency_max, client-id=([-.w]+)
fetch_rate kafka.consumer:type=consumer-fetch-manager-metrics, name=fetch_rate, client-id=([-.w]+)
fetch_size_avg kafka.consumer:type=consumer-fetch-manager-metrics, name=fetch_size_avg, client-id=([-.w]+)
fetch_size_max kafka.consumer:type=consumer-fetch-manager-metrics, name=fetch_size_max, client-id=([-.w]+)
fetch_throttle_time_avg kafka.consumer:type=consumer-fetch-manager-metrics, name=fetch_throttle_time_avg, client-id=([-.w]+)
fetch_throttle_time_max kafka.consumer:type=consumer-fetch-manager-metrics, name=fetch_throttle_time_max, client-id=([-.w]+)
heartbeat_rate kafka.consumer:type=consumer-coordinator-metrics, name=heartbeat_rate, client-id=([-.w]+)
heartbeat_response_time_max kafka.consumer:type=consumer-coordinator-metrics, name=heartbeat_response_time_max, client-id=([-.w]+)
in_byte_rate kafka.consumer:type=consumer-node-metrics, name= incoming_byte_rate, client-id=([-.w]+),node-id=([0-9]+)
io_ratio kafka.consumer:type=consumer-metrics, name=io_ratio, client-id=([-.w]+)
io_time_ns_avg kafka.consumer:type=consumer-metrics, name=io_time_ns_avg, client-id=([-.w]+)
io_wait_ratio kafka.consumer:type=consumer-metrics, name=io_wait_ratio, client-id=([-.w]+)
io_wait_time_ns_avg kafka.consumer:type=consumer-metrics, name=io_wait_time_ns_avg, client-id=([-.w]+)
join_rate kafka.consumer:type=consumer-coordinator-metrics, name=join_rate, client-id=([-.w]+)
join_time_avg kafka.consumer:type=consumer-coordinator-metrics, name=join_time_avg, client-id=([-.w]+)
join_time_max kafka.consumer:type=consumer-coordinator-metrics, name=join_time_max, client-id=([-.w]+)
last_heartbeat_seconds_ago kafka.consumer:type=consumer-coordinator-metrics, name=last_heartbeat_seconds_ago, client-id=([-.w]+)
out_byte_rate kafka.consumer:type=consumer-node-metrics, name= outgoing_byte_rate, client-id=([-.w]+),node-id=([0-9]+)
records_consumed_rate kafka.consumer:type=consumer-fetch-manager-metrics, name=records-consumed-rate, client-id=([-.w]+)
records_lag_max kafka.consumer:type=consumer-fetch-manager-metrics, name=records_lag_max, client-id=([-.w]+)
records_per_request_avg kafka.consumer:type=consumer-fetch-manager-metrics, name=records_per_request_avg, client-id=([-.w]+)
request_latency_avg kafka.consumer:type=consumer-metrics, name=request-latency_avg, client-id=([-.w]+)
request_latency_max kafka.consumer:type=consumer-metrics, name=request-latency_max, client-id=([-.w]+) 
request_rate kafka.consumer:type=consumer-node-metrics, name= request_rate, client-id=([-.w]+),node-id=([0-9]+)
request_size_avg kafka.consumer:type=consumer-node-metrics, name=request-size-avg, client-id=([-.w]+),node-id=([0-9]+)
request_size_max kafka.consumer:type=consumer-node-metrics, name=request-size-max, client-id=([-.w]+),node-id=([0-9]+)
response_rate kafka.consumer:type=consumer-node-metrics, name= response_rate, client-id=([-.w]+),node-id=([0-9]+)
select_rate kafka.consumer:type=consumer-metrics, name=select-rate, client-id=([-.w]+)
sync_rate kafka.consumer:type=consumer-coordinator-metrics, name=sync_rate, client-id=([-.w]+)
sync_time_avg kafka.consumer:type=consumer-coordinator-metrics, name=sync_time_avg, client-id=([-.w]+)
sync_time_max kafka.consumer:type=consumer-coordinator-metrics, name=sync_time_max, client-id=([-.w]+)

 

 

If you have any questions or comments about this article, feel free to contact us at support@opsclarity.com.

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.