Kafka Producer

 

Introduction

Here we'll go over the Kafka Producer JMX Data Source configuration fields and list Metrics for that source.  Please see our Kafka Overview article for details about other Kafka data sources besides Producer.

 

Requirements

Before we continue, you will need JMX enabled.  It is turned off by default, so you'll want to enable JMX it as follows:

1.)  Add the line "export JMX_PORT=9999" to the JMX settings section inside the $KAFKA_HOME/bin/kafka-run-class.sh file at this location, where "9999" is the port you wish to use:

# JMX settings
export JMX_PORT=9999
if [ -z "$KAFKA_JMX_OPTS" ]; then
  KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false "
fi
 
# JMX port to use
if [  $JMX_PORT ]; then
  KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT "
fi

2.)  Restart the Kafka Producer.

3.)  Observe that -Dcom.sun.management.jmxremote.port=9999 is now added to the command line.

4.)  Repeat steps 1 through 3 for all brokers in the cluster.

5.)  Replace the “PORT" string inside the OpsClarity Kafka configuration field labeled "JMX URL" with "9999" (or whatever port number you used) - see below for more details.

 

Configure Kafka Producer Data Sources

Open up the configuration modal for Kafka, and click on the Configure Data Sources tab.  In the Add a data source drop down menu [1], select Kafka Producer JMX.  Click on Edit [2] to the right to expose the fields that need to be configured [3]:

 

These are the field names (along with their default values) and a detailed description of what should go in each field under the Kafka Producer JMX data source:

Field Name Default Value Description
JMX URL service:jmx:rmi:///jndi/rmi://localhost:PORT/jmxrmi

When JMX is enabled at 9999 (for example), Opsclarity Recommends the following string:

service:jmx:rmi:///jndi/rmi://_HOST:9999_:9999/jmxrmi

When the agent sees this configuration, it will automatically figure out what address it should use to make a connection at the specified port.  This is extremely helpful in cases where listening service binds to local IP address instead of localhost.

JAVAHOME empty In most of the cases, the Opsclarity Agent will pick Up Java Home, but you can provide it here.
User Name empty If authentication is enabled, provide a username here.
Password empty If authentication is enabled, provide a password here. 
Classpath empty Input your Java classpath

 

 

Kafka Producer Metrics
 
OpsClarity Metric Name Source Metric Name
byte_rate kafka.producer:type=producer-topic-metrics,name=byte_rate, client-id=([-.w]+),topic=([-.w]+)
compression_rate

kafka.producer:type=producer-topic-metrics,name=compression_rate, client-id=([-.w]+),topic=([-.w]+)

connection_close_rate kafka.producer:type=producer-metrics, name=connection_close_rate, client-id=([-.w]+)
connection_count kafka.producer:type=producer-metrics, name=connection_count, client-id=([-.w]+)
connection_creation_rate kafka.producer:type=producer-metrics, name=connection_creation_rate, client-id=([-.w]+)
in_byte_rate kafka.producer:type=producer-node-metrics, name=incoming_byte_rate, client-id=([-.w]+),node-id=([0-9]+)
io_ratio kafka.producer:type=producer-metrics, name=io_ratio, client-id=([-.w]+)
io_time_ns_avg kafka.producer:type=producer-metrics, name=io_time_ns_avg, client-id=([-.w]+)
io_wait_ratio kafka.producer:type=producer-metrics, name=io_wait_ratio, client-id=([-.w]+)
io_wait_time_ns_avg kafka.producer:type=producer-metrics, name=io_wait_time_ns_avg, client-id=([-.w]+)
out_byte_rate kafka.producer:type=producer-node-metrics, name=outgoing_byte_rate, client-id=([-.w]+),node-id=([0-9]+)
record_error_rate kafka.producer:type=producer-topic-metrics,name=record_error_rate, client-id=([-.w]+),topic=([-.w]+)
record_retry_rate kafka.producer:type=producer-topic-metrics,name=record_retry_rate, client-id=([-.w]+),topic=([-.w]+)
record_send_rate kafka.producer:type=producer-topic-metrics,name=record_send_rate, client-id=([-.w]+),topic=([-.w]+)
request_rate kafka.producer:type=producer-node-metrics, name=request_rate, client-id=([-.w]+),node-id=([0-9]+)
request_size_avg kafka.producer:type=producer-node-metrics, name=request_size_avg, client-id=([-.w]+),node-id=([0-9]+)
request_size_max kafka.producer:type=producer-node-metrics, name=request_size_max, client-id=([-.w]+),node-id=([0-9]+)
response_rate kafka.producer:type=producer-node-metrics, name=response_rate, client-id=([-.w]+),node-id=([0-9]+)
select_rate kafka.producer:type=producer-metrics, name=select_rate, client-id=([-.w]+)

 

 

If you have any questions or comments about this article, feel free to contact us at support@opsclarity.com.

 

 
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.