kafka生产者性能监控：Monitor Kafka Producer for Performance

The producer sends data directly to the broker that is the leader for the partition without any intervening routing tier.

Optimization Approach

Batching is one of the big drivers of efficiency, and to enable batching the Kafka producer will attempt to accumulate data in memory and to send out larger batches in a single request. The batching can be configured to accumulate no more than a fixed number of messages and to wait no longer than some fixed latency bound (say 64k or 10 ms). This allows the accumulation of more bytes to send, and few larger I/O operations on the servers. This buffering is configurable and gives a mechanism to trade off a small amount of additional latency for better throughput.

In order to find the optimal batch size and latency, iterative test supported by producer statistics monitoring is needed.

Enable Monitoring

Start the producer with the JMX parameters enabled:

JMX_PORT=10102 bin/kafka-console-producer.sh --broker-list localhost:9092--topic testtopic