启停kafka服务的方法:
cd /data/kafka/bin
./kafka-server-start.sh
./kafka-server-stop.sh
注:观察和检查/data/kafka/logs下的各个日志文件,以确认无报错信息且各项服务日志输出正确。
9、kafka集群的常用管理命令
建议另外安排一个备机作为长期监控和测试kafka集群的主机。程序部署在/data/kafka下。
创建了以下用于服务和性能监控的topic:
./kafka-topics.sh --create --zookeeper 192.168.10.1:2181,192.168.10.2:2181,192.168.10.3:2181 --replication-factor 3 --partitions 6 --topic test-for-sys-monitor
查看已创建了的topics列表:
./kafka-topics.sh --list --zookeeper 192.168.10.1:2181
查看指定topic的详情:
./kafka-topics.sh --describe --zookeeper 192.168.10.1:2181 --topic test-for-sys-monitor
使用终端生产者命令进行测试:
./kafka-console-producer.sh --broker-list 192.168.10.1:9092,192.168.10.2:9092,192.168.10.3:9092 --topic test-for-sys-monitor
使用终端消费者命令进行测试:
./kafka-console-consumer.sh --zookeeper 192.168.10.1:2181 --topic test-for-sys-monitor
注:以���测试,在生产者侧输入的数据,会在消费者侧打印出来。
删除topic的命令:
./kafka-topics.sh --delete --zookeeper 192.168.10.1:2181 --topic test-for-sys-monitor
注:kafka集群中放开了删除topic的功能,请谨慎使用。
压测写500万条数据,每条1KB:
./kafka-producer-perf-test.sh --topic test-perf-20161220 --num-records 500000 --record-size 1000 --throughput 100000 --producer-props bootstrap.servers=192.168.10.1:9092,192.168.10.2:9092,192.168.10.3:9092
5000000 records sent, 53149.648149 records/sec (50.69 MB/sec), 569.30 ms avg latency, 2096.00 ms max latency, 8 ms 50th, 1759 ms 95th, 1874 ms 99th, 2044 ms 99.9th.
压测读6*100万条数据:
./kafka-consumer-perf-test.sh --zookeeper 192.168.10.1:2181,192.168.10.2:2181,192.168.10.3:2181 --messages 1000000 --topic test-perf-20161220 --threads 6
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2016-12-20 16:08:50:102, 2016-12-20 16:09:48:525, 5722.0459, 97.9417, 6000000, 102699.2794
注:压测后,请及时删除用于压测的topic,因为所产生的kafka日志数据很可观。
登录zookeeper shell中查看kafka创建的相关元数据:
./zookeeper-shell.sh 192.168.10.3:2181
Connecting to 192.168.10.3:2181
Welcome to ZooKeeper!
JLine support is disabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
ls /
[consumers, config, controller, isr_change_notification, brokers, admin, zookeeper, controller_epoch]
get /brokers/ids/1
{"jmx_port":-1,"timestamp":"1482223274389","endpoints":["PLAINTEXT://192.168.10.1:9092"],"host":"192.168.10.1","version":3,"port":9092}
cZxid = 0x900000009
ctime = Tue Dec 20 16:41:14 CST 2016
mZxid = 0x900000009
mtime = Tue Dec 20 16:41:14 CST 2016
pZxid = 0x900000009
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x1591b61880f0000
dataLength = 137
numChildren = 0
get /brokers/ids/2
{"jmx_port":-1,"timestamp":"1482223307625","endpoints":["PLAINTEXT://192.168.10.2:9092"],"host":"192.168.10.2","version":3,"port":9092}
cZxid = 0x900000013
ctime = Tue Dec 20 16:41:47 CST 2016
mZxid = 0x900000013
mtime = Tue Dec 20 16:41:47 CST 2016
pZxid = 0x900000013
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x1591b61880f0001
dataLength = 137
numChildren = 0
get /brokers/ids/3
{"jmx_port":-1,"timestamp":"1482223315746","endpoints":["PLAINTEXT://192.168.10.3:9092"],"host":"192.168.10.3","version":3,"port":9092}
cZxid = 0x900000020
ctime = Tue Dec 20 16:41:55 CST 2016
mZxid = 0x900000020
mtime = Tue Dec 20 16:41:55 CST 2016
pZxid = 0x900000020
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x3591b618ffe0001
dataLength = 137
numChildren = 0
quit
Quitting...
改变topic的分区数量
./kafka-topics.sh --alter --zookeeper 192.168.10.1:2181 --topic test_topic --partitions 4
增删改topic的配置参数
./kafka-topics.sh --alter --zookeeper 192.168.10.1:2181 --topic test_topic--config key=value
./kafka-topics.sh —alter --zookeeper 192.168.10.1:2181 --topic test_topic--deleteConfig key
查看topic的分区、副本状态,需要关注各个分区的负载分布是否均衡(即Leader角色的分布):
./kafka-topics.sh --describe --zookeeper 192.168.10.1:2181 --topic test-for-sys-monitor
Topic:test-for-sys-monitor PartitionCount:6 ReplicationFactor:3 Configs:
Topic: test-for-sys-monitor Partition: 0 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2
Topic: test-for-sys-monitor Partition: 1 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3
Topic: test-for-sys-monitor Partition: 2 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1
Topic: test-for-sys-monitor Partition: 3 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: test-for-sys-monitor Partition: 4 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
Topic: test-for-sys-monitor Partition: 5 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
执行leader分布的再平衡:
./kafka-preferred-replica-election.sh --zookeeper 192.168.10.1:2181