Kafka集群部署与配置手册(3)

启停kafka服务的方法:
cd /data/kafka/bin
./kafka-server-start.sh
./kafka-server-stop.sh
 
注:观察和检查/data/kafka/logs下的各个日志文件,以确认无报错信息且各项服务日志输出正确。
 
 
9、kafka集群的常用管理命令
建议另外安排一个备机作为长期监控和测试kafka集群的主机。程序部署在/data/kafka下。
创建了以下用于服务和性能监控的topic:
./kafka-topics.sh --create --zookeeper 192.168.10.1:2181,192.168.10.2:2181,192.168.10.3:2181 --replication-factor 3 --partitions 6 --topic test-for-sys-monitor
查看已创建了的topics列表:
./kafka-topics.sh --list --zookeeper 192.168.10.1:2181
查看指定topic的详情:
./kafka-topics.sh --describe --zookeeper 192.168.10.1:2181 --topic test-for-sys-monitor
使用终端生产者命令进行测试:
./kafka-console-producer.sh --broker-list 192.168.10.1:9092,192.168.10.2:9092,192.168.10.3:9092  --topic test-for-sys-monitor
使用终端消费者命令进行测试:
./kafka-console-consumer.sh --zookeeper 192.168.10.1:2181 --topic test-for-sys-monitor
注:以���测试,在生产者侧输入的数据,会在消费者侧打印出来。

删除topic的命令:
./kafka-topics.sh --delete --zookeeper 192.168.10.1:2181  --topic test-for-sys-monitor
注:kafka集群中放开了删除topic的功能,请谨慎使用。
 
压测写500万条数据,每条1KB:
./kafka-producer-perf-test.sh --topic test-perf-20161220 --num-records 500000 --record-size 1000 --throughput 100000 --producer-props bootstrap.servers=192.168.10.1:9092,192.168.10.2:9092,192.168.10.3:9092
5000000 records sent, 53149.648149 records/sec (50.69 MB/sec), 569.30 ms avg latency, 2096.00 ms max latency, 8 ms 50th, 1759 ms 95th, 1874 ms 99th, 2044 ms 99.9th.

压测读6*100万条数据:
 ./kafka-consumer-perf-test.sh --zookeeper 192.168.10.1:2181,192.168.10.2:2181,192.168.10.3:2181 --messages 1000000 --topic test-perf-20161220 --threads 6
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2016-12-20 16:08:50:102, 2016-12-20 16:09:48:525, 5722.0459, 97.9417, 6000000, 102699.2794

注:压测后,请及时删除用于压测的topic,因为所产生的kafka日志数据很可观。
 
登录zookeeper shell中查看kafka创建的相关元数据:

./zookeeper-shell.sh 192.168.10.3:2181
Connecting to 192.168.10.3:2181
Welcome to ZooKeeper!
JLine support is disabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
ls /
    [consumers, config, controller, isr_change_notification, brokers, admin, zookeeper, controller_epoch]
get /brokers/ids/1
{"jmx_port":-1,"timestamp":"1482223274389","endpoints":["PLAINTEXT://192.168.10.1:9092"],"host":"192.168.10.1","version":3,"port":9092}
cZxid = 0x900000009
ctime = Tue Dec 20 16:41:14 CST 2016
mZxid = 0x900000009
mtime = Tue Dec 20 16:41:14 CST 2016
pZxid = 0x900000009
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x1591b61880f0000
dataLength = 137
numChildren = 0
get /brokers/ids/2
{"jmx_port":-1,"timestamp":"1482223307625","endpoints":["PLAINTEXT://192.168.10.2:9092"],"host":"192.168.10.2","version":3,"port":9092}
cZxid = 0x900000013
ctime = Tue Dec 20 16:41:47 CST 2016
mZxid = 0x900000013
mtime = Tue Dec 20 16:41:47 CST 2016
pZxid = 0x900000013
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x1591b61880f0001
dataLength = 137
numChildren = 0
get /brokers/ids/3
{"jmx_port":-1,"timestamp":"1482223315746","endpoints":["PLAINTEXT://192.168.10.3:9092"],"host":"192.168.10.3","version":3,"port":9092}
cZxid = 0x900000020
ctime = Tue Dec 20 16:41:55 CST 2016
mZxid = 0x900000020
mtime = Tue Dec 20 16:41:55 CST 2016
pZxid = 0x900000020
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x3591b618ffe0001
dataLength = 137
numChildren = 0
quit
    Quitting...

改变topic的分区数量
./kafka-topics.sh --alter --zookeeper 192.168.10.1:2181 --topic test_topic --partitions 4
 
增删改topic的配置参数
./kafka-topics.sh --alter --zookeeper 192.168.10.1:2181 --topic test_topic--config key=value
./kafka-topics.sh —alter --zookeeper 192.168.10.1:2181 --topic test_topic--deleteConfig key
 
查看topic的分区、副本状态,需要关注各个分区的负载分布是否均衡(即Leader角色的分布):
./kafka-topics.sh --describe --zookeeper 192.168.10.1:2181 --topic test-for-sys-monitor
Topic:test-for-sys-monitor    PartitionCount:6    ReplicationFactor:3    Configs:
    Topic: test-for-sys-monitor    Partition: 0    Leader: 1    Replicas: 1,3,2    Isr: 1,3,2
    Topic: test-for-sys-monitor    Partition: 1    Leader: 2    Replicas: 2,1,3    Isr: 2,1,3
    Topic: test-for-sys-monitor    Partition: 2    Leader: 3    Replicas: 3,2,1    Isr: 3,2,1
    Topic: test-for-sys-monitor    Partition: 3    Leader: 1    Replicas: 1,2,3    Isr: 1,2,3
    Topic: test-for-sys-monitor    Partition: 4    Leader: 2    Replicas: 2,3,1    Isr: 2,3,1
    Topic: test-for-sys-monitor    Partition: 5    Leader: 3    Replicas: 3,1,2    Isr: 3,1,2

执行leader分布的再平衡:
./kafka-preferred-replica-election.sh --zookeeper 192.168.10.1:2181

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/7853397c2e5bf77c9414c0e80ec610f3.html