MySQL Binlog 解析工具 Maxwell 详解 (5)

实验一番,下面将引导 test 数据库中 test 表,首先是准备几条测试用的数据

INSERT INTO `test` VALUES (1, 1, '1'); INSERT INTO `test` VALUES (2, 2, '2'); INSERT INTO `test` VALUES (3, 3, '3'); INSERT INTO `test` VALUES (4, 4, '4');

然后 reset master; 清空binlog,删除 maxwell 库中的表。接着使用快速开始中的命令,启动Kafka、Maxwell和Kafka消费者,然后启动 maxwell-bootstrap

docker run -it --rm zendesk/maxwell bin/maxwell-bootstrap --user maxwell \ --password 123456 --host 10.100.97.246 --database test --table test --client_id maxwell

注意:--bootstrapper=sync 时,在处理bootstrap时,会阻塞正常的binlog解析;--bootstrapper=async 时,不会阻塞。

也可以执行下面的SQL,在 maxwell.bootstrap 表中插入记录,手动触发

insert into maxwell.bootstrap (database_name, table_name) values ('test', 'test');

就可以在 kafka 消费者端看见引导过来的数据了

{"database":"maxwell","table":"bootstrap","type":"insert","ts":1552199115,"xid":36738,"commit":true,"data":{"id":3,"database_name":"test","table_name":"test","where_clause":null,"is_complete":0,"inserted_rows":0,"total_rows":0,"created_at":null,"started_at":null,"completed_at":null,"binlog_file":null,"binlog_position":0,"client_id":"maxwell"}} {"database":"test","table":"test","type":"bootstrap-start","ts":1552199115,"data":{}} {"database":"test","table":"test","type":"bootstrap-insert","ts":1552199115,"data":{"id":1,"age":1,"name":"1"}} {"database":"test","table":"test","type":"bootstrap-insert","ts":1552199115,"data":{"id":2,"age":2,"name":"2"}} {"database":"test","table":"test","type":"bootstrap-insert","ts":1552199115,"data":{"id":3,"age":3,"name":"3"}} {"database":"test","table":"test","type":"bootstrap-insert","ts":1552199115,"data":{"id":4,"age":4,"name":"4"}} {"database":"maxwell","table":"bootstrap","type":"update","ts":1552199115,"xid":36756,"commit":true,"data":{"id":3,"database_name":"test","table_name":"test","where_clause":null,"is_complete":1,"inserted_rows":4,"total_rows":0,"created_at":null,"started_at":"2019-03-10 14:25:15","completed_at":"2019-03-10 14:25:15","binlog_file":"mysql-bin.000001","binlog_position":64446,"client_id":"maxwell"},"old":{"is_complete":0,"inserted_rows":1,"completed_at":null}} {"database":"test","table":"test","type":"bootstrap-complete","ts":1552199115,"data":{}}

中间的4条便是 test.test 的binlog数据了,注意这里的 type 是 bootstrap-insert,而不是 insert。

然后再一次查看binlog,show binlog events;,会发现只有与 maxwell 相关的binlog,并没有 test.test 相关的binlog,所以 maxwell-bootstrap 命令并不会产生多余的 binlog,当数据表的数量很大时,这个好处会更加明显

Bootstrap 的过程是 bootstrap-start -> bootstrap-insert -> bootstrap-complete,其中,start和complete的data字段为空,不携带数据。

在进行bootstrap过程中,如果maxwell崩溃,重启时,bootstrap会完全重新开始,不管之前进行到多少,若不希望这样,可以到数据库中设置 is_complete 字段值为1(表示完成),或者删除该行

Maxwell监控

Maxwell 提供了 base logging mechanism, JMX, HTTP or by push to Datadog 这四种监控方式,与监控相关的配置项有下列这些:

选项 参数值 描述 默认值
metrics_prefix   STRING   指标的前缀   MaxwellMetrics  
metrics_type   [slf4j | jmx | http | datadog]   发布指标的方式    
metrics_jvm   BOOLEAN   是否收集JVM信息   false  
metrics_slf4j_interval   SECONDS   将指标记录到日志的频率,metrics_type须配置为slf4j   60  
http_port   INT   metrics_type为http时,发布指标绑定的端口   8080  
http_path_prefix   STRING   http的路径前缀   /  
http_bind_address   STRING   http发布指标绑定的地址   all addresses  
http_diagnostic   BOOLEAN   http是否开启diagnostic后缀   false  
http_diagnostic_timeout   MILLISECONDS   http diagnostic 响应超时时间   10000  
metrics_datadog_type   [udp | http]   metrics_type为datadog时发布指标的方式   udp  
metrics_datadog_tags   STRING   提供给 datadog 的标签,如 tag1:value1,tag2:value2    
metrics_datadog_interval   INT   推指标到datadog的频率,单位秒   60  
metrics_datadog_apikey   STRING   当 metrics_datadog_type=http 时datadog用的api key    
metrics_datadog_host   STRING   当metrics_datadog_type=udp时推指标的目标地址   localhost  
metrics_datadog_port   INT   当metrics_datadog_type=udp 时推指标的端口   8125  

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/wpjggp.html