MySQL 逻辑备份mysqldumpmysqlpumpmydumper原理解析(3)

设置了多线程并行备份时，第一个线程会执行FTWRL，再开启一致性快照；等其它线程都开启RR隔离级别和一致性快照后并备份完非事务表后（若有）再 unlock tables 取消全局读锁（确保并行备份每个线程得到一致的数据）

--trx-consistency-only 选项开启后，创建完子线程就就释放FTWRL

第一个连接的线程加了全局读锁FTWRL后， 执行SHOW MASTER STATUS, SHOW SLAVE STATUS，获取当前备份 mysql已经执行了的事务gtid快照信息

支持多个线程以chunk的方式备份同一个表或多个表

添加了-r 参数后，会根据show table status 和select max(id),min(id) from table_name 的结果分成多个chunck，多个线程去dump

使用mydumper8线程并行备份

user=test1 passwd=test1 time mydumper -u $user -p $passwd -h 127.0.0.1 -P 3306 -B test -T sbtest1 -o /tmp -t 8 -r 800000 real 0m12.386s user 0m5.720s sys 0m1.729s

du -sh /tmp/test.sbtest1*
161M /tmp/test.sbtest1.00000.sql
162M /tmp/test.sbtest1.00001.sql
162M /tmp/test.sbtest1.00002.sql
162M /tmp/test.sbtest1.00003.sql
162M /tmp/test.sbtest1.00004.sql
162M /tmp/test.sbtest1.00005.sql
162M /tmp/test.sbtest1.00006.sql
162M /tmp/test.sbtest1.00007.sql
162M /tmp/test.sbtest1.00008.sql
162M /tmp/test.sbtest1.00009.sql
162M /tmp/test.sbtest1.00010.sql
162M /tmp/test.sbtest1.00011.sql
4.0K /tmp/test.sbtest1-schema.sql
19M /tmp/test.sbtest1.sql
dump出了多个sql文件

使用mydumper32线程并行备份

user=test1 passwd=test1 time mydumper -u $user -p $passwd -h 127.0.0.1 -P 3306 -B test -T sbtest1 -o /tmp -t 32 -r 800000 real 0m10.531s user 0m5.684s sys 0m1.751s

dump出的文件和8线程并行备份一样的

mydumper备份执行语句

user=test1 passwd=test1 time mydumper -u $user -p $passwd -h 127.0.0.1 -P 3306 -B test -T sbtest1 -o /tmp -t 8

2019-03-25T21:37:55.267321+08:00 442 Connect test1@127.0.0.1 on test using TCP/IP
2019-03-25T21:37:55.282783+08:00 442 Query SET SESSION wait_timeout = 2147483
2019-03-25T21:37:55.292308+08:00 442 Query SET SESSION net_write_timeout = 2147483
2019-03-25T21:37:55.294956+08:00 442 Query SHOW PROCESSLIST
2019-03-25T21:37:55.307138+08:00 442 Query FLUSH TABLES WITH READ LOCK
2019-03-25T21:37:55.321024+08:00 442 Query START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */
2019-03-25T21:37:55.347388+08:00 442 Query /!40101 SET NAMES binary/
2019-03-25T21:37:55.347552+08:00 442 Query SHOW MASTER STATUS
2019-03-25T21:37:55.347730+08:00 442 Query SHOW SLAVE STATUS
2019-03-25T21:37:55.348652+08:00 443 Connect test1@127.0.0.1 on using TCP/IP
2019-03-25T21:37:55.348941+08:00 443 Query SET SESSION wait_timeout = 2147483
2019-03-25T21:37:55.349067+08:00 443 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 2019-03-25T21:37:55.349156+08:00 443 Query START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */
2019-03-25T21:37:55.349235+08:00 443 Query /!40103 SET TIME_ZONE='+00:00' /
2019-03-25T21:37:55.349299+08:00 443 Query /!40101 SET NAMES binary/
2019-03-25T21:37:55.349853+08:00 444 Connect test1@127.0.0.1 on using TCP/IP
2019-03-25T21:37:55.350084+08:00 444 Query SET SESSION wait_timeout = 2147483
2019-03-25T21:37:55.350203+08:00 444 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 2019-03-25T21:37:55.350298+08:00 444 Query START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */
2019-03-25T21:37:55.350396+08:00 444 Query /!40103 SET TIME_ZONE='+00:00' /
2019-03-25T21:37:55.350486+08:00 444 Query /!40101 SET NAMES binary/
2019-03-25T21:37:55.351008+08:00 445 Connect test1@127.0.0.1 on using TCP/IP
2019-03-25T21:37:55.351181+08:00 445 Query SET SESSION wait_timeout = 2147483
2019-03-25T21:37:55.351268+08:00 445 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 2019-03-25T21:37:55.351334+08:00 445 Query START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */
2019-03-25T21:37:55.351404+08:00 445 Query /!40103 SET TIME_ZONE='+00:00' /
2019-03-25T21:37:55.351469+08:00 445 Query /!40101 SET NAMES binary/
2019-03-25T21:37:55.351891+08:00 446 Connect test1@127.0.0.1 on using TCP/IP
2019-03-25T21:37:55.352086+08:00 446 Query SET SESSION wait_timeout = 2147483
2019-03-25T21:37:55.352223+08:00 446 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 2019-03-25T21:37:55.352339+08:00 446 Query START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */
2019-03-25T21:37:55.352448+08:00 446 Query /!40103 SET TIME_ZONE='+00:00' /
2019-03-25T21:37:55.352549+08:00 446 Query /!40101 SET NAMES binary/
2019-03-25T21:37:55.353028+08:00 447 Connect test1@127.0.0.1 on using TCP/IP
2019-03-25T21:37:55.353172+08:00 447 Query SET SESSION wait_timeout = 2147483
2019-03-25T21:37:55.353244+08:00 447 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 2019-03-25T21:37:55.353301+08:00 447 Query START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */
2019-03-25T21:37:55.353361+08:00 447 Query /!40103 SET TIME_ZONE='+00:00' /
2019-03-25T21:37:55.353418+08:00 447 Query /!40101 SET NAMES binary/
2019-03-25T21:37:55.353874+08:00 448 Connect test1@127.0.0.1 on using TCP/IP
2019-03-25T21:37:55.354083+08:00 448 Query SET SESSION wait_timeout = 2147483
2019-03-25T21:37:55.354196+08:00 448 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 2019-03-25T21:37:55.354288+08:00 448 Query START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */
2019-03-25T21:37:55.354380+08:00 448 Query /!40103 SET TIME_ZONE='+00:00' /
2019-03-25T21:37:55.354468+08:00 448 Query /!40101 SET NAMES binary/
2019-03-25T21:37:55.354912+08:00 449 Connect test1@127.0.0.1 on using TCP/IP
2019-03-25T21:37:55.355066+08:00 449 Query SET SESSION wait_timeout = 2147483
2019-03-25T21:37:55.355151+08:00 449 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 2019-03-25T21:37:55.355216+08:00 449 Query START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */
2019-03-25T21:37:55.355283+08:00 449 Query /!40103 SET TIME_ZONE='+00:00' /
2019-03-25T21:37:55.355346+08:00 449 Query /!40101 SET NAMES binary/
2019-03-25T21:37:55.355783+08:00 450 Connect test1@127.0.0.1 on using TCP/IP
2019-03-25T21:37:55.355930+08:00 450 Query SET SESSION wait_timeout = 2147483
2019-03-25T21:37:55.356028+08:00 450 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 2019-03-25T21:37:55.356090+08:00 450 Query START TRANSACTION /*!40108 WITH CONSISTENT SNAPSHOT */
2019-03-25T21:37:55.356152+08:00 450 Query /!40103 SET TIME_ZONE='+00:00' /
2019-03-25T21:37:55.356211+08:00 450 Query /!40101 SET NAMES binary/
2019-03-25T21:37:55.356337+08:00 442 Init DB test
2019-03-25T21:37:55.356459+08:00 442 Query SHOW TABLE STATUS
2019-03-25T21:37:55.562238+08:00 442 Query SHOW CREATE DATABASE test
2019-03-25T21:37:55.562630+08:00 442 Query UNLOCK TABLES /* FTWRL */
2019-03-25T21:37:55.562751+08:00 443 Query SHOW CREATE TABLE test.sbtest1
2019-03-25T21:37:55.562864+08:00 449 Quit
2019-03-25T21:37:55.562941+08:00 448 Quit
2019-03-25T21:37:55.562995+08:00 446 Quit
2019-03-25T21:37:55.563010+08:00 445 Quit
2019-03-25T21:37:55.562942+08:00 447 Quit
2019-03-25T21:37:55.563100+08:00 442 Quit
2019-03-25T21:37:55.563207+08:00 450 Quit
2019-03-25T21:37:55.563207+08:00 443 Quit
2019-03-25T21:37:55.570808+08:00 444 Query select COLUMN_NAME from information_schema.COLUMNS where TABLE_SCHEMA='test' and TABLE_NAME='sbtest1' and extra like '%GENERATED%'
2019-03-25T21:37:55.578158+08:00 444 Query SELECT /!40001 SQL_NO_CACHE / * FROM test.sbtest1
2019-03-25T21:37:55.711160+08:00 444 Quit