记一次虚拟机无法挂载volume的怪异问题排查

使用nova volume-attach <server> <volume>命令挂载卷,命令没有返回错误,但是查看虚拟机状态,卷并没有挂载上。

 

故障原因

疑似虚拟机长时间运行(超过1年)后,libvirt无法执行live attach操作。

 

处理方法

将虚拟机关机,在关机状态下挂载卷,然后启动虚拟机。

 

排查过程

由于没有nova命令没有报错,基本确定问题出在计算节点,直接到计算节点查看日志,发现如下异常:

2018-06-05 13:40:32.337 160589 DEBUG nova.virt.libvirt.config [req-392fd85e-1853-4c6c-8248-310ca6289895 d31b768e1dbf4a0dbf2571234b4e2f5a 65ea11db9ebf49c69d 3c05bc38925617 - - -] Generated XML ('<disk type="network" device="disk">\n <driver type="raw" cache="none"/>\n <source protocol="rbd">\n <host port="6789"/>\n <host port="6789"/>\n <host port="6789"/>\n </source>\n <auth username="cinder">\n <secret type="ceph" uuid="90b0641c-a0d1-4103-ad8c-d580dd7da953"/>\n </auth>\n <target bus="virtio" dev="vdc"/>\n <serial>433ad82b-4d8d-4d5c-b1c0-d0c88e0c397b</serial>\n</disk>\n',) to_xml /usr/lib/python2.7/site-packages/nova/virt/libvirt/config.py:82 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [req-392fd85e-1853-4c6c-8248-310ca6289895 d31b768e1dbf4a0dbf2571234b4e2f5a 65ea11db9ebf49c69d 3c05bc38925617 - - -] [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] Failed to attach volume at mountpoint: /dev/vdc 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] Traceback (most recent call last): 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1121, in attach_volume 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] guest.attach_device(conf, persistent=True, live=live) 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 235, in attach_device 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] self._domain.attachDeviceFlags(conf.to_xml(), flags=flags) 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] result = proxy_call(self._autowrap, f, *args, **kwargs) 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] rv = execute(f, *args, **kwargs) 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] six.reraise(c, e, tb) 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] rv = meth(*args, **kwargs) 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 554, in attachDeviceFlags 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self) 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced] libvirtError: internal error: unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk2' could not be initialized 2018-06-05 13:40:32.386 160589 ERROR nova.virt.libvirt.driver [instance: 211437a1-e4c4-40e0-ade1-b167d6251ced]

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/zyfzzf.html