云服务器 virtblk_ioctl gpf 内核crash问题分析

前情提要

有客户反馈机器频繁出现重启，查看每次的堆栈都是virtio_check_driver_offered_feature访问非法地址的gpf报错，比较像是某个内核bug导致。

看到是fdisk进程触发，向客户了解到，问题机器都是执行了fdisk后跪了。

收集coredump，并根据以上现象着手进行排查。

问题原因

centos7 3.10.0-1160.6.1.el7.x86_64 以下的内核版本，在使用lvm后如果没清理vg后先umount并解挂盘，会导致内核中驱动相关的部分数据结构不完整，导致fdisk命令需要去访问到对应数据结构时，访问非法地址触发重启。

redhat对这个bug的官方说明：

https://access.redhat.com/solutions/4943981

对应修复的内核patch：

https://lore.kernel.org/linux-block/20200430140442.171016-1-stefanha@redhat.com/

分析过程

vmcore分析

堆栈：

代码语言：txt复制

crash> sys
      KERNEL: /usr/lib/debug/lib/modules/3.10.0-1062.9.1.el7.x86_64/vmlinux
    DUMPFILE: vmcore  [PARTIAL DUMP]
        CPUS: 2
        DATE: Tue Mar 30 12:44:11 2021
      UPTIME: 379 days, 13:03:40
LOAD AVERAGE: 0.29, 0.13, 0.08
       TASKS: 366
    NODENAME: mongodb-38
     RELEASE: 3.10.0-1062.9.1.el7.x86_64
     VERSION: #1 SMP Fri Dec 6 15:49:49 UTC 2019
     MACHINE: x86_64  (2394 Mhz)
      MEMORY: 16 GB
       PANIC: "general protection fault: 0000 [#1] SMP "
crash> bt
PID: 1370   TASK: ffff922804b441c0  CPU: 1   COMMAND: "fdisk"
 #0 [ffff9226746dbae8] machine_kexec at ffffffffba265b24 
 #1 [ffff9226746dbb48] __crash_kexec at ffffffffba322342 
 #2 [ffff9226746dbc18] crash_kexec at ffffffffba322430 
 #3 [ffff9226746dbc30] oops_end at ffffffffba985798 
 #4 [ffff9226746dbc58] die at ffffffffba230a7b 
 #5 [ffff9226746dbc88] do_general_protection at ffffffffba985092 
 #6 [ffff9226746dbcc0] general_protection at ffffffffba984718 
    [exception RIP: virtio_check_driver_offered_feature 16]
    RIP: ffffffffc031f450  RSP: ffff9226746dbd70  RFLAGS: 00010282
    RAX: ffff922498d6a300  RBX: ffff922498d6a180  RCX: dead000000000200
    RDX: 0000000000005331  RSI: 0000000000000007  RDI: ffff922498d6a180
    RBP: ffff9226746dbd70   R8: ffffffffbae4d900   R9: ffffffffc089dfb0
    R10: 00007f56af4ad7b8  R11: 0000000000000246  R12: ffff92280f34cd00
    R13: 000000000000001d  R14: 0000000000005331  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffff9226746dbd78] virtblk_ioctl at ffffffffc0498b4c  [virtio_blk]
 #8 [ffff9226746dbdb0] __blkdev_driver_ioctl at ffffffffba562a75 
 #9 [ffff9226746dbdc0] dm_blk_ioctl at ffffffffc089e024  [dm_mod]
#10 [ffff9226746dbe10] blkdev_ioctl at ffffffffba5634ba 
#11 [ffff9226746dbe70] block_ioctl at ffffffffba48a8b1 
#12 [ffff9226746dbe80] do_vfs_ioctl at ffffffffba45fb40 
#13 [ffff9226746dbf00] sys_ioctl at ffffffffba45fde1 
#14 [ffff9226746dbf50] system_call_fastpath at ffffffffba98dede 
    RIP: 00007f56af1db2b7  RSP: 00007fff1ee928e8  RFLAGS: 00000206
    RAX: 0000000000000010  RBX: 0000000001191310  RCX: ffffffffba98de21
    RDX: 0000000000000000  RSI: 0000000000005331  RDI: 0000000000000005
    RBP: 0000000001191030   R8: 0000000000000001   R9: 00007f56af4ad7b8
    R10: 00007f56af4ad7b8  R11: 0000000000000246  R12: 0000000000000000
    R13: 0000000001191550  R14: 0000000000000005  R15: 0000000000009bef
    ORIG_RAX: 0000000000000010  CS: 0033  SS: 002b

踩内存的地址：

代码语言：txt复制

crash> dis -rl ffffffffc031f450
0xffffffffc031f440 <virtio_check_driver_offered_feature>:       nopl   0x0(%rax,%rax,1) [FTRACE NOP]
0xffffffffc031f445 <virtio_check_driver_offered_feature 5>:     push   %rbp
0xffffffffc031f446 <virtio_check_driver_offered_feature 6>:     mov    0xa0(%rdi),%rcx
0xffffffffc031f44d <virtio_check_driver_offered_feature 13>:    mov    %rsp,%rbp
0xffffffffc031f450 <virtio_check_driver_offered_feature 16>:    mov    0x90(%rcx),�x

从偏移量

推导出踩内存的位置是vdev->dev.driver：

查看rdi和rcx：

RCX: dead000000000200

RDI: ffff922498d6a180

看到driver确实跪了：

往上追溯看看，是不是传过来的block_device就有问题？

从栈里找到block_device地址（省略找地址过程）：

代码语言：txt复制

crash> block_device.bd_disk ffff92280f34cd00
  bd_disk = 0xffff9227e7a75800
crash> gendisk.disk_name,private_data 0xffff9227e7a75800
  disk_name = "vdc000000000000000000000000000000000000000000000000000000c"
  private_data = 0xffff922498d6a300
crash> virtio_blk.vdev 0xffff922498d6a300
  vdev = 0xffff922498d6a180

看到地址能和rdi传进来的vdev对上，访问的是vdc盘。

但是，我们发现vdc盘，在系统里找不到了：