RAS(三)Intel MCA-CMCI

2023-08-21 15:47:29 浏览数 (2)

RAS(三)Intel MCA-CMCI

Corrected machine-check error interrupt (CMCI)是MCA的增强特性,它提供了一种threshold-based的错误上报方式。这种模式下,软件可以配置硬件corrected MC errors的阈值,硬件发生CE(Corrected Error)次数达到阈值后,会产生一个中断通知到软件处理。

值得一提的是,CMCI是随MCA加入的特性,最开始只能通过软件轮询方式获取CE信息。CMCI中断通知方式的优点是每个CE都会经过IRQ Handle处理,不会丢失任一CE;而轮询方式可能因为轮询频率低、存储空间有限等原因,导致丢失CE。但是并不是说CMCI最优,CMCI的缺点是大量CE会产生中断风暴,影响机器的性能。不幸的是在云服务器场景,CE风暴是比较常见的,那么当下Intel服务器是如何解决这个问题的呢?下面会讲到。

CMCI机制

CMCI默认是关闭的,软件需要通过配置IA32_MCG_CAP[10] = 1打开。

软件通过IA32_MCi_CTL2 MSR来控制对应Bank使能/关闭CMCI功能。

通过IA32_MCi_CTL2 Bit 14:0设置阈值,如果设置非0,则使用配置的阈值;如果CMCI不支持,则全0;

CMCI机制如下图

硬件通过比较IA32_MCi_CTL2 Bit 14:0和IA32_MCi_STATUS Bit 52:38,如果数值相等,那么overflow event发送到APIC的CMCI LVT entry。如果MC error涉及多个processors,那么CMCI中断会同时发送到这些processors,比如2个cpu共享的cache发生CE,那么这两个cpu都会收到CMCI。

CMCI初始化

以Linux v6.3分支为例,内核使能CMCI代码

C arch/x86/kernel/cpu/mce/intel.cvoid intel_init_cmci(void){ int banks; if (!cmci_supported(&banks)) return; mce_threshold_vector = intel_threshold_interrupt; cmci_discover(banks); /* * For CPU #0 this runs with still disabled APIC, but that's * ok because only the vector is set up. We still do another * check for the banks later for CPU #0 just to make sure * to not miss any events. */ apic_write(APIC_LVTCMCI, THRESHOLD_APIC_VECTOR|APIC_DM_FIXED); cmci_recheck();}

1.cmci_supported()函数主要事项包括

•根据内核启动参数"mce=no_cmci,ignore_ce"判断是否打开cmci和ce上报功能

•检查硬件是否支持cmci

•通过MCG_CMCI_P bit判断硬件是否使能cmci功能

2.mce_threshold_vector = intel_threshold_interrupt; 声明cmci的中断处理函数为intel_threshold_interrupt();

3.cmci_discover()函数主要完成

•遍历所有banks,通过配置IA32_MCi_CTL2寄存器使能所有bank的cmci功能;

C rdmsrl(MSR_IA32_MCx_CTL2(i), val); ... val |= MCI_CTL2_CMCI_EN; wrmsrl(MSR_IA32_MCx_CTL2(i), val); rdmsrl(MSR_IA32_MCx_CTL2(i), val);

•设置cmci threshold值,代码如下

C #define CMCI_THRESHOLD 1 if (!mca_cfg.bios_cmci_threshold) { val &= ~MCI_CTL2_CMCI_THRESHOLD_MASK; val |= CMCI_THRESHOLD; } else if (!(val & MCI_CTL2_CMCI_THRESHOLD_MASK)) { /* * If bios_cmci_threshold boot option was specified * but the threshold is zero, we'll try to initialize * it to 1. */ bios_zero_thresh = 1; val |= CMCI_THRESHOLD; }

如果用户未通过启动参数"mce=bios_cmci_threshold"配置值,则val = CMCI_THRESHOLD,为1;

如果启动参数"mce=bios_cmci_threshold"配置,那么表示bios已配置threshold值,即val & MCI_CTL2_CMCI_THRESHOLD_MASK不为0,跳过else if判断,采用bios配置值;如果bios未配置值,val & MCI_CTL2_CMCI_THRESHOLD_MASK为0,那么驱动初始化threshold为1。

4.cmci_recheck()

cmci_recheck函数通过调用machine_check_poll(),检查CPU #0是否有遗漏的CE&UCE events。

CMCI处理

cmci中断处理函数为intel_threshold_interrupt(),定义在arch/x86/kernel/cpu/mce/intel.c

C /* * The interrupt handler. This is called on every event. * Just call the poller directly to log any events. * This could in theory increase the threshold under high load, * but doesn't for now. */static void intel_threshold_interrupt(void){ if (cmci_storm_detect()) return; machine_check_poll(MCP_TIMESTAMP, this_cpu_ptr(&mce_banks_owned));}

1.cmci_storm_detect()函数主要是对cmci storm的处理,代码如下

C static bool cmci_storm_detect(void){ unsigned int cnt = __this_cpu_read(cmci_storm_cnt); unsigned long ts = __this_cpu_read(cmci_time_stamp); unsigned long now = jiffies; int r; if (__this_cpu_read(cmci_storm_state) != CMCI_STORM_NONE) return true; if (time_before_eq(now, ts CMCI_STORM_INTERVAL)) { cnt ; } else { cnt = 1; __this_cpu_write(cmci_time_stamp, now); } __this_cpu_write(cmci_storm_cnt, cnt); if (cnt <= CMCI_STORM_THRESHOLD) return false; cmci_toggle_interrupt_mode(false); __this_cpu_write(cmci_storm_state, CMCI_STORM_ACTIVE); r = atomic_add_return(1, &cmci_storm_on_cpus); mce_timer_kick(CMCI_STORM_INTERVAL); this_cpu_write(cmci_backoff_cnt, INITIAL_CHECK_INTERVAL); if (r == 1) pr_notice("CMCI storm detected: switching to poll moden"); return true;}

该函数通过jiffies,判断固定时间内发生的cmci次数是否大于CMCI_STORM_THRESHOLD(15),如果否则return,反之说明发生cmci storm,则执行cmci_toggle_interrupt_mode()关闭cmci功能, 切换为poll mode,通过轮询方式获取event;

2.非cmci storm情况下,通过machine_check_poll(MCP_TIMESTAMP, this_cpu_ptr(&mce_banks_owned))函数获取并记录故障信息

参数1定义如下,MCP_TIMESTAMP表示会记录当前TSC

C enum mcp_flags { MCP_TIMESTAMP = BIT(0), /* log time stamp */ MCP_UC = BIT(1), /* log uncorrected errors */ MCP_DONTLOG = BIT(2), /* only clear, don't log */};

machine_check_poll函数主要功能是通过读取IA32_MCG_STATUS、IA32_MCi_STATUS寄存器信息和CPU的ip、cs等相关信息,然后故障分类,将CE event或其他故障类型event记录到/dev/mcelog。用户可以通过读取/dev/mcelog获取错误记录。

执行流程如下,过程说明在代码注释中

C bool machine_check_poll(enum mcp_flags flags, mce_banks_t *b){ if (flags & MCP_TIMESTAMP) m.tsc = rdtsc(); // 记录当前TSC/*CE Error记录*/ /* If this entry is not valid, ignore it */ if (!(m.status & MCI_STATUS_VAL)) continue; /* * If we are logging everything (at CPU online) or this * is a corrected error, then we must log it. */ if ((flags & MCP_UC) || !(m.status & MCI_STATUS_UC)) goto log_it;/*UCNA Error记录*/ /* * Log UCNA (SDM: 15.6.3 "UCR Error Classification") * UC == 1 && PCC == 0 && S == 0 */ if (!(m.status & MCI_STATUS_PCC) && !(m.status & MCI_STATUS_S)) goto log_it;/*通过mce_log记录故障信息*/log_it: /* * Don't get the IP here because it's unlikely to * have anything to do with the actual error location. */ if (!(flags & MCP_DONTLOG) && !mca_cfg.dont_log_ce) mce_log(&m); else if (mce_usable_address(&m)) { /* * Although we skipped logging this, we still want * to take action. Add to the pool so the registered * notifiers will see it. */ if (!mce_gen_pool_add(&m)) mce_schedule_work(); } }

总结一下,CMCI是MCA的一个增强特性,主要用于将硬件CE、UCNA等类型故障通过中断方式上报到软件,软件收到中断后,执行中断处理函数intel_threshold_interrupt()采取irq mode或poll mode记录错误信息到/dev/mcelog,用户态可以通过/dev/mcelog获取硬件故障信息。

参考文档:《Intel® 64 and IA-32 Architectures Software Developer’s Manual 》

0 人点赞