记一次服务器性能问题排查
前几天发现有一个集群在流量高峰时期的访问失败率比较高。 通过小米监控发现有一台服务器在当天流量高峰时候网卡有net.if.in.dropped现象。并且这台机器的CPU软中断一直比其他服务器高，尤其是有问题的那个时间段：
通过其它的监控指标发现当时集群的网卡进出口流量一致，磁盘使用也一致，集群的负载是均衡的，唯一有差别的就是上面的两个指标。当时的场景没有抓到，事后我们主要用 vmstat，iostat，top 这三个命令来进行一下分析总结。vmstat命令可以动态的展示系统当前的内存使用，I/O吞吐，CPU使用情况。直接执行vmstat命令展示的数据说明如下：
Procs
r: The number of runnable processes (running or waiting for run time).
b: The number of processes in uninterruptible sleep.
Memory
swpd: the amount of virtual memory used.
free: the amount of idle memory.
buff: the amount of memory used as buffers.
cache: the amount of memory used as cache.
inact: the amount of inactive memory. (-a option)
active: the amount of active memory. (-a option)
Swap
si: Amount of memory swapped in from disk (/s).
so: Amount of memory swapped to disk (/s).
IO
bi: Blocks received from a block device (blocks/s).
bo: Blocks sent to a block device (blocks/s).
System
in: The number of interrupts per second, including the clock.
cs: The number of context switches per second.
CPU
These are percentages of total CPU time.
us: Time spent running non-kernel code. (user time, including nice time)
sy: Time spent running kernel code. (system time)
id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.
st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.
执行 vmstat 1 60 命令(每1秒采样一次，总共采样60次)。有问题服务器的结果如下：
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
8 0 0 1363892 268964 119842880 0 0 4935 1597 0 0 7 15 77 0 0
4 0 0 1356060 268972 119844024 0 0 170160 60000 162405 63983 10 23 68 0 0
3 0 0 1365568 268972 119844536 0 0 194800 67356 167479 66841 10 25 65 0 0
3 0 0 1372208 268972 119846128 0 0 194160 78864 168035 63354 11 26 63 0 0
5 0 0 1363208 268972 119846912 0 0 164560 71280 161833 61709 10 21 69 0 0
6 0 0 1330956 268972 119847760 0 0 176640 100756 171760 69361 11 28 62 0 0
12 0 0 1345788 268972 119849112 0 0 163040 103760 168653 62071 11 26 63 0 0
5 0 0 1351828 268972 119849776 0 0 159280 110160 169969 60950 11 25 64 0 0
1 0 0 1350516 268972 119851104 0 0 194800 81604 167894 62287 11 25 65 0 0
15 0 0 1339176 268972 119852336 0 0 209680 118596 171699 63858 11 28 61 0 0
5 0 0 1339564 268972 119853152 0 0 202080 113448 169312 66145 10 26 64 0 0
6 0 0 1328980 268972 119854208 0 0 196160 92656 166386 63560 10 25 65 0 0
7 0 0 1331616 268980 119855168 0 0 182960 69356 166491 61940 11 24 65 0 0
8 0 0 1294800 268980 119856112 0 0 199440 61320 169294 62949 11 26 63 0 0
9 0 0 1307652 268984 119857360 0 0 186400 85184 170619 61974 11 26 63 0 0
6 0 0 1312012 268984 119858144 0 0 161280 75280 167987 59314 11 25 65 0 0
10 0 0 1307996 268984 119859216 0 0 179440 105216 172963 59477 11 28 60 0 0
4 0 0 1318468 268988 119860608 0 0 181280 87596 170244 59577 11 26 63 0 0
4 0 0 1335068 268988 119860992 0 0 187440 88584 169176 62859 10 25 64 0 0
6 0 0 1341248 268988 119862656 0 0 166960 136996 170971 64830 10 26 63 0 0
6 0 0 1361224 268988 119863680 0 0 178160 135780 173369 71979 11 27 61 0 0
8 0 0 1366044 268988 119864272 0 0 182480 103648 169674 68866 11 26 63 0 0
3 0 0 1358212 268988 119865696 0 0 179600 87720 166669 66699 11 24 65 0 0
3 0 0 1351528 268988 119866624 0 0 174160 83952 170870 63858 11 26 63 0 0
8 0 0 1346232 268988 119867744 0 0 199120 110756 168965 65993 11 26 64 0 0
6 0 0 1350520 268988 119868872 0 0 192160 107408 170576 65287 11 26 63 0 0
6 0 0 1346920 268988 119869824 0 0 174480 111228 165635 65560 10 25 65 0 0
9 0 0 1331100 268988 119871168 0 0 169760 90852 168314 58844 11 26 63 0 0
4 0 0 1333852 268988 119872056 0 0 185360 111364 170081 62644 11 26 63 0 0
9 0 0 1342080 268988 119872880 0 0 180960 82828 164199 57328 10 23 68 0 0
6 0 0 1335656 268988 119873936 0 0 172000 90864 167503 57523 10 24 66 0 0
12 0 0 1333380 268988 119874992 0 0 151840 120400 167601 59765 11 25 64 0 0
4 0 0 1328596 268988 119875840 0 0 184240 93116 170450 61530 11 26 62 0 0
5 0 0 1355956 268988 119877088 0 0 179680 79144 171302 63731 11 26 63 0 0
4 0 0 1337092 268988 119878176 0 0 188400 82528 172773 64593 12 28 60 0 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
7 0 0 1336388 268988 119879136 0 0 196320 78744 172478 65928 11 28 61 0 0
6 0 0 1339660 268988 119880480 0 0 183120 96720 171711 67915 11 29 60 0 0
4 0 0 1316496 268988 119881136 0 0 213840 52872 171683 69725 11 28 61 0 0
12 0 0 1298836 268992 119882424 0 0 214160 57640 171877 64540 15 30 56 0 0
10 0 0 1341872 268992 119883264 0 0 229040 66952 170319 67409 13 32 55 0 0
10 0 0 1347632 268992 119884576 0 0 179680 105316 169700 62016 11 26 63 0 0
11 0 0 1339056 268992 119885672 0 0 226160 61216 168715 65290 10 26 64 0 0
2 0 0 1341484 268992 119886120 0 0 225600 73536 171622 64119 11 27 62 0 0
8 0 0 1348404 268992 119887760 0 0 183120 90628 170180 65838 11 26 63 0 0
3 0 0 1357300 268992 119884296 0 0 177280 77304 166134 65313 11 25 64 0 0
8 0 0 1355332 268992 119885360 0 0 194560 65464 168439 65062 11 25 64 0 0
42 0 0 1316796 268992 119886656 0 0 205760 73204 168823 67601 13 28 59 0 0
4 0 0 1321356 268992 119888080 0 0 194480 55240 170825 61484 22 32 46 0 0
7 0 0 1330468 268992 119888704 0 0 181680 65308 169781 63698 10 25 64 0 0
3 0 0 1314404 268992 119890232 0 0 189520 63568 168021 67659 10 26 64 0 0
5 0 0 1299968 268992 119891008 0 0 178640 53592 170838 67230 11 27 62 0 0
6 0 0 1299384 268992 119891856 0 0 172080 57800 170272 68717 11 27 62 0 0
8 0 0 1300984 268992 119893016 0 0 175280 70976 169275 64301 12 26 62 0 0
5 0 0 1293368 268996 119893592 0 0 165120 75932 166458 64532 10 25 65 0 0
9 0 0 1282432 268996 119895056 0 0 189760 57912 170060 66099 10 26 64 0 0
8 0 0 1279916 268996 119896184 0 0 182240 42956 168104 62231 11 26 63 0 0
14 0 0 1287636 268996 119897120 0 0 182080 56356 168438 62990 11 26 63 0 0
5 0 0 1284412 269008 119898392 0 0 200080 86536 169819 64652 11 26 63 0 0
4 0 0 1282696 269008 119899040 0 0 189040 83684 170384 64855 11 28 61 0 0
6 0 0 1303300 269008 119900440 0 0 173920 69120 167918 62939 11 25 64 0 0
正常的服务器结果如下：
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
7 0 0 1432956 280220 119618136 0 0 4935 1590 0 0 6 12 82 0 0
6 0 0 1465768 280220 119619168 0 0 184160 70532 164637 70572 9 22 69 0 0
4 0 0 1452600 280224 119620120 0 0 181360 95208 167543 76375 10 23 67 0 0
6 0 0 1458704 280224 119621328 0 0 168080 66356 164573 72164 9 20 71 0 0
2 0 0 1462256 280224 119622352 0 0 174880 58432 162123 74117 9 20 72 0 0
11 0 0 1463236 280224 119623232 0 0 175200 71972 164657 75265 9 20 71 0 0
3 0 0 1460012 280228 119624256 0 0 189760 89476 165881 74505 10 21 70 0 0
5 0 0 1465676 280232 119625472 0 0 155520 46720 160781 66475 8 18 73 0 0
5 0 0 1450120 280232 119626128 0 0 176240 89440 165782 73314 9 21 70 0 0
7 0 0 1439496 280232 119627592 0 0 211600 45940 165804 71051 9 22 69 0 0
4 0 0 1414760 280236 119628152 0 0 189920 100140 167497 73012 9 22 69 0 0
3 0 0 1399288 280236 119629648 0 0 176320 100628 167188 69257 9 22 69 0 0
7 0 0 1416724 280236 119630560 0 0 178960 89564 168472 73611 10 23 67 0 0
3 0 0 1422796 280236 119631232 0 0 171920 70892 163472 65205 9 20 72 0 0
4 0 0 1436236 280236 119632800 0 0 168000 80824 162848 67841 9 19 72 0 0
3 0 0 1428772 280240 119633616 0 0 170720 80968 164447 66245 9 20 71 0 0
4 0 0 1415552 280240 119634528 0 0 167360 104008 166701 69529 9 22 69 0 0
5 0 0 1419144 280240 119635760 0 0 196400 63864 164309 67556 8 20 72 0 0
9 0 0 1412948 280240 119632064 0 0 194000 65064 166030 70419 9 21 70 0 0
3 0 0 1385280 280240 119633328 0 0 191680 77280 167296 71985 9 23 68 0 0
7 0 0 1386332 280240 119634544 0 0 192000 61656 165662 71556 9 21 70 0 0
5 0 0 1408712 280248 119635600 0 0 183600 75396 166392 70483 9 21 70 0 0
12 0 0 1403020 280248 119636696 0 0 171200 63276 166451 73026 9 21 70 0 0
3 0 0 1419180 280248 119638048 0 0 191920 72924 167366 79526 10 23 68 0 0
4 0 0 1429724 280248 119639072 0 0 168320 95512 164707 78749 9 21 71 0 0
8 0 0 1429392 280248 119639760 0 0 155760 70128 163300 75082 9 20 71 0 0
6 0 0 1415576 280248 119641120 0 0 155760 71248 165516 67267 9 21 70 0 0
9 0 0 1433796 280252 119641968 0 0 166960 80712 162482 66827 9 20 71 0 0
8 0 0 1430748 280252 119643040 0 0 175440 74288 164925 71069 9 21 70 0 0
3 0 0 1424112 280252 119644000 0 0 166320 102568 165294 69742 10 21 70 0 0
2 0 0 1439392 280252 119645128 0 0 200320 80232 166113 72858 9 22 69 0 0
1 0 0 1430324 280252 119646304 0 0 185600 97584 165424 70817 9 21 71 0 0
4 0 0 1434020 280252 119647216 0 0 189680 93176 165508 69326 8 21 70 0 0
10 0 0 1413024 280252 119648288 0 0 196880 125668 168253 69284 9 23 67 0 0
8 0 0 1404968 280252 119649440 0 0 195120 100624 167747 72524 9 23 68 0 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
4 0 0 1404600 280252 119650368 0 0 201680 106068 169942 72652 9 24 67 0 0
6 0 0 1428024 280252 119651472 0 0 187360 79456 163193 73238 8 19 73 0 0
5 0 0 1412764 280252 119652352 0 0 162000 87924 165682 75857 9 22 69 0 0
2 2 0 1410620 280252 119653424 0 0 154724 205952 168696 76497 9 23 67 1 0
19 0 0 1404776 280252 119654680 0 0 187040 77760 164044 78910 9 21 69 0 0
6 0 0 1410524 280252 119655336 0 0 184800 76792 167502 81562 11 29 59 0 0
4 0 0 1423164 280252 119656960 0 0 176480 82748 168562 73045 9 23 68 0 0
3 0 0 1432608 280252 119657600 0 0 171040 79244 165631 68034 9 21 70 0 0
3 0 0 1421696 280252 119659008 0 0 169040 104024 163670 65359 9 20 71 0 0
7 0 0 1424236 280256 119660096 0 0 168640 125320 165326 68177 9 20 71 0 0
6 0 0 1421260 280256 119660560 0 0 151840 100332 164649 67403 9 21 70 0 0
5 0 0 1443264 280260 119662144 0 0 189920 107056 168534 75390 10 23 67 0 0
8 0 0 1436724 280260 119662992 0 0 171520 91448 164220 76516 8 20 71 0 0
6 0 0 1418436 280260 119663896 0 0 164640 105852 166848 80053 10 23 68 0 0
3 0 0 1383640 280264 119665488 0 0 196640 113096 166626 73147 19 26 54 0 0
6 0 0 1385380 280264 119666056 0 0 185680 89740 162566 73855 9 20 71 0 0
0 0 0 1389152 280264 119667408 0 0 196240 102140 165227 74665 9 21 70 0 0
5 0 0 1402344 280264 119668656 0 0 172080 104880 161882 72595 9 20 72 0 0
4 0 0 1409312 280264 119669136 0 0 178960 62748 158556 67730 8 18 74 0 0
5 0 0 1390740 280264 119670592 0 0 175360 69504 161059 71217 9 19 72 0 0
6 0 0 1407332 280264 119671592 0 0 198240 90960 162187 74631 9 20 71 0 0
3 0 0 1396088 280264 119672384 0 0 169360 76668 163721 71862 9 20 71 0 0
2 0 0 1362120 280264 119673824 0 0 166080 115656 165067 72769 10 21 70 0 0
3 0 0 1374292 280264 119674528 0 0 176480 97072 164634 73330 9 21 70 0 0
5 0 0 1381628 280264 119675824 0 0 202240 89932 165270 76687 10 22 69 0 0
Procs（进程）： r: 运行队列中进程数量。这个值也可以判断是否需要增加CPU，一般来说当这个值超过总的CPU线程数，就会出现CPU瓶颈。 b: 在非中断睡眠状态的进程数，即等待IO的进程数量。这个数量需要结合具体的情况来分析是服务器本身的问题或者是请求端的问题。
Memory（内存）： swpd: 使用虚拟内存大小。当系统被触发swap机制时，会把数据放在swap中，如果这个swpd数值在变动的时候，说明内存不够了（或有不够的风险）。如果swpd的值不为0，但是SI，SO的值长期为0，这种情况不会影响系统性能。 free: 空闲物理内存大小。这里提一下free 和 available ，available一般大于free，因为available包含了buffer/cache，buffer/cache在必要的时候是会被释放掉的。 buff: 用作缓冲的内存大小。中文一般翻译为 "缓冲区"。要理解缓冲区，必须明确另外两个概念："扇区" 和 "块"。扇区是设备的最小寻址单元，也叫 "硬扇区" 或 "设备块"。块是操作系统中文件系统的最小寻址单元，也叫 "文件块" 或 "I/O 块"。每个块包含一个或多个扇区，但大小不能超过一个页面，所以一个页可以容纳一个或多个内存中的块。当一个块被调入内存时，它要存储在一个缓冲区中。每个缓冲区与一个块对应，它相当于是磁盘块在内存中的表示。 cache: 用作缓存的内存大小。用来缓存打开的文件，提高文件的访问速度。在操作系统中指 page cache，中文一般翻译为 "页高速缓存"。页高速缓存是内核实现的磁盘缓存。它主要用来减少对磁盘的 I/O 操作。具体地讲，是通过把磁盘中的数据缓存到物理内存中，把对磁盘的访问变为对物理内存的访问。页高速缓存缓存的是内存页面。缓存中的页来自对普通文件、块设备文件(这个指的就是 buffer/cache )和内存映射文件的读写。 inact: 非活跃内存大小（当使用-a选项时显示） active: 活跃的内存大小（当使用-a选项时显示） 注意：如果cache的值大的时候，说明cache处的文件数多，如果频繁访问到的文件都能被cache处（内存命中），那么磁盘的读IO bi会非常小。
Swap（交换分区）： si: 每秒从交换区写到内存的大小，由磁盘调入内存 so: 每秒写入交换区的内存大小，由内存调入磁盘 注意：内存够用的时候，这2个值都是0，如果这2个值长期大于0时，系统性能会受到影响，磁盘IO和CPU资源都会被消耗。有些朋友看到空闲内存（free）很少的或接近于0时，就认为内存不够用了，不能光看这一点，还要结合si和so，如果free很少，但是si和so也很少（大多时候是0），那么不用担心，系统性能这时不会受到影响的，因为这里是Linux/Unix的聪明之处，把空闲的物理内存的一部分拿来做文件和目录的缓存，是为了提高程序执行的性能，当程序使用内存时，buffer/cached会很快地被使用。
IO：（现在的Linux版本块的大小为1kb） bi: 每秒读取的块数 bo: 每秒写入的块数 注意：随机磁盘读写的时候，这2个值越大（如超出1024k)，能看到CPU在IO等待的值也会越大。
system（系统）： in: 每秒中断数，包括时钟中断。 cs: 每秒上下文切换数。 注意：上面2个值越大，会看到由内核消耗的CPU时间会越大。
CPU（以百分比表示）：
us: 用户进程执行时间百分比(user time)。 us的值比较高时，说明用户进程消耗的CPU时间多，但是如果长期超50%的使用，那么我们就该考虑优化程序算法或者进行加速。
sy: 内核系统进程执行时间百分比(system time)。sy的值高时，说明系统内核消耗的CPU资源多，如果太高，表示系统调用时间长，例如IO操作频繁。
wa: IO等待时间百分比。wa的值高时，说明IO等待比较严重，这可能由于磁盘大量作随机访问造成，也有可能磁盘出现瓶颈（块操作）。
id: 空闲时间百分比。Linux内核中的一个空闲线程（idle thread）。cpu_idle()会在0号进程（process zero）中永远的运行下去。一旦有什么事情可做，比如有了一个活动就绪的进程（runnable process），0号进程就会激活CPU去执行这个任务，直到没有活动就绪的进程后才返回。所以这个值越大，说明CPU当前的负载越低。
wt ：CPU等待磁盘写入完成的时间。就是说前提是要进行IO操作，在进行IO操作的时候，CPU等待时间。比如某个程序，最后一步，从系统空间到dst硬盘空间的时候，如果程序是阻塞的，那么这个时候cpu就要等待数据写入磁盘才能完成写操作了。所以这个时候cpu等待的时间就是wa。
st：从虚拟机上偷取的时间。专门对虚拟机来说的，一台物理是可以虚拟化出几台虚拟机的。在其中一台虚拟机上查看发现st不为0，就说明本来有这么多个cpu时间是安排给我这个虚拟机的，但是由于某种虚拟技术，把这个cpu时间分配给了其他的虚拟机了。这就叫做偷取。服务器CPU都是12核心，24线程，内存126G。综合上面的数据分析，内存使用没有问题，磁盘I/O应该也是正常的（后面结合iostat来具体分析判定）。CPU使用方面us有问题的服务器略高，但是sy有问题的服务器就明显高于正常的服务器了，这还是在非高峰时段。
下面用top命令来查看下CPU的具体使用情况。有问题的服务器和正常的服务器：
可以看到有问题的服务器si（软中断）高于正常的服务器。有问题的服务器中断使用情况: cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23
0: 147 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 2-edge timer
8: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 8-edge rtc0
9: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 9-fasteoi acpi
16: 4185944508 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 16-fasteoi ahci[0000:00:11.4], ahci[0000:00:1f.2]
18: 61 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 18-fasteoi ehci_hcd:usb1, ehci_hcd:usb2
19: 40 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 19-fasteoi xhci-hcd:usb3
33: 0 616832492 0 0 0 0 1701 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67633152-edge mpt2sas0-msix0
34: 0 0 882245053 0 0 0 894 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67633153-edge mpt2sas0-msix1
35: 0 0 0 330191765 0 0 7849 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67633154-edge mpt2sas0-msix2
36: 0 0 0 0 378114622 0 195 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67633155-edge mpt2sas0-msix3
37: 0 0 0 0 0 721523631 814 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67633156-edge mpt2sas0-msix4
38: 0 0 0 0 0 0 413995958 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67633157-edge mpt2sas0-msix5
39: 0 0 0 0 0 0 32 750428987 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67633158-edge mpt2sas0-msix6
40: 0 0 0 0 0 0 42 0 624847678 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67633159-edge mpt2sas0-msix7
43: 159 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 524289-edge
44: 159 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 524290-edge
45: 159 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 524291-edge
46: 159 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 PCI-MSI 524292-edge
47: 159 0 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 PCI-MSI 524293-edge
48: 159 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 PCI-MSI 524294-edge
49: 159 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 PCI-MSI 524295-edge
50: 159 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 PCI-MSI 524296-edge
53: 159 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 PCI-MSI 526337-edge
54: 159 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 PCI-MSI 526338-edge
55: 159 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 PCI-MSI 526339-edge
56: 159 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19 0 PCI-MSI 526340-edge
57: 159 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19 PCI-MSI 526341-edge
58: 178 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 526342-edge
59: 159 19 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 526343-edge
60: 159 0 19 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 526344-edge
62: 14285 0 0 19921 0 0 0 0 0 4051253753 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1572864-edge eth2-TxRx-0
63: 1 7780 0 0 10096 0 0 0 0 0 4006506872 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1572865-edge eth2-TxRx-1
64: 1 0 8743 0 0 10818 0 0 0 0 0 3663293850 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1572866-edge eth2-TxRx-2
65: 1 0 0 11621 0 0 12785 0 0 0 0 0 3555724755 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1572867-edge eth2-TxRx-3
66: 1 0 0 0 10596 0 0 14969 0 0 0 0 0 3625001439 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1572868-edge eth2-TxRx-4
67: 1 0 0 0 0 9219 0 0 14465 0 0 0 0 0 3725854842 0 0 0 0 0 0 0 0 0 PCI-MSI 1572869-edge eth2-TxRx-5
68: 1 0 0 0 0 0 13058 0 0 16483 0 0 0 0 0 3677098790 0 0 0 0 0 0 0 0 PCI-MSI 1572870-edge eth2-TxRx-6
69: 1 0 0 0 0 0 0 8697 0 0 15163 0 0 0 0 0 3688148186 0 0 0 0 0 0 0 PCI-MSI 1572871-edge eth2-TxRx-7
70: 1 0 0 0 0 0 0 0 9012 0 0 16550 0 0 0 0 0 3658702427 0 0 0 0 0 0 PCI-MSI 1572872-edge eth2-TxRx-8
71: 1 0 0 0 0 0 0 0 0 10183 0 0 15603 0 0 0 0 0 3749469305 0 0 0 0 0 PCI-MSI 1572873-edge eth2-TxRx-9
72: 1 0 0 0 0 0 0 0 0 0 10229 0 0 17304 0 0 0 0 0 4028227773 0 0 0 0 PCI-MSI 1572874-edge eth2-TxRx-10
73: 1 0 0 0 0 0 0 0 0 0 0 10721 0 0 16318 0 0 0 0 0 3978988204 0 0 0 PCI-MSI 1572875-edge eth2-TxRx-11
74: 1 0 0 0 0 0 0 0 0 0 0 0 10700 0 0 16073 0 0 0 0 0 3676339114 0 0 PCI-MSI 1572876-edge eth2-TxRx-12
75: 1 0 0 0 0 0 0 0 0 0 0 0 0 8681 0 0 15610 0 0 0 0 0 3660092097 0 PCI-MSI 1572877-edge eth2-TxRx-13
76: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 10199 0 0 15682 0 0 0 0 0 3709949847 PCI-MSI 1572878-edge eth2-TxRx-14
77: 3837431257 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10815 0 0 15187 0 0 0 0 0 PCI-MSI 1572879-edge eth2-TxRx-15
78: 1 3592075450 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3038 0 0 13214 0 0 0 0 PCI-MSI 1572880-edge eth2-TxRx-16
79: 1 0 3560947694 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2625 0 0 11841 0 0 0 PCI-MSI 1572881-edge eth2-TxRx-17
80: 1 0 0 3623257445 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2179 0 0 13037 0 0 PCI-MSI 1572882-edge eth2-TxRx-18
81: 1 0 0 0 3663837363 0 0 0 0 0 0 0 0 0 0 0 0 0 0 38974 0 0 10726 0 PCI-MSI 1572883-edge eth2-TxRx-19
82: 1 0 0 0 0 3645849525 0 0 0 0 0 0 0 0 0 0 0 0 0 0 33666 0 0 8684 PCI-MSI 1572884-edge eth2-TxRx-20
83: 7639 0 0 0 0 0 3573581815 0 0 0 0 0 0 0 0 0 0 0 0 0 0 147770 0 0 PCI-MSI 1572885-edge eth2-TxRx-21
84: 1 8751 0 0 0 0 0 3630178014 0 0 0 0 0 0 0 0 0 0 0 0 0 0 640 0 PCI-MSI 1572886-edge eth2-TxRx-22
85: 1 0 7456 0 0 0 0 0 3689360962 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1325 PCI-MSI 1572887-edge eth2-TxRx-23
86: 5 0 0 2 0 0 0 0 0 529440 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1572888-edge eth2
88: 6377 0 0 0 11081 0 0 0 0 0 4029740654 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1574912-edge eth3-TxRx-0
89: 1 5102 0 0 0 9730 0 0 0 0 0 3984172705 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1574913-edge eth3-TxRx-1
90: 1 0 6034 0 0 0 10968 0 0 0 0 0 3716952288 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1574914-edge eth3-TxRx-2
91: 1 0 0 6351 0 0 0 11174 0 0 0 0 0 3583072151 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1574915-edge eth3-TxRx-3
92: 1 0 0 0 7674 0 0 0 14764 0 0 0 0 0 3625276429 0 0 0 0 0 0 0 0 0 PCI-MSI 1574916-edge eth3-TxRx-4
93: 1 0 0 0 0 5566 0 0 0 12282 0 0 0 0 0 3716269844 0 0 0 0 0 0 0 0 PCI-MSI 1574917-edge eth3-TxRx-5
94: 1 0 0 0 0 0 8006 0 0 0 14197 0 0 0 0 0 3671011734 0 0 0 0 0 0 0 PCI-MSI 1574918-edge eth3-TxRx-6
95: 1 0 0 0 0 0 0 6184 0 0 0 13934 0 0 0 0 0 3672346704 0 0 0 0 0 0 PCI-MSI 1574919-edge eth3-TxRx-7
96: 1 0 0 0 0 0 0 0 6954 0 0 0 15527 0 0 0 0 0 3630949275 0 0 0 0 0 PCI-MSI 1574920-edge eth3-TxRx-8
97: 1 0 0 0 0 0 0 0 0 6951 0 0 0 13983 0 0 0 0 0 3770902656 0 0 0 0 PCI-MSI 1574921-edge eth3-TxRx-9
98: 1 0 0 0 0 0 0 0 0 0 5968 0 0 0 14806 0 0 0 0 0 4027759407 0 0 0 PCI-MSI 1574922-edge eth3-TxRx-10
99: 1 0 0 0 0 0 0 0 0 0 0 6409 0 0 0 14353 0 0 0 0 0 3968031173 0 0 PCI-MSI 1574923-edge eth3-TxRx-11
100: 1 0 0 0 0 0 0 0 0 0 0 0 6138 0 0 0 14810 0 0 0 0 0 3654158637 0 PCI-MSI 1574924-edge eth3-TxRx-12
101: 1 0 0 0 0 0 0 0 0 0 0 0 0 4785 0 0 0 14832 0 0 0 0 0 3638355400 PCI-MSI 1574925-edge eth3-TxRx-13
102: 3762866907 0 0 0 0 0 0 0 0 0 0 0 0 0 6024 0 0 0 15446 0 0 0 0 0 PCI-MSI 1574926-edge eth3-TxRx-14
103: 1 3848661478 0 0 0 0 0 0 0 0 0 0 0 0 0 5405 0 0 0 13395 0 0 0 0 PCI-MSI 1574927-edge eth3-TxRx-15
104: 1 0 3585952899 0 0 0 0 0 0 0 0 0 0 0 0 0 1675 0 0 0 11718 0 0 0 PCI-MSI 1574928-edge eth3-TxRx-16
105: 1 0 0 3556259119 0 0 0 0 0 0 0 0 0 0 0 0 0 1392 0 0 0 11025 0 0 PCI-MSI 1574929-edge eth3-TxRx-17
106: 1 0 0 0 3617073506 0 0 0 0 0 0 0 0 0 0 0 0 0 1337 0 0 0 11409 0 PCI-MSI 1574930-edge eth3-TxRx-18
107: 1 0 0 0 0 3646695596 0 0 0 0 0 0 0 0 0 0 0 0 0 14832 0 0 0 9222 PCI-MSI 1574931-edge eth3-TxRx-19
108: 8420 0 0 0 0 0 3618177208 0 0 0 0 0 0 0 0 0 0 0 0 0 36750 0 0 0 PCI-MSI 1574932-edge eth3-TxRx-20
109: 1 7050 0 0 0 0 0 3603482608 0 0 0 0 0 0 0 0 0 0 0 0 0 170911 0 0 PCI-MSI 1574933-edge eth3-TxRx-21
110: 1 0 7731 0 0 0 0 0 3627596847 0 0 0 0 0 0 0 0 0 0 0 0 0 340 0 PCI-MSI 1574934-edge eth3-TxRx-22
111: 1 0 0 6564 0 0 0 0 0 3677188209 0 0 0 0 0 0 0 0 0 0 0 0 0 1217 PCI-MSI 1574935-edge eth3-TxRx-23
112: 5 0 0 0 2 0 0 0 0 0 529115 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1574936-edge eth3
NMI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Non-maskable interrupts
LOC: 3437727303 3330712413 2977312369 2551350781 2432292316 2582526537 2251387175 2411670152 2309463943 2303713819 2546077042 2525041842 2585177601 2627796384 2694235576 2762312235 2731795624 2667247519 2307806843 2402825197 2477180949 2424008647 2336282244 2321585242 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Performance monitoring interrupts
IWI: 1 0 3 2 1 1 0 0 0 0 0 0 0 2 7 2 1 3 0 0 0 0 0 0 IRQ work interrupts
RTR: 23 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 APIC ICR read retries
RES: 2212937038 1642191915 1452636061 1371660034 1190185594 1011809552 827170255 722580577 702988819 695910979 687471370 690619702 681430587 688903006 717509823 714058078 697796966 692369925 625335386 590544007 598884893 604242597 607774517 621117270 Rescheduling interrupts
CAL: 493960896 464297756 432423735 412337257 403711257 402674547 867316133 867612442 861612014 873878530 889983097 897813278 405799237 407549084 406932642 538843361 540174492 539953297 754770256 759485923 763979229 763600439 762040453 770806887 Function call interrupts
TLB: 490210373 460481040 428588194 408482700 399848016 398808815 409593187 405037451 403395744 408779560 413254707 406996454 402234415 403760047 403136187 401138446 397902842 395778629 411112617 414995888 416864210 416288621 415774753 415353883 TLB shootdowns
TRM: 6781209 6781168 6781137 6781153 6781154 6781159 0 0 0 0 0 0 6781120 6781137 6781167 6781158 6781177 6781153 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 Machine check polls
ERR: 0
MIS: 0
PIN: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Posted-interrupt notification event
PIW: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Posted-interrupt wakeup event
cat /proc/stat
cpu 1642623836 0 1161743037 17528736506 8247617 0 2346885347 0 0 0
cpu0 94717962 0 59878881 684159368 773823 0 104668306 0 0 0
cpu1 77192000 0 58178703 720349957 984970 0 96235824 0 0 0
cpu2 64666962 0 53893433 741346482 447582 0 90271194 0 0 0
cpu3 61719977 0 44870534 751885468 331111 0 87590880 0 0 0
cpu4 60235918 0 43305683 751001386 274678 0 89234030 0 0 0
cpu5 58684246 0 46152476 743788939 259702 0 95286429 0 0 0
cpu6 67458963 0 50971227 721695927 276535 0 100888925 0 0 0
cpu7 63584778 0 53368044 734001270 522956 0 94228015 0 0 0
cpu8 63696248 0 51322376 737302935 570726 0 92255661 0 0 0
cpu9 69480131 0 46930859 728500250 397451 0 98227366 0 0 0
cpu10 71986551 0 47254607 713878646 342846 0 110238881 0 0 0
cpu11 66850174 0 46714066 712922297 320522 0 115363758 0 0 0
cpu12 63251094 0 46681499 736904762 162104 0 98387873 0 0 0
cpu13 64781245 0 47304696 746006434 193350 0 89728690 0 0 0
cpu14 65408320 0 47029806 746548143 195393 0 89323975 0 0 0
cpu15 64992093 0 45163902 746075472 259086 0 91801401 0 0 0
cpu16 63638429 0 43325354 744783171 268564 0 94862592 0 0 0
cpu17 62130432 0 42487288 740678188 259419 0 99424750 0 0 0
cpu18 69857890 0 49713152 718045666 247428 0 103966563 0 0 0
cpu19 72695931 0 49027377 725770695 248585 0 97178597 0 0 0
cpu20 74693525 0 48304283 723885172 238587 0 98338712 0 0 0
cpu21 74294012 0 46664518 723250524 226819 0 100369408 0 0 0
cpu22 73631859 0 46518216 719975482 223086 0 102816004 0 0 0
cpu23 72975087 0 46682047 715979862 222282 0 106197504 0 0 0
intr 178446540687 147 0 0 0 0 0 0 0 1 3 0 0 0 0 0 0 4185951469 0 61 40 0 0 0 0 0 0 0 0 0 0 0 0 0 616834582 882246562 330199886 378115020 721524993 413996215 750429347 624848054 0 0 178 178 178 178 178 178 178 178 0 0 178 178 178 178 178 178 178 178 0 4051301056 4006538163 3663326199 3555762036 3625039830 3725891576 3677141537 3688185178 3658740909 3749507982 4028268795 3979028472 3676378884 3660129539 3709988565 3837470381 3592104727 3560975011 3623285434 3663900101 3645904763 3573750143 3630200721 3689382626 529447 0 4029771491 3984200769 3716982258 3583102455 3625311608 3716300637 3671046794 3672380008 3630984596 3770936340 4027793470 3968065211 3654192834 3638387918 3762901374 3848693526 3585979160 3556284314 3617099002 3646732505 3618235214 3603673594 3627618130 3677208752 529123 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 429803998363
btime 1578681262
processes 161558166
procs_running 3
procs_blocked 0
softirq 271394548400 0 2366235965 1288699814 770066312 0 0 3918293385 3888487167 0 1464727997
cat /proc/softirqs
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23
HI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
TIMER: 933586917 823590426 507867089 161302411 59017710 162014787 196517951 211938185 153098429 149596741 293490074 265684574 244802067 284657114 294428597 310504296 255164775 200976561 242326731 189996044 254023345 224350456 139275764 103176198
NET_TX: 54170373 53521934 52025338 50645907 51469475 54099985 52940117 51993638 50193980 55139997 60723451 61571774 53424830 51785320 50664632 51127022 52046572 54294612 53592964 52346601 51362965 53938204 56968447 58653567
NET_RX: 3493142749 3371825040 3073809598 3035412656 3110806639 3128123114 2955286117 3077052843 3152845861 3516306059 3801103601 3382823596 3057119248 3038990124 3190263113 3233872590 3187665447 3129958884 3142535886 3596783696 3807128200 3430083197 3077271927 3090829289
BLOCK: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
IRQ_POLL: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
TASKLET: 2200925591 2287368619 2225930422 2198851099 2234679633 2131064128 2036104163 2098912082 2192820421 2248119880 2215882732 1998554084 2092692988 2194575001 2280333716 2212949513 2071721063 1953539906 2039157977 2247942351 2245037368 1993792580 1855902592 1906561009
SCHED: 3664886555 3304819803 3056666037 2836269462 2630915740 2532209441 2410216102 2400999841 2340376365 2309620186 2362048798 2341282317 2336910805 2334840875 2346120707 2352738028 2329119313 2305424145 2232269946 2263635446 2288125376 2270863351 2240828138 2231974138
HRTIMER: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
RCU: 3336441212 3328017338 3208393067 3066621364 3012848018 3075986694 3017948160 3130675033 3102998125 3111122282 3159131790 3150519818 3147869208 3185666186 3193132118 3197905031 3167232257 3139484797 3105825112 3168421618 3197127701 3174040728 1979226060 3122673183
正常服务器的中断使用情况：cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23
0: 95 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC 2-edge timer
8: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC 8-edge rtc0
9: 9771501 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC 9-fasteoi acpi
16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC 16-fasteoi ahci[0000:00:11.4], ahci[0000:00:1f.2]
18: 134 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC 18-fasteoi ehci_hcd:usb1, ehci_hcd:usb2
54: 3479 1136291758 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 524288-edge mpt3sas0-msix0
55: 0 3127 934705390 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 524289-edge mpt3sas0-msix1
56: 0 0 1718 947829505 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 524290-edge mpt3sas0-msix2
57: 0 0 0 1728 892855598 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 524291-edge mpt3sas0-msix3
58: 0 0 0 0 2857 865359037 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 524292-edge mpt3sas0-msix4
59: 0 0 0 0 0 3466 373629218 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 524293-edge mpt3sas0-msix5
60: 0 0 0 0 0 0 1915 770639585 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 524294-edge mpt3sas0-msix6
61: 0 0 0 0 0 0 0 4750 629939555 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 524295-edge mpt3sas0-msix7
62: 0 0 0 0 0 0 0 0 1848 652043791 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 524296-edge mpt3sas0-msix8
63: 0 0 0 0 0 0 0 0 0 459 636908649 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 524297-edge mpt3sas0-msix9
64: 0 0 0 0 0 0 0 0 0 0 1088 635073178 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 524298-edge mpt3sas0-msix10
65: 0 0 0 0 0 0 0 0 0 0 0 1973 280062460 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 524299-edge mpt3sas0-msix11
66: 0 0 0 0 0 0 0 0 0 0 0 0 2414 1077444987 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 524300-edge mpt3sas0-msix12
67: 0 0 0 0 0 0 0 0 0 0 0 0 0 1015 859212748 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 524301-edge mpt3sas0-msix13
68: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4520 879413695 0 0 0 0 0 0 0 0 IR-PCI-MSI 524302-edge mpt3sas0-msix14
69: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 582 850659418 0 0 0 0 0 0 0 IR-PCI-MSI 524303-edge mpt3sas0-msix15
70: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1283 841547352 0 0 0 0 0 0 IR-PCI-MSI 524304-edge mpt3sas0-msix16
71: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 831 365500787 0 0 0 0 0 IR-PCI-MSI 524305-edge mpt3sas0-msix17
72: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 DMAR-MSI 0-edge dmar0
73: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 DMAR-MSI 1-edge dmar1
74: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1199 824084145 0 0 0 0 IR-PCI-MSI 524306-edge mpt3sas0-msix18
75: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1388 688546000 0 0 0 IR-PCI-MSI 524307-edge mpt3sas0-msix19
76: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2010 731191197 0 0 IR-PCI-MSI 524308-edge mpt3sas0-msix20
77: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3566 720110281 0 IR-PCI-MSI 524309-edge mpt3sas0-msix21
78: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1107 720623692 IR-PCI-MSI 524310-edge mpt3sas0-msix22
79: 283770408 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9263 IR-PCI-MSI 524311-edge mpt3sas0-msix23
82: 134 0 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5242881-edge
83: 134 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5242882-edge
84: 134 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5242883-edge
85: 134 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5242884-edge
86: 134 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5242885-edge
87: 134 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5242886-edge
88: 134 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5242887-edge
89: 134 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5242888-edge
92: 134 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5244929-edge
93: 134 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5244930-edge
94: 134 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5244931-edge
95: 134 0 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5244932-edge
96: 134 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 IR-PCI-MSI 5244933-edge
97: 134 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 IR-PCI-MSI 5244934-edge
98: 134 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 IR-PCI-MSI 5244935-edge
99: 134 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 IR-PCI-MSI 5244936-edge
100: 6773 3670352419 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16860 0 0 0 0 IR-PCI-MSI 2621440-edge eth2-TxRx-0
101: 0 131312 3671377758 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16331 0 0 0 IR-PCI-MSI 2621441-edge eth2-TxRx-1
102: 0 0 31168 3857854613 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16806 0 0 IR-PCI-MSI 2621442-edge eth2-TxRx-2
103: 0 0 0 9683 3845422064 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15013 0 IR-PCI-MSI 2621443-edge eth2-TxRx-3
104: 0 0 0 0 5766 3698141622 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15223 IR-PCI-MSI 2621444-edge eth2-TxRx-4
105: 14982 0 0 0 0 6313 3626748875 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2621445-edge eth2-TxRx-5
106: 0 15727 0 0 0 0 31916 3991209838 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2621446-edge eth2-TxRx-6
107: 0 0 13937 0 0 0 0 15640 3905858625 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2621447-edge eth2-TxRx-7
108: 0 0 0 13681 0 0 0 0 9179 3944701613 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2621448-edge eth2-TxRx-8
109: 0 0 0 0 15283 0 0 0 0 10808 3941987973 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2621449-edge eth2-TxRx-9
110: 0 0 0 0 0 15101 0 0 0 0 9500 3871809467 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2621450-edge eth2-TxRx-10
111: 0 0 0 0 0 0 11752 0 0 0 0 9602 3879754085 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2621451-edge eth2-TxRx-11
112: 0 0 0 0 0 0 0 9609 0 0 0 0 21203 4014698079 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2621452-edge eth2-TxRx-12
113: 0 0 0 0 0 0 0 0 11179 0 0 0 0 12436 3996040824 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2621453-edge eth2-TxRx-13
114: 0 0 0 0 0 0 0 0 0 12129 0 0 0 0 10640 4033936791 0 0 0 0 0 0 0 0 IR-PCI-MSI 2621454-edge eth2-TxRx-14
115: 0 0 0 0 0 0 0 0 0 0 11631 0 0 0 0 22585 4021375393 0 0 0 0 0 0 0 IR-PCI-MSI 2621455-edge eth2-TxRx-15
116: 0 0 0 0 0 0 0 0 0 0 0 7347 0 0 0 0 85079 3636482624 0 0 0 0 0 0 IR-PCI-MSI 2621456-edge eth2-TxRx-16
117: 0 0 0 0 0 0 0 0 0 0 0 0 7680 0 0 0 0 2191 3393256853 0 0 0 0 0 IR-PCI-MSI 2621457-edge eth2-TxRx-17
118: 0 0 0 0 0 0 0 0 0 0 0 0 0 7391 0 0 0 0 1670 3524848310 0 0 0 0 IR-PCI-MSI 2621458-edge eth2-TxRx-18
119: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9771 0 0 0 0 1990 3337289559 0 0 0 IR-PCI-MSI 2621459-edge eth2-TxRx-19
120: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 12249 0 0 0 0 2637 3271508797 0 0 IR-PCI-MSI 2621460-edge eth2-TxRx-20
121: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 12539 0 0 0 0 1489 3184015563 0 IR-PCI-MSI 2621461-edge eth2-TxRx-21
122: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11701 0 0 0 0 3882 3040271774 IR-PCI-MSI 2621462-edge eth2-TxRx-22
123: 3001179314 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11302 0 0 0 0 760 IR-PCI-MSI 2621463-edge eth2-TxRx-23
124: 4 525381 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 IR-PCI-MSI 2621464-edge eth2
126: 7793 0 3663221269 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 22675 0 0 0 IR-PCI-MSI 2623488-edge eth3-TxRx-0
127: 0 48562 0 3653392425 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15473 0 0 IR-PCI-MSI 2623489-edge eth3-TxRx-1
128: 0 0 17607 0 3840745431 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16286 0 IR-PCI-MSI 2623490-edge eth3-TxRx-2
129: 0 0 0 7015 0 3824050932 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15320 IR-PCI-MSI 2623491-edge eth3-TxRx-3
130: 15199 0 0 0 3683 0 3643731565 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2623492-edge eth3-TxRx-4
131: 0 15721 0 0 0 4409 0 3647614440 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2623493-edge eth3-TxRx-5
132: 0 0 16483 0 0 0 34455 0 3977971405 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2623494-edge eth3-TxRx-6
133: 0 0 0 14235 0 0 0 14334 0 3897467707 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2623495-edge eth3-TxRx-7
134: 0 0 0 0 14589 0 0 0 5436 0 3935317108 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2623496-edge eth3-TxRx-8
135: 0 0 0 0 0 15315 0 0 0 7309 0 3919048480 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2623497-edge eth3-TxRx-9
136: 0 0 0 0 0 0 15341 0 0 0 6675 0 3855236657 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2623498-edge eth3-TxRx-10
137: 0 0 0 0 0 0 0 11611 0 0 0 6827 0 3883364729 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2623499-edge eth3-TxRx-11
138: 0 0 0 0 0 0 0 0 9637 0 0 0 12925 0 4007136271 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2623500-edge eth3-TxRx-12
139: 0 0 0 0 0 0 0 0 0 10164 0 0 0 7409 0 3983955487 0 0 0 0 0 0 0 0 IR-PCI-MSI 2623501-edge eth3-TxRx-13
140: 0 0 0 0 0 0 0 0 0 0 12053 0 0 0 6317 0 4019389955 0 0 0 0 0 0 0 IR-PCI-MSI 2623502-edge eth3-TxRx-14
141: 0 0 0 0 0 0 0 0 0 0 0 10663 0 0 0 13937 0 3998640903 0 0 0 0 0 0 IR-PCI-MSI 2623503-edge eth3-TxRx-15
142: 0 0 0 0 0 0 0 0 0 0 0 0 7674 0 0 0 29714 0 3575162085 0 0 0 0 0 IR-PCI-MSI 2623504-edge eth3-TxRx-16
143: 0 0 0 0 0 0 0 0 0 0 0 0 0 8440 0 0 0 1367 0 3420142317 0 0 0 0 IR-PCI-MSI 2623505-edge eth3-TxRx-17
144: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8067 0 0 0 1010 0 3517642983 0 0 0 IR-PCI-MSI 2623506-edge eth3-TxRx-18
145: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9275 0 0 0 1525 0 3329174581 0 0 IR-PCI-MSI 2623507-edge eth3-TxRx-19
146: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 12400 0 0 0 3287 0 3256818247 0 IR-PCI-MSI 2623508-edge eth3-TxRx-20
147: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11864 0 0 0 882 0 3165745163 IR-PCI-MSI 2623509-edge eth3-TxRx-21
148: 3031718696 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11939 0 0 0 2441 0 IR-PCI-MSI 2623510-edge eth3-TxRx-22
149: 0 3009414283 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11021 0 0 0 712 IR-PCI-MSI 2623511-edge eth3-TxRx-23
150: 4 0 525971 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 IR-PCI-MSI 2623512-edge eth3
NMI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Non-maskable interrupts
LOC: 2752451343 2531115056 2677952583 2703073595 2733841969 2709280072 2445042996 2458343653 2500534984 2492974625 2487960572 2449128378 2610995930 2652983076 2719103349 2727741801 2742060726 2574848940 2015135035 1929970714 1906893565 1859470197 1849861153 1821709031 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IRQ work interrupts
RTR: 23 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 APIC ICR read retries
RES: 1428456762 946043812 560802855 379923470 272624658 209228846 193736165 138873235 110436813 90825656 79168459 66362524 62198722 53666435 48414342 44994616 41960395 42387993 40222148 39128201 34183205 34608178 31209382 30220377 Rescheduling interrupts
CAL: 331383383 291234017 289072423 289684063 294933786 640595300 316124114 300804025 300614993 299476294 301917266 562373770 295772733 289363303 292914064 293295201 297336973 631877571 289750249 279126099 276451372 274969331 276341337 537404029 Function call interrupts
TLB: 324950485 284973903 281990180 281648847 283905615 282867750 310629076 295387659 294709276 293092048 293370675 291779315 289525742 283645701 286174414 285894940 286928796 280806080 284144334 273814436 270278544 267888338 266773004 264389432 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 29884 Machine check polls
ERR: 0
MIS: 0
PIN: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Posted-interrupt notification event
PIW: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Posted-interrupt wakeup event
cpu 1334913711 0 855474006 18797777478 2631813 0 1849437810 0 0 0
cpu0 66472980 0 39817132 765629107 102745 0 79223920 0 0 0
cpu1 51178815 0 35295133 791955051 103861 0 73295429 0 0 0
cpu2 54415085 0 35129469 791318921 102334 0 72999895 0 0 0
cpu3 56489569 0 35095664 789434900 109391 0 73470221 0 0 0
cpu4 58052946 0 36453251 784607273 114279 0 75413702 0 0 0
cpu5 56922031 0 36602272 780097319 204971 0 79612271 0 0 0
cpu6 65999728 0 37556484 751990561 73094 0 93249368 0 0 0
cpu7 61488351 0 35815275 772922711 77393 0 81791985 0 0 0
cpu8 62528376 0 36066991 777391307 88116 0 77412355 0 0 0
cpu9 62467569 0 36021253 778019218 89691 0 76937034 0 0 0
cpu10 62458452 0 37096446 776049709 97523 0 77580402 0 0 0
cpu11 61011447 0 37244443 772323612 166613 0 81066230 0 0 0
cpu12 59519072 0 34955698 774322977 85931 0 82572523 0 0 0
cpu13 58255512 0 34613555 783591544 89450 0 76733413 0 0 0
cpu14 59230538 0 35208836 785824181 94904 0 74450996 0 0 0
cpu15 59073731 0 35180344 785898573 100568 0 74583214 0 0 0
cpu16 59046898 0 36357640 782989104 106123 0 76204534 0 0 0
cpu17 53612465 0 36098789 783776332 196557 0 78472835 0 0 0
cpu18 52293873 0 34791673 769863720 72065 0 87433581 0 0 0
cpu19 45839924 0 33768498 792438277 81296 0 75504636 0 0 0
cpu20 43394533 0 33639858 801375187 90454 0 70436705 0 0 0
cpu21 42437435 0 33427002 803734645 93848 0 69295812 0 0 0
cpu22 41905535 0 34514412 803020269 105541 0 69286221 0 0 0
cpu23 40818836 0 34723875 799202971 185056 0 72410516 0 0 0
intr 145353125723 95 0 0 0 0 0 0 0 1 9771506 0 0 0 0 0 0 0 0 134 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1136295782 934708890 947831568 892857680 865362387 373632797 770642016 629944713 652045992 636909469 635074675 280064609 1077447784 859214096 879418590 850660351 841548991 365501733 0 0 824085837 688547837 731193606 720114219 720625087 283779797 0 0 154 154 154 154 154 154 154 154 0 0 154 154 154 154 154 154 154 154 3670387597 3671536962 3857914411 3845458535 3698174310 3626781943 3991269227 3905899882 3944736298 3942025915 3871845700 3879786877 4014740680 3996076344 4033971429 4021421495 3636586686 3393278400 3524868906 3337312604 3271535168 3184040868 3040298686 3001202407 525387 0 3663263259 3653467963 3840791180 3824084994 3643762091 3647646372 3978034070 3897507929 3935348912 3919083007 3855270385 3883394821 4007170400 3983984950 4019420002 3998677612 3575210949 3420163662 3517663713 3329196711 3256845325 3165769156 3031744557 3009437158 525977 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 476400246010
btime 1578681303
processes 164574551
procs_running 2
procs_blocked 0
softirq 299088869740 1 3373306296 1305697733 162793096 0 0 3264051591 2751270008 0 468942183
cat /proc/stat
cpu 1334913711 0 855474006 18797777478 2631813 0 1849437810 0 0 0
cpu0 66472980 0 39817132 765629107 102745 0 79223920 0 0 0
cpu1 51178815 0 35295133 791955051 103861 0 73295429 0 0 0
cpu2 54415085 0 35129469 791318921 102334 0 72999895 0 0 0
cpu3 56489569 0 35095664 789434900 109391 0 73470221 0 0 0
cpu4 58052946 0 36453251 784607273 114279 0 75413702 0 0 0
cpu5 56922031 0 36602272 780097319 204971 0 79612271 0 0 0
cpu6 65999728 0 37556484 751990561 73094 0 93249368 0 0 0
cpu7 61488351 0 35815275 772922711 77393 0 81791985 0 0 0
cpu8 62528376 0 36066991 777391307 88116 0 77412355 0 0 0
cpu9 62467569 0 36021253 778019218 89691 0 76937034 0 0 0
cpu10 62458452 0 37096446 776049709 97523 0 77580402 0 0 0
cpu11 61011447 0 37244443 772323612 166613 0 81066230 0 0 0
cpu12 59519072 0 34955698 774322977 85931 0 82572523 0 0 0
cpu13 58255512 0 34613555 783591544 89450 0 76733413 0 0 0
cpu14 59230538 0 35208836 785824181 94904 0 74450996 0 0 0
cpu15 59073731 0 35180344 785898573 100568 0 74583214 0 0 0
cpu16 59046898 0 36357640 782989104 106123 0 76204534 0 0 0
cpu17 53612465 0 36098789 783776332 196557 0 78472835 0 0 0
cpu18 52293873 0 34791673 769863720 72065 0 87433581 0 0 0
cpu19 45839924 0 33768498 792438277 81296 0 75504636 0 0 0
cpu20 43394533 0 33639858 801375187 90454 0 70436705 0 0 0
cpu21 42437435 0 33427002 803734645 93848 0 69295812 0 0 0
cpu22 41905535 0 34514412 803020269 105541 0 69286221 0 0 0
cpu23 40818836 0 34723875 799202971 185056 0 72410516 0 0 0
intr 145353125723 95 0 0 0 0 0 0 0 1 9771506 0 0 0 0 0 0 0 0 134 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1136295782 934708890 947831568 892857680 865362387 373632797 770642016 629944713 652045992 636909469 635074675 280064609 1077447784 859214096 879418590 850660351 841548991 365501733 0 0 824085837 688547837 731193606 720114219 720625087 283779797 0 0 154 154 154 154 154 154 154 154 0 0 154 154 154 154 154 154 154 154 3670387597 3671536962 3857914411 3845458535 3698174310 3626781943 3991269227 3905899882 3944736298 3942025915 3871845700 3879786877 4014740680 3996076344 4033971429 4021421495 3636586686 3393278400 3524868906 3337312604 3271535168 3184040868 3040298686 3001202407 525387 0 3663263259 3653467963 3840791180 3824084994 3643762091 3647646372 3978034070 3897507929 3935348912 3919083007 3855270385 3883394821 4007170400 3983984950 4019420002 3998677612 3575210949 3420163662 3517663713 3329196711 3256845325 3165769156 3031744557 3009437158 525977 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 476400246010
btime 1578681303
processes 164574551
procs_running 2
procs_blocked 0
softirq 299088869740 1 3373306296 1305697733 162793096 0 0 3264051591 2751270008 0 468942183
cat /proc/softirqs
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23
HI: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
TIMER: 420334007 153114972 276956181 277234858 308128410 262752629 274364864 175987029 218185904 184861673 184654447 146591304 276956736 247313034 320893228 315986436 325586890 169859626 4248817473 4102888563 4107942367 4054784510 4056552699 4032446282
NET_TX: 52859472 53441399 54325291 53381230 54208847 55914655 59749349 55196484 52652296 53126096 53769813 55612969 56064776 53629394 52860066 53464983 54390687 56171150 59932244 55300691 52243406 52199344 52134253 53069640
NET_RX: 1925930128 2576564734 3243529390 3419301755 3593487679 3416221978 3090008560 3499482339 3764180831 3726465731 3759809603 3660682938 3601659963 3794178579 3913655162 3926266205 3946915323 3521648712 2771816080 2789463729 2715282198 2460572867 2300171848 2055609045
BLOCK: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
IRQ_POLL: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
TASKLET: 1742798234 2034796667 2247595198 2239905136 2313495222 2205421009 1788029026 2233805938 2423788180 2407943279 2457952205 2408796520 2320409315 2525084146 2656183825 2662856137 2689342390 2567607484 2023012851 2312976280 2337381634 2182552613 2094090084 1928055077
SCHED: 3723906457 3171973605 2850564938 2680864969 2589463594 2515871656 2463414472 2409634114 2393502365 2368503094 2350838187 2324702798 2355202082 2344268270 2363639605 2362707925 2363317097 2297532398 2157887551 2123345940 2112058152 2093346000 2090189740 2079155373
HRTIMER: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
RCU: 3268553842 3209744695 3275195818 3311442497 3321976912 3301516157 3168234526 3259325185 3287192874 3294129827 3281446837 3260122835 3264595114 3317217063 3377922559 3365205232 3379218185 3285137947 3078246110 3109261997 3112126240 3101905332 3087242903 3061458028
通过对比可以看出有问题的服务器处理软中断的数目甚至是少于正常服务器的，但是它的si占用却高，根据之前的一些经验，估计是CPU性能有问题了。怎么去实锤，我们再用iostat命令看下I/O情况。有问题的服务器：
avg-cpu: %user %nice %system %iowait %steal %idle
7.30 0.00 15.03 0.00 0.00 77.68
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdc 0.00 24.00 39.00 30.00 3120.00 3888.00 203.13 0.10 1.48 1.03 7.10
sdh 0.00 6.00 94.00 11.00 7520.00 1000.00 162.29 0.93 8.84 3.34 35.10
sde 0.00 16.00 92.00 18.00 7360.00 2432.00 178.04 1.61 14.89 5.25 57.80
sdb 0.00 0.00 18.00 6.00 1440.00 408.00 154.00 0.07 2.71 2.33 5.60
sdd 0.00 10.00 58.00 15.00 4640.00 1640.00 172.05 0.10 1.19 1.23 9.00
sdf 0.00 44.00 6.00 48.00 480.00 6784.00 269.04 0.14 2.54 1.85 10.00
sda 0.00 10.00 38.00 20.00 3040.00 1680.00 162.76 0.17 2.88 2.38 13.80
sdg 0.00 38.00 30.00 43.00 2400.00 5976.00 229.48 0.14 2.15 1.48 10.80
sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdj 0.00 0.00 280.00 66.00 22400.00 5280.00 160.00 0.18 0.53 0.48 16.60
sdk 0.00 0.00 277.00 43.00 22160.00 3440.00 160.00 0.18 0.57 0.51 16.20
sdl 0.00 0.00 271.00 53.00 21680.00 4240.00 160.00 0.18 0.56 0.50 16.30
sdm 0.00 33.00 67.00 40.00 5360.00 5336.00 199.93 0.38 3.52 2.01 21.50
sdn 0.00 8.00 39.00 12.00 3120.00 1312.00 173.80 0.12 2.35 2.10 10.70
sdo 0.00 14.00 60.00 18.00 4800.00 2272.00 181.33 0.59 7.53 3.26 25.40
sdp 0.00 141.00 42.00 160.00 3360.00 23032.00 261.31 0.38 1.82 1.00 20.10
avg-cpu: %user %nice %system %iowait %steal %idle
7.00 0.00 16.73 0.04 0.00 76.22
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdc 0.00 8.00 56.00 9.00 4480.00 1288.00 177.48 0.32 4.91 3.51 22.80
sdh 0.00 8.00 48.00 15.00 3840.00 1480.00 168.89 0.31 4.95 3.02 19.00
sde 0.00 15.00 38.00 11.00 3040.00 1792.00 197.22 0.14 2.90 2.27 11.10
sdb 0.00 0.00 21.00 10.00 1680.00 368.00 132.13 0.10 3.29 2.90 9.00
sdd 0.00 17.00 24.00 19.00 1920.00 2448.00 203.16 0.19 4.72 3.47 14.90
sdf 0.00 9.00 24.00 10.00 1920.00 1304.00 189.65 0.06 1.85 1.65 5.60
sda 0.00 46.00 78.00 62.00 6240.00 7632.00 198.17 1.13 8.16 3.20 44.80
sdg 0.00 8.00 74.00 15.00 5920.00 1480.00 166.29 0.10 1.18 1.06 9.40
sdi 0.00 0.00 0.00 6.00 0.00 160.00 53.33 0.00 0.00 0.00 0.00
sdj 0.00 0.00 323.00 72.00 25840.00 5760.00 160.00 0.22 0.55 0.49 19.20
sdk 0.00 212.00 324.00 315.00 25920.00 9400.00 110.55 0.49 0.77 0.32 20.20
sdl 0.00 0.00 324.00 67.00 25920.00 5360.00 160.00 0.22 0.56 0.51 20.10
sdm 0.00 3.00 5.00 4.00 400.00 488.00 197.33 0.04 4.44 4.22 3.80
sdn 0.00 2.00 17.00 4.00 1360.00 336.00 161.52 0.14 6.71 5.29 11.10
sdo 0.00 1.00 48.00 1.00 3840.00 160.00 163.27 0.16 3.27 2.92 14.30
sdp 0.00 40.00 73.00 43.00 5840.00 6280.00 208.97 0.30 2.66 2.01 23.30
正常的服务器：
avg-cpu: %user %nice %system %iowait %steal %idle
5.81 0.00 12.10 0.00 0.00 82.09
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 2.00 0.00 84.00 84.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 301.00 41.00 24080.00 3280.00 160.00 0.22 0.63 0.55 18.70
sdc 0.00 0.00 282.00 49.00 22560.00 3920.00 160.00 0.19 0.58 0.52 17.10
sdd 0.00 0.00 299.00 40.00 23920.00 3200.00 160.00 0.19 0.56 0.49 16.60
sdl 0.00 1.00 66.00 2.00 5280.00 168.00 160.24 0.14 2.10 2.09 14.20
sdp 0.00 27.00 45.00 34.00 3600.00 4304.00 200.10 0.27 3.42 2.39 18.90
sdk 0.00 1.00 17.00 2.00 1360.00 168.00 160.84 0.09 4.95 4.37 8.30
sdi 0.00 7.00 39.00 12.00 3120.00 1304.00 173.49 0.14 2.78 2.14 10.90
sdj 0.00 22.00 15.00 32.00 1200.00 3600.00 204.26 0.17 3.70 1.43 6.70
sdf 0.00 15.00 16.00 25.00 1280.00 2480.00 183.41 0.15 3.76 1.93 7.90
sdn 0.00 16.00 26.00 22.00 2080.00 2608.00 195.33 0.18 3.77 2.21 10.60
sde 0.00 30.00 81.00 35.00 6480.00 4984.00 197.66 0.24 2.09 1.33 15.40
sdm 0.00 72.00 31.00 82.00 2480.00 11744.00 251.75 0.30 2.73 1.21 13.70
sdg 0.00 8.00 48.00 13.00 3840.00 1320.00 169.18 0.18 3.02 2.38 14.50
sdh 0.00 6.00 31.00 9.00 2480.00 984.00 173.20 0.11 2.70 2.20 8.80
sdo 0.00 40.00 127.00 47.00 10160.00 6456.00 190.99 0.84 4.99 2.49 43.30
avg-cpu: %user %nice %system %iowait %steal %idle
6.20 0.00 12.65 0.00 0.00 81.15
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 4.00 0.00 80.00 40.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 276.00 59.00 22080.00 4720.00 160.00 0.20 0.58 0.52 17.50
sdc 0.00 0.00 282.00 76.00 22560.00 6080.00 160.00 0.22 0.63 0.54 19.20
sdd 0.00 0.00 284.00 56.00 22720.00 4480.00 160.00 0.21 0.62 0.55 18.80
sdl 0.00 1.00 46.00 2.00 3680.00 168.00 160.33 0.11 2.19 2.17 10.40
sdp 0.00 1.00 60.00 1.00 4800.00 160.00 162.62 0.14 2.23 2.23 13.60
sdk 0.00 1.00 6.00 1.00 480.00 160.00 182.86 0.07 9.86 9.86 6.90
sdi 0.00 5.00 56.00 7.00 4480.00 816.00 168.13 0.34 5.37 3.59 22.60
sdj 0.00 16.00 29.00 27.00 2320.00 2792.00 182.57 0.16 2.84 1.73 9.70
sdf 0.00 1.00 50.00 1.00 4000.00 160.00 163.14 0.10 1.94 1.94 9.90
sdn 0.00 71.00 49.00 82.00 3920.00 11304.00 232.43 0.53 4.01 2.01 26.30
sde 0.00 27.00 42.00 26.00 3360.00 4024.00 217.18 0.20 2.94 1.87 12.70
sdm 0.00 14.00 48.00 18.00 3840.00 2272.00 185.21 0.20 2.97 2.02 13.30
sdg 0.00 1.00 43.00 1.00 3440.00 160.00 163.64 0.11 2.55 2.55 11.20
sdh 0.00 26.00 35.00 34.00 2800.00 4224.00 203.59 0.29 4.04 3.00 20.70
sdo 0.00 7.00 200.00 12.00 16000.00 1304.00 163.25 0.93 4.43 2.38 50.40
可以看到磁盘I/O没什么问题，有问题的服务器user和system的CPU使用率高。前面还有个网卡丢包，关于网卡数据包的接收和处理，之前有同学分享过：
1.驱动加载和初始化
2.数据包到达网卡控制器(NIC)
3.数据包被复制到内核空间( DMA-> ring buffer )
4.产生硬件中断，通知系统数据可读
5.驱动调用NAPI激活poll循环(如果该循环处于休眠状态)
6. ksoftirpd 调用驱动注册的poll函数，读取ring buffer中的数据包
7. ring buffer 对应的内存区域被解除映射( memory region unmapped )
8.数据包被封装为 skb 结构体，准备传递到上层协议栈
9. 如果开启网卡多队列，数据帧会被负载均衡到多个CPU进行处理
10.数据帧经由队列，递交上层协议栈
网卡一般可以自行通过 DMA 将网络包传送到内核空间的固定内存中（ring buffer），但是在数据传送完成之后还是需要通过发起中断通知 CPU，让其将 ring buffer 中的数据取走，比如移动到网络栈的更上层进行下一步处理。但是一旦 CPU 处理太慢，就可能导致 buffer 满，从而发生丢包。这个时候可以调整ring buffer大一些（付出的代价是包的处理速度会变慢），如果调 ring buffer 没用，因为在进入 ring buffer 前就丢了，这样原因就复杂了，有可能是硬件中断问题或 PCIE 带宽不够或者网卡，模块，光纤，交换机都有可能！先做能做的吧，用ethtool -S eth2|grep errors来查看下： 有问题的服务器：
rx_errors: 0
tx_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 217401
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_csum_offload_errors: 0
正常的服务器：
rx_errors: 0
tx_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_csum_offload_errors: 0
如果看到了不少 rx_no_buffer_count，则表明需要增加 ring buffer；而如果看到 tx_deferred_ok 表示传输被延迟，不算 errors； rx_missed_errors, rx_fw_discards 这个表示包满了，多出的全被 drop 掉了。
接着用一个运算脚本对CPU进行下简单的压测：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
|
#!/usr/bin/env python#-*- coding:utf8 -*-import osimport timeimport affinityimport multiprocessingdef bench_cpu(idx): print 'benchmark cpu%d' % idx pid = os.getpid() affinity.set_process_affinity_mask(pid, 2**idx) res = 0 start = time.time() for idx in xrange(10000000): res = res + idx end = time.time() return (end-start)def main(): result_list = [] for idx in range(multiprocessing.cpu_count()): spent_time = bench_cpu(idx) result_list.append((idx, spent_time)) print print 'benchmark over.' print for result in result_list: idx, spent_time = result print 'cpu%d: %f' % (idx, spent_time)if __name__ == '__main__': main() |
结果也发现有问题的服务器的确是耗时较长，这样就排除了很多可能，剩下就是硬中断或者CPU性能问题。从上面的采样数据来看，不论是us还是sy，CPU都要花费比较长的时间来处理，先更换CPU来试试吧（其实这个问题可以进一步写测试程序来挖，但是不懂C语言，就先用替换硬件的方法）。
后续：
这个集群的服务器应用主要是两个，一个是使用内存和CPU的，一个是使用硬盘的。所以我们把使用硬盘的程序都停掉，接着把nginx的程序也停掉，观察他们的CPU使用情况，结果如下（红色线为之前有问题的，蓝色为之前正常的）。
此时在用脚本测试下CPU的计算耗时如下。
之前有问题的：
cpu0: 0.453358
cpu1: 0.455238
cpu2: 0.452720
cpu3: 0.455165
cpu4: 0.453572
cpu5: 0.458413
cpu6: 0.458023
cpu7: 0.455635
cpu8: 0.454871
cpu9: 0.455738
cpu10: 0.454763
cpu11: 0.454787
cpu12: 0.455693
cpu13: 0.453935
cpu14: 0.454519
cpu15: 0.454309
cpu16: 0.453584
cpu17: 0.454185
cpu18: 0.454524
cpu19: 0.453513
cpu20: 0.454963
cpu21: 0.454353
cpu22: 0.453824
cpu23: 0.455402
正常的：
cpu0: 0.418269
cpu1: 0.418326
cpu2: 0.418120
cpu3: 0.419714
cpu4: 0.418277
cpu5: 0.418985
cpu6: 0.418227
cpu7: 0.419253
cpu8: 0.417472
cpu9: 0.418434
cpu10: 0.417896
cpu11: 0.418111
cpu12: 0.418881
cpu13: 0.418430
cpu14: 0.419276
cpu15: 0.420470
cpu16: 0.419735
cpu17: 0.421145
cpu18: 0.419259
cpu19: 0.420511
cpu20: 0.420068
cpu21: 0.420064
cpu22: 0.418365
cpu23: 0.419380
查看/var/log/messages日志，并发现了CPU温度的报警，而其他服务器却没有。如下：
Mar 18 08:00:03 localhost kernel: CPU1: Package temperature above threshold, cpu clock throttled (total events = 82819894)
Mar 18 08:00:03 localhost kernel: CPU12: Package temperature above threshold, cpu clock throttled (total events = 82820006)
Mar 18 08:00:03 localhost kernel: CPU13: Package temperature above threshold, cpu clock throttled (total events = 82819435)
Mar 18 08:00:03 localhost kernel: CPU17: Package temperature above threshold, cpu clock throttled (total events = 82819498)
Mar 18 08:00:03 localhost kernel: CPU5: Package temperature above threshold, cpu clock throttled (total events = 82819307)
Mar 18 08:00:03 localhost kernel: CPU2: Package temperature above threshold, cpu clock throttled (total events = 82819369)
Mar 18 08:00:03 localhost kernel: CPU14: Package temperature above threshold, cpu clock throttled (total events = 82819315)
Mar 18 08:00:03 localhost kernel: CPU16: Package temperature above threshold, cpu clock throttled (total events = 82819313)
Mar 18 08:00:03 localhost kernel: CPU15: Package temperature above threshold, cpu clock throttled (total events = 82819182)
Mar 18 08:00:03 localhost kernel: CPU4: Package temperature above threshold, cpu clock throttled (total events = 82818996)
Mar 18 08:00:03 localhost kernel: CPU3: Package temperature above threshold, cpu clock throttled (total events = 82819343)
Mar 18 08:00:03 localhost kernel: CPU0: Package temperature above threshold, cpu clock throttled (total events = 82820622)
Mar 18 08:00:03 localhost kernel: CPU13: Package temperature/speed normal
Mar 18 08:00:03 localhost kernel: CPU5: Package temperature/speed normal
Mar 18 08:00:03 localhost kernel: CPU17: Package temperature/speed normal
Mar 18 08:00:03 localhost kernel: CPU14: Package temperature/speed normal
Mar 18 08:00:03 localhost kernel: CPU12: Package temperature/speed normal
Mar 18 08:00:03 localhost kernel: CPU0: Package temperature/speed normal
Mar 18 08:00:03 localhost kernel: CPU4: Package temperature/speed normal
Mar 18 08:00:03 localhost kernel: CPU2: Package temperature/speed normal
Mar 18 08:00:03 localhost kernel: CPU3: Package temperature/speed normal
Mar 18 08:00:03 localhost kernel: CPU15: Package temperature/speed normal
Mar 18 08:00:03 localhost kernel: CPU16: Package temperature/speed normal
Mar 18 08:00:03 localhost kernel: CPU1: Package temperature/speed normal
Apr 29 08:00:11 localhost kernel: CPU12: Package temperature above threshold, cpu clock throttled (total events = 125932667)
Apr 29 08:00:11 localhost kernel: CPU15: Package temperature above threshold, cpu clock throttled (total events = 125931379)
Apr 29 08:00:11 localhost kernel: CPU3: Package temperature above threshold, cpu clock throttled (total events = 125931484)
Apr 29 08:00:11 localhost kernel: CPU14: Package temperature above threshold, cpu clock throttled (total events = 125931530)
Apr 29 08:00:11 localhost kernel: CPU1: Package temperature above threshold, cpu clock throttled (total events = 125932494)
Apr 29 08:00:11 localhost kernel: CPU13: Package temperature above threshold, cpu clock throttled (total events = 125931774)
Apr 29 08:00:11 localhost kernel: CPU5: Package temperature above threshold, cpu clock throttled (total events = 125931586)
Apr 29 08:00:11 localhost kernel: CPU17: Package temperature above threshold, cpu clock throttled (total events = 125931885)
Apr 29 08:00:11 localhost kernel: CPU2: Package temperature above threshold, cpu clock throttled (total events = 125931776)
Apr 29 08:00:11 localhost kernel: CPU0: Package temperature above threshold, cpu clock throttled (total events = 125933796)
Apr 29 08:00:11 localhost kernel: CPU4: Package temperature above threshold, cpu clock throttled (total events = 125931076)
Apr 29 08:00:11 localhost kernel: CPU16: Package temperature above threshold, cpu clock throttled (total events = 125931603)
Apr 29 08:00:11 localhost kernel: CPU17: Package temperature/speed normal
Apr 29 08:00:11 localhost kernel: CPU15: Package temperature/speed normal
Apr 29 08:00:11 localhost kernel: CPU0: Package temperature/speed normal
Apr 29 08:00:11 localhost kernel: CPU2: Package temperature/speed normal
Apr 29 08:00:11 localhost kernel: CPU16: Package temperature/speed normal
Apr 29 08:00:11 localhost kernel: CPU3: Package temperature/speed normal
Apr 29 08:00:11 localhost kernel: CPU14: Package temperature/speed normal
Apr 29 08:00:11 localhost kernel: CPU5: Package temperature/speed normal
Apr 29 08:00:11 localhost kernel: CPU1: Package temperature/speed normal
Apr 29 08:00:11 localhost kernel: CPU13: Package temperature/speed normal
Apr 29 08:00:11 localhost kernel: CPU4: Package temperature/speed normal
Apr 29 08:00:11 localhost kernel: CPU12: Package temperature/speed normal
这下结果应该比较清晰了。CPU有问题…更换之
相关链接：
分类: Linux
评论 (0)
发表评论