Info

Permalink
Function
smp_call_function_many
First occurrence
2020-12-27
Last occurrence
2023-08-14
State
NEW
Components
kernel
Quality
-19

Statistics

Operating system Count
CentOS 7 95
Architecture Count
x86_64 95
Related packages Count
kernel 95
 0:3.10.0-957.el7 70
 0:3.10.0-957.12.2.el7 19
 0:3.10.0-693.el7 3
 0:3.10.0-957.27.2.el7 2
 0:3.10.0-862.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1253945
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9ed1173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9ec7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9ec7a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffff9ede4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffff9ede5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
zap_page_range
vmlinux 0xffffffff9ede87c2
Build id: 3.10.0-957.el7.x86_64
0x142
7
? wake_up_q
vmlinux 0xffffffff9ecd6768
Build id: 3.10.0-957.el7.x86_64
0x58
8
SyS_madvise
vmlinux 0xffffffff9ede368d
Build id: 3.10.0-957.el7.x86_64
0x49d
9
tracesys
vmlinux 0xffffffff9f37505b
Build id: 3.10.0-957.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [agent:8364]
Modules linked in: xt_REDIRECT nf_nat_redirect ip_vs_rr xt_ipvs ip_vs xt_nat veth vxlan ip6_udp_tunnel udp_tunnel xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter binfmt_misc overlay(T) vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi
crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmwgfx e1000 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm floppy ata_piix mptspi libata scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 8364 Comm: agent Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8b2736c1b0c0 ti: ffff8b2e29b8c000 task.ti: ffff8b2e29b8c000
RIP: 0010:[<ffffffff9ed1173a>]  [<ffffffff9ed1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b2e29b8fc78  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff8b2a2fc1fdd8
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff8b2e29b8fcb0 R08: ffff8b277a2e3800 R09: ffffffff9ef755c9
R10: ffff8b2a2fc5f160 R11: ffffce848e60f400 R12: 0000000000000296
R13: ffff8b2e29b8fc28 R14: 0000000000000001 R15: ffffce848e454bc0
FS:  000000c000500090(0000) GS:ffff8b2a2fc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0007ff000 CR3: 000000042cb52000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9ec7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9ec7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff9ede4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff9ede5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff9ede87c2>] zap_page_range+0x142/0x190
[<ffffffff9ecd6768>] ? wake_up_q+0x58/0x80
[<ffffffff9ede368d>] SyS_madvise+0x49d/0xac0
[<ffffffff9f37505b>] tracesys+0xa3/0xc9
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 94 9f f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1197549
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
6
zap_page_range
vmlinux 0xffffffff811b08fa
Build id: 3.10.0-693.el7.x86_64
0x13a
7
? wake_up_q
vmlinux 0xffffffff810c47c8
Build id: 3.10.0-693.el7.x86_64
0x58
8
SyS_madvise
vmlinux 0xffffffff811ac71d
Build id: 3.10.0-693.el7.x86_64
0x3cd
9
? set_next_entity
vmlinux 0xffffffff810cb62c
Build id: 3.10.0-693.el7.x86_64
0x3c
10
? SyS_futex
vmlinux 0xffffffff810f8040
Build id: 3.10.0-693.el7.x86_64
0x80
11
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [trantorAgent:17521]
Modules linked in: qax_tq_base(OE) uinput veth xt_nat xt_CHECKSUM iptable_mangle nf_conntrack_netlink nfnetlink ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat xt_addrtype nf_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack br_netfilter ipt_REJECT nf_reject_ipv4 overlay(T) tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter sunrpc cirrus ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm i2c_piix4 virtio_net virtio_console i2c_core sg virtio_scsi virtio_balloon snd_hda_codec_generic snd_hda_intel joydev snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer virtio_pci snd virtio_ring shpchp virtio soundcore ip_vs_ftp nf_nat ip_vs_sed ip_vs_nq ip_vs_sh ip_vs_dh ip_vs_lblcr ip_vs_lblc pcspkr ip_vs_wrr
ip_vs_rr ip_vs_wlc ip_vs_lc ip_vs nf_conntrack ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi ata_piix libata serio_raw floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: qax_tq_base]
CPU: 6 PID: 17521 Comm: trantorAgent Tainted: G           OEL ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: inspur Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-2.module_kos5+70+5e037788 04/01/2014
task: ffff8803da2b2f70 ti: ffff880370ba4000 task.ti: ffff880370ba4000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff880370ba7cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008106e4a0 RCX: ffff88043fc1ae30
RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff880370ba7ce8 R08: ffff88017a360400 R09: ffffffff813227d9
R10: ffff88043fd99c80 R11: ffffea0002d92400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff880370ba7c60 R15: ffffffff81f579c0
FS:  00007f92bd7fa700(0000) GS:ffff88043fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c001e25010 CR3: 0000000356a9d000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000101e25fff ffff880356abed88 ffff880356abea40 0000000000000000
ffffffffffffffff ffff880356abed88 ffff880356abea40 ffff880370ba7d38
ffffffff8106e668 ffff880356abea40 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b08fa>] zap_page_range+0x13a/0x180
[<ffffffff810c47c8>] ? wake_up_q+0x58/0x80
[<ffffffff811ac71d>] SyS_madvise+0x3cd/0x9c0
[<ffffffff810cb62c>] ? set_next_entity+0x3c/0xe0
[<ffffffff810f8040>] ? SyS_futex+0x80/0x180
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1196963
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a6
Build id: 3.10.0-693.el7.x86_64
0x206
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
6
zap_page_range
vmlinux 0xffffffff811b08fa
Build id: 3.10.0-693.el7.x86_64
0x13a
7
? wake_up_q
vmlinux 0xffffffff810c47c8
Build id: 3.10.0-693.el7.x86_64
0x58
8
SyS_madvise
vmlinux 0xffffffff811ac71d
Build id: 3.10.0-693.el7.x86_64
0x3cd
9
? set_next_entity
vmlinux 0xffffffff810cb62c
Build id: 3.10.0-693.el7.x86_64
0x3c
10
? SyS_futex
vmlinux 0xffffffff810f8040
Build id: 3.10.0-693.el7.x86_64
0x80
11
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trantorAgent:25249]
Modules linked in: qax_tq_base(OE) binfmt_misc xt_set ip_set_hash_ip cfg80211 rfkill uinput gsch(OE) redirfs(OE) nf_conntrack_netlink xt_addrtype br_netfilter dsa_filter(POE) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ilocker(OE) overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter cirrus ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops snd_hda_codec_generic snd_hda_intel snd_hda_codec
drm virtio_net snd_hda_core snd_hwdep snd_seq snd_seq_device virtio_console pcspkr i2c_piix4 snd_pcm joydev snd_timer virtio_balloon virtio_scsi sg shpchp i2c_core snd soundcore virtio_pci virtio_ring virtio nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi ata_piix libata serio_raw floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: qax_tq_base]
CPU: 1 PID: 25249 Comm: trantorAgent Tainted: P           OE  ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: inspur Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-2.module_kos5+70+5e037788 04/01/2014
task: ffff8800b71d9fa0 ti: ffff8800877d0000 task.ti: ffff8800877d0000
RIP: 0010:[<ffffffff810f99a6>]  [<ffffffff810f99a6>] smp_call_function_many+0x206/0x260
RSP: 0018:ffff8800877d3cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008106e4a0 RCX: ffff88043fc1a6c8
RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8800877d3ce8 R08: ffff88017a207400 R09: ffffffff813227d9
R10: ffff88043fc59c80 R11: ffffea000df4c000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8800877d3c60 R15: 0000000000000001
FS:  00007f40867fc700(0000) GS:ffff88043fc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00167b010 CR3: 00000001e2755000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
000000010169dfff ffff8803e96d6108 ffff8803e96d5dc0 0000000000000000
ffffffffffffffff ffff8803e96d6108 ffff8803e96d5dc0 ffff8800877d3d38
ffffffff8106e668 ffff8803e96d5dc0 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b08fa>] zap_page_range+0x13a/0x180
[<ffffffff810c47c8>] ? wake_up_q+0x58/0x80
[<ffffffff811ac71d>] SyS_madvise+0x3cd/0x9c0
[<ffffffff810cb62c>] ? set_next_entity+0x3c/0xe0
[<ffffffff810f8040>] ? SyS_futex+0x80/0x180
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #1016176
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffaf11173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffaf07a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffaf07a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffffaf1e4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffaf1e5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
zap_page_range
vmlinux 0xffffffffaf1e87c2
Build id: 3.10.0-957.el7.x86_64
0x142
7
? wake_up_q
vmlinux 0xffffffffaf0d6768
Build id: 3.10.0-957.el7.x86_64
0x58
8
SyS_madvise
vmlinux 0xffffffffaf1e368d
Build id: 3.10.0-957.el7.x86_64
0x49d
9
system_call_fastpath
vmlinux 0xffffffffaf774ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [serverLibDown:25860]
Modules linked in: tcp_lp nls_utf8 isofs fuse binfmt_misc tcp_diag inet_diag xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw
e1000 nfit libnvdimm ata_generic vmwgfx pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi ata_piix mptscsih drm libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 25860 Comm: serverLibDown Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8f5455f68000 ti: ffff8f564cee4000 task.ti: ffff8f564cee4000
RIP: 0010:[<ffffffffaf11173a>]  [<ffffffffaf11173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8f564cee7c78  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff8f566d6204b8
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8f564cee7cb0 R08: ffff8f53b9cb3800 R09: ffffffffaf3755c9
R10: ffff8f566d69f160 R11: fffff8818f59a400 R12: 0000000000000296
R13: 0000000000000018 R14: ffff8f564cee7be8 R15: 0000000000000296
FS:  000000c000400090(0000) GS:ffff8f566d680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000b3e6f8 CR3: 00000003dabf0000 CR4: 00000000003607e0
Call Trace:
[<ffffffffaf07a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaf07a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffaf1e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffaf1e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffaf1e87c2>] zap_page_range+0x142/0x190
[<ffffffffaf0d6768>] ? wake_up_q+0x58/0x80
[<ffffffffaf1e368d>] SyS_madvise+0x49d/0xac0
[<ffffffffaf774ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 d4 af f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #820424
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb991173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb987a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb987a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffffb99e4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffb99e5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
zap_page_range
vmlinux 0xffffffffb99e87c2
Build id: 3.10.0-957.el7.x86_64
0x142
7
? wake_up_q
vmlinux 0xffffffffb98d6768
Build id: 3.10.0-957.el7.x86_64
0x58
8
SyS_madvise
vmlinux 0xffffffffb99e368d
Build id: 3.10.0-957.el7.x86_64
0x49d
9
? wake_up_state
vmlinux 0xffffffffb98d67b0
Build id: 3.10.0-957.el7.x86_64
0x20
10
system_call_fastpath
vmlinux 0xffffffffb9f74ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [process-exporte:9936]
Modules linked in: nfnetlink_queue nfnetlink_log bluetooth rfkill xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq joydev snd_seq_device snd_pcm pcspkr snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic
pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm e1000 ata_piix mptspi scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 9936 Comm: process-exporte Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff8a6dc1049040 ti: ffff8a6db0b3c000 task.ti: ffff8a6db0b3c000
RIP: 0010:[<ffffffffb991173a>]  [<ffffffffb991173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8a6db0b3fc78  EFLAGS: 00000202
RAX: 0000000000000007 RBX: 00000000000000fc RCX: ffff8a6dc51e0650
RDX: 0000000000000007 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8a6db0b3fcb0 R08: ffff8a6af86ccc00 R09: ffffffffb9b755c9
R10: ffff8a6dc515f160 R11: ffffe859006ab200 R12: 0000000000000296
R13: ffff8a6db0b3fc28 R14: 0000000000000001 R15: ffffe8590dff43c0
FS:  00007faf3166d700(0000) GS:ffff8a6dc5140000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6dac3d91bc CR3: 0000000430e40000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb987a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb987a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb99e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffb99e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffb99e87c2>] zap_page_range+0x142/0x190
[<ffffffffb98d6768>] ? wake_up_q+0x58/0x80
[<ffffffffb99e368d>] SyS_madvise+0x49d/0xac0
[<ffffffffb98d67b0>] ? wake_up_state+0x20/0x20
[<ffffffffb9f74ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 54 ba f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #705142
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8d71173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8d67a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8d67a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffff8d7e4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffff8d7e5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
zap_page_range
vmlinux 0xffffffff8d7e87c2
Build id: 3.10.0-957.el7.x86_64
0x142
7
? wake_up_q
vmlinux 0xffffffff8d6d6768
Build id: 3.10.0-957.el7.x86_64
0x58
8
SyS_madvise
vmlinux 0xffffffff8d7e368d
Build id: 3.10.0-957.el7.x86_64
0x49d
9
tracesys
vmlinux 0xffffffff8dd7505b
Build id: 3.10.0-957.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [minio:34157]
Modules linked in: binfmt_misc veth xt_nat xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock fuse vmhgfs(OE) overlay(T) sunrpc snd_seq_midi snd_seq_midi_event sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel snd_ens1371 lrw snd_rawmidi gf128mul ppdev glue_helper snd_ac97_codec ablk_helper cryptd ac97_bus vmw_balloon snd_seq snd_seq_device snd_pcm snd_timer snd joydev soundcore pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata mptspi scsi_transport_spi mptscsih e1000 mptbase nfit libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 34157 Comm: minio Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8ce63331b0c0 ti: ffff8ce810b0c000 task.ti: ffff8ce810b0c000
RIP: 0010:[<ffffffff8d71173a>]  [<ffffffff8d71173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8ce810b0fc78  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff8ce9ad6204b8
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8ce810b0fcb0 R08: ffff8ce6f9cb3800 R09: ffffffff8d9755c9
R10: ffff8ce9ad69f160 R11: fffff56809f1c800 R12: 0000000000000296
R13: ffff8ce810b0fc28 R14: 0000000000000001 R15: fffff5680101bec0
FS:  000000c00008f890(0000) GS:ffff8ce9ad680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa75391f700 CR3: 00000000a95ca000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8d67a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8d67a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff8d7e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff8d7e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff8d7e87c2>] zap_page_range+0x142/0x190
[<ffffffff8d6d6768>] ? wake_up_q+0x58/0x80
[<ffffffff8d7e368d>] SyS_madvise+0x49d/0xac0
[<ffffffff8dd7505b>] tracesys+0xa3/0xc9
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 34 8e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #193502
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa5f07e3e
Build id: 3.10.0-862.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffffa5e77018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa5e77088
Build id: 3.10.0-862.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffffa5fc0497
Build id: 3.10.0-862.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffa5fc1c75
Build id: 3.10.0-862.el7.x86_64
0x55
6
zap_page_range
vmlinux 0xffffffffa5fc4a02
Build id: 3.10.0-862.el7.x86_64
0x142
7
? wake_up_q
vmlinux 0xffffffffa5ecee38
Build id: 3.10.0-862.el7.x86_64
0x58
8
SyS_madvise
vmlinux 0xffffffffa5fbfafd
Build id: 3.10.0-862.el7.x86_64
0x3cd
9
? wake_up_state
vmlinux 0xffffffffa5ecee80
Build id: 3.10.0-862.el7.x86_64
0x20
10
system_call_fastpath
vmlinux 0xffffffffa651f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 32s! [node_exporter:1561]
Modules linked in: iptable_filter ip_tables binfmt_misc ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg pcspkr joydev vmw_balloon parport_pc parport shpchp vmw_vmci i2c_piix4 xfs libcrc32c sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper sr_mod cdrom ata_generic syscopyarea pata_acpi sysfillrect sysimgblt fb_sys_fops ttm mptsas drm ata_piix scsi_transport_sas crct10dif_pclmul crct10dif_common libata mptscsih serio_raw crc32c_intel mptbase vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 3 PID: 1561 Comm: node_exporter Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8a9a35e99fa0 ti: ffff8a9a0b590000 task.ti: ffff8a9a0b590000
RIP: 0010:[<ffffffffa5f07e3e>]  [<ffffffffa5f07e3e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8a9a0b593c78  EFLAGS: 00000202
RAX: 0000000000000005 RBX: 00000000000000fc RCX: ffff8a9e3fd5cf00
RDX: 0000000000000005 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8a9a0b593cb0 R08: ffff8a9b7a357400 R09: ffffffffa614a3d9
R10: ffff8a9e3fcdbb00 R11: ffffda679089f400 R12: 0000000000000296
R13: ffff8a9a0b593c28 R14: 0000000000000001 R15: ffffda6781628700
FS:  000000c000054e90(0000) GS:ffff8a9e3fcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00076f000 CR3: 00000003a49d6000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa5e77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa5e77088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa5fc0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffa5fc1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffa5fc4a02>] zap_page_range+0x142/0x190
[<ffffffffa5ecee38>] ? wake_up_q+0x58/0x80
[<ffffffffa5fbfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffffa5ecee80>] ? wake_up_state+0x20/0x20
[<ffffffffa651f7d5>] system_call_fastpath+0x1c/0x21
Code: b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 b3 a6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #23323
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffaa911a0a
Build id: 3.10.0-957.12.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffaa87a628
Build id: 3.10.0-957.12.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffaa87a698
Build id: 3.10.0-957.12.2.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffffaa9e4607
Build id: 3.10.0-957.12.2.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffaa9e5f05
Build id: 3.10.0-957.12.2.el7.x86_64
0x55
6
zap_page_range
vmlinux 0xffffffffaa9e8d52
Build id: 3.10.0-957.12.2.el7.x86_64
0x142
7
? wake_up_q
vmlinux 0xffffffffaa8d6a98
Build id: 3.10.0-957.12.2.el7.x86_64
0x58
8
SyS_madvise
vmlinux 0xffffffffaa9e3c1d
Build id: 3.10.0-957.12.2.el7.x86_64
0x49d
9
system_call_fastpath
vmlinux 0xffffffffaaf75ddb
Build id: 3.10.0-957.12.2.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [mysqld:73675]
Modules linked in: nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter binfmt_misc overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event coretemp iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 ppdev snd_rawmidi snd_ac97_codec ac97_bus snd_seq pcspkr snd_seq_device snd_pcm snd_timer snd soundcore vmw_balloon joydev sg parport_pc parport vmw_vmci i2c_piix4 ip_tables
xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops serio_raw ttm drm mptspi scsi_transport_spi mptscsih mptbase e1000 ata_piix libata drm_panel_orientation_quirks nfit libnvdimm
CPU: 6 PID: 73675 Comm: mysqld Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.12.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9158d9caa080 ti: ffff9159f668c000 task.ti: ffff9159f668c000
RIP: 0010:[<ffffffffaa911a0a>]  [<ffffffffaa911a0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9159f668fc78  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00000000000000fc RCX: ffff915a2d720678
RDX: 0000000000000004 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9159f668fcb0 R08: ffff915779cb4800 R09: ffffffffaab75d69
R10: ffff915a2d79f160 R11: fffffd339076ec00 R12: 0000000000000296
R13: ffff9159f668fc28 R14: 0000000000000001 R15: fffffd3382ba1d80
FS:  00007f3bf6805700(0000) GS:ffff915a2d780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3bf766df90 CR3: 00000003793d4000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaa87a628>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaa87a698>] flush_tlb_mm_range+0x68/0x140
[<ffffffffaa9e4607>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffaa9e5f05>] tlb_finish_mmu+0x55/0x60
[<ffffffffaa9e8d52>] zap_page_range+0x142/0x190
[<ffffffffaa8d6a98>] ? wake_up_q+0x58/0x80
[<ffffffffaa9e3c1d>] SyS_madvise+0x49d/0xac0
[<ffffffffaaf75ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 7e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 bc 54 ab f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 83 c4 00 eb b7 0f b6 4d cc 4c 89