Info

Permalink
Function
smp_call_function_many
First occurrence
2020-11-07
Last occurrence
2020-11-07
State
NEW
Components
kernel
Quality
-18

Statistics

Operating system Count
CentOS 7 150
Architecture Count
x86_64 150
Related packages Count
kernel 150
 0:3.10.0-1062.el7 72
 0:3.10.0-1160.el7 18
 0:3.10.0-1160.88.1.el7 14
 0:3.10.0-1160.11.1.el7 7
 0:3.10.0-1062.18.1.el7 5
 0:3.10.0-1127.el7 4
 0:3.10.0-1160.83.1.el7 4
 0:3.10.0-1127.19.1.el7 3
 0:3.10.0-1160.95.1.el7 3
 0:3.10.0-1160.24.1.el7 2
 0:3.10.0-1160.25.1.el7 2
 0:3.10.0-1160.76.1.el7 2
 0:3.10.0-1160.15.2.el7 2
 0:3.10.0-1062.1.2.el7 2
 0:3.10.0-1160.49.1.el7 2
 0:3.10.0-1160.42.2.el7 2
 0:3.10.0-1160.119.1.el7 1
 0:3.10.0-1160.45.1.el7 1
 0:3.10.0-1062.9.1.el7 1
 0:3.10.0-1160.92.1.el7 1
 0:3.10.0-1160.31.1.el7 1
 0:3.10.0-1160.102.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1397850
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb2f1e23a
Build id: 3.10.0-1160.102.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb2e81fa8
Build id: 3.10.0-1160.102.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb2e8201b
Build id: 3.10.0-1160.102.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb2ffa07e
Build id: 3.10.0-1160.102.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb2ffbbff
Build id: 3.10.0-1160.102.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb2ffbd33
Build id: 3.10.0-1160.102.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffb2ff8766
Build id: 3.10.0-1160.102.1.el7.x86_64
0x116
8
? wake_up_q
vmlinux 0xffffffffb2ee1118
Build id: 3.10.0-1160.102.1.el7.x86_64
0x58
9
? futex_wake
vmlinux 0xffffffffb2f19690
Build id: 3.10.0-1160.102.1.el7.x86_64
0x90
10
SyS_madvise
vmlinux 0xffffffffb2ff96a1
Build id: 3.10.0-1160.102.1.el7.x86_64
0x4d1
11
? lockref_put_or_lock
vmlinux 0xffffffffb31aee88
Build id: 3.10.0-1160.102.1.el7.x86_64
0x48
12
system_call_fastpath
vmlinux 0xffffffffb35c539a
Build id: 3.10.0-1160.102.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [node_exporter:2197]
Modules linked in: xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype nf_conntrack_netlink iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables veth br_netfilter overlay(T) tcp_lp binfmt_misc vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) tun devlink ipmi_devintf ipmi_msghandler bridge stp llc ip_set vmw_vsock_vmci_transport vsock sunrpc squashfs loop ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev sg parport_pc vmw_vmci parport i2c_piix4 xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sd_mod sysimgblt fb_sys_fops crc_t10dif ttm crct10dif_generic ahci nfit drm libahci ata_piix libnvdimm libata crct10dif_pclmul
crct10dif_common crc32c_intel vmxnet3 serio_raw vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ip_tables]
CPU: 7 PID: 2197 Comm: node_exporter Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-1160.102.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9a7dca95b180 ti: ffff9a7db0844000 task.ti: ffff9a7db0844000
RIP: 0010:[<ffffffffb2f1e23a>]  [<ffffffffb2f1e23a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9a7db0847c70  EFLAGS: 00000202
RAX: 0000000000000011 RBX: 00080000b2e81b10 RCX: ffff9a7dcf4606f0
RDX: 0000000000000011 RSI: 0000000000000018 RDI: 0000000000000000
RBP: ffff9a7db0847ca8 R08: ffff9a7321556400 R09: ffffffffb319c7b9
R10: ffff9a7dcf1df160 R11: fffffeabafcba400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9a7db0847c20 R15: 000000c0006e4000
FS:  000000c000080490(0000) GS:ffff9a7dcf1c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5ac5ef4000 CR3: 0000000bf1962000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffb2e81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb2e8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb2ffa07e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffb2ffbbff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb2ffbd33>] tlb_finish_mmu+0x23/0x40
[<ffffffffb2ff8766>] madvise_free_single_vma+0x116/0x180
[<ffffffffb2ee1118>] ? wake_up_q+0x58/0x90
[<ffffffffb2f19690>] ? futex_wake+0x90/0x190
[<ffffffffb2ff96a1>] SyS_madvise+0x4d1/0xac0
[<ffffffffb31aee88>] ? lockref_put_or_lock+0x48/0x80
[<ffffffffb35c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 4e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 19 b5 b3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1196862
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff815170da
Build id: 3.10.0-1160.49.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8147e258
Build id: 3.10.0-1160.49.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8147e2cb
Build id: 3.10.0-1160.49.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff815ee676
Build id: 3.10.0-1160.49.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff815f013f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff815f0263
Build id: 3.10.0-1160.49.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffff815ecd16
Build id: 3.10.0-1160.49.1.el7.x86_64
0x116
8
? wake_up_q
vmlinux 0xffffffff814dada8
Build id: 3.10.0-1160.49.1.el7.x86_64
0x58
9
? futex_wake
vmlinux 0xffffffff81512670
Build id: 3.10.0-1160.49.1.el7.x86_64
0x90
10
SyS_madvise
vmlinux 0xffffffff815edc51
Build id: 3.10.0-1160.49.1.el7.x86_64
0x4d1
11
system_call_fastpath
vmlinux 0xffffffff81b95f92
Build id: 3.10.0-1160.49.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [snmp_exporter:1731]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink tun vmw_vsock_vmci_transport vsock sunrpc vfat fat sb_edac nfit ppdev iosf_mbi libnvdimm crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg parport_pc parport tpm_crb i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi scsi_transport_spi mptscsih e1000 crct10dif_pclmul crct10dif_common crc32c_intel mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ipmi_msghandler]
CPU: 1 PID: 1731 Comm: snmp_exporter Tainted: G        W    L ------------   3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.16707776.B64.2008070230 08/07/2020
task: ffff944b284a3180 ti: ffff944b2af8c000 task.ti: ffff944b2af8c000
RIP: 0010:[<ffffffff815170da>]  [<ffffffff815170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff944b2af8fc70  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 000800008147ddd0 RCX: ffff944b2bd20558
RDX: 0000000000000004 RSI: 000000000000000c RDI: 0000000000000000
RBP: ffff944b2af8fca8 R08: ffff944772bc1400 R09: ffffffff81786bf9
R10: ffff944b2bc5f160 R11: ffffebfe50eb5800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff944b2af8fc20 R15: 000000c001c59000
FS:  000000c000500490(0000) GS:ffff944b2bc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055636d744130 CR3: 000000052a2dc000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8147e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8147e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff815ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff815f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff815f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff815ecd16>] madvise_free_single_vma+0x116/0x180
[<ffffffff814dada8>] ? wake_up_q+0x58/0x80
[<ffffffff81512670>] ? futex_wake+0x90/0x180
[<ffffffff815edc51>] SyS_madvise+0x4d1/0xac0
[<ffffffff81b95f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 15 82 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #901619
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb5f1e1ca
Build id: 3.10.0-1160.95.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb5e81fa8
Build id: 3.10.0-1160.95.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb5e8201b
Build id: 3.10.0-1160.95.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb5ffa00e
Build id: 3.10.0-1160.95.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb5ffbb8f
Build id: 3.10.0-1160.95.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb5ffbcc3
Build id: 3.10.0-1160.95.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffb5ff86f6
Build id: 3.10.0-1160.95.1.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffffb5ff9631
Build id: 3.10.0-1160.95.1.el7.x86_64
0x4d1
9
system_call_fastpath
vmlinux 0xffffffffb65c539a
Build id: 3.10.0-1160.95.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [influxd:3371]
Modules linked in: ipt_REJECT nf_reject_ipv4 xt_multiport ipt_rpfilter xt_set iptable_raw ip_set_hash_ip ip_set_hash_net ip_set vxlan ip6_udp_tunnel udp_tunnel ip6t_MASQUERADE nf_nat_masquerade_ipv6 xt_mark xt_comment ip6table_filter ip6table_mangle iptable_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables nf_tables veth xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_addrtype iptable_filter br_netfilter bridge stp llc overlay(T) ip_vs nf_conntrack vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq sb_edac iosf_mbi btusb crc32_pclmul snd_seq_device ghash_clmulni_intel btrtl btbcm snd_pcm aesni_intel
lrw btintel gf128mul snd_timer glue_helper snd ablk_helper bluetooth soundcore cryptd ppdev joydev vmw_vmci i2c_piix4 rfkill sg pcspkr vmw_balloon parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common nfit crc32c_intel libata nvme serio_raw libnvdimm e1000 nvme_core drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod synosnap(OE)
CPU: 8 PID: 3371 Comm: influxd Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-1160.95.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff921599a94200 ti: ffff921599aac000 task.ti: ffff921599aac000
RIP: 0010:[<ffffffffb5f1e1ca>]  [<ffffffffb5f1e1ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff921599aafc70  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000b5e81b10 RCX: ffff921601e60718
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff921599aafca8 R08: ffff92117095d400 R09: ffffffffb619c749
R10: ffff92160201f160 R11: ffffdca497f4c600 R12: 0000000000000282
R13: 0000000000000282 R14: ffff921599aafc20 R15: 000000c086e89000
FS:  00007f09f4fd9700(0000) GS:ffff921602000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffe119c6ff8 CR3: 00000005a3a64000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb5e81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb5e8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb5ffa00e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffb5ffbb8f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb5ffbcc3>] tlb_finish_mmu+0x23/0x40
[<ffffffffb5ff86f6>] madvise_free_single_vma+0x116/0x180
[<ffffffffb5ff9631>] SyS_madvise+0x4d1/0xac0
[<ffffffffb65c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 7e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 b5 b6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #897666
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbbf16c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbbe7e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbbe7e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbbfedd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbbfef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbbfef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffbbfec3a6
Build id: 3.10.0-1160.el7.x86_64
0x116
8
? __enqueue_entity
vmlinux 0xffffffffbbedfb48
Build id: 3.10.0-1160.el7.x86_64
0x78
9
? put_prev_entity
vmlinux 0xffffffffbbee5f1b
Build id: 3.10.0-1160.el7.x86_64
0x8b
10
SyS_madvise
vmlinux 0xffffffffbbfed2e1
Build id: 3.10.0-1160.el7.x86_64
0x4d1
11
system_call_fastpath
vmlinux 0xffffffffbc593f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [AsyncMetrics:19757]
Modules linked in: tcp_diag inet_diag tcp_lp nls_utf8 isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper vmw_balloon cryptd pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi drm nfit ata_piix mptscsih libata crct10dif_pclmul crct10dif_common
drm_panel_orientation_quirks libnvdimm mptbase serio_raw e1000 crc32c_intel dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 19757 Comm: AsyncMetrics Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9d1575c65280 ti: ffff9d14719f0000 task.ti: ffff9d14719f0000
RIP: 0010:[<ffffffffbbf16c0a>]  [<ffffffffbbf16c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9d14719f3c70  EFLAGS: 00000202
RAX: 0000000000000009 RBX: 00080000bbe7e6a0 RCX: ffff9d1618660600
RDX: 0000000000000009 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9d14719f3ca8 R08: ffff9d107b37b800 R09: ffffffffbc185f99
R10: ffff9d161845f160 R11: ffffdd595b3b0600 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9d14719f3c20 R15: 00007f38cc979000
FS:  00007f38bc9c8700(0000) GS:ffff9d1618440000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb19e075000 CR3: 000000063981c000 CR4: 0000000000360fe0
Call Trace:
[<ffffffffbbe7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbbe7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbbfedd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffbbfef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbbfef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffffbbfec3a6>] madvise_free_single_vma+0x116/0x180
[<ffffffffbbedfb48>] ? __enqueue_entity+0x78/0x80
[<ffffffffbbee5f1b>] ? put_prev_entity+0x8b/0x400
[<ffffffffbbfed2e1>] SyS_madvise+0x4d1/0xac0
[<ffffffffbc593f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 b5 bc f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #840994
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8db16bda
Build id: 3.10.0-1160.31.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8da7e258
Build id: 3.10.0-1160.31.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8da7e2cb
Build id: 3.10.0-1160.31.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8dbee176
Build id: 3.10.0-1160.31.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8dbefc3f
Build id: 3.10.0-1160.31.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8dbefd63
Build id: 3.10.0-1160.31.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffff8dbec816
Build id: 3.10.0-1160.31.1.el7.x86_64
0x116
8
? __check_object_size
vmlinux 0xffffffff8dc4ac2a
Build id: 3.10.0-1160.31.1.el7.x86_64
0x1ca
9
SyS_madvise
vmlinux 0xffffffff8dbed751
Build id: 3.10.0-1160.31.1.el7.x86_64
0x4d1
10
? putname
vmlinux 0xffffffff8dc5fe3d
Build id: 3.10.0-1160.31.1.el7.x86_64
0x3d
11
system_call_fastpath
vmlinux 0xffffffff8e195f92
Build id: 3.10.0-1160.31.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [zabbix_agent2:1753]
Modules linked in: vmw_vsock_vmci_transport vsock iosf_mbi ppdev crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea crc_t10dif sysfillrect crct10dif_generic sysimgblt fb_sys_fops ttm drm ata_piix libata nfit crct10dif_pclmul crct10dif_common crc32c_intel serio_raw libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 12 PID: 1753 Comm: zabbix_agent2 Kdump: loaded Not tainted 3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffffa11a17273180 ti: ffffa1199a9a8000 task.ti: ffffa1199a9a8000
RIP: 0010:[<ffffffff8db16bda>]  [<ffffffff8db16bda>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa1199a9abc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008da7ddd0 RCX: ffffa11a1f2207b8
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffffa1199a9abca8 R08: ffffa1136c5c9c00 R09: ffffffff8dd866a9
R10: ffffa11a1f51f160 R11: fffffa63c02a9a00 R12: 0000000000000282
R13: 0000000000000282 R14: ffffa1199a9abc20 R15: 000000c000817000
FS:  00007fd4c2ffd700(0000) GS:ffffa11a1f500000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f55d43abf90 CR3: 000000081d694000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff8da7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8da7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8dbee176>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8dbefc3f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8dbefd63>] tlb_finish_mmu+0x23/0x30
[<ffffffff8dbec816>] madvise_free_single_vma+0x116/0x180
[<ffffffff8dc4ac2a>] ? __check_object_size+0x1ca/0x250
[<ffffffff8dbed751>] SyS_madvise+0x4d1/0xac0
[<ffffffff8dc5fe3d>] ? putname+0x3d/0x60
[<ffffffff8e195f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 8a c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 75 8e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 8a c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #817779
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa8716cba
Build id: 3.10.0-1062.18.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa867e608
Build id: 3.10.0-1062.18.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa867e67b
Build id: 3.10.0-1062.18.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa87ebc86
Build id: 3.10.0-1062.18.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa87ed74f
Build id: 3.10.0-1062.18.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa87ed873
Build id: 3.10.0-1062.18.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffa87ea326
Build id: 3.10.0-1062.18.1.el7.x86_64
0x116
8
? wake_up_q
vmlinux 0xffffffffa86db5b8
Build id: 3.10.0-1062.18.1.el7.x86_64
0x58
9
? futex_wake
vmlinux 0xffffffffa8712110
Build id: 3.10.0-1062.18.1.el7.x86_64
0x90
10
SyS_madvise
vmlinux 0xffffffffa87eb261
Build id: 3.10.0-1062.18.1.el7.x86_64
0x4d1
11
tracesys
vmlinux 0xffffffffa8d8e172
Build id: 3.10.0-1062.18.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kafkabeat:26105]
Modules linked in: tun ip6t_MASQUERADE nf_nat_masquerade_ipv6 xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_mark ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg vmw_vmci parport_pc i2c_piix4 parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic
pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common ahci crc32c_intel ata_piix drm libahci nfit serio_raw libata libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 26105 Comm: kafkabeat Kdump: loaded Tainted: G             L ------------   3.10.0-1062.18.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
task: ffff8afd81b0c1c0 ti: ffff8b0159478000 task.ti: ffff8b0159478000
RIP: 0010:[<ffffffffa8716cba>]  [<ffffffffa8716cba>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b015947bc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a867e440 RCX: ffff8b03bfc20650
RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff8b015947bca8 R08: ffff8afef640cc00 R09: ffffffffa89819f9
R10: ffff8b03bfcdf160 R11: ffffe5c55894d400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8b015947bc20 R15: 000000c141e5b000
FS:  00007f0934b7e700(0000) GS:ffff8b03bfcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7e8645b5f0 CR3: 0000000366bbe000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffa867e608>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa867e67b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa87ebc86>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa87ed74f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa87ed873>] tlb_finish_mmu+0x23/0x30
[<ffffffffa87ea326>] madvise_free_single_vma+0x116/0x180
[<ffffffffa86db5b8>] ? wake_up_q+0x58/0x80
[<ffffffffa8712110>] ? futex_wake+0x90/0x180
[<ffffffffa87eb261>] SyS_madvise+0x4d1/0xac0
[<ffffffffa8d8e172>] tracesys+0xa6/0xcc
Code: 48 63 35 ee 5c c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 e8 34 a9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5c c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #787108
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9c3150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9c27d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9c27d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9c3ea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9c3ebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9c3ebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffff9c3e8866
Build id: 3.10.0-1062.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffff9c3e97a1
Build id: 3.10.0-1062.el7.x86_64
0x4d1
9
? pick_next_task_fair
vmlinux 0xffffffff9c2e166f
Build id: 3.10.0-1062.el7.x86_64
0x5f
10
? sched_clock_cpu
vmlinux 0xffffffff9c2dd425
Build id: 3.10.0-1062.el7.x86_64
0x85
11
system_call_fastpath
vmlinux 0xffffffff9c98bede
Build id: 3.10.0-1062.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 67s! [containerd:9073]
Modules linked in: tcp_diag udp_diag inet_diag fuse nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter nf_conntrack_ftp overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 nfs fscache arc4 md4 nls_utf8 cifs ccm dns_resolver ext4 mbcache jbd2 nfit sg libnvdimm ppdev virtio_balloon joydev parport_pc parport i2c_piix4 pcspkr iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net
virtio_scsi virtio_console virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel virtio_pci virtio_ring serio_raw floppy virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 9073 Comm: containerd Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff91ff94a20000 ti: ffff91ff8abe4000 task.ti: ffff91ff8abe4000
RIP: 0010:[<ffffffff9c3150ca>]  [<ffffffff9c3150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff91ff8abe7c70  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000800009c27d1d0 RCX: ffff91fffcee0628
RDX: 0000000000000003 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff91ff8abe7ca8 R08: ffff91e193053800 R09: ffffffff9c57f629
R10: ffff91fffcf1f160 R11: ffffe491471a9a00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff91ff8abe7c20 R15: 000000c000d1d000
FS:  00007f4b297fa700(0000) GS:ffff91fffcf00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4613b379e5 CR3: 0000001f7e24c000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff9c27d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9c27d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9c3ea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9c3ebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9c3ebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffff9c3e8866>] madvise_free_single_vma+0x116/0x180
[<ffffffff9c3e97a1>] SyS_madvise+0x4d1/0xac0
[<ffffffff9c2e166f>] ? pick_next_task_fair+0x5f/0x1b0
[<ffffffff9c2dd425>] ? sched_clock_cpu+0x85/0xc0
[<ffffffff9c98bede>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 f4 9c f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #748678
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffaa11718a
Build id: 3.10.0-1160.76.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffaa07e258
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffaa07e2cb
Build id: 3.10.0-1160.76.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffaa1ee6a6
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffaa1f016f
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffaa1f0293
Build id: 3.10.0-1160.76.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffaa1ecd46
Build id: 3.10.0-1160.76.1.el7.x86_64
0x116
8
? wake_up_q
vmlinux 0xffffffffaa0db298
Build id: 3.10.0-1160.76.1.el7.x86_64
0x58
9
? futex_wake
vmlinux 0xffffffffaa112720
Build id: 3.10.0-1160.76.1.el7.x86_64
0x90
10
SyS_madvise
vmlinux 0xffffffffaa1edc81
Build id: 3.10.0-1160.76.1.el7.x86_64
0x4d1
11
? user_path_at
vmlinux 0xffffffffaa261401
Build id: 3.10.0-1160.76.1.el7.x86_64
0x11
12
? vfs_fstatat
vmlinux 0xffffffffaa253e53
Build id: 3.10.0-1160.76.1.el7.x86_64
0x63
13
system_call_fastpath
vmlinux 0xffffffffaa799f92
Build id: 3.10.0-1160.76.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [influxd:45817]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock bnep sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus sb_edac snd_seq snd_seq_device iosf_mbi crc32_pclmul snd_pcm ghash_clmulni_intel
aesni_intel snd_timer btusb lrw snd btrtl btbcm btintel gf128mul bluetooth soundcore glue_helper rfkill ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix nfit mptspi libnvdimm libata scsi_transport_spi serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks synosnap(OE) fuse
CPU: 6 PID: 45817 Comm: influxd Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffffa0ec7fee1080 ti: ffffa0f0fd3b8000 task.ti: ffffa0f0fd3b8000
RIP: 0010:[<ffffffffaa11718a>]  [<ffffffffaa11718a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa0f0fd3bbc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000aa07ddd0 RCX: ffffa0f11d6206c8
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffffa0f0fd3bbca8 R08: ffffa0ea6c55cc00 R09: ffffffffaa3869f9
R10: ffffa0f11d79f160 R11: ffffeafb96a1a200 R12: 0000000000000282
R13: 0000000000000282 R14: ffffa0f0fd3bbc20 R15: 000000c07535d000
FS:  00007f6d077fe700(0000) GS:ffffa0f11d780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc7082029d8 CR3: 000000081b9ac000 CR4: 00000000003607e0
Call Trace:
[<ffffffffaa07e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaa07e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaa1ee6a6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffaa1f016f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaa1f0293>] tlb_finish_mmu+0x23/0x30
[<ffffffffaa1ecd46>] madvise_free_single_vma+0x116/0x180
[<ffffffffaa0db298>] ? wake_up_q+0x58/0x80
[<ffffffffaa112720>] ? futex_wake+0x90/0x180
[<ffffffffaa1edc81>] SyS_madvise+0x4d1/0xac0
[<ffffffffaa261401>] ? user_path_at+0x11/0x20
[<ffffffffaa253e53>] ? vfs_fstatat+0x63/0xc0
[<ffffffffaa799f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 be 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 d5 aa f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 8d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #650106
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffabf1e0ba
Build id: 3.10.0-1160.83.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffabe81f88
Build id: 3.10.0-1160.83.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffabe81ffb
Build id: 3.10.0-1160.83.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffabff9f4e
Build id: 3.10.0-1160.83.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffabffba8f
Build id: 3.10.0-1160.83.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffabffbbc3
Build id: 3.10.0-1160.83.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffabff85d6
Build id: 3.10.0-1160.83.1.el7.x86_64
0x116
8
? wake_up_q
vmlinux 0xffffffffabee1008
Build id: 3.10.0-1160.83.1.el7.x86_64
0x58
9
? futex_wake
vmlinux 0xffffffffabf19510
Build id: 3.10.0-1160.83.1.el7.x86_64
0x90
10
SyS_madvise
vmlinux 0xffffffffabff9511
Build id: 3.10.0-1160.83.1.el7.x86_64
0x4d1
11
system_call_fastpath
vmlinux 0xffffffffac5c539a
Build id: 3.10.0-1160.83.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 21s! [weaver:74508]
Modules linked in: xt_NFLOG dummy vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch ip_set_hash_net xt_physdev xt_state nfnetlink_log xt_statistic nf_tables ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs ip_set_hash_ip xt_set veth xt_nat nf_conntrack_netlink xt_addrtype ip6t_MASQUERADE nf_nat_masquerade_ipv6 ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_comment xt_mark mfe_fileaccess_100714877(OE) mfe_aac_100714877(OE) nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache cbsensor_2_5_1230541(OE) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security
iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ppdev vmw_balloon pcspkr sg joydev parport_pc parport vmw_vmci i2c_piix4 nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit ata_piix crc32c_intel libnvdimm mptspi libata scsi_transport_spi serio_raw mptscsih mptbase vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse br_netfilter bridge stp llc overlay(T)
CPU: 8 PID: 74508 Comm: weaver Tainted: G        W  OEL ------------ T 3.10.0-1160.83.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
task: ffff99529bd85280 ti: ffff994e6797c000 task.ti: ffff994e6797c000
RIP: 0010:[<ffffffffabf1e0ba>]  [<ffffffffabf1e0ba>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff994e6797fc70  EFLAGS: 00000202
RAX: 000000000000000f RBX: 00080000abe81af0 RCX: ffff9953d39e06f0
RDX: 000000000000000f RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff994e6797fca8 R08: ffff994aa469cc00 R09: ffffffffac19c179
R10: ffff9953d381f160 R11: ffffeaeb0477da00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff994e6797fc20 R15: 000000c0005d3000
FS:  00007f2700ff9700(0000) GS:ffff9953d3800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fac185f8010 CR3: 0000000a78fe4000 CR4: 00000000000007e0
Call Trace:
[<ffffffffabe81f88>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffabe81ffb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffabff9f4e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffabffba8f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffabffbbc3>] tlb_finish_mmu+0x23/0x40
[<ffffffffabff85d6>] madvise_free_single_vma+0x116/0x180
[<ffffffffabee1008>] ? wake_up_q+0x58/0x90
[<ffffffffabf19510>] ? futex_wake+0x90/0x190
[<ffffffffabff9511>] SyS_madvise+0x4d1/0xac0
[<ffffffffac5c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 8e 19 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 b5 ac f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d 19 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #625482
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff943170da
Build id: 3.10.0-1160.42.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9427e258
Build id: 3.10.0-1160.42.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9427e2cb
Build id: 3.10.0-1160.42.2.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff943ee676
Build id: 3.10.0-1160.42.2.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff943f013f
Build id: 3.10.0-1160.42.2.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff943f0263
Build id: 3.10.0-1160.42.2.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffff943ecd16
Build id: 3.10.0-1160.42.2.el7.x86_64
0x116
8
? wake_up_q
vmlinux 0xffffffff942dada8
Build id: 3.10.0-1160.42.2.el7.x86_64
0x58
9
? futex_wake
vmlinux 0xffffffff94312670
Build id: 3.10.0-1160.42.2.el7.x86_64
0x90
10
SyS_madvise
vmlinux 0xffffffff943edc51
Build id: 3.10.0-1160.42.2.el7.x86_64
0x4d1
11
? wake_up_state
vmlinux 0xffffffff942dadf0
Build id: 3.10.0-1160.42.2.el7.x86_64
0x20
12
system_call_fastpath
vmlinux 0xffffffff94995f92
Build id: 3.10.0-1160.42.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [containerd:1549]
Modules linked in: binfmt_misc ip_vs_rr xt_ipvs ip_vs xt_REDIRECT nf_nat_redirect vxlan ip6_udp_tunnel udp_tunnel xt_mark xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper
cryptd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 CyProtectDrv(POE) CyProtectDrvOpen(OE) ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea sysfillrect sysimgblt fb_sys_fops crc_t10dif ahci ttm crct10dif_generic ata_piix libahci drm crct10dif_pclmul nfit crct10dif_common crc32c_intel libata libnvdimm vmxnet3 serio_raw vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 13 PID: 1549 Comm: containerd Kdump: loaded Tainted: P        W  OEL ------------ T 3.10.0-1160.42.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff99909b5eb180 ti: ffff99908c028000 task.ti: ffff99908c028000
RIP: 0010:[<ffffffff943170da>]  [<ffffffff943170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff99908c02bc70  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 000800009427ddd0 RCX: ffff9990ad7a07e0
RDX: 0000000000000006 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff99908c02bca8 R08: ffff998df5c6e000 R09: ffffffff94586bd9
R10: ffff9990ad95f160 R11: ffffe55dcee77c00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff99908c02bc20 R15: 000000c0009e7000
FS:  00007fe907fff700(0000) GS:ffff9990ad940000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00003bfee8d2a000 CR3: 00000004159e0000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9427e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9427e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff943ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff943f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff943f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff943ecd16>] madvise_free_single_vma+0x116/0x180
[<ffffffff942dada8>] ? wake_up_q+0x58/0x80
[<ffffffff94312670>] ? futex_wake+0x90/0x180
[<ffffffff943edc51>] SyS_madvise+0x4d1/0xac0
[<ffffffff942dadf0>] ? wake_up_state+0x20/0x20
[<ffffffff94995f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 f5 94 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #625263
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8d9170da
Build id: 3.10.0-1160.49.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8d87e258
Build id: 3.10.0-1160.49.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8d87e2cb
Build id: 3.10.0-1160.49.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8d9ee676
Build id: 3.10.0-1160.49.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8d9f013f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8d9f0263
Build id: 3.10.0-1160.49.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffff8d9ecd16
Build id: 3.10.0-1160.49.1.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffff8d9edc51
Build id: 3.10.0-1160.49.1.el7.x86_64
0x4d1
9
? pick_next_task_fair
vmlinux 0xffffffff8d8e9d2e
Build id: 3.10.0-1160.49.1.el7.x86_64
0x61e
10
system_call_fastpath
vmlinux 0xffffffff8df95f92
Build id: 3.10.0-1160.49.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [jemalloc_bg_thd:3716]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac coretemp snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus iosf_mbi snd_seq crc32_pclmul snd_seq_device ghash_clmulni_intel snd_pcm aesni_intel snd_timer lrw gf128mul snd glue_helper ppdev soundcore ablk_helper cryptd pcspkr sg joydev vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm crct10dif_pclmul drm crct10dif_common crc32c_intel ata_piix mptspi libata scsi_transport_spi serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 3716 Comm: jemalloc_bg_thd Kdump: loaded Tainted: G             L ------------   3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
task: ffff914a8404d280 ti: ffff914af26b0000 task.ti: ffff914af26b0000
RIP: 0010:[<ffffffff8d9170da>]  [<ffffffff8d9170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff914af26b3c70  EFLAGS: 00000202
RAX: 0000000000000003 RBX: ffffffff8d87ddd0 RCX: ffff914abe8e0788
RDX: 0000000000000003 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff914af26b3ca8 R08: ffff914abdbab400 R09: ffffffff8db86bf9
R10: ffff914abe89f160 R11: ffffce6b407e4400 R12: 0000000000000286
R13: ffff914af26b3c20 R14: ffffce6b44d12400 R15: 00007fca8444f000
FS:  00007fca7c2b9700(0000) GS:ffff914abe880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000007259e50 CR3: 0000000072500000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8d87e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8d87e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8d9ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8d9f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8d9f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff8d9ecd16>] madvise_free_single_vma+0x116/0x180
[<ffffffff8d9edc51>] SyS_madvise+0x4d1/0xac0
[<ffffffff8d8e9d2e>] ? pick_next_task_fair+0x61e/0x870
[<ffffffff8df95f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 55 8e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #600355
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffab116a6a
Build id: 3.10.0-1062.9.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffab07e478
Build id: 3.10.0-1062.9.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffab07e4eb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffab1ec026
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffab1edaef
Build id: 3.10.0-1062.9.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffab1edc13
Build id: 3.10.0-1062.9.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffab1ea726
Build id: 3.10.0-1062.9.1.el7.x86_64
0x116
8
? wake_up_q
vmlinux 0xffffffffab0db478
Build id: 3.10.0-1062.9.1.el7.x86_64
0x58
9
? futex_wake
vmlinux 0xffffffffab111ec0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x90
10
SyS_madvise
vmlinux 0xffffffffab1eb661
Build id: 3.10.0-1062.9.1.el7.x86_64
0x4d1
11
? wake_up_state
vmlinux 0xffffffffab0db4c0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x20
12
system_call_fastpath
vmlinux 0xffffffffab78dede
Build id: 3.10.0-1062.9.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [gitea:962]
Modules linked in: xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 coretemp snd_rawmidi snd_ac97_codec iosf_mbi crc32_pclmul ac97_bus ghash_clmulni_intel aesni_intel snd_seq lrw gf128mul snd_seq_device snd_pcm glue_helper ppdev vmw_balloon ablk_helper cryptd snd_timer snd soundcore pcspkr sg joydev parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi e1000 crct10dif_pclmul crct10dif_common scsi_transport_spi mptscsih crc32c_intel serio_raw drm_panel_orientation_quirks mptbase floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 962 Comm: gitea Kdump: loaded Tainted: G               ------------ T 3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
task: ffff973a75d9a0e0 ti: ffff973afb01c000 task.ti: ffff973afb01c000
RIP: 0010:[<ffffffffab116a6a>]  [<ffffffffab116a6a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff973afb01fc70  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffffffab07e2b0 RCX: ffff973b7a660650
RDX: 0000000000000001 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff973afb01fca8 R08: ffff973b785ac800 R09: ffffffffab3817e9
R10: ffff973b7a6df160 R11: fffffaaec0d52200 R12: 0000000000000286
R13: ffff973afb01fc20 R14: fffffaaec1eb47c0 R15: 000000c00e31f000
FS:  00007f4ff9dad700(0000) GS:ffff973b7a6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f661b70a000 CR3: 0000000132fa6000 CR4: 00000000001607e0
Call Trace:
[<ffffffffab07e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffab07e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffab1ec026>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffab1edaef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffab1edc13>] tlb_finish_mmu+0x23/0x30
[<ffffffffab1ea726>] madvise_free_single_vma+0x116/0x180
[<ffffffffab0db478>] ? wake_up_q+0x58/0x80
[<ffffffffab111ec0>] ? futex_wake+0x90/0x180
[<ffffffffab1eb661>] SyS_madvise+0x4d1/0xac0
[<ffffffffab0db4c0>] ? wake_up_state+0x20/0x20
[<ffffffffab78dede>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 fe 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 d4 ab f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #498105
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8f9150ca
Build id: 3.10.0-1062.1.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8f87d398
Build id: 3.10.0-1062.1.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8f87d40b
Build id: 3.10.0-1062.1.2.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8f9ea166
Build id: 3.10.0-1062.1.2.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8f9ebc2f
Build id: 3.10.0-1062.1.2.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8f9ebd53
Build id: 3.10.0-1062.1.2.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffff8f9e8866
Build id: 3.10.0-1062.1.2.el7.x86_64
0x116
8
? futex_wake
vmlinux 0xffffffff8f910560
Build id: 3.10.0-1062.1.2.el7.x86_64
0x90
9
SyS_madvise
vmlinux 0xffffffff8f9e97a1
Build id: 3.10.0-1062.1.2.el7.x86_64
0x4d1
10
tracesys
vmlinux 0xffffffff8ff8d172
Build id: 3.10.0-1062.1.2.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 33s! [1_scheduler:56289]
Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd parport_pc pcspkr sg joydev parport i2c_piix4 vmw_vmci xfs libcrc32c sr_mod ata_generic cdrom pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea crc_t10dif sysfillrect crct10dif_generic sysimgblt fb_sys_fops ttm ahci drm libahci ata_piix nfit libata crct10dif_pclmul libnvdimm crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy
dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 0 PID: 56289 Comm: 1_scheduler Tainted: G             L ------------ T 3.10.0-1062.1.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff9aadfb29a0e0 ti: ffff9aade73bc000 task.ti: ffff9aade73bc000
RIP: 0010:[<ffffffff8f9150ca>]  [<ffffffff8f9150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9aade73bfc70  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffffffff8f87d1d0 RCX: ffff9aae7fc1fc10
RDX: 0000000000000008 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff9aade73bfca8 R08: ffff9aacbfd53c00 R09: ffffffff8fb7f629
R10: ffff9aacf9c1f160 R11: fffff75e894bc000 R12: 0000000000000286
R13: ffff9aade73bfc20 R14: 00007fe9c7e00000 R15: 00007fe9c7cb1000
FS:  00007fea658bf700(0000) GS:ffff9aacf9c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5a6d07a000 CR3: 00000002a6b4a000 CR4: 00000000001607f0
Call Trace:
[<ffffffff8f87d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8f87d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8f9ea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8f9ebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8f9ebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffff8f9e8866>] madvise_free_single_vma+0x116/0x180
[<ffffffff8f910560>] ? futex_wake+0x90/0x180
[<ffffffff8f9e97a1>] SyS_madvise+0x4d1/0xac0
[<ffffffff8ff8d172>] tracesys+0xa6/0xcc
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 54 90 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #454811
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbab1760a
Build id: 3.10.0-1127.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbaa7e8f8
Build id: 3.10.0-1127.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbaa7e96b
Build id: 3.10.0-1127.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbabeddb6
Build id: 3.10.0-1127.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbabef87f
Build id: 3.10.0-1127.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbabef9a3
Build id: 3.10.0-1127.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffbabec466
Build id: 3.10.0-1127.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffffbabed3a1
Build id: 3.10.0-1127.el7.x86_64
0x4d1
9
? sched_clock_cpu
vmlinux 0xffffffffbaadebe5
Build id: 3.10.0-1127.el7.x86_64
0x85
10
system_call_fastpath
vmlinux 0xffffffffbb192ed2
Build id: 3.10.0-1127.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [postgres_export:112040]
Modules linked in: tcp_lp nls_utf8 isofs fuse nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM tun devlink binfmt_misc overlay(T) ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi iosf_mbi snd_ac97_codec kvm_intel kvm irqbypass crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq
snd_seq_device aesni_intel snd_pcm snd_timer lrw gf128mul glue_helper snd ablk_helper soundcore pcspkr sg cryptd joydev ppdev vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata nfit libnvdimm crct10dif_pclmul crct10dif_common mptspi crc32c_intel e1000 scsi_transport_spi mptscsih serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 112040 Comm: postgres_export Kdump: loaded Tainted: G             L ------------ T 3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff99aff2fe1070 ti: ffff99b1a18d0000 task.ti: ffff99b1a18d0000
RIP: 0010:[<ffffffffbab1760a>]  [<ffffffffbab1760a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff99b1a18d3c70  EFLAGS: 00000202
RAX: 000000000000000d RBX: 0000000000000003 RCX: ffff99b1b5960650
RDX: 000000000000000d RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff99b1a18d3ca8 R08: ffff99b0fc88e400 R09: ffffffffbad85099
R10: ffff99b1b56df160 R11: fffff95f04969c00 R12: 0000000000013378
R13: 000000fcbad84fef R14: 0000000000000282 R15: 0000000000000282
FS:  00007f59ebfff700(0000) GS:ffff99b1b56c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa312d23000 CR3: 000000021c93a000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffbaa7e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbaa7e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbabeddb6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffbabef87f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbabef9a3>] tlb_finish_mmu+0x23/0x30
[<ffffffffbabec466>] madvise_free_single_vma+0x116/0x180
[<ffffffffbabed3a1>] SyS_madvise+0x4d1/0xab0
[<ffffffffbaadebe5>] ? sched_clock_cpu+0x85/0xc0
[<ffffffffbb192ed2>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 1e 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 0f 75 bb f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 7a c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #101538
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffaaf150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffaae7d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffaae7d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffaafea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffaafebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffaafebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffaafe8866
Build id: 3.10.0-1062.el7.x86_64
0x116
8
? wake_up_q
vmlinux 0xffffffffaaeda068
Build id: 3.10.0-1062.el7.x86_64
0x58
9
? futex_wake
vmlinux 0xffffffffaaf10560
Build id: 3.10.0-1062.el7.x86_64
0x90
10
SyS_madvise
vmlinux 0xffffffffaafe97a1
Build id: 3.10.0-1062.el7.x86_64
0x4d1
11
system_call_fastpath
vmlinux 0xffffffffab58bede
Build id: 3.10.0-1062.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [influxd:2629]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg vmw_vmci joydev i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod vmwgfx cdrom ata_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops pata_acpi ttm nfit mptspi drm crct10dif_pclmul ata_piix
scsi_transport_spi crct10dif_common libnvdimm crc32c_intel e1000 mptscsih libata serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 2629 Comm: influxd Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9d5b4d6c9070 ti: ffff9d5b61740000 task.ti: ffff9d5b61740000
RIP: 0010:[<ffffffffaaf150ca>]  [<ffffffffaaf150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9d5b61743c70  EFLAGS: 00000202
RAX: 0000000000000007 RBX: 00080000aae7d1d0 RCX: ffff9d5b757e0600
RDX: 0000000000000007 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9d5b61743ca8 R08: ffff9d5abc84cc00 R09: ffffffffab17f629
R10: ffff9d5b756df160 R11: fffff9924884be00 R12: 0000000000000282
R13: 0000000000000018 R14: ffff9d5b61743be0 R15: 0000000000000282
FS:  00007fd5a3bff700(0000) GS:ffff9d5b756c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0bd3177000 CR3: 00000000afc56000 CR4: 00000000001607e0
Call Trace:
[<ffffffffaae7d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaae7d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaafea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffaafebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaafebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffffaafe8866>] madvise_free_single_vma+0x116/0x180
[<ffffffffaaeda068>] ? wake_up_q+0x58/0x80
[<ffffffffaaf10560>] ? futex_wake+0x90/0x180
[<ffffffffaafe97a1>] SyS_madvise+0x4d1/0xac0
[<ffffffffab58bede>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 b4 ab f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #19110
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9611761a
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9607e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9607e96b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff961ede36
Build id: 3.10.0-1127.19.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff961ef8ff
Build id: 3.10.0-1127.19.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff961efa23
Build id: 3.10.0-1127.19.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffff961ec4e6
Build id: 3.10.0-1127.19.1.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffff961ed421
Build id: 3.10.0-1127.19.1.el7.x86_64
0x4d1
9
system_call_fastpath
vmlinux 0xffffffff96792ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [calico-node:14562]
Modules linked in: bnep btusb btrtl btbcm btintel bluetooth rfkill binfmt_misc xt_set ipt_rpfilter xt_multiport ip_set_hash_ip ip_set_hash_net vxlan ip6_udp_tunnel udp_tunnel xt_comment xt_mark ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport
vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 iosf_mbi snd_rawmidi crc32_pclmul snd_ac97_codec ppdev ghash_clmulni_intel ac97_bus snd_seq snd_seq_device aesni_intel lrw gf128mul snd_pcm glue_helper vmw_balloon ablk_helper cryptd snd_timer snd joydev pcspkr soundcore vmw_vmci sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit drm crct10dif_pclmul crct10dif_common mptspi crc32c_intel ata_piix libnvdimm scsi_transport_spi mptscsih mptbase libata serio_raw e1000 drm_panel_orientation_quirks
CPU: 3 PID: 14562 Comm: calico-node Tainted: G             L ------------ T 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff940243a9b150 ti: ffff940243a38000 task.ti: ffff940243a38000
RIP: 0010:[<ffffffff9611761a>]  [<ffffffff9611761a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff940243a3bc70  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000800009607e730 RCX: ffff9404c02a0650
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff940243a3bca8 R08: ffff9403bb9cb800 R09: ffffffff963851b9
R10: ffff9404c02df160 R11: fffff88c87137000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff940243a3bc20 R15: 000000c000dcf000
FS:  00007fa499b51700(0000) GS:ffff9404c02c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbf9f5c0000 CR3: 00000002270be000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9607e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9607e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff961ede36>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff961ef8ff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff961efa23>] tlb_finish_mmu+0x23/0x30
[<ffffffff961ec4e6>] madvise_free_single_vma+0x116/0x180
[<ffffffff961ed421>] SyS_madvise+0x4d1/0xab0
[<ffffffff96792ed2>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 d5 96 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 7b c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #13808
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa511761a
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa507e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa507e96b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa51ede36
Build id: 3.10.0-1127.19.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa51ef8ff
Build id: 3.10.0-1127.19.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa51efa23
Build id: 3.10.0-1127.19.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffa51ec4e6
Build id: 3.10.0-1127.19.1.el7.x86_64
0x116
8
? ep_poll
vmlinux 0xffffffffa529c39e
Build id: 3.10.0-1127.19.1.el7.x86_64
0x31e
9
SyS_madvise
vmlinux 0xffffffffa51ed421
Build id: 3.10.0-1127.19.1.el7.x86_64
0x4d1
10
? wake_up_state
vmlinux 0xffffffffa50db990
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20
11
system_call_fastpath
vmlinux 0xffffffffa5792ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [telegraf:1595]
Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper vmwgfx cryptd ttm ppdev drm_kms_helper joydev pcspkr vmw_balloon sg
syscopyarea sysfillrect sysimgblt fb_sys_fops parport_pc drm vmw_vmci drm_panel_orientation_quirks parport i2c_piix4 auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic ahci libahci ata_piix crct10dif_pclmul crct10dif_common crc32c_intel nfit serio_raw libata vmxnet3 vmw_pvscsi libnvdimm
CPU: 2 PID: 1595 Comm: telegraf Kdump: loaded Tainted: G               ------------ T 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9d8c33cbc1c0 ti: ffff9d8d2b240000 task.ti: ffff9d8d2b240000
RIP: 0010:[<ffffffffa511761a>]  [<ffffffffa511761a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9d8d2b243c70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a507e730 RCX: ffff9d8d3fc20628
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff9d8d2b243ca8 R08: ffff9d8cfd721c00 R09: ffffffffa53851b9
R10: ffff9d8d3fd1f160 R11: fffffda6c62d6000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9d8d2b243c20 R15: 000000c0010bd000
FS:  00007fe84fc22700(0000) GS:ffff9d8d3fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000560292e42ea0 CR3: 00000000b8a58000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa507e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa507e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa51ede36>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa51ef8ff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa51efa23>] tlb_finish_mmu+0x23/0x30
[<ffffffffa51ec4e6>] madvise_free_single_vma+0x116/0x180
[<ffffffffa529c39e>] ? ep_poll+0x31e/0x360
[<ffffffffa51ed421>] SyS_madvise+0x4d1/0xab0
[<ffffffffa50db990>] ? wake_up_state+0x20/0x20
[<ffffffffa5792ed2>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 d5 a5 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 7b c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #752474
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffba9150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffba87d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffba87d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffba9ea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffba9ebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffba9ebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffba9e8866
Build id: 3.10.0-1062.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffffba9e97a1
Build id: 3.10.0-1062.el7.x86_64
0x4d1
9
? pick_next_task_fair
vmlinux 0xffffffffba8e166f
Build id: 3.10.0-1062.el7.x86_64
0x5f
10
? sched_clock_cpu
vmlinux 0xffffffffba8dd425
Build id: 3.10.0-1062.el7.x86_64
0x85
11
? system_call_after_swapgs
vmlinux 0xffffffffbaf8be21
Build id: 3.10.0-1062.el7.x86_64
0xae
12
tracesys
vmlinux 0xffffffffbaf8c172
Build id: 3.10.0-1062.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [kube-apiserver:92719]
Modules linked in: qax_tq_base(OE) binfmt_misc rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_common iscsi_tcp libiscsi_tcp libiscsi xt_recent xt_set ipt_rpfilter xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net vxlan ip6_udp_tunnel udp_tunnel xt_statistic ipt_REJECT nf_reject_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_comment xt_mark xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables br_netfilter bridge stp llc overlay(T) scsi_transport_iscsi ip_set nfnetlink ppdev pcspkr virtio_balloon joydev sg i2c_piix4 parport_pc parport
xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci floppy serio_raw virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: qax_tq_base]
CPU: 7 PID: 92719 Comm: kube-apiserver Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff90e6bc0241c0 ti: ffff90e38d370000 task.ti: ffff90e38d370000
RIP: 0010:[<ffffffffba9150ca>]  [<ffffffffba9150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff90e38d373c70  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00080000ba87d1d0 RCX: ffff90e6bcea04a0
RDX: 0000000000000002 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff90e38d373ca8 R08: ffff90d8295d4400 R09: ffffffffbab7f629
R10: ffff90e6bcfdf160 R11: ffffd38610b0ce00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff90e38d373c20 R15: 000000c02029b000
FS:  000000c000280790(0000) GS:ffff90e6bcfc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000625ff8 CR3: 0000000c1eef2000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffba87d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffba87d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffba9ea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffba9ebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffba9ebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffffba9e8866>] madvise_free_single_vma+0x116/0x180
[<ffffffffba9e97a1>] SyS_madvise+0x4d1/0xac0
[<ffffffffba8e166f>] ? pick_next_task_fair+0x5f/0x1b0
[<ffffffffba8dd425>] ? sched_clock_cpu+0x85/0xc0
[<ffffffffbaf8be21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffbaf8c172>] tracesys+0xa6/0xcc
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 54 bb f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #542276
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffaf1150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffaf07d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffaf07d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffaf1ea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffaf1ebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffaf1ebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffaf1e8866
Build id: 3.10.0-1062.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffffaf1e97a1
Build id: 3.10.0-1062.el7.x86_64
0x4d1
9
? system_call_after_swapgs
vmlinux 0xffffffffaf78be21
Build id: 3.10.0-1062.el7.x86_64
0xae
10
tracesys
vmlinux 0xffffffffaf78c172
Build id: 3.10.0-1062.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 24s! [nsqd:3948]
Modules linked in: xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay(T) sunrpc ppdev joydev sg virtio_balloon parport_pc pcspkr parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom virtio_console virtio_scsi virtio_blk virtio_net ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci serio_raw virtio_ring virtio floppy drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 3948 Comm: nsqd Kdump: loaded Tainted: G               ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff8b137a305230 ti: ffff8b10305d8000 task.ti: ffff8b10305d8000
RIP: 0010:[<ffffffffaf1150ca>]  [<ffffffffaf1150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b10305dbc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000af07d1d0 RCX: ffff8b13ade20450
RDX: 0000000000000000 RSI: 0000000000000060 RDI: 0000000000000000
RBP: ffff8b10305dbca8 R08: ffff8b10fa3a4400 R09: ffffffffaf37f629
R10: ffff8b13adf5f160 R11: fffffb39c280c800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8b10305dbc20 R15: 000000c000c93000
FS:  000000c00004e090(0000) GS:ffff8b13adf40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f493035d9e5 CR3: 00000000b0732000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaf07d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaf07d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaf1ea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffaf1ebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaf1ebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffffaf1e8866>] madvise_free_single_vma+0x116/0x180
[<ffffffffaf1e97a1>] SyS_madvise+0x4d1/0xac0
[<ffffffffaf78be21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffaf78c172>] tracesys+0xa6/0xcc
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 d4 af f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89