Info

Permalink
Function
smp_call_function_many
First occurrence
2020-11-07
Last occurrence
2020-11-07
State
NEW
Components
kernel
Quality
-18

Statistics

Operating system Count
CentOS 7 89
Architecture Count
x86_64 89
Related packages Count
kernel 89
 0:3.10.0-1062.el7 45
 0:3.10.0-1160.el7 12
 0:3.10.0-1160.11.1.el7 7
 0:3.10.0-1127.19.1.el7 4
 0:3.10.0-1062.18.1.el7 4
 0:3.10.0-1127.el7 4
 0:3.10.0-1160.119.1.el7 3
 0:3.10.0-1160.95.1.el7 3
 0:3.10.0-1160.24.1.el7 2
 0:3.10.0-1160.25.1.el7 1
 0:3.10.0-1160.45.1.el7 1
 0:3.10.0-1160.76.1.el7 1
 0:3.10.0-1160.49.1.el7 1
 0:3.10.0-1160.31.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1414563
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa7b1e25a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa7a81fa8
Build id: 3.10.0-1160.119.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa7a8201b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa7bfa0ce
Build id: 3.10.0-1160.119.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa7bfbc4f
Build id: 3.10.0-1160.119.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa7bfbd83
Build id: 3.10.0-1160.119.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffa7bff2ca
Build id: 3.10.0-1160.119.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffa7bf96bd
Build id: 3.10.0-1160.119.1.el7.x86_64
0x49d
9
? hrtimer_get_res
vmlinux 0xffffffffa7acf340
Build id: 3.10.0-1160.119.1.el7.x86_64
0x50
10
? do_nanosleep
vmlinux 0xffffffffa81b6e1b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x5b
11
system_call_fastpath
vmlinux 0xffffffffa81c539a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [transport:13908]
Modules linked in: uinput nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm ppdev snd_timer sg parport_pc snd parport pcspkr joydev soundcore i2c_piix4 virtio_balloon ip_tables xfs libcrc32c sd_mod
crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi virtio_console ata_piix libata e1000 crc32c_intel virtio_pci serio_raw virtio_ring floppy virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 6 PID: 13908 Comm: transport Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: SANGFOR Standard PC (i440FX + PIIX, 1996)/Bochs, BIOS Bochs 01/01/2011
task: ffff9278f0ec4200 ti: ffff92793620c000 task.ti: ffff92793620c000
RIP: 0010:[<ffffffffa7b1e25a>]  [<ffffffffa7b1e25a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff92793620fc70  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000a7a81b10 RCX: ffff92793fc604c8
RDX: 0000000000000001 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff92793620fca8 R08: ffff92787aee1c00 R09: ffffffffa7d9c7f9
R10: ffff92793fd9f160 R11: ffffcf0687229000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff92793620fc20 R15: 0000000000000001
FS:  00007f7b9b524700(0000) GS:ffff92793fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff7ef147b90 CR3: 00000000866fa000 CR4: 00000000000206e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa7a81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa7a8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa7bfa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffa7bfbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa7bfbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffffa7bff2ca>] zap_page_range+0xea/0x140
[<ffffffffa7bf96bd>] SyS_madvise+0x49d/0xac0
[<ffffffffa7acf340>] ? hrtimer_get_res+0x50/0x50
[<ffffffffa81b6e1b>] ? do_nanosleep+0x5b/0x100
[<ffffffffa81c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ae 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 c0 19 75 a8 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 7d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #901619
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb5f1e1ca
Build id: 3.10.0-1160.95.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb5e81fa8
Build id: 3.10.0-1160.95.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb5e8201b
Build id: 3.10.0-1160.95.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb5ffa00e
Build id: 3.10.0-1160.95.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb5ffbb8f
Build id: 3.10.0-1160.95.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb5ffbcc3
Build id: 3.10.0-1160.95.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffb5ff86f6
Build id: 3.10.0-1160.95.1.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffffb5ff9631
Build id: 3.10.0-1160.95.1.el7.x86_64
0x4d1
9
system_call_fastpath
vmlinux 0xffffffffb65c539a
Build id: 3.10.0-1160.95.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [influxd:3371]
Modules linked in: ipt_REJECT nf_reject_ipv4 xt_multiport ipt_rpfilter xt_set iptable_raw ip_set_hash_ip ip_set_hash_net ip_set vxlan ip6_udp_tunnel udp_tunnel ip6t_MASQUERADE nf_nat_masquerade_ipv6 xt_mark xt_comment ip6table_filter ip6table_mangle iptable_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables nf_tables veth xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_addrtype iptable_filter br_netfilter bridge stp llc overlay(T) ip_vs nf_conntrack vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq sb_edac iosf_mbi btusb crc32_pclmul snd_seq_device ghash_clmulni_intel btrtl btbcm snd_pcm aesni_intel
lrw btintel gf128mul snd_timer glue_helper snd ablk_helper bluetooth soundcore cryptd ppdev joydev vmw_vmci i2c_piix4 rfkill sg pcspkr vmw_balloon parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common nfit crc32c_intel libata nvme serio_raw libnvdimm e1000 nvme_core drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod synosnap(OE)
CPU: 8 PID: 3371 Comm: influxd Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-1160.95.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff921599a94200 ti: ffff921599aac000 task.ti: ffff921599aac000
RIP: 0010:[<ffffffffb5f1e1ca>]  [<ffffffffb5f1e1ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff921599aafc70  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000b5e81b10 RCX: ffff921601e60718
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff921599aafca8 R08: ffff92117095d400 R09: ffffffffb619c749
R10: ffff92160201f160 R11: ffffdca497f4c600 R12: 0000000000000282
R13: 0000000000000282 R14: ffff921599aafc20 R15: 000000c086e89000
FS:  00007f09f4fd9700(0000) GS:ffff921602000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffe119c6ff8 CR3: 00000005a3a64000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb5e81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb5e8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb5ffa00e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffb5ffbb8f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb5ffbcc3>] tlb_finish_mmu+0x23/0x40
[<ffffffffb5ff86f6>] madvise_free_single_vma+0x116/0x180
[<ffffffffb5ff9631>] SyS_madvise+0x4d1/0xac0
[<ffffffffb65c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 7e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 b5 b6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #840994
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8db16bda
Build id: 3.10.0-1160.31.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8da7e258
Build id: 3.10.0-1160.31.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8da7e2cb
Build id: 3.10.0-1160.31.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8dbee176
Build id: 3.10.0-1160.31.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8dbefc3f
Build id: 3.10.0-1160.31.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8dbefd63
Build id: 3.10.0-1160.31.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffff8dbec816
Build id: 3.10.0-1160.31.1.el7.x86_64
0x116
8
? __check_object_size
vmlinux 0xffffffff8dc4ac2a
Build id: 3.10.0-1160.31.1.el7.x86_64
0x1ca
9
SyS_madvise
vmlinux 0xffffffff8dbed751
Build id: 3.10.0-1160.31.1.el7.x86_64
0x4d1
10
? putname
vmlinux 0xffffffff8dc5fe3d
Build id: 3.10.0-1160.31.1.el7.x86_64
0x3d
11
system_call_fastpath
vmlinux 0xffffffff8e195f92
Build id: 3.10.0-1160.31.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [zabbix_agent2:1753]
Modules linked in: vmw_vsock_vmci_transport vsock iosf_mbi ppdev crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea crc_t10dif sysfillrect crct10dif_generic sysimgblt fb_sys_fops ttm drm ata_piix libata nfit crct10dif_pclmul crct10dif_common crc32c_intel serio_raw libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 12 PID: 1753 Comm: zabbix_agent2 Kdump: loaded Not tainted 3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffffa11a17273180 ti: ffffa1199a9a8000 task.ti: ffffa1199a9a8000
RIP: 0010:[<ffffffff8db16bda>]  [<ffffffff8db16bda>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa1199a9abc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008da7ddd0 RCX: ffffa11a1f2207b8
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffffa1199a9abca8 R08: ffffa1136c5c9c00 R09: ffffffff8dd866a9
R10: ffffa11a1f51f160 R11: fffffa63c02a9a00 R12: 0000000000000282
R13: 0000000000000282 R14: ffffa1199a9abc20 R15: 000000c000817000
FS:  00007fd4c2ffd700(0000) GS:ffffa11a1f500000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f55d43abf90 CR3: 000000081d694000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff8da7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8da7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8dbee176>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8dbefc3f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8dbefd63>] tlb_finish_mmu+0x23/0x30
[<ffffffff8dbec816>] madvise_free_single_vma+0x116/0x180
[<ffffffff8dc4ac2a>] ? __check_object_size+0x1ca/0x250
[<ffffffff8dbed751>] SyS_madvise+0x4d1/0xac0
[<ffffffff8dc5fe3d>] ? putname+0x3d/0x60
[<ffffffff8e195f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 8a c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 75 8e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 8a c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #787108
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9c3150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9c27d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9c27d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9c3ea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9c3ebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9c3ebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffff9c3e8866
Build id: 3.10.0-1062.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffff9c3e97a1
Build id: 3.10.0-1062.el7.x86_64
0x4d1
9
? pick_next_task_fair
vmlinux 0xffffffff9c2e166f
Build id: 3.10.0-1062.el7.x86_64
0x5f
10
? sched_clock_cpu
vmlinux 0xffffffff9c2dd425
Build id: 3.10.0-1062.el7.x86_64
0x85
11
system_call_fastpath
vmlinux 0xffffffff9c98bede
Build id: 3.10.0-1062.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 67s! [containerd:9073]
Modules linked in: tcp_diag udp_diag inet_diag fuse nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter nf_conntrack_ftp overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 nfs fscache arc4 md4 nls_utf8 cifs ccm dns_resolver ext4 mbcache jbd2 nfit sg libnvdimm ppdev virtio_balloon joydev parport_pc parport i2c_piix4 pcspkr iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net
virtio_scsi virtio_console virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel virtio_pci virtio_ring serio_raw floppy virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 9073 Comm: containerd Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff91ff94a20000 ti: ffff91ff8abe4000 task.ti: ffff91ff8abe4000
RIP: 0010:[<ffffffff9c3150ca>]  [<ffffffff9c3150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff91ff8abe7c70  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000800009c27d1d0 RCX: ffff91fffcee0628
RDX: 0000000000000003 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff91ff8abe7ca8 R08: ffff91e193053800 R09: ffffffff9c57f629
R10: ffff91fffcf1f160 R11: ffffe491471a9a00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff91ff8abe7c20 R15: 000000c000d1d000
FS:  00007f4b297fa700(0000) GS:ffff91fffcf00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4613b379e5 CR3: 0000001f7e24c000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff9c27d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9c27d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9c3ea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9c3ebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9c3ebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffff9c3e8866>] madvise_free_single_vma+0x116/0x180
[<ffffffff9c3e97a1>] SyS_madvise+0x4d1/0xac0
[<ffffffff9c2e166f>] ? pick_next_task_fair+0x5f/0x1b0
[<ffffffff9c2dd425>] ? sched_clock_cpu+0x85/0xc0
[<ffffffff9c98bede>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 f4 9c f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #752474
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffba9150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffba87d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffba87d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffba9ea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffba9ebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffba9ebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffba9e8866
Build id: 3.10.0-1062.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffffba9e97a1
Build id: 3.10.0-1062.el7.x86_64
0x4d1
9
? pick_next_task_fair
vmlinux 0xffffffffba8e166f
Build id: 3.10.0-1062.el7.x86_64
0x5f
10
? sched_clock_cpu
vmlinux 0xffffffffba8dd425
Build id: 3.10.0-1062.el7.x86_64
0x85
11
? system_call_after_swapgs
vmlinux 0xffffffffbaf8be21
Build id: 3.10.0-1062.el7.x86_64
0xae
12
tracesys
vmlinux 0xffffffffbaf8c172
Build id: 3.10.0-1062.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [kube-apiserver:92719]
Modules linked in: qax_tq_base(OE) binfmt_misc rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_common iscsi_tcp libiscsi_tcp libiscsi xt_recent xt_set ipt_rpfilter xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net vxlan ip6_udp_tunnel udp_tunnel xt_statistic ipt_REJECT nf_reject_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_comment xt_mark xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables br_netfilter bridge stp llc overlay(T) scsi_transport_iscsi ip_set nfnetlink ppdev pcspkr virtio_balloon joydev sg i2c_piix4 parport_pc parport
xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci floppy serio_raw virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: qax_tq_base]
CPU: 7 PID: 92719 Comm: kube-apiserver Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff90e6bc0241c0 ti: ffff90e38d370000 task.ti: ffff90e38d370000
RIP: 0010:[<ffffffffba9150ca>]  [<ffffffffba9150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff90e38d373c70  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00080000ba87d1d0 RCX: ffff90e6bcea04a0
RDX: 0000000000000002 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff90e38d373ca8 R08: ffff90d8295d4400 R09: ffffffffbab7f629
R10: ffff90e6bcfdf160 R11: ffffd38610b0ce00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff90e38d373c20 R15: 000000c02029b000
FS:  000000c000280790(0000) GS:ffff90e6bcfc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000625ff8 CR3: 0000000c1eef2000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffba87d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffba87d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffba9ea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffba9ebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffba9ebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffffba9e8866>] madvise_free_single_vma+0x116/0x180
[<ffffffffba9e97a1>] SyS_madvise+0x4d1/0xac0
[<ffffffffba8e166f>] ? pick_next_task_fair+0x5f/0x1b0
[<ffffffffba8dd425>] ? sched_clock_cpu+0x85/0xc0
[<ffffffffbaf8be21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffbaf8c172>] tracesys+0xa6/0xcc
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 54 bb f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #657280
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffab51761a
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffab47e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffab47e96b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffab5ede36
Build id: 3.10.0-1127.19.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffab5ef8ff
Build id: 3.10.0-1127.19.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffab5efa23
Build id: 3.10.0-1127.19.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffab5ec4e6
Build id: 3.10.0-1127.19.1.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffffab5ed421
Build id: 3.10.0-1127.19.1.el7.x86_64
0x4d1
9
? hrtimer_get_res
vmlinux 0xffffffffab4ca2d0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x50
10
? do_nanosleep
vmlinux 0xffffffffabb84dbb
Build id: 3.10.0-1127.19.1.el7.x86_64
0x5b
11
system_call_fastpath
vmlinux 0xffffffffabb92ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [dockerd:1749]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 sb_edac snd_rawmidi snd_ac97_codec iosf_mbi ppdev crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq aesni_intel lrw gf128mul glue_helper vmw_balloon snd_seq_device ablk_helper snd_pcm cryptd snd_timer joydev snd pcspkr soundcore sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common
crc32c_intel nfit ata_piix libnvdimm mptspi serio_raw libata e1000 scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 1749 Comm: dockerd Kdump: loaded Tainted: G             L ------------ T 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9b21c71f3150 ti: ffff9b23d1eb4000 task.ti: ffff9b23d1eb4000
RIP: 0010:[<ffffffffab51761a>]  [<ffffffffab51761a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9b23d1eb7c70  EFLAGS: 00000202
RAX: 0000000000000007 RBX: 00080000ab47e730 RCX: ffff9b23ed7e06c8
RDX: 0000000000000007 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9b23d1eb7ca8 R08: ffff9b2139ccc400 R09: ffffffffab7851b9
R10: ffff9b23ed79f160 R11: ffffe04f8e809c00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9b23d1eb7c20 R15: 000000c001fa7000
FS:  00007f6be6fed700(0000) GS:ffff9b23ed780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055c43ead2000 CR3: 000000041eaa8000 CR4: 00000000003607e0
Call Trace:
[<ffffffffab47e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffab47e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffab5ede36>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffab5ef8ff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffab5efa23>] tlb_finish_mmu+0x23/0x30
[<ffffffffab5ec4e6>] madvise_free_single_vma+0x116/0x180
[<ffffffffab5ed421>] SyS_madvise+0x4d1/0xab0
[<ffffffffab4ca2d0>] ? hrtimer_get_res+0x50/0x50
[<ffffffffabb84dbb>] ? do_nanosleep+0x5b/0x100
[<ffffffffabb92ed2>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 15 ac f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 7b c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #625263
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8d9170da
Build id: 3.10.0-1160.49.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8d87e258
Build id: 3.10.0-1160.49.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8d87e2cb
Build id: 3.10.0-1160.49.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8d9ee676
Build id: 3.10.0-1160.49.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8d9f013f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8d9f0263
Build id: 3.10.0-1160.49.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffff8d9ecd16
Build id: 3.10.0-1160.49.1.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffff8d9edc51
Build id: 3.10.0-1160.49.1.el7.x86_64
0x4d1
9
? pick_next_task_fair
vmlinux 0xffffffff8d8e9d2e
Build id: 3.10.0-1160.49.1.el7.x86_64
0x61e
10
system_call_fastpath
vmlinux 0xffffffff8df95f92
Build id: 3.10.0-1160.49.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [jemalloc_bg_thd:3716]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac coretemp snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus iosf_mbi snd_seq crc32_pclmul snd_seq_device ghash_clmulni_intel snd_pcm aesni_intel snd_timer lrw gf128mul snd glue_helper ppdev soundcore ablk_helper cryptd pcspkr sg joydev vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm crct10dif_pclmul drm crct10dif_common crc32c_intel ata_piix mptspi libata scsi_transport_spi serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 3716 Comm: jemalloc_bg_thd Kdump: loaded Tainted: G             L ------------   3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
task: ffff914a8404d280 ti: ffff914af26b0000 task.ti: ffff914af26b0000
RIP: 0010:[<ffffffff8d9170da>]  [<ffffffff8d9170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff914af26b3c70  EFLAGS: 00000202
RAX: 0000000000000003 RBX: ffffffff8d87ddd0 RCX: ffff914abe8e0788
RDX: 0000000000000003 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff914af26b3ca8 R08: ffff914abdbab400 R09: ffffffff8db86bf9
R10: ffff914abe89f160 R11: ffffce6b407e4400 R12: 0000000000000286
R13: ffff914af26b3c20 R14: ffffce6b44d12400 R15: 00007fca8444f000
FS:  00007fca7c2b9700(0000) GS:ffff914abe880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000007259e50 CR3: 0000000072500000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8d87e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8d87e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8d9ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8d9f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8d9f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff8d9ecd16>] madvise_free_single_vma+0x116/0x180
[<ffffffff8d9edc51>] SyS_madvise+0x4d1/0xac0
[<ffffffff8d8e9d2e>] ? pick_next_task_fair+0x61e/0x870
[<ffffffff8df95f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 55 8e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #542276
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffaf1150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffaf07d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffaf07d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffaf1ea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffaf1ebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffaf1ebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffaf1e8866
Build id: 3.10.0-1062.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffffaf1e97a1
Build id: 3.10.0-1062.el7.x86_64
0x4d1
9
? system_call_after_swapgs
vmlinux 0xffffffffaf78be21
Build id: 3.10.0-1062.el7.x86_64
0xae
10
tracesys
vmlinux 0xffffffffaf78c172
Build id: 3.10.0-1062.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 24s! [nsqd:3948]
Modules linked in: xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay(T) sunrpc ppdev joydev sg virtio_balloon parport_pc pcspkr parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom virtio_console virtio_scsi virtio_blk virtio_net ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci serio_raw virtio_ring virtio floppy drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 3948 Comm: nsqd Kdump: loaded Tainted: G               ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff8b137a305230 ti: ffff8b10305d8000 task.ti: ffff8b10305d8000
RIP: 0010:[<ffffffffaf1150ca>]  [<ffffffffaf1150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b10305dbc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000af07d1d0 RCX: ffff8b13ade20450
RDX: 0000000000000000 RSI: 0000000000000060 RDI: 0000000000000000
RBP: ffff8b10305dbca8 R08: ffff8b10fa3a4400 R09: ffffffffaf37f629
R10: ffff8b13adf5f160 R11: fffffb39c280c800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8b10305dbc20 R15: 000000c000c93000
FS:  000000c00004e090(0000) GS:ffff8b13adf40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f493035d9e5 CR3: 00000000b0732000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaf07d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaf07d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaf1ea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffaf1ebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaf1ebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffffaf1e8866>] madvise_free_single_vma+0x116/0x180
[<ffffffffaf1e97a1>] SyS_madvise+0x4d1/0xac0
[<ffffffffaf78be21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffaf78c172>] tracesys+0xa6/0xcc
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 d4 af f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #454811
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbab1760a
Build id: 3.10.0-1127.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbaa7e8f8
Build id: 3.10.0-1127.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbaa7e96b
Build id: 3.10.0-1127.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbabeddb6
Build id: 3.10.0-1127.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbabef87f
Build id: 3.10.0-1127.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbabef9a3
Build id: 3.10.0-1127.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffbabec466
Build id: 3.10.0-1127.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffffbabed3a1
Build id: 3.10.0-1127.el7.x86_64
0x4d1
9
? sched_clock_cpu
vmlinux 0xffffffffbaadebe5
Build id: 3.10.0-1127.el7.x86_64
0x85
10
system_call_fastpath
vmlinux 0xffffffffbb192ed2
Build id: 3.10.0-1127.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [postgres_export:112040]
Modules linked in: tcp_lp nls_utf8 isofs fuse nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM tun devlink binfmt_misc overlay(T) ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi iosf_mbi snd_ac97_codec kvm_intel kvm irqbypass crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq
snd_seq_device aesni_intel snd_pcm snd_timer lrw gf128mul glue_helper snd ablk_helper soundcore pcspkr sg cryptd joydev ppdev vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata nfit libnvdimm crct10dif_pclmul crct10dif_common mptspi crc32c_intel e1000 scsi_transport_spi mptscsih serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 112040 Comm: postgres_export Kdump: loaded Tainted: G             L ------------ T 3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff99aff2fe1070 ti: ffff99b1a18d0000 task.ti: ffff99b1a18d0000
RIP: 0010:[<ffffffffbab1760a>]  [<ffffffffbab1760a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff99b1a18d3c70  EFLAGS: 00000202
RAX: 000000000000000d RBX: 0000000000000003 RCX: ffff99b1b5960650
RDX: 000000000000000d RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff99b1a18d3ca8 R08: ffff99b0fc88e400 R09: ffffffffbad85099
R10: ffff99b1b56df160 R11: fffff95f04969c00 R12: 0000000000013378
R13: 000000fcbad84fef R14: 0000000000000282 R15: 0000000000000282
FS:  00007f59ebfff700(0000) GS:ffff99b1b56c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa312d23000 CR3: 000000021c93a000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffbaa7e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbaa7e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbabeddb6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffbabef87f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbabef9a3>] tlb_finish_mmu+0x23/0x30
[<ffffffffbabec466>] madvise_free_single_vma+0x116/0x180
[<ffffffffbabed3a1>] SyS_madvise+0x4d1/0xab0
[<ffffffffbaadebe5>] ? sched_clock_cpu+0x85/0xc0
[<ffffffffbb192ed2>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 1e 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 0f 75 bb f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 7a c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #19110
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9611761a
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9607e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9607e96b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff961ede36
Build id: 3.10.0-1127.19.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff961ef8ff
Build id: 3.10.0-1127.19.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff961efa23
Build id: 3.10.0-1127.19.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffff961ec4e6
Build id: 3.10.0-1127.19.1.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffff961ed421
Build id: 3.10.0-1127.19.1.el7.x86_64
0x4d1
9
system_call_fastpath
vmlinux 0xffffffff96792ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [calico-node:14562]
Modules linked in: bnep btusb btrtl btbcm btintel bluetooth rfkill binfmt_misc xt_set ipt_rpfilter xt_multiport ip_set_hash_ip ip_set_hash_net vxlan ip6_udp_tunnel udp_tunnel xt_comment xt_mark ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport
vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 iosf_mbi snd_rawmidi crc32_pclmul snd_ac97_codec ppdev ghash_clmulni_intel ac97_bus snd_seq snd_seq_device aesni_intel lrw gf128mul snd_pcm glue_helper vmw_balloon ablk_helper cryptd snd_timer snd joydev pcspkr soundcore vmw_vmci sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit drm crct10dif_pclmul crct10dif_common mptspi crc32c_intel ata_piix libnvdimm scsi_transport_spi mptscsih mptbase libata serio_raw e1000 drm_panel_orientation_quirks
CPU: 3 PID: 14562 Comm: calico-node Tainted: G             L ------------ T 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff940243a9b150 ti: ffff940243a38000 task.ti: ffff940243a38000
RIP: 0010:[<ffffffff9611761a>]  [<ffffffff9611761a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff940243a3bc70  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000800009607e730 RCX: ffff9404c02a0650
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff940243a3bca8 R08: ffff9403bb9cb800 R09: ffffffff963851b9
R10: ffff9404c02df160 R11: fffff88c87137000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff940243a3bc20 R15: 000000c000dcf000
FS:  00007fa499b51700(0000) GS:ffff9404c02c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbf9f5c0000 CR3: 00000002270be000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9607e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9607e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff961ede36>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff961ef8ff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff961efa23>] tlb_finish_mmu+0x23/0x30
[<ffffffff961ec4e6>] madvise_free_single_vma+0x116/0x180
[<ffffffff961ed421>] SyS_madvise+0x4d1/0xab0
[<ffffffff96792ed2>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 d5 96 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 7b c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #13808
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa511761a
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa507e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa507e96b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa51ede36
Build id: 3.10.0-1127.19.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa51ef8ff
Build id: 3.10.0-1127.19.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa51efa23
Build id: 3.10.0-1127.19.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffa51ec4e6
Build id: 3.10.0-1127.19.1.el7.x86_64
0x116
8
? ep_poll
vmlinux 0xffffffffa529c39e
Build id: 3.10.0-1127.19.1.el7.x86_64
0x31e
9
SyS_madvise
vmlinux 0xffffffffa51ed421
Build id: 3.10.0-1127.19.1.el7.x86_64
0x4d1
10
? wake_up_state
vmlinux 0xffffffffa50db990
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20
11
system_call_fastpath
vmlinux 0xffffffffa5792ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [telegraf:1595]
Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper vmwgfx cryptd ttm ppdev drm_kms_helper joydev pcspkr vmw_balloon sg
syscopyarea sysfillrect sysimgblt fb_sys_fops parport_pc drm vmw_vmci drm_panel_orientation_quirks parport i2c_piix4 auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic ahci libahci ata_piix crct10dif_pclmul crct10dif_common crc32c_intel nfit serio_raw libata vmxnet3 vmw_pvscsi libnvdimm
CPU: 2 PID: 1595 Comm: telegraf Kdump: loaded Tainted: G               ------------ T 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9d8c33cbc1c0 ti: ffff9d8d2b240000 task.ti: ffff9d8d2b240000
RIP: 0010:[<ffffffffa511761a>]  [<ffffffffa511761a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9d8d2b243c70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a507e730 RCX: ffff9d8d3fc20628
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff9d8d2b243ca8 R08: ffff9d8cfd721c00 R09: ffffffffa53851b9
R10: ffff9d8d3fd1f160 R11: fffffda6c62d6000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9d8d2b243c20 R15: 000000c0010bd000
FS:  00007fe84fc22700(0000) GS:ffff9d8d3fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000560292e42ea0 CR3: 00000000b8a58000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa507e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa507e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa51ede36>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa51ef8ff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa51efa23>] tlb_finish_mmu+0x23/0x30
[<ffffffffa51ec4e6>] madvise_free_single_vma+0x116/0x180
[<ffffffffa529c39e>] ? ep_poll+0x31e/0x360
[<ffffffffa51ed421>] SyS_madvise+0x4d1/0xab0
[<ffffffffa50db990>] ? wake_up_state+0x20/0x20
[<ffffffffa5792ed2>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 d5 a5 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 7b c4 00 eb b7 0f b6 4d cc 4c 89