Info

Permalink
Function
smp_call_function_many
First occurrence
2020-09-26
Last occurrence
2026-10-07
State
NEW
Components
kernel
Quality
-27

Statistics

Operating system Count
CentOS 7 1,179
Architecture Count
x86_64 1,179
Related packages Count
kernel 1,179
 0:3.10.0-1062.12.1.el7 706
 0:3.10.0-1160.el7 225
 0:3.10.0-1160.59.1.el7 127
 0:3.10.0-1160.25.1.el7 20
 0:3.10.0-1160.42.2.el7 16
 0:3.10.0-1160.76.1.el7 16
 0:3.10.0-1160.71.1.el7 15
 0:3.10.0-1160.6.1.el7 12
 0:3.10.0-1160.62.1.el7 11
 0:3.10.0-1062.el7 9
 0:3.10.0-1160.66.1.el7 4
 0:3.10.0-1160.53.1.el7 4
 0:3.10.0-1160.31.1.el7 4
 0:3.10.0-1062.9.1.el7 3
 0:3.10.0-1160.45.1.el7 2
 0:3.10.0-1160.49.1.el7 2
 0:3.10.0-1160.11.1.el7 1
 0:3.10.0-1160.2.2.el7 1
 0:3.10.0-1160.15.2.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #812858
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9fd1720a
Build id: 3.10.0-1160.71.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9fc7e258
Build id: 3.10.0-1160.71.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9fc7e2cb
Build id: 3.10.0-1160.71.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9fdee6c6
Build id: 3.10.0-1160.71.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9fdf018f
Build id: 3.10.0-1160.71.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9fdf02b3
Build id: 3.10.0-1160.71.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff9fdf377a
Build id: 3.10.0-1160.71.1.el7.x86_64
0xea
8
? do_mmap
vmlinux 0xffffffff9fdfd36c
Build id: 3.10.0-1160.71.1.el7.x86_64
0x39c
9
SyS_madvise
vmlinux 0xffffffff9fdedc6d
Build id: 3.10.0-1160.71.1.el7.x86_64
0x49d
10
? __seccomp_filter
vmlinux 0xffffffff9fd573fb
Build id: 3.10.0-1160.71.1.el7.x86_64
0x5b
11
? system_call_after_swapgs
vmlinux 0xffffffffa0399ed5
Build id: 3.10.0-1160.71.1.el7.x86_64
0xa2
12
? system_call_after_swapgs
vmlinux 0xffffffffa0399ec9
Build id: 3.10.0-1160.71.1.el7.x86_64
0x96
13
tracesys
vmlinux 0xffffffffa039a226
Build id: 3.10.0-1160.71.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [nacos-grpc-clie:55956]
Modules linked in: veth binfmt_misc xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) ext4 ppdev mbcache jbd2 pcspkr joydev virtio_balloon sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net net_failover virtio_console virtio_scsi virtio_blk failover bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci floppy virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 55956 Comm: nacos-grpc-clie Kdump: loaded Tainted: G               ------------ T 3.10.0-1160.71.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff8952722e6300 ti: ffff89511d084000 task.ti: ffff89511d084000
RIP: 0010:[<ffffffff9fd1720a>]  [<ffffffff9fd1720a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff89511d087c70  EFLAGS: 00000202
RAX: 0000000000000005 RBX: 000c00009fc7ddd0 RCX: ffff8952acf60428
RDX: 0000000000000005 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff89511d087ca8 R08: ffff894ff5c42c00 R09: ffffffff9ff86a19
R10: ffff8952ace9f160 R11: ffffbf6287382200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff89511d087c20 R15: 0000000000000001
FS:  00007f70e0de5640(0000) GS:ffff8952ace80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1b3a1e3d70 CR3: 00000003f210c000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9fc7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9fc7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9fdee6c6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9fdf018f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9fdf02b3>] tlb_finish_mmu+0x23/0x30
[<ffffffff9fdf377a>] zap_page_range+0xea/0x130
[<ffffffff9fdfd36c>] ? do_mmap+0x39c/0x590
[<ffffffff9fdedc6d>] SyS_madvise+0x49d/0xac0
[<ffffffff9fd573fb>] ? __seccomp_filter+0x5b/0x300
[<ffffffffa0399ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffa0399ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffa039a226>] tracesys+0xa6/0xcc
Code: 48 63 35 9e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 95 a0 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 6d 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #407480
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8251722a
Build id: 3.10.0-1160.62.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8247e258
Build id: 3.10.0-1160.62.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8247e2cb
Build id: 3.10.0-1160.62.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff825ee666
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff825f012f
Build id: 3.10.0-1160.62.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff825f0253
Build id: 3.10.0-1160.62.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff825f371a
Build id: 3.10.0-1160.62.1.el7.x86_64
0xea
8
? wake_up_q
vmlinux 0xffffffff824daef8
Build id: 3.10.0-1160.62.1.el7.x86_64
0x58
9
SyS_madvise
vmlinux 0xffffffff825edc0d
Build id: 3.10.0-1160.62.1.el7.x86_64
0x49d
10
? __seccomp_filter
vmlinux 0xffffffff825573eb
Build id: 3.10.0-1160.62.1.el7.x86_64
0x5b
11
? system_call_after_swapgs
vmlinux 0xffffffff82b99ed5
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa2
12
? system_call_after_swapgs
vmlinux 0xffffffff82b99ec9
Build id: 3.10.0-1160.62.1.el7.x86_64
0x96
13
tracesys
vmlinux 0xffffffff82b9a226
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [portainer:23352]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc vfat fat snd_intel8x0 snd_ac97_codec ac97_bus iTCO_wdt iTCO_vendor_support snd_seq snd_seq_device snd_pcm snd_timer ppdev lpc_ich snd parport_pc soundcore sg video parport i2c_piix4 pcspkr ip_tables xfs libcrc32c sd_mod
crc_t10dif crct10dif_generic sr_mod cdrom ata_generic crct10dif_common pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci virtio_net ata_piix libahci serio_raw net_failover drm failover crc32c_intel libata virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 23352 Comm: portainer Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff933bf572b180 ti: ffff933bf5e44000 task.ti: ffff933bf5e44000
RIP: 0010:[<ffffffff8251722a>]  [<ffffffff8251722a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff933bf5e47c70  EFLAGS: 00010202
RAX: 0000000000000000 RBX: ffff933bf5e47c30 RCX: ffff933c63c20400
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff933bf5e47ca8 R08: ffff9339b5c29800 R09: ffffffff827869d9
R10: ffff933c63c9f160 R11: ffffe2df8ed7f600 R12: ffff933bf5e47c40
R13: ffffffff8278692f R14: ffff933bf5d4cc00 R15: 0000000000000001
FS:  000000c000080090(0000) GS:ffff933c63c80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006e6bd4 CR3: 000000041139c000 CR4: 00000000000006e0
Call Trace:
[<ffffffff8247e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8247e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff825ee666>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff825f012f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff825f0253>] tlb_finish_mmu+0x23/0x30
[<ffffffff825f371a>] zap_page_range+0xea/0x130
[<ffffffff824daef8>] ? wake_up_q+0x58/0x80
[<ffffffff825edc0d>] SyS_madvise+0x49d/0xac0
[<ffffffff825573eb>] ? __seccomp_filter+0x5b/0x300
[<ffffffff82b99ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff82b99ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff82b9a226>] tracesys+0xa6/0xcc
Code: 48 63 35 7e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 15 83 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #349777
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff953170de
Build id: 3.10.0-1160.49.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffff9527e258
Build id: 3.10.0-1160.49.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9527e2cb
Build id: 3.10.0-1160.49.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff953ee676
Build id: 3.10.0-1160.49.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff953f013f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff953f0263
Build id: 3.10.0-1160.49.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff953f372a
Build id: 3.10.0-1160.49.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff953edc1d
Build id: 3.10.0-1160.49.1.el7.x86_64
0x49d
9
? __seccomp_filter
vmlinux 0xffffffff9535729b
Build id: 3.10.0-1160.49.1.el7.x86_64
0x5b
10
? poll_select_copy_remaining
vmlinux 0xffffffff95464924
Build id: 3.10.0-1160.49.1.el7.x86_64
0x144
11
? system_call_after_swapgs
vmlinux 0xffffffff95995ed5
Build id: 3.10.0-1160.49.1.el7.x86_64
0xa2
12
? system_call_after_swapgs
vmlinux 0xffffffff95995ec9
Build id: 3.10.0-1160.49.1.el7.x86_64
0x96
13
tracesys
vmlinux 0xffffffff95996226
Build id: 3.10.0-1160.49.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [grafana-server:5405]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix nfit libata drm libnvdimm crct10dif_pclmul
crct10dif_common crc32c_intel mptspi scsi_transport_spi mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 7 PID: 5405 Comm: grafana-server Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018
task: ffff9db2d4526300 ti: ffff9dafda9d4000 task.ti: ffff9dafda9d4000
RIP: 0010:[<ffffffff953170de>]  [<ffffffff953170de>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff9dafda9d7c70  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffffff9527ddd0 RCX: ffff9db2eee606f0
RDX: 0000000000000001 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff9dafda9d7ca8 R08: ffff9db035c04400 R09: ffffffff95586bf9
R10: ffff9db2eefdf160 R11: fffff954c272ca00 R12: 0000000000000286
R13: ffff9dafda9d7c20 R14: ffff9db2ed8e9800 R15: 0000000000000001
FS:  00007f201a96e700(0000) GS:ffff9db2eefc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3890401d70 CR3: 000000011a070000 CR4: 00000000000407e0
Call Trace:
[<ffffffff9527e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9527e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff953ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff953f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff953f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff953f372a>] zap_page_range+0xea/0x130
[<ffffffff953edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff9535729b>] ? __seccomp_filter+0x5b/0x300
[<ffffffff95464924>] ? poll_select_copy_remaining+0x144/0x180
[<ffffffff95995ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff95995ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff95996226>] tracesys+0xa6/0xcc
Code: 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 f5 95 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #93568
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa8b16e4e
Build id: 3.10.0-1160.15.2.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffffa8a7e868
Build id: 3.10.0-1160.15.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa8a7e8db
Build id: 3.10.0-1160.15.2.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa8bede86
Build id: 3.10.0-1160.15.2.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa8bef94f
Build id: 3.10.0-1160.15.2.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa8befa73
Build id: 3.10.0-1160.15.2.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffa8bf2f3a
Build id: 3.10.0-1160.15.2.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffa8bed42d
Build id: 3.10.0-1160.15.2.el7.x86_64
0x49d
9
? __seccomp_filter
vmlinux 0xffffffffa8b56ffb
Build id: 3.10.0-1160.15.2.el7.x86_64
0x5b
10
? system_call_after_swapgs
vmlinux 0xffffffffa9194ed5
Build id: 3.10.0-1160.15.2.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffffa9194ec9
Build id: 3.10.0-1160.15.2.el7.x86_64
0x96
12
tracesys
vmlinux 0xffffffffa9195226
Build id: 3.10.0-1160.15.2.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [elasticsearch[n:86179]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev snd_timer snd soundcore vmw_balloon coretemp pcspkr joydev vmw_vmci sg i2c_piix4 parport_pc parport sch_fq ip_tables xfs libcrc32c
sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect crc32c_intel sysimgblt fb_sys_fops ttm ata_piix drm nfit libata mptspi serio_raw scsi_transport_spi libnvdimm e1000 mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 86179 Comm: elasticsearch[n Kdump: loaded Not tainted 3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8bce06be5280 ti: ffff8bd18ea9c000 task.ti: ffff8bd18ea9c000
RIP: 0010:[<ffffffffa8b16e4e>]  [<ffffffffa8b16e4e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8bd18ea9fc70  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00080000a8a7e6a0 RCX: ffff8bd4dd6e0600
RDX: 0000000000000003 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8bd18ea9fca8 R08: ffff8bce32f53c00 R09: ffffffffa8d86479
R10: ffff8bd4dd69f160 R11: ffffee4e8586ec00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8bd18ea9fc20 R15: 0000000000000001
FS:  00007fb0ec7c5700(0000) GS:ffff8bd4dd680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f25de8ba000 CR3: 000000014dbf2000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa8a7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa8a7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa8bede86>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa8bef94f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa8befa73>] tlb_finish_mmu+0x23/0x30
[<ffffffffa8bf2f3a>] zap_page_range+0xea/0x130
[<ffffffffa8bed42d>] SyS_madvise+0x49d/0xac0
[<ffffffffa8b56ffb>] ? __seccomp_filter+0x5b/0x300
[<ffffffffa9194ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffa9194ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffa9195226>] tracesys+0xa6/0xcc
Code: 86 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 75 a9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 ad 86 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #792281
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb0b1722e
Build id: 3.10.0-1160.62.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffffb0a7e258
Build id: 3.10.0-1160.62.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb0a7e2cb
Build id: 3.10.0-1160.62.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb0bee666
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb0bf012f
Build id: 3.10.0-1160.62.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb0bf0253
Build id: 3.10.0-1160.62.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffb0bf371a
Build id: 3.10.0-1160.62.1.el7.x86_64
0xea
8
? wake_up_q
vmlinux 0xffffffffb0adaef8
Build id: 3.10.0-1160.62.1.el7.x86_64
0x58
9
SyS_madvise
vmlinux 0xffffffffb0bedc0d
Build id: 3.10.0-1160.62.1.el7.x86_64
0x49d
10
? wake_up_state
vmlinux 0xffffffffb0adaf40
Build id: 3.10.0-1160.62.1.el7.x86_64
0x20
11
? system_call_after_swapgs
vmlinux 0xffffffffb1199ed5
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa2
12
? system_call_after_swapgs
vmlinux 0xffffffffb1199ec9
Build id: 3.10.0-1160.62.1.el7.x86_64
0x96
13
system_call_fastpath
vmlinux 0xffffffffb1199f92
Build id: 3.10.0-1160.62.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffffb1199ed5
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [calico-node:23623]
Modules linked in: ipt_rpfilter xt_set xt_multiport iptable_raw ip_set_hash_ip vxlan ip_set_hash_net ip6_udp_tunnel udp_tunnel nfsv3 nfs_acl nfs lockd grace fscache veth nf_conntrack_netlink xt_recent xt_statistic xt_nat xt_addrtype ipt_REJECT nf_reject_ipv4 nf_tables ip_set ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6table_mangle ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_filter ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_conntrack xt_comment iptable_filter xt_mark iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon
joydev pcspkr sg i2c_piix4 parport_pc vmw_vmci parport auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm ata_piix libahci nfit libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse br_netfilter bridge stp llc overlay(T)
CPU: 7 PID: 23623 Comm: calico-node Kdump: loaded Tainted: G        W      ------------ T 3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff99f6ac2ab180 ti: ffff99f5410f8000 task.ti: ffff99f5410f8000
RIP: 0010:[<ffffffffb0b1722e>]  [<ffffffffb0b1722e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff99f5410fbc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000b0a7ddd0 RCX: ffff99f533c20728
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff99f5410fbca8 R08: ffff99f3783a9400 R09: ffffffffb0d869d9
R10: ffff99f533ddf160 R11: fffffc8980404800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff99f5410fbc20 R15: 0000000000000001
FS:  00007ff1acff9700(0000) GS:ffff99f533dc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5a86ebefc0 CR3: 00000002d4a7c000 CR4: 00000000000007e0
Call Trace:
[<ffffffffb0a7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb0a7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb0bee666>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb0bf012f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb0bf0253>] tlb_finish_mmu+0x23/0x30
[<ffffffffb0bf371a>] zap_page_range+0xea/0x130
[<ffffffffb0adaef8>] ? wake_up_q+0x58/0x80
[<ffffffffb0bedc0d>] SyS_madvise+0x49d/0xac0
[<ffffffffb0adaf40>] ? wake_up_state+0x20/0x20
[<ffffffffb1199ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb1199ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb1199f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb1199ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 75 b1 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 4d 87 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #536461
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffafb1718e
Build id: 3.10.0-1160.76.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffffafa7e258
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffafa7e2cb
Build id: 3.10.0-1160.76.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffafbee6a6
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffafbf016f
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffafbf0293
Build id: 3.10.0-1160.76.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffafbf375a
Build id: 3.10.0-1160.76.1.el7.x86_64
0xea
8
? wake_up_q
vmlinux 0xffffffffafadb298
Build id: 3.10.0-1160.76.1.el7.x86_64
0x58
9
SyS_madvise
vmlinux 0xffffffffafbedc4d
Build id: 3.10.0-1160.76.1.el7.x86_64
0x49d
10
? system_call_after_swapgs
vmlinux 0xffffffffb0199ed5
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffffb0199ec9
Build id: 3.10.0-1160.76.1.el7.x86_64
0x96
12
system_call_fastpath
vmlinux 0xffffffffb0199f92
Build id: 3.10.0-1160.76.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffffb0199ed5
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kubelet:1494]
Modules linked in: sd_mod crc_t10dif crct10dif_generic crct10dif_common iscsi_tcp libiscsi_tcp libiscsi ib_core xt_NFLOG veth vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch ip_set_hash_net nf_conntrack_netlink ip_set_hash_ipportip ip_set_bitmap_port ip_set_hash_ipportnet ip_set_hash_ipport dummy xt_addrtype xt_physdev xt_state nfnetlink_log nf_tables ip_set_hash_ip xt_set ip_set ip6table_mangle ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_filter ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_conntrack xt_comment iptable_filter xt_mark iptable_nat nf_nat_ipv4 nf_nat scsi_transport_iscsi overlay(T) nf_conntrack_ipv4 nf_defrag_ipv4 br_netfilter bridge stp llc ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack
ext4 mbcache jbd2 ppdev sg virtio_balloon parport_pc parport i2c_piix4 pcspkr joydev ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net net_failover virtio_console virtio_scsi virtio_blk failover cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci virtio_ring virtio floppy drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 1494 Comm: kubelet Kdump: loaded Tainted: G               ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff91f8f5b7c200 ti: ffff9200fb118000 task.ti: ffff9200fb118000
RIP: 0010:[<ffffffffafb1718e>]  [<ffffffffafb1718e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff9200fb11bc70  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00080000afa7ddd0 RCX: ffff9200fcea0450
RDX: 0000000000000002 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9200fb11bca8 R08: ffff91fa2c4d3800 R09: ffffffffafd869f9
R10: ffff9200fcedf160 R11: fffff0d3debe1400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9200fb11bc20 R15: 0000000000000001
FS:  00007febc17fa700(0000) GS:ffff9200fcec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f38e957a000 CR3: 00000008088f4000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffafa7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffafa7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffafbee6a6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffafbf016f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffafbf0293>] tlb_finish_mmu+0x23/0x30
[<ffffffffafbf375a>] zap_page_range+0xea/0x130
[<ffffffffafadb298>] ? wake_up_q+0x58/0x80
[<ffffffffafbedc4d>] SyS_madvise+0x49d/0xac0
[<ffffffffb0199ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb0199ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb0199f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb0199ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 75 b0 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 8d 88 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #430789
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa2b16eea
Build id: 3.10.0-1160.25.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa2a7e868
Build id: 3.10.0-1160.25.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa2a7e8db
Build id: 3.10.0-1160.25.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa2bedf56
Build id: 3.10.0-1160.25.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa2befa1f
Build id: 3.10.0-1160.25.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa2befb43
Build id: 3.10.0-1160.25.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffa2bf300a
Build id: 3.10.0-1160.25.1.el7.x86_64
0xea
8
? wake_up_q
vmlinux 0xffffffffa2adada8
Build id: 3.10.0-1160.25.1.el7.x86_64
0x58
9
SyS_madvise
vmlinux 0xffffffffa2bed4fd
Build id: 3.10.0-1160.25.1.el7.x86_64
0x49d
10
? system_call_after_swapgs
vmlinux 0xffffffffa3195ed5
Build id: 3.10.0-1160.25.1.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffffa3195ec9
Build id: 3.10.0-1160.25.1.el7.x86_64
0x96
12
system_call_fastpath
vmlinux 0xffffffffa3195f92
Build id: 3.10.0-1160.25.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffffa3195ed5
Build id: 3.10.0-1160.25.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [containerd:1356]
<EOI> 
Modules linked in:
tcp_lp nls_utf8 isofs veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vboxsf(OE) sunrpc snd_intel8x0 snd_ac97_codec ppdev ac97_bus hid_multitouch snd_seq snd_seq_device joydev snd_pcm vboxguest(OE) snd_timer sg snd pcspkr iosf_mbi soundcore parport_pc i2c_piix4 parport video ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea ahci sysfillrect sysimgblt fb_sys_fops libahci ata_piix ttm serio_raw
drm e1000 libata crc32c_intel drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 1356 Comm: containerd Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff8824f8fda100 ti: ffff8824f8fe8000 task.ti: ffff8824f8fe8000
RIP: 0010:[<ffffffffa2b16eea>]  [<ffffffffa2b16eea>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8824f8febc70  EFLAGS: 00010202
RAX: 0000000000000000 RBX: 0000000000000010 RCX: ffff8825a0020400
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff8824f8febca8 R08: ffff8824fc77d800 R09: ffffffffa2d86549
R10: ffff8825a009f160 R11: ffffd7bcc6c5bb00 R12: ffffffffffffff10
R13: 0000000000000282 R14: 00000000000000fc R15: 00000000000008fc
FS:  00007fed6679f700(0000) GS:ffff8825a0080000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0007fd000 CR3: 00000000d93a4000 CR4: 00000000000206e0
Call Trace:
[<ffffffffa2a7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa2a7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa2bedf56>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa2befa1f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa2befb43>] tlb_finish_mmu+0x23/0x30
[<ffffffffa2bf300a>] zap_page_range+0xea/0x130
[<ffffffffa2adada8>] ? wake_up_q+0x58/0x80
[<ffffffffa2bed4fd>] SyS_madvise+0x49d/0xac0
[<ffffffffa3195ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffa3195ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffa3195f92>] system_call_fastpath+0x25/0x2a
[<ffffffffa3195ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 fe 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 15 75 a3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #389334
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffaed16c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffaec7e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffaec7e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffaededd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffaedef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffaedef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffaedf2dba
Build id: 3.10.0-1160.el7.x86_64
0xea
8
? wake_up_q
vmlinux 0xffffffffaecdaaa8
Build id: 3.10.0-1160.el7.x86_64
0x58
9
SyS_madvise
vmlinux 0xffffffffaeded2ad
Build id: 3.10.0-1160.el7.x86_64
0x49d
10
? wake_up_state
vmlinux 0xffffffffaecdaaf0
Build id: 3.10.0-1160.el7.x86_64
0x20
11
? system_call_after_swapgs
vmlinux 0xffffffffaf393ed5
Build id: 3.10.0-1160.el7.x86_64
0xa2
12
? system_call_after_swapgs
vmlinux 0xffffffffaf393ec9
Build id: 3.10.0-1160.el7.x86_64
0x96
13
system_call_fastpath
vmlinux 0xffffffffaf393f92
Build id: 3.10.0-1160.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffffaf393ed5
Build id: 3.10.0-1160.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [docker:3699]
Modules linked in: veth nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) tcp_lp nls_utf8 isofs xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc joydev vboxguest(OE) ppdev sg iosf_mbi parport_pc parport video i2c_piix4 pcspkr ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic crct10dif_common pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm e1000 ata_piix ahci libahci drm_panel_orientation_quirks crc32c_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 3699 Comm: docker Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff989602ac6300 ti: ffff989604a88000 task.ti: ffff989604a88000
RIP: 0010:[<ffffffffaed16c0a>]  [<ffffffffaed16c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff989604a8bc70  EFLAGS: 00010202
RAX: 0000000000000002 RBX: ffff989604a8bc30 RCX: ffff98960e720450
RDX: 0000000000000002 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff989604a8bca8 R08: ffff989531ffa000 R09: ffffffffaef85f99
R10: ffff98960e79f160 R11: ffffdc1403112100 R12: ffff989604a8bc40
R13: ffffffffaef85eef R14: ffff98953fd52c00 R15: 0000000000000001
FS:  00007fa017fff700(0000) GS:ffff98960e780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa01c2b24d8 CR3: 00000000c47ec000 CR4: 00000000000206e0
Call Trace:
[<ffffffffaec7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaec7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaededd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffaedef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaedef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffffaedf2dba>] zap_page_range+0xea/0x130
[<ffffffffaecdaaa8>] ? wake_up_q+0x58/0x80
[<ffffffffaeded2ad>] SyS_madvise+0x49d/0xac0
[<ffffffffaecdaaf0>] ? wake_up_state+0x20/0x20
[<ffffffffaf393ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffaf393ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffaf393f92>] system_call_fastpath+0x25/0x2a
[<ffffffffaf393ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 95 af f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:33:plane-0] flip_done timed out
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:37:crtc-0] flip_done timed out
Complete report #371747
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff913170da
Build id: 3.10.0-1160.59.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9127e258
Build id: 3.10.0-1160.59.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9127e2cb
Build id: 3.10.0-1160.59.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff913ee676
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff913f013f
Build id: 3.10.0-1160.59.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff913f0263
Build id: 3.10.0-1160.59.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff913f372a
Build id: 3.10.0-1160.59.1.el7.x86_64
0xea
8
? wake_up_q
vmlinux 0xffffffff912dada8
Build id: 3.10.0-1160.59.1.el7.x86_64
0x58
9
SyS_madvise
vmlinux 0xffffffff913edc1d
Build id: 3.10.0-1160.59.1.el7.x86_64
0x49d
10
? system_call_after_swapgs
vmlinux 0xffffffff91999ed5
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffff91999ec9
Build id: 3.10.0-1160.59.1.el7.x86_64
0x96
12
system_call_fastpath
vmlinux 0xffffffff91999f92
Build id: 3.10.0-1160.59.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffff91999ed5
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [node_exporter:2086]
Modules linked in: tcp_lp vboxsf(OE) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter tun sunrpc snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer joydev snd ppdev pcspkr vboxguest(OE) sg soundcore parport_pc i2c_piix4 parport video ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ata_piix ttm libahci crc32c_intel drm serio_raw libata e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 2086 Comm: node_exporter Kdump: loaded Tainted: G           OE  ------------   3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff96cb90ef3180 ti: ffff96cb90e10000 task.ti: ffff96cb90e10000
RIP: 0010:[<ffffffff913170da>]  [<ffffffff913170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff96cb90e13c70  EFLAGS: 00010202
RAX: 0000000000000000 RBX: ffff96cb90e13c00 RCX: ffff96cb9fc20400
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff96cb90e13ca8 R08: ffff96c9f82f9800 R09: ffffffff91586c09
R10: ffff96cb9fc9f160 R11: ffffeebb4c42a900 R12: 0000000000000010
R13: ffffffff91264ce3 R14: ffffffffffffff10 R15: 0000000000000293
FS:  000000c000428090(0000) GS:ffff96cb9fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f58b9b42000 CR3: 00000000db036000 CR4: 00000000000206e0
Call Trace:
[<ffffffff9127e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9127e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff913ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff913f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff913f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff913f372a>] zap_page_range+0xea/0x130
[<ffffffff912dada8>] ? wake_up_q+0x58/0x80
[<ffffffff913edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff91999ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff91999ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff91999f92>] system_call_fastpath+0x25/0x2a
[<ffffffff91999ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 f5 91 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #367846
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff82f16c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff82e7e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff82e7e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff82fedd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff82fef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff82fef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff82ff2dba
Build id: 3.10.0-1160.el7.x86_64
0xea
8
? wake_up_q
vmlinux 0xffffffff82edaaa8
Build id: 3.10.0-1160.el7.x86_64
0x58
9
SyS_madvise
vmlinux 0xffffffff82fed2ad
Build id: 3.10.0-1160.el7.x86_64
0x49d
10
? system_call_after_swapgs
vmlinux 0xffffffff83593ed5
Build id: 3.10.0-1160.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffff83593ec9
Build id: 3.10.0-1160.el7.x86_64
0x96
12
system_call_fastpath
vmlinux 0xffffffff83593f92
Build id: 3.10.0-1160.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffff83593ed5
Build id: 3.10.0-1160.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 66s! [containerd:1285]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) sunrpc snd_hda_codec_idt snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm ppdev snd_timer snd pcspkr parport_pc soundcore sg joydev i2c_piix4 parport video ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ahci libahci serio_raw libata e1000 crc32c_intel dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 1285 Comm: containerd Kdump: loaded Tainted: G        W      ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff96ce98662100 ti: ffff96cf581f0000 task.ti: ffff96cf581f0000
RIP: 0010:[<ffffffff82f16c0a>]  [<ffffffff82f16c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff96cf581f3c70  EFLAGS: 00010202
RAX: 0000000000000000 RBX: ffffffff82e7e6a0 RCX: ffff96cf5f420428
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff96cf581f3ca8 R08: ffff96cf343bdc00 R09: ffffffff83185f99
R10: ffff96cf5f51f160 R11: ffffdf2d45075200 R12: ffff96cf581f3c40
R13: ffffffff83185eef R14: ffff96ce9877a800 R15: 0000000000000001
FS:  00007ff6ae61c700(0000) GS:ffff96cf5f500000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0007ff000 CR3: 000000019890a000 CR4: 00000000000006e0
Call Trace:
[<ffffffff82e7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff82e7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff82fedd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff82fef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff82fef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff82ff2dba>] zap_page_range+0xea/0x130
[<ffffffff82edaaa8>] ? wake_up_q+0x58/0x80
[<ffffffff82fed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffff83593ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff83593ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff83593f92>] system_call_fastpath+0x25/0x2a
[<ffffffff83593ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 b5 83 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #212765
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb3916bda
Build id: 3.10.0-1160.31.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb387e258
Build id: 3.10.0-1160.31.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb387e2cb
Build id: 3.10.0-1160.31.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb39ee176
Build id: 3.10.0-1160.31.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb39efc3f
Build id: 3.10.0-1160.31.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb39efd63
Build id: 3.10.0-1160.31.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffb39f322a
Build id: 3.10.0-1160.31.1.el7.x86_64
0xea
8
? wake_up_q
vmlinux 0xffffffffb38dad78
Build id: 3.10.0-1160.31.1.el7.x86_64
0x58
9
SyS_madvise
vmlinux 0xffffffffb39ed71d
Build id: 3.10.0-1160.31.1.el7.x86_64
0x49d
10
? wake_up_state
vmlinux 0xffffffffb38dadc0
Build id: 3.10.0-1160.31.1.el7.x86_64
0x20
11
? system_call_after_swapgs
vmlinux 0xffffffffb3f95ed5
Build id: 3.10.0-1160.31.1.el7.x86_64
0xa2
12
? system_call_after_swapgs
vmlinux 0xffffffffb3f95ec9
Build id: 3.10.0-1160.31.1.el7.x86_64
0x96
13
system_call_fastpath
vmlinux 0xffffffffb3f95f92
Build id: 3.10.0-1160.31.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffffb3f95ed5
Build id: 3.10.0-1160.31.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [nordvpnd:44767]
Modules linked in: binfmt_misc tun xt_hl ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm snd_timer sg snd pcspkr vmw_balloon soundcore joydev i2c_piix4 vmw_vmci parport_pc parport
ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit drm ata_piix mptspi libnvdimm scsi_transport_spi mptscsih libata serio_raw e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 4 PID: 44767 Comm: nordvpnd Tainted: G             L ------------   3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff9b78af960000 ti: ffff9b75029f4000 task.ti: ffff9b75029f4000
RIP: 0010:[<ffffffffb3916bda>]  [<ffffffffb3916bda>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9b75029f7c70  EFLAGS: 00000202
RAX: 0000000000000009 RBX: 00080000b387ddd0 RCX: ffff9b799d860650
RDX: 0000000000000009 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9b75029f7ca8 R08: ffff9b72ec554400 R09: ffffffffb3b866a9
R10: ffff9b799d71f160 R11: ffffdffcdc054e00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9b75029f7c20 R15: 0000000000000001
FS:  00007f32197fa700(0000) GS:ffff9b799d700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc16a26736c CR3: 00000008051b4000 CR4: 00000000000007e0
Call Trace:
[<ffffffffb387e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb387e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb39ee176>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb39efc3f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb39efd63>] tlb_finish_mmu+0x23/0x30
[<ffffffffb39f322a>] zap_page_range+0xea/0x130
[<ffffffffb38dad78>] ? wake_up_q+0x58/0x80
[<ffffffffb39ed71d>] SyS_madvise+0x49d/0xac0
[<ffffffffb38dadc0>] ? wake_up_state+0x20/0x20
[<ffffffffb3f95ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb3f95ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb3f95f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb3f95ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 ce 8a c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 55 b4 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 8a c4 00 eb b7 0f b6 4d cc 4c 89 
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:33:plane-0] flip_done timed out
Complete report #97751
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbbf16a6a
Build id: 3.10.0-1062.9.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbbe7e478
Build id: 3.10.0-1062.9.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbbe7e4eb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbbfec026
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbbfedaef
Build id: 3.10.0-1062.9.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbbfedc13
Build id: 3.10.0-1062.9.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffbbff091a
Build id: 3.10.0-1062.9.1.el7.x86_64
0xea
8
? futex_wake
vmlinux 0xffffffffbbf11ec0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x90
9
SyS_madvise
vmlinux 0xffffffffbbfeb62d
Build id: 3.10.0-1062.9.1.el7.x86_64
0x49d
10
? system_call_after_swapgs
vmlinux 0xffffffffbc58de21
Build id: 3.10.0-1062.9.1.el7.x86_64
0xae
11
? system_call_after_swapgs
vmlinux 0xffffffffbc58de15
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa2
12
system_call_fastpath
vmlinux 0xffffffffbc58dede
Build id: 3.10.0-1062.9.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffffbc58de21
Build id: 3.10.0-1062.9.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 33s! [mysqld:10169]
Modules linked in: tun ppdev vmw_balloon sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi scsi_transport_spi libata crct10dif_pclmul crct10dif_common mptscsih crc32c_intel serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 10169 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9f395f7d0000 ti: ffff9f39bb424000 task.ti: ffff9f39bb424000
RIP: 0010:[<ffffffffbbf16a6a>]  [<ffffffffbbf16a6a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9f39bb427c70  EFLAGS: 00000202
RAX: 000000000000000b RBX: ffffffffbbe7e2b0 RCX: ffff9f43d54dfc10
RDX: 000000000000000b RSI: 000000000000000c RDI: 0000000000000000
RBP: ffff9f39bb427ca8 R08: ffff9f3abfd58000 R09: ffffffffbc1817e9
R10: ffff9f43d521f160 R11: ffffe605c22b0c00 R12: 0000000000000286
R13: ffff9f39bb427c20 R14: ffff9f43d5200000 R15: ffffffffbbe2b621
FS:  00007f5ba7cc6700(0000) GS:ffff9f43d5200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5d00285030 CR3: 0000000a92e9c000 CR4: 00000000000407f0
Call Trace:
[<ffffffffbbe7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbbe7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbbfec026>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffbbfedaef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbbfedc13>] tlb_finish_mmu+0x23/0x30
[<ffffffffbbff091a>] zap_page_range+0xea/0x130
[<ffffffffbbf11ec0>] ? futex_wake+0x90/0x180
[<ffffffffbbfeb62d>] SyS_madvise+0x49d/0xac0
[<ffffffffbc58de21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffbc58de15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffbc58dede>] system_call_fastpath+0x25/0x2a
[<ffffffffbc58de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 fe 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 b4 bc f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1109867
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbc516c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbc47e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbc47e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbc5edd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbc5ef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbc5ef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffbc5f2dba
Build id: 3.10.0-1160.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffbc5ed2ad
Build id: 3.10.0-1160.el7.x86_64
0x49d
9
? vm_mmap_pgoff
vmlinux 0xffffffffbc5de1bb
Build id: 3.10.0-1160.el7.x86_64
0xeb
10
? system_call_after_swapgs
vmlinux 0xffffffffbcb93ed5
Build id: 3.10.0-1160.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffffbcb93ec9
Build id: 3.10.0-1160.el7.x86_64
0x96
12
system_call_fastpath
vmlinux 0xffffffffbcb93f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [java:75317]
Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter uio_ivshmem(OE) uio gsch(OE) redirfs(OE) sunrpc ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr virtio_balloon sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console net_failover virtio_scsi virtio_blk failover cirrus drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci virtio_ring drm_panel_orientation_quirks virtio floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 6 PID: 75317 Comm: java Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
task: ffff9e9211bbc200 ti: ffff9e97e3750000 task.ti: ffff9e97e3750000
RIP: 0010:[<ffffffffbc516c0a>]  [<ffffffffbc516c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9e97e3753c70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000bc47e6a0 RCX: ffff9e98dce204c8
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9e97e3753ca8 R08: ffff9e9232354400 R09: ffffffffbc785f99
R10: ffff9e98dcf9f160 R11: ffffd76ac5ffa800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9e97e3753c20 R15: 0000000000000001
FS:  00007f835dfe1700(0000) GS:ffff9e98dcf80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5efdeee000 CR3: 0000000172394000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffbc47e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbc47e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbc5edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffbc5ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbc5ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffffbc5f2dba>] zap_page_range+0xea/0x130
[<ffffffffbc5ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffffbc5de1bb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffbcb93ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffbcb93ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffbcb93f92>] system_call_fastpath+0x25/0x2a
Complete report #904440
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb791718a
Build id: 3.10.0-1160.76.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb787e258
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb787e2cb
Build id: 3.10.0-1160.76.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb79ee6a6
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb79f016f
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb79f0293
Build id: 3.10.0-1160.76.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffb79f375a
Build id: 3.10.0-1160.76.1.el7.x86_64
0xea
8
? vm_mmap_pgoff
vmlinux 0xffffffffb79deddb
Build id: 3.10.0-1160.76.1.el7.x86_64
0xeb
9
SyS_madvise
vmlinux 0xffffffffb79edc4d
Build id: 3.10.0-1160.76.1.el7.x86_64
0x49d
10
? system_call_after_swapgs
vmlinux 0xffffffffb7f99ed5
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffffb7f99ec9
Build id: 3.10.0-1160.76.1.el7.x86_64
0x96
12
tracesys
vmlinux 0xffffffffb7f9a226
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [java:16097]
Modules linked in: ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sisap(POE) sisevt(PE) sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon ppdev vmw_vmci sg joydev i2c_piix4 pcspkr parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic drm nfit ata_piix libnvdimm crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 7 PID: 16097 Comm: java Kdump: loaded Tainted: P           OEL ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9eb57722c200 ti: ffff9eb1550d8000 task.ti: ffff9eb1550d8000
RIP: 0010:[<ffffffffb791718a>]  [<ffffffffb791718a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9eb1550dbc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffffb787ddd0 RCX: ffff9eb743a206f0
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff9eb1550dbca8 R08: ffff9eb2b0990800 R09: ffffffffb7b869f9
R10: ffff9eb743bdf160 R11: fffff0be8400aa00 R12: 0000000000000286
R13: ffff9eb1550dbc20 R14: ffff9eb2bfd5c400 R15: 0000000000000001
FS:  00007fe0acf6f700(0000) GS:ffff9eb743bc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd5e7d7d000 CR3: 00000004ecd88000 CR4: 00000000000407e0
Call Trace:
[<ffffffffb787e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb787e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb79ee6a6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb79f016f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb79f0293>] tlb_finish_mmu+0x23/0x30
[<ffffffffb79f375a>] zap_page_range+0xea/0x130
[<ffffffffb79deddb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffb79edc4d>] SyS_madvise+0x49d/0xac0
[<ffffffffb7f99ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb7f99ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb7f9a226>] tracesys+0xa6/0xcc
Code: 48 63 35 be 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 55 b8 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 8d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #572091
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb8116c5a
Build id: 3.10.0-1160.6.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb807e868
Build id: 3.10.0-1160.6.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb807e8db
Build id: 3.10.0-1160.6.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb81edd46
Build id: 3.10.0-1160.6.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb81ef80f
Build id: 3.10.0-1160.6.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb81ef933
Build id: 3.10.0-1160.6.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffb81f2dfa
Build id: 3.10.0-1160.6.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffb81ed2ed
Build id: 3.10.0-1160.6.1.el7.x86_64
0x49d
9
? vm_mmap_pgoff
vmlinux 0xffffffffb81de1fb
Build id: 3.10.0-1160.6.1.el7.x86_64
0xeb
10
? system_call_after_swapgs
vmlinux 0xffffffffb8794ed5
Build id: 3.10.0-1160.6.1.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffffb8794ec9
Build id: 3.10.0-1160.6.1.el7.x86_64
0x96
12
system_call_fastpath
vmlinux 0xffffffffb8794f92
Build id: 3.10.0-1160.6.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffffb8794ed5
Build id: 3.10.0-1160.6.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [qtp366590980-19:7026]
Modules linked in: iptable_filter xt_multiport sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper sg ppdev ablk_helper cryptd vmw_balloon pcspkr joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm ata_piix crc32c_intel mptspi scsi_transport_spi serio_raw e1000 libata mptscsih vmxnet3 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 7026 Comm: qtp366590980-19 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff8d40f9244200 ti: ffff8d41a59ec000 task.ti: ffff8d41a59ec000
RIP: 0010:[<ffffffffb8116c5a>]  [<ffffffffb8116c5a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8d41a59efc70  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 00080000b807e6a0 RCX: ffff8d42bfda0600
RDX: 0000000000000006 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8d41a59efca8 R08: ffff8d41fd361000 R09: ffffffffb8385fd9
R10: ffff8d42bfc5f160 R11: ffffede80286fa00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8d41a59efc20 R15: 0000000000000001
FS:  00007f4f673ad700(0000) GS:ffff8d42bfc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff9437b4000 CR3: 00000000b5940000 CR4: 00000000000607e0
Call Trace:
[<ffffffffb807e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb807e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb81edd46>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb81ef80f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb81ef933>] tlb_finish_mmu+0x23/0x30
[<ffffffffb81f2dfa>] zap_page_range+0xea/0x130
[<ffffffffb81ed2ed>] SyS_madvise+0x49d/0xac0
[<ffffffffb81de1fb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffb8794ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb8794ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb8794f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb8794ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 d5 b8 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #361372
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff87316a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8727e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8727e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff873ec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff873edb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff873edc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff873f098a
Build id: 3.10.0-1062.12.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff873eb69d
Build id: 3.10.0-1062.12.1.el7.x86_64
0x49d
9
? vm_mmap_pgoff
vmlinux 0xffffffff873ddffb
Build id: 3.10.0-1062.12.1.el7.x86_64
0xeb
10
? system_call_after_swapgs
vmlinux 0xffffffff8798de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
11
? system_call_after_swapgs
vmlinux 0xffffffff8798de15
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa2
12
system_call_fastpath
vmlinux 0xffffffff8798dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffff8798de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [java:149601]
Modules linked in: tcp_lp fuse kbase(OE) xt_NFQUEUE nfnetlink_log nfnetlink nfring(OE) vmsecmod(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc ppdev sg parport_pc pcspkr parport virtio_balloon joydev i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci virtio_ring serio_raw floppy virtio
drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vmsec_nfq]
CPU: 11 PID: 149601 Comm: java Kdump: loaded Tainted: GF          OEL ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9ea84a7fe2a0 ti: ffff9ea7f9780000 task.ti: ffff9ea7f9780000
RIP: 0010:[<ffffffff87316a7a>]  [<ffffffff87316a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9ea7f9783c70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008727e2b0 RCX: ffff9eaf7ce20590
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9ea7f9783ca8 R08: ffff9ea0e95dd400 R09: ffffffff87581839
R10: ffff9eaf7d0df160 R11: ffffdd7de275ba00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9ea7f9783c20 R15: 0000000000000001
FS:  00007f2de1c72700(0000) GS:ffff9eaf7d0c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2de1c5e688 CR3: 000000070ab94000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8727e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8727e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff873ec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff873edb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff873edc83>] tlb_finish_mmu+0x23/0x30
[<ffffffff873f098a>] zap_page_range+0xea/0x130
[<ffffffff873eb69d>] SyS_madvise+0x49d/0xac0
[<ffffffff873ddffb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff8798de21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff8798de15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffff8798dede>] system_call_fastpath+0x25/0x2a
[<ffffffff8798de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 f4 87 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #42711
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8c716a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8c67e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8c67e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8c7ec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8c7edb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8c7edc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff8c7f098a
Build id: 3.10.0-1062.12.1.el7.x86_64
0xea
8
? futex_wake
vmlinux 0xffffffff8c711ed0
Build id: 3.10.0-1062.12.1.el7.x86_64
0x90
9
SyS_madvise
vmlinux 0xffffffff8c7eb69d
Build id: 3.10.0-1062.12.1.el7.x86_64
0x49d
10
? vm_mmap_pgoff
vmlinux 0xffffffff8c7ddffb
Build id: 3.10.0-1062.12.1.el7.x86_64
0xeb
11
? system_call_after_swapgs
vmlinux 0xffffffff8cd8de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
12
? system_call_after_swapgs
vmlinux 0xffffffff8cd8de15
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa2
13
system_call_fastpath
vmlinux 0xffffffff8cd8dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffff8cd8de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 21s! [Messages View G:31136]
Modules linked in: tcp_diag udp_diag inet_diag tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack fuse nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_set ip_set nfnetlink iptable_filter sunrpc snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore ppdev sg i2c_piix4 virtio_balloon pcspkr joydev parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common virtio_net virtio_blk ata_generic pata_acpi cirrus drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata virtio_pci floppy virtio_ring serio_raw virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 31136 Comm: Messages View G Kdump: loaded Tainted: G             L ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff9a13ce361070 ti: ffff9a13d6378000 task.ti: ffff9a13d6378000
RIP: 0010:[<ffffffff8c716a7a>]  [<ffffffff8c716a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9a13d637bc70  EFLAGS: 00000202
RAX: 0000000000000007 RBX: ffffffff8c67e2b0 RCX: ffff9a144b9e0450
RDX: 0000000000000007 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff9a13d637bca8 R08: ffff9a03e1a1f400 R09: ffffffff8c981839
R10: ffff9a144b8df160 R11: ffffdac41a89ac00 R12: ffff9a13d637bc40
R13: ffffffff8c98178f R14: ffff9a03ffd56000 R15: 0000000000000001
FS:  00007fdf410ff700(0000) GS:ffff9a144b8c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000013eddf000 CR3: 00000008342cc000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8c67e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8c67e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8c7ec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8c7edb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8c7edc83>] tlb_finish_mmu+0x23/0x30
[<ffffffff8c7f098a>] zap_page_range+0xea/0x130
[<ffffffff8c711ed0>] ? futex_wake+0x90/0x180
[<ffffffff8c7eb69d>] SyS_madvise+0x49d/0xac0
[<ffffffff8c7ddffb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff8cd8de21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff8cd8de15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffff8cd8dede>] system_call_fastpath+0x25/0x2a
[<ffffffff8cd8de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 34 8d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #16602
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9f716c0a
Build id: 3.10.0-1160.2.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9f67e868
Build id: 3.10.0-1160.2.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9f67e8db
Build id: 3.10.0-1160.2.2.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9f7edd06
Build id: 3.10.0-1160.2.2.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9f7ef7cf
Build id: 3.10.0-1160.2.2.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9f7ef8f3
Build id: 3.10.0-1160.2.2.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff9f7f2dba
Build id: 3.10.0-1160.2.2.el7.x86_64
0xea
8
? do_mmap
vmlinux 0xffffffff9f7fc9d8
Build id: 3.10.0-1160.2.2.el7.x86_64
0x378
9
SyS_madvise
vmlinux 0xffffffff9f7ed2ad
Build id: 3.10.0-1160.2.2.el7.x86_64
0x49d
10
? vm_mmap_pgoff
vmlinux 0xffffffff9f7de1bb
Build id: 3.10.0-1160.2.2.el7.x86_64
0xeb
11
? system_call_after_swapgs
vmlinux 0xffffffff9fd93ed5
Build id: 3.10.0-1160.2.2.el7.x86_64
0xa2
12
? system_call_after_swapgs
vmlinux 0xffffffff9fd93ec9
Build id: 3.10.0-1160.2.2.el7.x86_64
0x96
13
system_call_fastpath
vmlinux 0xffffffff9fd93f92
Build id: 3.10.0-1160.2.2.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffff9fd93ed5
Build id: 3.10.0-1160.2.2.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 25s! [qtp366590980-12:26965]
Modules linked in: veeamsnap(OE) sunrpc dm_mirror dm_region_hash dm_log dm_mod pcspkr joydev i2c_piix4 sg virtio_balloon ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi virtio_net net_failover virtio_console failover virtio_scsi qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy
CPU: 3 PID: 26965 Comm: qtp366590980-12 Tainted: G           OEL ------------   3.10.0-1160.2.2.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-48-gd9c812dda519-prebuilt.qemu.org 04/01/2014
task: ffff95b4bdf60000 ti: ffff95b50cba4000 task.ti: ffff95b50cba4000
RIP: 0010:[<ffffffff9f716c0a>]  [<ffffffff9f716c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff95b50cba7c70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800009f67e6a0 RCX: ffff95b57fc20450
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff95b50cba7ca8 R08: ffff95b4bcb7e000 R09: ffffffff9f985f99
R10: ffff95b57fd9f160 R11: fffff449c8d53700 R12: 0000000000000282
R13: 0000000000000282 R14: ffff95b50cba7c20 R15: 0000000000000001
FS:  00007fdde4313700(0000) GS:ffff95b57fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fdde42fea48 CR3: 00000000a7f6a000 CR4: 00000000000006e0
Call Trace:
[<ffffffff9f67e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9f67e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9f7edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9f7ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9f7ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff9f7f2dba>] zap_page_range+0xea/0x130
[<ffffffff9f7fc9d8>] ? do_mmap+0x378/0x530
[<ffffffff9f7ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffff9f7de1bb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff9fd93ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff9fd93ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff9fd93f92>] system_call_fastpath+0x25/0x2a
[<ffffffff9fd93ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 35 a0 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #12271
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb7b16a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb7a7e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb7a7e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb7bec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb7bedb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb7bedc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffb7bf098a
Build id: 3.10.0-1062.12.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffb7beb69d
Build id: 3.10.0-1062.12.1.el7.x86_64
0x49d
9
? vm_mmap_pgoff
vmlinux 0xffffffffb7bddffb
Build id: 3.10.0-1062.12.1.el7.x86_64
0xeb
10
? system_call_after_swapgs
vmlinux 0xffffffffb818de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
11
? system_call_after_swapgs
vmlinux 0xffffffffb818de15
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa2
12
system_call_fastpath
vmlinux 0xffffffffb818dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffffb818de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [java:42012]
Modules linked in: xt_NFQUEUE nfnetlink_log nfnetlink nfring(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc ppdev sg virtio_balloon joydev i2c_piix4 pcspkr parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata virtio_pci serio_raw virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 42012 Comm: java Kdump: loaded Tainted: G           OE  ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff98587fe162a0 ti: ffff98589864c000 task.ti: ffff98589864c000
RIP: 0010:[<ffffffffb7b16a7a>]  [<ffffffffb7b16a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff98589864fc70  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00080000b7a7e2b0 RCX: ffff985e5cf204f0
RDX: 0000000000000004 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff98589864fca8 R08: ffff9857b44acc00 R09: ffffffffb7d81839
R10: ffff985e5cfdf160 R11: ffffd77a46be1400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff98589864fc20 R15: 0000000000000001
FS:  00007f4354080700(0000) GS:ffff985e5cfc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f435406c798 CR3: 00000007edbc4000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb7a7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb7a7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb7bec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb7bedb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb7bedc83>] tlb_finish_mmu+0x23/0x30
[<ffffffffb7bf098a>] zap_page_range+0xea/0x130
[<ffffffffb7beb69d>] SyS_madvise+0x49d/0xac0
[<ffffffffb7bddffb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffb818de21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffb818de15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffb818dede>] system_call_fastpath+0x25/0x2a
[<ffffffffb818de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 74 b8 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #5745
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9e116a6a
Build id: 3.10.0-1062.9.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9e07e478
Build id: 3.10.0-1062.9.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9e07e4eb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9e1ec026
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9e1edaef
Build id: 3.10.0-1062.9.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9e1edc13
Build id: 3.10.0-1062.9.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff9e1f091a
Build id: 3.10.0-1062.9.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff9e1eb62d
Build id: 3.10.0-1062.9.1.el7.x86_64
0x49d
9
? vm_mmap_pgoff
vmlinux 0xffffffff9e1ddfdb
Build id: 3.10.0-1062.9.1.el7.x86_64
0xeb
10
? system_call_after_swapgs
vmlinux 0xffffffff9e78de21
Build id: 3.10.0-1062.9.1.el7.x86_64
0xae
11
? system_call_after_swapgs
vmlinux 0xffffffff9e78de15
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa2
12
system_call_fastpath
vmlinux 0xffffffff9e78dede
Build id: 3.10.0-1062.9.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffff9e78de21
Build id: 3.10.0-1062.9.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [qtp1225197672-2:1500]
Modules linked in: nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dm_mirror dm_region_hash dm_log dm_mod ppdev aesni_intel lrw gf128mul sg glue_helper ablk_helper cryptd pcspkr joydev i2c_piix4 virtio_balloon parport_pc parport ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_common sr_mod cdrom virtio_net virtio_console virtio_scsi
ata_generic pata_acpi bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm ata_piix libata drm_panel_orientation_quirks virtio_pci serio_raw virtio_ring virtio floppy
CPU: 2 PID: 1500 Comm: qtp1225197672-2 Tainted: G             L ------------   3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014
task: ffff9706f161b150 ti: ffff970a9dcb0000 task.ti: ffff970a9dcb0000
RIP: 0010:[<ffffffff9e116a6a>]  [<ffffffff9e116a6a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff970a9dcb3c70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800009e07e2b0 RCX: ffff97055e020428
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff970a9dcb3ca8 R08: ffff97057a80b800 R09: ffffffff9e3817e9
R10: ffff97057fc1f160 R11: fffffd88cabc0d00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff970a9dcb3c20 R15: 0000000000000001
FS:  00007fb77ad91700(0000) GS:ffff97057fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000f1af2920 CR3: 00000002d669a000 CR4: 00000000000006e0
Call Trace:
[<ffffffff9e07e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9e07e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9e1ec026>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9e1edaef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9e1edc13>] tlb_finish_mmu+0x23/0x30
[<ffffffff9e1f091a>] zap_page_range+0xea/0x130
[<ffffffff9e1eb62d>] SyS_madvise+0x49d/0xac0
[<ffffffff9e1ddfdb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff9e78de21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff9e78de15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffff9e78dede>] system_call_fastpath+0x25/0x2a
[<ffffffff9e78de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 fe 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 d4 9e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #2387
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbdb16a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbda7e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbda7e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbdbec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbdbedb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbdbedc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffbdbf098a
Build id: 3.10.0-1062.12.1.el7.x86_64
0xea
8
? do_mmap
vmlinux 0xffffffffbdbfad78
Build id: 3.10.0-1062.12.1.el7.x86_64
0x378
9
SyS_madvise
vmlinux 0xffffffffbdbeb69d
Build id: 3.10.0-1062.12.1.el7.x86_64
0x49d
10
? vm_mmap_pgoff
vmlinux 0xffffffffbdbddffb
Build id: 3.10.0-1062.12.1.el7.x86_64
0xeb
11
? system_call_after_swapgs
vmlinux 0xffffffffbe18de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
12
? system_call_after_swapgs
vmlinux 0xffffffffbe18de15
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa2
13
system_call_fastpath
vmlinux 0xffffffffbe18dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffffbe18de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [java:18290]
Modules linked in: arc4 md4 nls_utf8 cifs ccm dns_resolver binfmt_misc ext4 mbcache jbd2 ppdev sg i2c_piix4 parport_pc virtio_balloon joydev parport pcspkr ip_tables xfs libcrc32c sr_mod sd_mod crc_t10dif cdrom crct10dif_generic crct10dif_common ata_generic pata_acpi virtio_net virtio_console bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 18290 Comm: java Kdump: loaded Tainted: G             L ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014
task: ffff9f41f5fda0e0 ti: ffff9f45e263c000 task.ti: ffff9f45e263c000
RIP: 0010:[<ffffffffbdb16a7a>]  [<ffffffffbdb16a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9f45e263fc70  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000c0000bda7e2b0 RCX: ffff9f45ffda0400
RDX: 0000000000000003 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff9f45e263fca8 R08: ffff9f433a30cc00 R09: ffffffffbdd81839
R10: ffff9f45ffc9f160 R11: ffffead850058300 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9f45e263fc20 R15: 0000000000000001
FS:  00007f6a45da6700(0000) GS:ffff9f45ffc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fcc7f5ad000 CR3: 000000041e64e000 CR4: 00000000000006e0
Call Trace:
[<ffffffffbda7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbda7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbdbec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffbdbedb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbdbedc83>] tlb_finish_mmu+0x23/0x30
[<ffffffffbdbf098a>] zap_page_range+0xea/0x130
[<ffffffffbdbfad78>] ? do_mmap+0x378/0x530
[<ffffffffbdbeb69d>] SyS_madvise+0x49d/0xac0
[<ffffffffbdbddffb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffbe18de21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffbe18de15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffbe18dede>] system_call_fastpath+0x25/0x2a
[<ffffffffbe18de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 74 be f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #534898
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa5f170de
Build id: 3.10.0-1160.53.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffffa5e7e258
Build id: 3.10.0-1160.53.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa5e7e2cb
Build id: 3.10.0-1160.53.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa5fee676
Build id: 3.10.0-1160.53.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa5ff013f
Build id: 3.10.0-1160.53.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa5ff0263
Build id: 3.10.0-1160.53.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffa5ff372a
Build id: 3.10.0-1160.53.1.el7.x86_64
0xea
8
? handle_mm_fault
vmlinux 0xffffffffa5ff66a2
Build id: 3.10.0-1160.53.1.el7.x86_64
0xb22
9
? do_mmap
vmlinux 0xffffffffa5ffd36c
Build id: 3.10.0-1160.53.1.el7.x86_64
0x39c
10
SyS_madvise
vmlinux 0xffffffffa5fedc1d
Build id: 3.10.0-1160.53.1.el7.x86_64
0x49d
11
? system_call_after_swapgs
vmlinux 0xffffffffa6595ed5
Build id: 3.10.0-1160.53.1.el7.x86_64
0xa2
12
? system_call_after_swapgs
vmlinux 0xffffffffa6595ec9
Build id: 3.10.0-1160.53.1.el7.x86_64
0x96
13
system_call_fastpath
vmlinux 0xffffffffa6595f92
Build id: 3.10.0-1160.53.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffffa6595ed5
Build id: 3.10.0-1160.53.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 45s! [qtp2076287037-4:7949]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr vmw_balloon joydev sg parport_pc parport vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic
pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix nfit crct10dif_pclmul libata crct10dif_common libnvdimm mptspi e1000 crc32c_intel scsi_transport_spi mptscsih serio_raw mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 7949 Comm: qtp2076287037-4 Not tainted 3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8a5fb08f5280 ti: ffff8a60712fc000 task.ti: ffff8a60712fc000
RIP: 0010:[<ffffffffa5f170de>]  [<ffffffffa5f170de>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8a60712ffc70  EFLAGS: 00000202
RAX: 0000000000000006 RBX: ffffffffa5e7ddd0 RCX: ffff8a622f59fc10
RDX: 0000000000000006 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8a60712ffca8 R08: ffff8a60bfd55000 R09: ffffffffa6186bf9
R10: ffff8a622f41f160 R11: ffffef67c8453e00 R12: ffff8a60712ffc40
R13: ffffffffa6186b4f R14: ffff8a60bfd55c00 R15: 0000000000000001
FS:  00007fb954882700(0000) GS:ffff8a622f400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb9786d52b0 CR3: 00000000a74f0000 CR4: 00000000000407f0
Call Trace:
[<ffffffffa5e7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa5e7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa5fee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa5ff013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa5ff0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffa5ff372a>] zap_page_range+0xea/0x130
[<ffffffffa5ff66a2>] ? handle_mm_fault+0xb22/0xfb0
[<ffffffffa5ffd36c>] ? do_mmap+0x39c/0x590
[<ffffffffa5fedc1d>] SyS_madvise+0x49d/0xac0
[<ffffffffa6595ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffa6595ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffa6595f92>] system_call_fastpath+0x25/0x2a
[<ffffffffa6595ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 b5 a6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee