Info

Permalink
Function
generic_exec_single
First occurrence
2020-09-25
Last occurrence
2020-09-25
State
NEW
Components
kernel
Quality
-40

Statistics

Operating system Count
CentOS 7 652
Architecture Count
x86_64 652
Related packages Count
kernel 652
 0:3.10.0-862.el7 332
 0:3.10.0-1127.el7 91
 0:3.10.0-1160.81.1.el7 52
 0:3.10.0-1160.el7 35
 0:3.10.0-1160.25.1.el7 30
 0:3.10.0-1160.41.1.el7 13
 0:3.10.0-1160.62.1.el7 12
 0:3.10.0-1160.88.1.el7 11
 0:3.10.0-1127.19.1.el7 11
 0:3.10.0-957.el7 11
 0:3.10.0-862.14.4.el7 8
 0:3.10.0-1160.71.1.el7 6
 0:3.10.0-957.1.3.el7 5
 0:3.10.0-1160.108.1.el7 5
 0:3.10.0-1160.15.2.el7 5
 0:3.10.0-1160.76.1.el7 3
 0:3.10.0-1160.45.1.el7 3
 0:3.10.0-1160.59.1.el7 2
 0:3.10.0-1160.6.1.el7 2
 0:3.10.0-1062.18.1.el7 2
 0:3.10.0-693.17.1.el7 2
 0:3.10.0-1062.el7 2
 0:3.10.0-1062.12.1.el7 1
 0:3.10.0-1062.9.1.el7 1
 0:3.10.0-1160.11.1.el7 1
 0:3.10.0-1160.83.1.el7 1
 0:3.10.0-1160.99.1.el7 1
 0:3.10.0-1127.10.1.el7 1
 0:3.10.0-1160.42.2.el7 1
 0:3.10.0-1160.66.1.el7 1
 0:3.10.0-1160.53.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #853902
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9d316ae2
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffff9d27ddd0
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffff9d27ddd0
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffff9d27ddd0
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffff9d316bff
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9d586845
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff9d3171ab
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9d27e258
Build id: 3.10.0-1160.76.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9d27e2cb
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9d3ee6a6
Build id: 3.10.0-1160.76.1.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff9d3f016f
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff9d3f0293
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x23
13
zap_page_range
vmlinux 0xffffffff9d3f375a
Build id: 3.10.0-1160.76.1.el7.x86_64 		0xea
14
SyS_madvise
vmlinux 0xffffffff9d3edc4d
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x49d
15
? __seccomp_filter
vmlinux 0xffffffff9d35737b
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x5b
16
? system_call_after_swapgs
vmlinux 0xffffffff9d999ed5
Build id: 3.10.0-1160.76.1.el7.x86_64 		0xa2
17
? system_call_after_swapgs
vmlinux 0xffffffff9d999ec9
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x96
18
tracesys
vmlinux 0xffffffff9d99a226
Build id: 3.10.0-1160.76.1.el7.x86_64 		0xa6 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [mysqld:7980]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay(T) sunrpc ppdev joydev virtio_balloon sg parport_pc parport i2c_piix4 pcspkr ip_tables xfs libcrc32c virtio_net virtio_blk net_failover virtio_console virtio_scsi failover sr_mod cdrom ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci virtio_ring drm_panel_orientation_quirks floppy virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 7980 Comm: mysqld Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9f5794be4200 ti: ffff9f5387ae4000 task.ti: ffff9f5387ae4000
RIP: 0010:[<ffffffff9d316ae2>]  [<ffffffff9d316ae2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9f5387ae7be0  EFLAGS: 00000202
RAX: 0000000000000060 RBX: ffff9f5387ae7bb0 RCX: 0000000000000020
RDX: 00000000ffffffff RSI: 0000000000000060 RDI: 0000000000000282
RBP: ffff9f5387ae7c30 R08: ffffffff9da146c8 R09: 00003ffffffff000
R10: ffff9f569c327950 R11: 00007fe103df1000 R12: ffff9f5387ae7bc8
R13: ffff9f571a2e2000 R14: ffff9f555cdea4f8 R15: ffffffffc077e4da
FS:  00007fe103e31700(0000) GS:ffff9f57ade00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb8aa853000 CR3: 000000008663e000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9d27ddd0>] ? leave_mm+0x120/0x120
[<ffffffff9d27ddd0>] ? leave_mm+0x120/0x120
[<ffffffff9d27ddd0>] ? leave_mm+0x120/0x120
[<ffffffff9d316bff>] smp_call_function_single+0x5f/0xa0
[<ffffffff9d586845>] ? cpumask_next_and+0x35/0x50
[<ffffffff9d3171ab>] smp_call_function_many+0x22b/0x270
[<ffffffff9d27e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9d27e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9d3ee6a6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9d3f016f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9d3f0293>] tlb_finish_mmu+0x23/0x30
[<ffffffff9d3f375a>] zap_page_range+0xea/0x130
[<ffffffff9d3edc4d>] SyS_madvise+0x49d/0xac0
[<ffffffff9d35737b>] ? __seccomp_filter+0x5b/0x300
[<ffffffff9d999ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff9d999ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff9d99a226>] tracesys+0xa6/0xcc
Code: 89 de 48 03 14 c5 00 19 f5 9d 48 89 df e8 d7 6e 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #211060
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffbbf0779a
Build id: 3.10.0-862.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffffbbe76e50
Build id: 3.10.0-862.el7.x86_64 		0x110
3
? leave_mm
vmlinux 0xffffffffbbe76e50
Build id: 3.10.0-862.el7.x86_64 		0x110
4
? leave_mm
vmlinux 0xffffffffbbe76e50
Build id: 3.10.0-862.el7.x86_64 		0x110
5
smp_call_function_single
vmlinux 0xffffffffbbf078af
Build id: 3.10.0-862.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffbc14a275
Build id: 3.10.0-862.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffbbf07e5b
Build id: 3.10.0-862.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffbbe77018
Build id: 3.10.0-862.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffbbe77088
Build id: 3.10.0-862.el7.x86_64 		0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffffbbfc0497
Build id: 3.10.0-862.el7.x86_64 		0x37
11
tlb_finish_mmu
vmlinux 0xffffffffbbfc1c75
Build id: 3.10.0-862.el7.x86_64 		0x55
12
zap_page_range
vmlinux 0xffffffffbbfc4a02
Build id: 3.10.0-862.el7.x86_64 		0x142
13
SyS_madvise
vmlinux 0xffffffffbbfbfafd
Build id: 3.10.0-862.el7.x86_64 		0x3cd
14
? poll_select_copy_remaining
vmlinux 0xffffffffbc030801
Build id: 3.10.0-862.el7.x86_64 		0x121
15
? system_call_after_swapgs
vmlinux 0xffffffffbc51f721
Build id: 3.10.0-862.el7.x86_64 		0xae
16
? system_call_after_swapgs
vmlinux 0xffffffffbc51f715
Build id: 3.10.0-862.el7.x86_64 		0xa2
17
system_call_fastpath
vmlinux 0xffffffffbc51f7d5
Build id: 3.10.0-862.el7.x86_64 		0x1c
18
? system_call_after_swapgs
vmlinux 0xffffffffbc51f721
Build id: 3.10.0-862.el7.x86_64 		0xae 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [flanneld:6949]
Modules linked in: veth vxlan ip6_udp_tunnel udp_tunnel rpcsec_gss_krb5 auth_rpcgss ip6table_nat nfsv4 nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 dns_resolver nfs lockd grace fscache nf_conntrack_netlink br_netfilter overlay(T) xt_addrtype xt_set xt_mark ip_set_hash_ipportnet ip_set_hash_ipport ip_set_hash_ipportip ip_set_bitmap_port ip_set nfnetlink dummy xt_comment xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper
cryptd pcspkr sg joydev vmw_balloon vmw_vmci parport_pc parport i2c_piix4 shpchp sch_fq_codel ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci mptspi ata_piix scsi_transport_spi libahci drm crct10dif_pclmul crct10dif_common mptscsih libata crc32c_intel serio_raw mptbase vmxnet3 i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 6949 Comm: flanneld Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff909e26b08000 ti: ffff909ddff2c000 task.ti: ffff909ddff2c000
RIP: 0010:[<ffffffffbbf0779a>]  [<ffffffffbbf0779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff909ddff2fbf0  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff909ddff2fbc0 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff909ddff2fc38 R08: ffffffffbc6138e0 R09: ffff909de9b221b0
R10: 0000000000002000 R11: 0000000000000246 R12: ffff909e0ef4ec00
R13: ffff909a35bd8cc0 R14: 00000000f8e12044 R15: ffff909e16b42130
FS:  00007f80ed581700(0000) GS:ffff909e3fc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f7c82d60000 CR3: 00000003ea7c6000 CR4: 00000000000007e0
Call Trace:
[<ffffffffbbe76e50>] ? leave_mm+0x110/0x110
[<ffffffffbbe76e50>] ? leave_mm+0x110/0x110
[<ffffffffbbe76e50>] ? leave_mm+0x110/0x110
[<ffffffffbbf078af>] smp_call_function_single+0x5f/0xa0
[<ffffffffbc14a275>] ? cpumask_next_and+0x35/0x50
[<ffffffffbbf07e5b>] smp_call_function_many+0x22b/0x270
[<ffffffffbbe77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbbe77088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffbbfc0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffbbfc1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffbbfc4a02>] zap_page_range+0x142/0x190
[<ffffffffbbfbfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffffbc030801>] ? poll_select_copy_remaining+0x121/0x150
[<ffffffffbc51f721>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffbc51f715>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffbc51f7d5>] system_call_fastpath+0x1c/0x21
[<ffffffffbc51f721>] ? system_call_after_swapgs+0xae/0x146
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 b3 bc 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #198925
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff87916842
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffff8787e6a0
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffff8787e6a0
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffff8787e6a0
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffff8791695f
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff87b86395
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff87916f0b
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8787e868
Build id: 3.10.0-1160.25.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8787e8db
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff879edf56
Build id: 3.10.0-1160.25.1.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff879efa1f
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff879efb43
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x23
13
zap_page_range
vmlinux 0xffffffff879f300a
Build id: 3.10.0-1160.25.1.el7.x86_64 		0xea
14
? do_mmap
vmlinux 0xffffffff879fcc4c
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x39c
15
SyS_madvise
vmlinux 0xffffffff879ed4fd
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x49d
16
? __seccomp_filter
vmlinux 0xffffffff879570ab
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x5b
17
? system_call_after_swapgs
vmlinux 0xffffffff87f95ed5
Build id: 3.10.0-1160.25.1.el7.x86_64 		0xa2
18
? system_call_after_swapgs
vmlinux 0xffffffff87f95ec9
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x96
19
tracesys
vmlinux 0xffffffff87f96226
Build id: 3.10.0-1160.25.1.el7.x86_64 		0xa6 
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 21s! [hz.ReferenceDat:31769]
Modules linked in: ip_vs_rr xt_ipvs ip_vs xt_REDIRECT nf_nat_redirect vxlan ip6_udp_tunnel udp_tunnel xt_mark xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM tun devlink rpcsec_gss_krb5 cmac auth_rpcgss nfsv4 arc4 md4 nfs lockd nls_utf8 grace cifs fscache ccm dns_resolver ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash
dm_log dm_mod nfit libnvdimm ppdev snd_intel8x0 iosf_mbi crc32_pclmul snd_ac97_codec ac97_bus ghash_clmulni_intel snd_seq snd_seq_device snd_pcm joydev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg snd_timer parport_pc snd parport i2c_piix4 pcspkr video soundcore ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul e1000 crct10dif_common crc32c_intel serio_raw drm_panel_orientation_quirks fuse
CPU: 8 PID: 31769 Comm: hz.ReferenceDat Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff8ae17f832100 ti: ffff8adfbcac4000 task.ti: ffff8adfbcac4000
RIP: 0010:[<ffffffff87916842>]  [<ffffffff87916842>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8adfbcac7be0  EFLAGS: 00000202
RAX: 000000000000000c RBX: ffff8adfbcac7bb0 RCX: 0000000000000830
RDX: 000000000000000c RSI: 000000000000000c RDI: 0000000000000282
RBP: ffff8adfbcac7c30 R08: ffffffff88015340 R09: 00003ffffffff000
R10: ffff8ad4438dca20 R11: 00007f3e190db000 R12: 00000000c57c7c00
R13: ffffd49b5038eaf0 R14: 00007f3e190db000 R15: 0000000100400030
FS:  00007f3e191d7700(0000) GS:ffff8ae398e00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0c24336000 CR3: 0000000167790000 CR4: 00000000000606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8787e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8787e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8787e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8791695f>] smp_call_function_single+0x5f/0xa0
[<ffffffff87b86395>] ? cpumask_next_and+0x35/0x50
[<ffffffff87916f0b>] smp_call_function_many+0x22b/0x270
[<ffffffff8787e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8787e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff879edf56>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff879efa1f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff879efb43>] tlb_finish_mmu+0x23/0x30
[<ffffffff879f300a>] zap_page_range+0xea/0x130
[<ffffffff879fcc4c>] ? do_mmap+0x39c/0x590
[<ffffffff879ed4fd>] SyS_madvise+0x49d/0xac0
[<ffffffff879570ab>] ? __seccomp_filter+0x5b/0x300
[<ffffffff87f95ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff87f95ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff87f96226>] tracesys+0xa6/0xcc
Code: 89 de 48 03 14 c5 a0 15 55 88 48 89 df e8 c7 6c 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #774646
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff85f16a32
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffff85e7ddd0
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffff85e7ddd0
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffff85e7ddd0
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffff85f16b4f
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff86186a45
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff85f170fb
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff85e7e258
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff85e7e2cb
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff85fee676
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff85ff013f
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff85ff0263
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x23
13
zap_page_range
vmlinux 0xffffffff85ff372a
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xea
14
? wake_up_q
vmlinux 0xffffffff85edada8
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x58
15
SyS_madvise
vmlinux 0xffffffff85fedc1d
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x49d
16
? __switch_to
vmlinux 0xffffffff85e2b621
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x151
17
? system_call_after_swapgs
vmlinux 0xffffffff86595ed5
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xa2
18
? system_call_after_swapgs
vmlinux 0xffffffff86595ec9
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x96
19
system_call_fastpath
vmlinux 0xffffffff86595f92
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x25
20
? system_call_after_swapgs
vmlinux 0xffffffff86595ed5
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [mysqld:56338]
Modules linked in: xt_nat ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) binfmt_misc iptable_filter nls_utf8 isofs vmw_vsock_vmci_transport vsock raid0 coretemp crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod ata_piix crc_t10dif crct10dif_generic drm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3
vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 4 PID: 56338 Comm: mysqld Tainted: G             L ------------ T 3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff898f5e9da100 ti: ffff8990e44c0000 task.ti: ffff8990e44c0000
RIP: 0010:[<ffffffff85f16a32>]  [<ffffffff85f16a32>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8990e44c3be0  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff8990e44c3bb0 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff8990e44c3c30 R08: ffffffff86613a40 R09: 00003ffffffff000
R10: ffff898edd8115f0 R11: 0000000000000202 R12: ffff898f5e9da100
R13: 0000000300000048 R14: ffff8990e44c3cc0 R15: ffffffff85fc862f
FS:  00007f3b66fcd700(0000) GS:ffff8991ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd87fff8a08 CR3: 0000000330c12000 CR4: 00000000000007e0
Call Trace:
[<ffffffff85e7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff85e7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff85e7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff85f16b4f>] smp_call_function_single+0x5f/0xa0
[<ffffffff86186a45>] ? cpumask_next_and+0x35/0x50
[<ffffffff85f170fb>] smp_call_function_many+0x22b/0x270
[<ffffffff85e7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff85e7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff85fee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff85ff013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff85ff0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff85ff372a>] zap_page_range+0xea/0x130
[<ffffffff85edada8>] ? wake_up_q+0x58/0x80
[<ffffffff85fedc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff85e2b621>] ? __switch_to+0x151/0x580
[<ffffffff86595ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff86595ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff86595f92>] system_call_fastpath+0x25/0x2a
[<ffffffff86595ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 60 15 b5 86 48 89 df e8 87 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #525601
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff84116b62
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffff8407ddd0
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffff8407ddd0
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffff8407ddd0
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffff84116c7f
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff84386865
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff8411722b
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8407e258
Build id: 3.10.0-1160.71.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8407e2cb
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff841ee6c6
Build id: 3.10.0-1160.71.1.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff841f018f
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff841f02b3
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x23
13
zap_page_range
vmlinux 0xffffffff841f377a
Build id: 3.10.0-1160.71.1.el7.x86_64 		0xea
14
? wake_up_q
vmlinux 0xffffffff840daed8
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x58
15
SyS_madvise
vmlinux 0xffffffff841edc6d
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x49d
16
? system_call_after_swapgs
vmlinux 0xffffffff84799ed5
Build id: 3.10.0-1160.71.1.el7.x86_64 		0xa2
17
? system_call_after_swapgs
vmlinux 0xffffffff84799ec9
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x96
18
system_call_fastpath
vmlinux 0xffffffff84799f92
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x25
19
? system_call_after_swapgs
vmlinux 0xffffffff84799ed5
Build id: 3.10.0-1160.71.1.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [docker:5910]
Modules linked in: vhost_net vhost macvtap macvlan hci_vhci bluetooth vfio_iommu_type1 vfio uinput uhid ppp_generic slhc lp loop btrfs raid6_pq xor veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) tcp_lp tcp_diag inet_diag vboxsf(OE) xt_CHECKSUM iptable_mangle cfg80211 ipt_MASQUERADE nf_nat_masquerade_ipv4 rfkill iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink sunrpc snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev snd_timer snd iosf_mbi joydev vboxguest(OE) sg video pcspkr soundcore parport_pc i2c_piix4 parport sch_fq_codel ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic
crct10dif_common vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci serio_raw libahci drm drm_panel_orientation_quirks libata crc32c_intel e1000 dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 5910 Comm: docker Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-1160.71.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff88e43f3a0000 ti: ffff88e440904000 task.ti: ffff88e440904000
RIP: 0010:[<ffffffff84116b62>]  [<ffffffff84116b62>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff88e440907be0  EFLAGS: 00010202
RAX: 00000000000008fd RBX: ffffffff840e108e RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000286
RBP: ffff88e440907c30 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffff88e595a6ad80 R11: 0000000000000202 R12: 000003804e0d5fe7
R13: ffffffff840e495c R14: ffff88e440907b80 R15: 0000000000000087
FS:  00007fd16d54f700(0000) GS:ffff88e59fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f283f3d99d0 CR3: 00000001f5fc4000 CR4: 00000000000206e0
Call Trace:
[<ffffffff8407ddd0>] ? leave_mm+0x120/0x120
[<ffffffff8407ddd0>] ? leave_mm+0x120/0x120
[<ffffffff8407ddd0>] ? leave_mm+0x120/0x120
[<ffffffff84116c7f>] smp_call_function_single+0x5f/0xa0
[<ffffffff84386865>] ? cpumask_next_and+0x35/0x50
[<ffffffff8411722b>] smp_call_function_many+0x22b/0x270
[<ffffffff8407e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8407e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff841ee6c6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff841f018f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff841f02b3>] tlb_finish_mmu+0x23/0x30
[<ffffffff841f377a>] zap_page_range+0xea/0x130
[<ffffffff840daed8>] ? wake_up_q+0x58/0x80
[<ffffffff841edc6d>] SyS_madvise+0x49d/0xac0
[<ffffffff84799ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff84799ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff84799f92>] system_call_fastpath+0x25/0x2a
[<ffffffff84799ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 60 18 d5 84 48 89 df e8 77 6e 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #432697
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb9116b82
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffffb907ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffffb907ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffffb907ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffffb9116c9f
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb9386825
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffb911724b
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb907e258
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb907e2cb
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb91ee666
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb91f012f
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb91f0253
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x23
13
zap_page_range
vmlinux 0xffffffffb91f371a
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xea
14
? wake_up_q
vmlinux 0xffffffffb90daef8
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x58
15
SyS_madvise
vmlinux 0xffffffffb91edc0d
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x49d
16
? system_call_after_swapgs
vmlinux 0xffffffffb9799ed5
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xa2
17
? system_call_after_swapgs
vmlinux 0xffffffffb9799ec9
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x96
18
system_call_fastpath
vmlinux 0xffffffffb9799f92
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x25
19
? system_call_after_swapgs
vmlinux 0xffffffffb9799ed5
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [docker:17513]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) sunrpc snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev snd_timer snd soundcore sg pcspkr parport_pc iosf_mbi video i2c_piix4 parport ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea ahci sysfillrect sysimgblt fb_sys_fops ttm libahci ata_piix drm serio_raw crc32c_intel drm_panel_orientation_quirks libata e1000
dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 17513 Comm: docker Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff9e0d6bda5280 ti: ffff9e0d83a60000 task.ti: ffff9e0d83a60000
RIP: 0010:[<ffffffffb9116b82>]  [<ffffffffb9116b82>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9e0d83a63be0  EFLAGS: 00010202
RAX: 00000000000008fd RBX: 00000000000000fb RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000286
RBP: ffff9e0d83a63c30 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffff9e0d9313c438 R11: 0000000000000202 R12: 0000000000000000
R13: 00000000000008fd R14: 0000000000000001 R15: 00003ffffffff000
FS:  00007fadebfff700(0000) GS:ffff9e0e1fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000056034ff5e000 CR3: 00000000d75e6000 CR4: 00000000000206f0
Call Trace:
[<ffffffffb907ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb907ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb907ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb9116c9f>] smp_call_function_single+0x5f/0xa0
[<ffffffffb9386825>] ? cpumask_next_and+0x35/0x50
[<ffffffffb911724b>] smp_call_function_many+0x22b/0x270
[<ffffffffb907e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb907e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb91ee666>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb91f012f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb91f0253>] tlb_finish_mmu+0x23/0x30
[<ffffffffb91f371a>] zap_page_range+0xea/0x130
[<ffffffffb90daef8>] ? wake_up_q+0x58/0x80
[<ffffffffb91edc0d>] SyS_madvise+0x49d/0xac0
[<ffffffffb9799ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb9799ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb9799f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb9799ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 60 18 d5 b9 48 89 df e8 17 6e 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #405345
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb2916b82
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffffb287ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffffb287ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffffb287ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffffb2916c9f
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb2b86825
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffb291724b
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb287e258
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb287e2cb
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb29ee666
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb29f012f
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb29f0253
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x23
13
zap_page_range
vmlinux 0xffffffffb29f371a
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xea
14
? wake_up_q
vmlinux 0xffffffffb28daef8
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x58
15
SyS_madvise
vmlinux 0xffffffffb29edc0d
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x49d
16
? wake_up_state
vmlinux 0xffffffffb28daf40
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x20
17
? system_call_after_swapgs
vmlinux 0xffffffffb2f99ed5
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xa2
18
? system_call_after_swapgs
vmlinux 0xffffffffb2f99ec9
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x96
19
tracesys
vmlinux 0xffffffffb2f9a226
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xa6 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [containerd:4049]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) tcp_lp xt_owner xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter sunrpc joydev ppdev parport_pc vboxguest(OE) iosf_mbi i2c_piix4 sg video parport pcspkr binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_common ata_generic pata_acpi vmwgfx virtio_net net_failover failover drm_kms_helper ahci virtio_pci ata_piix libahci syscopyarea sysfillrect sysimgblt virtio_ring fb_sys_fops serio_raw ttm drm crc32c_intel virtio libata e1000 drm_panel_orientation_quirks
dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 4049 Comm: containerd Kdump: loaded Tainted: G        W  OE  ------------ T 3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff89e8b8085280 ti: ffff89e8d4f5c000 task.ti: ffff89e8d4f5c000
RIP: 0010:[<ffffffffb2916b82>]  [<ffffffffb2916b82>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff89e8d4f5fbe0  EFLAGS: 00010202
RAX: 00000000000008fd RBX: ffff89e8d4f5fbd8 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000286
RBP: ffff89e8d4f5fc30 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffff89e8d4dfaaf8 R11: 0000000000000202 R12: ffff89e8d4f5fb60
R13: 0000000000000001 R14: ffff89e92069acc0 R15: ffffffffb28d3d6a
FS:  00007feb118e7700(0000) GS:ffff89e920680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7060785000 CR3: 0000000481a06000 CR4: 00000000000206e0
Call Trace:
[<ffffffffb287ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb287ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb287ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb2916c9f>] smp_call_function_single+0x5f/0xa0
[<ffffffffb2b86825>] ? cpumask_next_and+0x35/0x50
[<ffffffffb291724b>] smp_call_function_many+0x22b/0x270
[<ffffffffb287e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb287e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb29ee666>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb29f012f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb29f0253>] tlb_finish_mmu+0x23/0x30
[<ffffffffb29f371a>] zap_page_range+0xea/0x130
[<ffffffffb28daef8>] ? wake_up_q+0x58/0x80
[<ffffffffb29edc0d>] SyS_madvise+0x49d/0xac0
[<ffffffffb28daf40>] ? wake_up_state+0x20/0x20
[<ffffffffb2f99ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb2f99ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb2f9a226>] tracesys+0xa6/0xcc
Code: 89 de 48 03 14 c5 60 18 55 b3 48 89 df e8 17 6e 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #397331
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb6916b82
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffffb687ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffffb687ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffffb687ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffffb6916c9f
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb6b86825
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffb691724b
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb687e258
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb687e2cb
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb69ee666
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb69f012f
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb69f0253
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x23
13
zap_page_range
vmlinux 0xffffffffb69f371a
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xea
14
? wake_up_q
vmlinux 0xffffffffb68daef8
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x58
15
SyS_madvise
vmlinux 0xffffffffb69edc0d
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x49d
16
? system_call_after_swapgs
vmlinux 0xffffffffb6f99ed5
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xa2
17
? system_call_after_swapgs
vmlinux 0xffffffffb6f99ec9
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x96
18
system_call_fastpath
vmlinux 0xffffffffb6f99f92
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x25
19
? system_call_after_swapgs
vmlinux 0xffffffffb6f99ed5
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [go:6288]
Modules linked in: tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm snd_timer iosf_mbi snd pcspkr soundcore vboxguest(OE) sg i2c_piix4 joydev parport_pc parport video ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common vmwgfx drm_kms_helper ahci syscopyarea
sysfillrect sr_mod sysimgblt cdrom fb_sys_fops ttm ata_generic pata_acpi drm crc32c_intel serio_raw libahci ata_piix e1000 libata drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 6288 Comm: go Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff9946443bb180 ti: ffff9946433e8000 task.ti: ffff9946433e8000
RIP: 0010:[<ffffffffb6916b82>]  [<ffffffffb6916b82>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9946433ebbe0  EFLAGS: 00010202
RAX: 00000000000008fd RBX: ffffffffb68e10ae RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000286
RBP: ffff9946433ebc30 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffff9946435b7e58 R11: 0000000000000202 R12: 0000007c13cfc520
R13: ffffffffb68e497c R14: ffff9946433ebb80 R15: ffff99464439b180
FS:  00007fc1eccd3700(0000) GS:ffff99475fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000174a00e31c20 CR3: 000000000323a000 CR4: 00000000000206e0
Call Trace:
[<ffffffffb687ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb687ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb687ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb6916c9f>] smp_call_function_single+0x5f/0xa0
[<ffffffffb6b86825>] ? cpumask_next_and+0x35/0x50
[<ffffffffb691724b>] smp_call_function_many+0x22b/0x270
[<ffffffffb687e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb687e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb69ee666>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb69f012f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb69f0253>] tlb_finish_mmu+0x23/0x30
[<ffffffffb69f371a>] zap_page_range+0xea/0x130
[<ffffffffb68daef8>] ? wake_up_q+0x58/0x80
[<ffffffffb69edc0d>] SyS_madvise+0x49d/0xac0
[<ffffffffb6f99ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb6f99ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb6f99f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb6f99ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 60 18 55 b7 48 89 df e8 17 6e 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #374937
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb8116a32
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffffb807ddd0
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffffb807ddd0
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffffb807ddd0
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffffb8116b4f
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb8386a55
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffb81170fb
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb807e258
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb807e2cb
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb81ee676
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb81f013f
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb81f0263
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x23
13
zap_page_range
vmlinux 0xffffffffb81f372a
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xea
14
? wake_up_q
vmlinux 0xffffffffb80dada8
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x58
15
SyS_madvise
vmlinux 0xffffffffb81edc1d
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x49d
16
? wake_up_state
vmlinux 0xffffffffb80dadf0
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x20
17
? system_call_after_swapgs
vmlinux 0xffffffffb8799ed5
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xa2
18
? system_call_after_swapgs
vmlinux 0xffffffffb8799ec9
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x96
19
system_call_fastpath
vmlinux 0xffffffffb8799f92
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x25
20
? system_call_after_swapgs
vmlinux 0xffffffffb8799ed5
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [prometheus:770]
Modules linked in: tcp_lp vboxsf(OE) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter tun sunrpc snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer ppdev joydev sg snd parport_pc vboxguest(OE) parport soundcore pcspkr video i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_common ata_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm ata_piix libahci libata drm serio_raw e1000 crc32c_intel drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 770 Comm: prometheus Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff8f6291965280 ti: ffff8f628f5d8000 task.ti: ffff8f628f5d8000
RIP: 0010:[<ffffffffb8116a32>]  [<ffffffffb8116a32>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8f628f5dbbe0  EFLAGS: 00010202
RAX: 00000000000008fd RBX: 0000000000000000 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000286
RBP: ffff8f628f5dbc30 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffff8f5fb5409360 R11: 0000000000000202 R12: 0000000000000002
R13: ffff8f629fd1acc0 R14: ffff8f60f83f1c00 R15: ffff8f60f83f3800
FS:  000000c00006c890(0000) GS:ffff8f629fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f267111f268 CR3: 00000003109be000 CR4: 00000000000206f0
Call Trace:
[<ffffffffb807ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb807ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb807ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb8116b4f>] smp_call_function_single+0x5f/0xa0
[<ffffffffb8386a55>] ? cpumask_next_and+0x35/0x50
[<ffffffffb81170fb>] smp_call_function_many+0x22b/0x270
[<ffffffffb807e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb807e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb81ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb81f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb81f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffb81f372a>] zap_page_range+0xea/0x130
[<ffffffffb80dada8>] ? wake_up_q+0x58/0x80
[<ffffffffb81edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffffb80dadf0>] ? wake_up_state+0x20/0x20
[<ffffffffb8799ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb8799ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb8799f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb8799ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 60 18 d5 b8 48 89 df e8 97 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #366707
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffad916562
Build id: 3.10.0-1160.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffffad87e6a0
Build id: 3.10.0-1160.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffffad87e6a0
Build id: 3.10.0-1160.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffffad87e6a0
Build id: 3.10.0-1160.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffffad91667f
Build id: 3.10.0-1160.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffadb85de5
Build id: 3.10.0-1160.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffad916c2b
Build id: 3.10.0-1160.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffad87e868
Build id: 3.10.0-1160.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffad87e8db
Build id: 3.10.0-1160.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffad9edd06
Build id: 3.10.0-1160.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffad9ef7cf
Build id: 3.10.0-1160.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffad9ef8f3
Build id: 3.10.0-1160.el7.x86_64 		0x23
13
zap_page_range
vmlinux 0xffffffffad9f2dba
Build id: 3.10.0-1160.el7.x86_64 		0xea
14
? wake_up_q
vmlinux 0xffffffffad8daaa8
Build id: 3.10.0-1160.el7.x86_64 		0x58
15
SyS_madvise
vmlinux 0xffffffffad9ed2ad
Build id: 3.10.0-1160.el7.x86_64 		0x49d
16
? wake_up_state
vmlinux 0xffffffffad8daaf0
Build id: 3.10.0-1160.el7.x86_64 		0x20
17
? system_call_after_swapgs
vmlinux 0xffffffffadf93ed5
Build id: 3.10.0-1160.el7.x86_64 		0xa2
18
? system_call_after_swapgs
vmlinux 0xffffffffadf93ec9
Build id: 3.10.0-1160.el7.x86_64 		0x96
19
system_call_fastpath
vmlinux 0xffffffffadf93f92
Build id: 3.10.0-1160.el7.x86_64 		0x25
20
? system_call_after_swapgs
vmlinux 0xffffffffadf93ed5
Build id: 3.10.0-1160.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [dockerd:3005]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc snd_hda_codec_idt snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer ppdev snd parport_pc soundcore joydev sg i2c_piix4 video parport pcspkr ip_tables
xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ahci libahci crc32c_intel libata serio_raw e1000 dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 3005 Comm: dockerd Kdump: loaded Tainted: G               ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff9094c363b180 ti: ffff909598340000 task.ti: ffff909598340000
RIP: 0010:[<ffffffffad916562>]  [<ffffffffad916562>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff909598343be0  EFLAGS: 00010202
RAX: 00000000000008fd RBX: 00000000000000fb RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000286
RBP: ffff909598343c30 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffff9094d5177948 R11: 0000000000000202 R12: 0000000000000000
R13: 00000000000008fd R14: 0000000000000001 R15: 00003ffffffff000
FS:  00007fce567fc700(0000) GS:ffff90959f500000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9c66338e40 CR3: 0000000198130000 CR4: 00000000000006e0
Call Trace:
[<ffffffffad87e6a0>] ? leave_mm+0x120/0x120
[<ffffffffad87e6a0>] ? leave_mm+0x120/0x120
[<ffffffffad87e6a0>] ? leave_mm+0x120/0x120
[<ffffffffad91667f>] smp_call_function_single+0x5f/0xa0
[<ffffffffadb85de5>] ? cpumask_next_and+0x35/0x50
[<ffffffffad916c2b>] smp_call_function_many+0x22b/0x270
[<ffffffffad87e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffad87e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffad9edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffad9ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffad9ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffffad9f2dba>] zap_page_range+0xea/0x130
[<ffffffffad8daaa8>] ? wake_up_q+0x58/0x80
[<ffffffffad9ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffffad8daaf0>] ? wake_up_state+0x20/0x20
[<ffffffffadf93ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffadf93ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffadf93f92>] system_call_fastpath+0x25/0x2a
[<ffffffffadf93ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 e0 13 55 ae 48 89 df e8 f7 69 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #199355
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff85d0779a
Build id: 3.10.0-862.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff85c76e50
Build id: 3.10.0-862.el7.x86_64 		0x110
3
? leave_mm
vmlinux 0xffffffff85c76e50
Build id: 3.10.0-862.el7.x86_64 		0x110
4
? leave_mm
vmlinux 0xffffffff85c76e50
Build id: 3.10.0-862.el7.x86_64 		0x110
5
smp_call_function_single
vmlinux 0xffffffff85d078af
Build id: 3.10.0-862.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff85f4a275
Build id: 3.10.0-862.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff85d07e5b
Build id: 3.10.0-862.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff85c77018
Build id: 3.10.0-862.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff85c77088
Build id: 3.10.0-862.el7.x86_64 		0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffff85dc0497
Build id: 3.10.0-862.el7.x86_64 		0x37
11
tlb_finish_mmu
vmlinux 0xffffffff85dc1c75
Build id: 3.10.0-862.el7.x86_64 		0x55
12
zap_page_range
vmlinux 0xffffffff85dc4a02
Build id: 3.10.0-862.el7.x86_64 		0x142
13
? wake_up_q
vmlinux 0xffffffff85ccee38
Build id: 3.10.0-862.el7.x86_64 		0x58
14
SyS_madvise
vmlinux 0xffffffff85dbfafd
Build id: 3.10.0-862.el7.x86_64 		0x3cd
15
? system_call_after_swapgs
vmlinux 0xffffffff8631f721
Build id: 3.10.0-862.el7.x86_64 		0xae
16
? system_call_after_swapgs
vmlinux 0xffffffff8631f715
Build id: 3.10.0-862.el7.x86_64 		0xa2
17
system_call_fastpath
vmlinux 0xffffffff8631f7d5
Build id: 3.10.0-862.el7.x86_64 		0x1c
18
? system_call_after_swapgs
vmlinux 0xffffffff8631f721
Build id: 3.10.0-862.el7.x86_64 		0xae 
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [xfsaild/dm-0:545]
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [containerd-shim:27711]
Modules linked in: veth vxlan ip6_udp_tunnel udp_tunnel nfsv4 dns_resolver nfs ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 fscache rpcsec_gss_krb5 nf_conntrack_netlink br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype xt_set tun bridge stp llc xt_mark ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_bitmap_port ip_set_hash_ipport ip_set nfnetlink dummy xt_comment iptable_nat nf_nat_ipv4 ebtable_filter ebtables nf_nat ip6table_filter ip6_tables iptable_filter devlink nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack vmw_vsock_vmci_transport vsock crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd sg parport_pc
parport vmw_balloon joydev pcspkr vmw_vmci i2c_piix4 shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc sch_fq_codel ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci ata_piix mptspi i2c_core libata scsi_transport_spi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw mptscsih vmxnet3 mptbase dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 27711 Comm: containerd-shim Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff949967f9eeb0 ti: ffff9498af738000 task.ti: ffff9498af738000
RIP: 0010:[<ffffffff85d0779a>]  [<ffffffff85d0779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9498af73bbf0  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff9498af73bbc0 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff9498af73bc38 R08: ffffffff86413b60 R09: ffffdf82d0948340
R10: ffffdf82d0948340 R11: 0000000000000246 R12: ffff949967f9eeb0
R13: ffff949967f9eeb0 R14: ffff9498af73bba0 R15: 0000000000000001
FS:  000000c420031890(0000) GS:ffff94997fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c420091010 CR3: 00000003ee3f8000 CR4: 00000000000007f0
Call Trace:
[<ffffffff85c76e50>] ? leave_mm+0x110/0x110
[<ffffffff85c76e50>] ? leave_mm+0x110/0x110
[<ffffffff85c76e50>] ? leave_mm+0x110/0x110
[<ffffffff85d078af>] smp_call_function_single+0x5f/0xa0
[<ffffffff85f4a275>] ? cpumask_next_and+0x35/0x50
[<ffffffff85d07e5b>] smp_call_function_many+0x22b/0x270
[<ffffffff85c77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff85c77088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff85dc0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff85dc1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff85dc4a02>] zap_page_range+0x142/0x190
[<ffffffff85ccee38>] ? wake_up_q+0x58/0x80
[<ffffffff85dbfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffff8631f721>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff8631f715>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffff8631f7d5>] system_call_fastpath+0x1c/0x21
[<ffffffff8631f721>] ? system_call_after_swapgs+0xae/0x146
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 93 86 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #16333
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff92916612
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffff9287e440
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x110
3
? leave_mm
vmlinux 0xffffffff9287e440
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x110
4
? leave_mm
vmlinux 0xffffffff9287e440
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x110
5
smp_call_function_single
vmlinux 0xffffffff9291672f
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff92b81845
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff92916cdb
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9287e608
Build id: 3.10.0-1062.18.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9287e67b
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x6b
10
zap_page_range
vmlinux 0xffffffff929f0d28
Build id: 3.10.0-1062.18.1.el7.x86_64 		0xd8
11
? wake_up_q
vmlinux 0xffffffff928db5b8
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x58
12
SyS_madvise
vmlinux 0xffffffff929eb22d
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x49d
13
? wake_up_state
vmlinux 0xffffffff928db600
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x20
14
? system_call_after_swapgs
vmlinux 0xffffffff92f8de21
Build id: 3.10.0-1062.18.1.el7.x86_64 		0xae
15
? system_call_after_swapgs
vmlinux 0xffffffff92f8de15
Build id: 3.10.0-1062.18.1.el7.x86_64 		0xa2
16
system_call_fastpath
vmlinux 0xffffffff92f8dede
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x25
17
? system_call_after_swapgs
vmlinux 0xffffffff92f8de21
Build id: 3.10.0-1062.18.1.el7.x86_64 		0xae 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kubelet:2073]
Modules linked in: xt_statistic nf_conntrack_netlink nfnetlink xt_nat ipt_REJECT nf_reject_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc vmw_vsock_vmci_transport vsock overlay(T) sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq pcspkr ppdev vmw_balloon snd_seq_device snd_pcm snd_timer snd joydev soundcore sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit
crc32c_intel mptspi libnvdimm scsi_transport_spi mptscsih ata_piix e1000 mptbase serio_raw libata drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 2073 Comm: kubelet Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.18.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff91029bfce2a0 ti: ffff91029a810000 task.ti: ffff91029a810000
RIP: 0010:[<ffffffff92916612>]  [<ffffffff92916612>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff91029a813c30  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff91029a813c00 RCX: 0000000000000004
RDX: ffffffff93013cd0 RSI: 0000000000000080 RDI: 0000000000000286
RBP: ffff91029a813c80 R08: ffffffff93013cc8 R09: 00003ffffffff000
R10: ffff9102b60d41b0 R11: 0000000000000246 R12: ffff91029a813bc0
R13: 0000000000000000 R14: ffff9102bb69ac80 R15: ffffffff928de7f5
FS:  00007fd3a12c7700(0000) GS:ffff9102bb680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fce31bafab8 CR3: 0000000078962000 CR4: 00000000000007e0
Call Trace:
[<ffffffff9287e440>] ? leave_mm+0x110/0x110
[<ffffffff9287e440>] ? leave_mm+0x110/0x110
[<ffffffff9287e440>] ? leave_mm+0x110/0x110
[<ffffffff9291672f>] smp_call_function_single+0x5f/0xa0
[<ffffffff92b81845>] ? cpumask_next_and+0x35/0x50
[<ffffffff92916cdb>] smp_call_function_many+0x22b/0x270
[<ffffffff9287e608>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9287e67b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff929f0d28>] zap_page_range+0xd8/0x150
[<ffffffff928db5b8>] ? wake_up_q+0x58/0x80
[<ffffffff929eb22d>] SyS_madvise+0x49d/0xac0
[<ffffffff928db600>] ? wake_up_state+0x20/0x20
[<ffffffff92f8de21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff92f8de15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffff92f8dede>] system_call_fastpath+0x25/0x2a
[<ffffffff92f8de21>] ? system_call_after_swapgs+0xae/0x146
Code: 89 de 48 03 14 c5 60 e8 54 93 48 89 df e8 b7 22 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #12780
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffae70779a
Build id: 3.10.0-862.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffffae676e50
Build id: 3.10.0-862.el7.x86_64 		0x110
3
? leave_mm
vmlinux 0xffffffffae676e50
Build id: 3.10.0-862.el7.x86_64 		0x110
4
? leave_mm
vmlinux 0xffffffffae676e50
Build id: 3.10.0-862.el7.x86_64 		0x110
5
smp_call_function_single
vmlinux 0xffffffffae7078af
Build id: 3.10.0-862.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffae94a275
Build id: 3.10.0-862.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffae707e5b
Build id: 3.10.0-862.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffae677018
Build id: 3.10.0-862.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffae677088
Build id: 3.10.0-862.el7.x86_64 		0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffffae7c0497
Build id: 3.10.0-862.el7.x86_64 		0x37
11
tlb_finish_mmu
vmlinux 0xffffffffae7c1c75
Build id: 3.10.0-862.el7.x86_64 		0x55
12
zap_page_range
vmlinux 0xffffffffae7c4a02
Build id: 3.10.0-862.el7.x86_64 		0x142
13
? wake_up_q
vmlinux 0xffffffffae6cee38
Build id: 3.10.0-862.el7.x86_64 		0x58
14
SyS_madvise
vmlinux 0xffffffffae7bfafd
Build id: 3.10.0-862.el7.x86_64 		0x3cd
15
? system_call_after_swapgs
vmlinux 0xffffffffaed1f721
Build id: 3.10.0-862.el7.x86_64 		0xae
16
? system_call_after_swapgs
vmlinux 0xffffffffaed1f715
Build id: 3.10.0-862.el7.x86_64 		0xa2
17
system_call_fastpath
vmlinux 0xffffffffaed1f7d5
Build id: 3.10.0-862.el7.x86_64 		0x1c
18
? system_call_after_swapgs
vmlinux 0xffffffffaed1f721
Build id: 3.10.0-862.el7.x86_64 		0xae 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [dockerd:1566]
Modules linked in: gsch(OE) redirfs(OE) veth tcp_diag udp_diag inet_diag xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink dinet(OE) difile(OE) overlay(T) dsa_filter(POE) sunrpc ppdev sg virtio_balloon joydev parport_pc parport i2c_piix4 pcspkr nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata i2c_core virtio_pci virtio_ring serio_raw floppy virtio dm_mirror dm_region_hash
dm_log dm_mod
CPU: 2 PID: 1566 Comm: dockerd Kdump: loaded Tainted: P        W  OE  ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.11.0-2.el7 04/01/2014
task: ffff8becd4209fa0 ti: ffff8beccc2e0000 task.ti: ffff8beccc2e0000
RIP: 0010:[<ffffffffae70779a>]  [<ffffffffae70779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8beccc2e3bf0  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff8beccc2e3bc0 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff8beccc2e3c38 R08: ffffffffaee13160 R09: ffff8bece23866c0
R10: 00007fffffffffff R11: 0000000000000246 R12: 00000000000000d9
R13: ffffffffae6d865c R14: ffff8beccc2e3ba0 R15: 0000000000000001
FS:  00007fd24d7fa700(0000) GS:ffff8bed3fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff861ffafb8 CR3: 00000007e56be000 CR4: 00000000000006e0
Call Trace:
[<ffffffffae676e50>] ? leave_mm+0x110/0x110
[<ffffffffae676e50>] ? leave_mm+0x110/0x110
[<ffffffffae676e50>] ? leave_mm+0x110/0x110
[<ffffffffae7078af>] smp_call_function_single+0x5f/0xa0
[<ffffffffae94a275>] ? cpumask_next_and+0x35/0x50
[<ffffffffae707e5b>] smp_call_function_many+0x22b/0x270
[<ffffffffae677018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffae677088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffae7c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffae7c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffae7c4a02>] zap_page_range+0x142/0x190
[<ffffffffae6cee38>] ? wake_up_q+0x58/0x80
[<ffffffffae7bfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffffaed1f721>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffaed1f715>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffaed1f7d5>] system_call_fastpath+0x1c/0x21
[<ffffffffaed1f721>] ? system_call_after_swapgs+0xae/0x146
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 33 af 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #1088355
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff98116f62
Build id: 3.10.0-1127.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffff9807e730
Build id: 3.10.0-1127.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffff9807e730
Build id: 3.10.0-1127.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffff9807e730
Build id: 3.10.0-1127.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffff9811707f
Build id: 3.10.0-1127.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff98384ee5
Build id: 3.10.0-1127.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff9811762b
Build id: 3.10.0-1127.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9807e8f8
Build id: 3.10.0-1127.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9807e96b
Build id: 3.10.0-1127.el7.x86_64 		0x6b
10
zap_page_range
vmlinux 0xffffffff981f2e58
Build id: 3.10.0-1127.el7.x86_64 		0xd8
11
? futex_wake
vmlinux 0xffffffff98112a60
Build id: 3.10.0-1127.el7.x86_64 		0x90
12
SyS_madvise
vmlinux 0xffffffff981ed36d
Build id: 3.10.0-1127.el7.x86_64 		0x49d
13
? system_call_after_swapgs
vmlinux 0xffffffff98792e15
Build id: 3.10.0-1127.el7.x86_64 		0xa2
14
? system_call_after_swapgs
vmlinux 0xffffffff98792e09
Build id: 3.10.0-1127.el7.x86_64 		0x96
15
system_call_fastpath
vmlinux 0xffffffff98792ed2
Build id: 3.10.0-1127.el7.x86_64 		0x25
16
? system_call_after_swapgs
vmlinux 0xffffffff98792e15
Build id: 3.10.0-1127.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [sdatad:24280]
Modules linked in: tcp_lp fuse sfs(OE) binfmt_misc nls_utf8 isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink sunrpc xfs libcrc32c ppdev pcspkr joydev parport_pc sg parport virtio_balloon i2c_piix4 ip_tables ext4 mbcache jbd2 sr_mod cdrom virtio_blk virtio_net virtio_console virtio_scsi net_failover failover ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dattobd]
CPU: 1 PID: 24280 Comm: sdatad Kdump: loaded Tainted: G           OEL ------------   3.10.0-1127.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9aefb4848000 ti: ffff9af59e86c000 task.ti: ffff9af59e86c000
RIP: 0010:[<ffffffff98116f62>]  [<ffffffff98116f62>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9af59e86fc30  EFLAGS: 00000202
RAX: 0000000000000030 RBX: ffff9af59e86fc00 RCX: 0000000000000010
RDX: 0000ffffffffffff RSI: 0000000000000030 RDI: 0000000000000286
RBP: ffff9af59e86fc80 R08: ffffffff98813600 R09: 0000000001a4f000
R10: 0000000000000000 R11: 0000000000000206 R12: 0000000000000001
R13: ffff9af1aea5acc0 R14: ffff9af1aea5ad40 R15: ffff9aefb4848068
FS:  00007fbc9b7e6700(0000) GS:ffff9af1aea40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8f7bffe008 CR3: 000000056fdb4000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9807e730>] ? leave_mm+0x120/0x120
[<ffffffff9807e730>] ? leave_mm+0x120/0x120
[<ffffffff9807e730>] ? leave_mm+0x120/0x120
[<ffffffff9811707f>] smp_call_function_single+0x5f/0xa0
[<ffffffff98384ee5>] ? cpumask_next_and+0x35/0x50
[<ffffffff9811762b>] smp_call_function_many+0x22b/0x270
[<ffffffff9807e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9807e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff981f2e58>] zap_page_range+0xd8/0x150
[<ffffffff98112a60>] ? futex_wake+0x90/0x180
[<ffffffff981ed36d>] SyS_madvise+0x49d/0xab0
[<ffffffff98792e15>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff98792e09>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff98792ed2>] system_call_fastpath+0x25/0x2a
[<ffffffff98792e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 e0 0f d5 98 48 89 df e8 17 51 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #1072176
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9ab0779a
Build id: 3.10.0-862.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff9aa76e50
Build id: 3.10.0-862.el7.x86_64 		0x110
3
? leave_mm
vmlinux 0xffffffff9aa76e50
Build id: 3.10.0-862.el7.x86_64 		0x110
4
? leave_mm
vmlinux 0xffffffff9aa76e50
Build id: 3.10.0-862.el7.x86_64 		0x110
5
smp_call_function_single
vmlinux 0xffffffff9ab078af
Build id: 3.10.0-862.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9ad4a275
Build id: 3.10.0-862.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff9ab07e5b
Build id: 3.10.0-862.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9aa77018
Build id: 3.10.0-862.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9aa77088
Build id: 3.10.0-862.el7.x86_64 		0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffff9abc0497
Build id: 3.10.0-862.el7.x86_64 		0x37
11
tlb_finish_mmu
vmlinux 0xffffffff9abc1c75
Build id: 3.10.0-862.el7.x86_64 		0x55
12
zap_page_range
vmlinux 0xffffffff9abc4a02
Build id: 3.10.0-862.el7.x86_64 		0x142
13
? wake_up_q
vmlinux 0xffffffff9aacee38
Build id: 3.10.0-862.el7.x86_64 		0x58
14
? futex_wake
vmlinux 0xffffffff9ab033a0
Build id: 3.10.0-862.el7.x86_64 		0x90
15
SyS_madvise
vmlinux 0xffffffff9abbfafd
Build id: 3.10.0-862.el7.x86_64 		0x3cd
16
? wake_up_state
vmlinux 0xffffffff9aacee80
Build id: 3.10.0-862.el7.x86_64 		0x20
17
? system_call_after_swapgs
vmlinux 0xffffffff9b11f721
Build id: 3.10.0-862.el7.x86_64 		0xae
18
? system_call_after_swapgs
vmlinux 0xffffffff9b11f715
Build id: 3.10.0-862.el7.x86_64 		0xa2
19
system_call_fastpath
vmlinux 0xffffffff9b11f7d5
Build id: 3.10.0-862.el7.x86_64 		0x1c
20
? system_call_after_swapgs
vmlinux 0xffffffff9b11f721
Build id: 3.10.0-862.el7.x86_64 		0xae 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kubelet:1282]
Modules linked in: xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc edac_mce_amd crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd sg pcspkr joydev vmw_balloon nfit libnvdimm shpchp vmw_vmci parport_pc parport i2c_piix4 nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops crct10dif_pclmul crct10dif_common ttm ahci crc32c_intel drm serio_raw libahci ata_piix libata vmxnet3 vmw_pvscsi i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 1282 Comm: kubelet Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9ac237e26eb0 ti: ffff9ac107c88000 task.ti: ffff9ac107c88000
RIP: 0010:[<ffffffff9ab0779a>]  [<ffffffff9ab0779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9ac107c8bbf0  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff9ac107c8bbc0 RCX: 000000000000003c
RDX: 000000000000000f RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff9ac107c8bc38 R08: ffffffff9b213160 R09: ffff9ac19da5b1b0
R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000091
R13: ffff9ac23fd98b80 R14: ffff9ac1bbb9b780 R15: 0000000000000001
FS:  00007efd8a2e4700(0000) GS:ffff9ac23fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007efd880ad0d8 CR3: 0000000003110000 CR4: 00000000000407e0
Call Trace:
[<ffffffff9aa76e50>] ? leave_mm+0x110/0x110
[<ffffffff9aa76e50>] ? leave_mm+0x110/0x110
[<ffffffff9aa76e50>] ? leave_mm+0x110/0x110
[<ffffffff9ab078af>] smp_call_function_single+0x5f/0xa0
[<ffffffff9ad4a275>] ? cpumask_next_and+0x35/0x50
[<ffffffff9ab07e5b>] smp_call_function_many+0x22b/0x270
[<ffffffff9aa77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9aa77088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff9abc0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff9abc1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff9abc4a02>] zap_page_range+0x142/0x190
[<ffffffff9aacee38>] ? wake_up_q+0x58/0x80
[<ffffffff9ab033a0>] ? futex_wake+0x90/0x180
[<ffffffff9abbfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffff9aacee80>] ? wake_up_state+0x20/0x20
[<ffffffff9b11f721>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff9b11f715>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffff9b11f7d5>] system_call_fastpath+0x1c/0x21
[<ffffffff9b11f721>] ? system_call_after_swapgs+0xae/0x146
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 73 9b 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #485268
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb0b16b62
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffffb0a7ddd0
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffffb0a7ddd0
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffffb0a7ddd0
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffffb0b16c7f
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb0d86865
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffb0b1722b
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb0a7e258
Build id: 3.10.0-1160.71.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb0a7e2cb
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb0bee6c6
Build id: 3.10.0-1160.71.1.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb0bf018f
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb0bf02b3
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x23
13
zap_page_range
vmlinux 0xffffffffb0bf377a
Build id: 3.10.0-1160.71.1.el7.x86_64 		0xea
14
? futex_wake
vmlinux 0xffffffffb0b127a0
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x90
15
SyS_madvise
vmlinux 0xffffffffb0bedc6d
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x49d
16
? system_call_after_swapgs
vmlinux 0xffffffffb1199ed5
Build id: 3.10.0-1160.71.1.el7.x86_64 		0xa2
17
? system_call_after_swapgs
vmlinux 0xffffffffb1199ec9
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x96
18
system_call_fastpath
vmlinux 0xffffffffb1199f92
Build id: 3.10.0-1160.71.1.el7.x86_64 		0x25
19
? system_call_after_swapgs
vmlinux 0xffffffffb1199ed5
Build id: 3.10.0-1160.71.1.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [popen.rb:42:13111]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink arc4 md4 nls_utf8 cifs ccm dns_resolver sunrpc ppdev pcspkr virtio_balloon sg i2c_piix4 parport_pc parport joydev ip_tables xfs libcrc32c sr_mod cdrom virtio_net virtio_blk net_failover failover virtio_console virtio_scsi ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata serio_raw virtio_pci drm_panel_orientation_quirks virtio_ring floppy virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 13111 Comm: popen.rb:42 Tainted: G             L ------------   3.10.0-1160.71.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
task: ffff8abe2a11e300 ti: ffff8abd79da0000 task.ti: ffff8abd79da0000
RIP: 0010:[<ffffffffb0b16b62>]  [<ffffffffb0b16b62>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8abd79da3be0  EFLAGS: 00000202
RAX: 0000000000000020 RBX: ffff8abd79da3bb0 RCX: 0000000000000020
RDX: 00000000ffffffff RSI: 0000000000000020 RDI: 0000000000000282
RBP: ffff8abd79da3c30 R08: ffffffffb12137c0 R09: 00003ffffffff000
R10: ffff8abef660f290 R11: 0000000000000206 R12: 0000360ec6e8bfcf
R13: ffffffffb0ae495c R14: ffff8abd79da3b80 R15: ffff8ab4bfd11000
FS:  00007f09626d0700(0000) GS:ffff8abc5aec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0986fdd000 CR3: 0000000be30fe000 CR4: 00000000000006e0
Call Trace:
[<ffffffffb0a7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb0a7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb0a7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb0b16c7f>] smp_call_function_single+0x5f/0xa0
[<ffffffffb0d86865>] ? cpumask_next_and+0x35/0x50
[<ffffffffb0b1722b>] smp_call_function_many+0x22b/0x270
[<ffffffffb0a7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb0a7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb0bee6c6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb0bf018f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb0bf02b3>] tlb_finish_mmu+0x23/0x30
[<ffffffffb0bf377a>] zap_page_range+0xea/0x130
[<ffffffffb0b127a0>] ? futex_wake+0x90/0x180
[<ffffffffb0bedc6d>] SyS_madvise+0x49d/0xac0
[<ffffffffb1199ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb1199ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb1199f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb1199ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 60 18 75 b1 48 89 df e8 77 6e 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #251363
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff92916f62
Build id: 3.10.0-1127.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffff9287e730
Build id: 3.10.0-1127.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffff9287e730
Build id: 3.10.0-1127.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffff9287e730
Build id: 3.10.0-1127.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffff9291707f
Build id: 3.10.0-1127.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff92b84ee5
Build id: 3.10.0-1127.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff9291762b
Build id: 3.10.0-1127.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9287e8f8
Build id: 3.10.0-1127.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9287e96b
Build id: 3.10.0-1127.el7.x86_64 		0x6b
10
zap_page_range
vmlinux 0xffffffff929f2e58
Build id: 3.10.0-1127.el7.x86_64 		0xd8
11
? futex_wake
vmlinux 0xffffffff92912a60
Build id: 3.10.0-1127.el7.x86_64 		0x90
12
SyS_madvise
vmlinux 0xffffffff929ed36d
Build id: 3.10.0-1127.el7.x86_64 		0x49d
13
? system_call_after_swapgs
vmlinux 0xffffffff92f92e15
Build id: 3.10.0-1127.el7.x86_64 		0xa2
14
? system_call_after_swapgs
vmlinux 0xffffffff92f92e09
Build id: 3.10.0-1127.el7.x86_64 		0x96
15
system_call_fastpath
vmlinux 0xffffffff92f92ed2
Build id: 3.10.0-1127.el7.x86_64 		0x25
16
? system_call_after_swapgs
vmlinux 0xffffffff92f92e15
Build id: 3.10.0-1127.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [popen.rb:41:25208]
Modules linked in: tcp_lp btrfs raid6_pq xor vfat fat tcp_diag inet_diag ext4 mbcache jbd2 fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc uas coretemp usb_storage crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon
sg pcspkr joydev vmw_vmci parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw crct10dif_pclmul crct10dif_common crc32c_intel drm mptspi ata_piix scsi_transport_spi mptscsih libata mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 25208 Comm: popen.rb:41 Kdump: loaded Tainted: G             L ------------   3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
task: ffff9c85815c20e0 ti: ffff9c84ef00c000 task.ti: ffff9c84ef00c000
RIP: 0010:[<ffffffff92916f62>]  [<ffffffff92916f62>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9c84ef00fc30  EFLAGS: 00000202
RAX: 00000000000008fd RBX: ffff9c85815c2148 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000286
RBP: ffff9c84ef00fc80 R08: 0000000000000001 R09: 00007f3c1cfac000
R10: 00007f3c39f0a140 R11: 0000000000000206 R12: ffff9c85bfc1acc0
R13: ffffffff928e103c R14: ffff9c84ef00fbb8 R15: ffffffff928e2e89
FS:  00007f3c1d0ab700(0000) GS:ffff9c85bfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3c2e6f13e0 CR3: 00000000a5918000 CR4: 00000000000007f0
Call Trace:
[<ffffffff9287e730>] ? leave_mm+0x120/0x120
[<ffffffff9287e730>] ? leave_mm+0x120/0x120
[<ffffffff9287e730>] ? leave_mm+0x120/0x120
[<ffffffff9291707f>] smp_call_function_single+0x5f/0xa0
[<ffffffff92b84ee5>] ? cpumask_next_and+0x35/0x50
[<ffffffff9291762b>] smp_call_function_many+0x22b/0x270
[<ffffffff9287e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9287e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff929f2e58>] zap_page_range+0xd8/0x150
[<ffffffff92912a60>] ? futex_wake+0x90/0x180
[<ffffffff929ed36d>] SyS_madvise+0x49d/0xab0
[<ffffffff92f92e15>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff92f92e09>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff92f92ed2>] system_call_fastpath+0x25/0x2a
[<ffffffff92f92e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 e0 0f 55 93 48 89 df e8 17 51 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #180478
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810fb21a
Build id: 3.10.0-693.17.1.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff8106f340
Build id: 3.10.0-693.17.1.el7.x86_64 		0x110
3
? leave_mm
vmlinux 0xffffffff8106f340
Build id: 3.10.0-693.17.1.el7.x86_64 		0x110
4
? leave_mm
vmlinux 0xffffffff8106f340
Build id: 3.10.0-693.17.1.el7.x86_64 		0x110
5
smp_call_function_single
vmlinux 0xffffffff810fb31f
Build id: 3.10.0-693.17.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff81324bd5
Build id: 3.10.0-693.17.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810fb8b3
Build id: 3.10.0-693.17.1.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106f508
Build id: 3.10.0-693.17.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106f5d8
Build id: 3.10.0-693.17.1.el7.x86_64 		0x68
10
tlb_flush_mmu.part.63
vmlinux 0xffffffff811af0f3
Build id: 3.10.0-693.17.1.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811b0415
Build id: 3.10.0-693.17.1.el7.x86_64 		0x55
12
zap_page_range
vmlinux 0xffffffff811b299a
Build id: 3.10.0-693.17.1.el7.x86_64 		0x13a
13
? futex_wake
vmlinux 0xffffffff810f6e60
Build id: 3.10.0-693.17.1.el7.x86_64 		0x90
14
SyS_madvise
vmlinux 0xffffffff811ae7bd
Build id: 3.10.0-693.17.1.el7.x86_64 		0x3cd
15
? system_call_after_swapgs
vmlinux 0xffffffff816b895a
Build id: 3.10.0-693.17.1.el7.x86_64 		0x187
16
system_call_fastpath
vmlinux 0xffffffff816b89fd
Build id: 3.10.0-693.17.1.el7.x86_64 		0x16
17
? system_call_after_swapgs
vmlinux 0xffffffff816b889d
Build id: 3.10.0-693.17.1.el7.x86_64 		0xca 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [Plex Media Serv:1571]
Modules linked in: arc4 md4 rpcsec_gss_krb5 nfsv4 nfs fscache nls_utf8 cifs dns_resolver ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ppdev pcspkr sg joydev virtio_balloon shpchp parport_pc parport i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi virtio_net virtio_scsi bochs_drm drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm serio_raw virtio_pci virtio_ring virtio ata_piix libata i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 1571 Comm: Plex Media Serv Tainted: G             L ------------   3.10.0-693.17.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
task: ffff88005989cf10 ti: ffff8800610cc000 task.ti: ffff8800610cc000
RIP: 0010:[<ffffffff810fb21a>]  [<ffffffff810fb21a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8800610cfbf0  EFLAGS: 00000202
RAX: 0000000000000002 RBX: ffff8800610cfbc0 RCX: 0000000000000830
RDX: 0000000000000002 RSI: 0000000000000002 RDI: 0000000000000282
RBP: ffff8800610cfc40 R08: ffffffff816d69e0 R09: 00007f7adf8e2000
R10: 0000000000000000 R11: 0000000000000206 R12: ffffffff816b9bff
R13: ffffffff816b9c06 R14: ffffffff816b9c0d R15: ffffffff816b9c14
FS:  00007f7ad7ff7700(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7adc1c6138 CR3: 0000000078b56000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8106f340>] ? leave_mm+0x110/0x110
[<ffffffff8106f340>] ? leave_mm+0x110/0x110
[<ffffffff8106f340>] ? leave_mm+0x110/0x110
[<ffffffff810fb31f>] smp_call_function_single+0x5f/0xa0
[<ffffffff81324bd5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810fb8b3>] smp_call_function_many+0x223/0x260
[<ffffffff8106f508>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106f5d8>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811af0f3>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811b0415>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b299a>] zap_page_range+0x13a/0x180
[<ffffffff810f6e60>] ? futex_wake+0x90/0x180
[<ffffffff811ae7bd>] SyS_madvise+0x3cd/0x9c0
[<ffffffff816b895a>] ? system_call_after_swapgs+0x187/0x214
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
[<ffffffff816b889d>] ? system_call_after_swapgs+0xca/0x214
Code: 40 95 01 00 48 89 de 48 03 14 c5 20 8e b1 81 48 89 df e8 5a e0 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #1109994
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffba716562
Build id: 3.10.0-1160.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffffba67e6a0
Build id: 3.10.0-1160.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffffba67e6a0
Build id: 3.10.0-1160.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffffba67e6a0
Build id: 3.10.0-1160.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffffba71667f
Build id: 3.10.0-1160.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffba985de5
Build id: 3.10.0-1160.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffba716c2b
Build id: 3.10.0-1160.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffba67e868
Build id: 3.10.0-1160.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffba67e8db
Build id: 3.10.0-1160.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffba7edd06
Build id: 3.10.0-1160.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffba7ef7cf
Build id: 3.10.0-1160.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffba7ef8f3
Build id: 3.10.0-1160.el7.x86_64 		0x23
13
zap_page_range
vmlinux 0xffffffffba7f2dba
Build id: 3.10.0-1160.el7.x86_64 		0xea
14
? do_mmap
vmlinux 0xffffffffba7fc9d8
Build id: 3.10.0-1160.el7.x86_64 		0x378
15
SyS_madvise
vmlinux 0xffffffffba7ed2ad
Build id: 3.10.0-1160.el7.x86_64 		0x49d
16
? vm_mmap_pgoff
vmlinux 0xffffffffba7de1bb
Build id: 3.10.0-1160.el7.x86_64 		0xeb
17
? system_call_after_swapgs
vmlinux 0xffffffffbad93ed5
Build id: 3.10.0-1160.el7.x86_64 		0xa2
18
? system_call_after_swapgs
vmlinux 0xffffffffbad93ec9
Build id: 3.10.0-1160.el7.x86_64 		0x96
19
system_call_fastpath
vmlinux 0xffffffffbad93f92
Build id: 3.10.0-1160.el7.x86_64 		0x25
20
? system_call_after_swapgs
vmlinux 0xffffffffbad93ed5
Build id: 3.10.0-1160.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 67s! [java:62394]
Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables uio_ivshmem(OE) uio gsch(OE) redirfs(OE) bridge stp llc ip_set nfnetlink sunrpc ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr virtio_balloon sg i2c_piix4 parport_pc parport xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net net_failover virtio_console virtio_blk virtio_scsi failover cirrus drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ip_tables]
CPU: 6 PID: 62394 Comm: java Kdump: loaded Tainted: G        W  OEL ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
task: ffff9bd406ef8000 ti: ffff9bd207aa8000 task.ti: ffff9bd207aa8000
RIP: 0010:[<ffffffffba716562>]  [<ffffffffba716562>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9bd207aabbe0  EFLAGS: 00000202
RAX: 00000000000000a0 RBX: ffff9bd207aabbb0 RCX: 0000000000000020
RDX: 00000000ffffffff RSI: 00000000000000a0 RDI: 0000000000000282
RBP: ffff9bd207aabc30 R08: ffffffffbae14408 R09: 00003ffffffff000
R10: ffff9bceb5e5a1b0 R11: 00007fa9c8e5e000 R12: 00000000bac570b5
R13: fffff196d2c04c30 R14: 00007fa9c8e5e000 R15: 0000000180400022
FS:  00007fa9c8f5b700(0000) GS:ffff9bd69cf80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8440c78100 CR3: 00000006f0e36000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffba67e6a0>] ? leave_mm+0x120/0x120
[<ffffffffba67e6a0>] ? leave_mm+0x120/0x120
[<ffffffffba67e6a0>] ? leave_mm+0x120/0x120
[<ffffffffba71667f>] smp_call_function_single+0x5f/0xa0
[<ffffffffba985de5>] ? cpumask_next_and+0x35/0x50
[<ffffffffba716c2b>] smp_call_function_many+0x22b/0x270
[<ffffffffba67e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffba67e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffba7edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffba7ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffba7ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffffba7f2dba>] zap_page_range+0xea/0x130
[<ffffffffba7fc9d8>] ? do_mmap+0x378/0x530
[<ffffffffba7ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffffba7de1bb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffbad93ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffbad93ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffbad93f92>] system_call_fastpath+0x25/0x2a
[<ffffffffbad93ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 e0 13 35 bb 48 89 df e8 f7 69 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #451568
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb9516b82
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffffb947ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffffb947ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffffb947ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffffb9516c9f
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb9786825
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffb951724b
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb947e258
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb947e2cb
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb95ee666
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb95f012f
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb95f0253
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x23
13
zap_page_range
vmlinux 0xffffffffb95f371a
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xea
14
? do_mmap
vmlinux 0xffffffffb95fd35c
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x39c
15
SyS_madvise
vmlinux 0xffffffffb95edc0d
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x49d
16
? vm_mmap_pgoff
vmlinux 0xffffffffb95ded9b
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xeb
17
? system_call_after_swapgs
vmlinux 0xffffffffb9b99ed5
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xa2
18
? system_call_after_swapgs
vmlinux 0xffffffffb9b99ec9
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x96
19
system_call_fastpath
vmlinux 0xffffffffb9b99f92
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x25
20
? system_call_after_swapgs
vmlinux 0xffffffffb9b99ed5
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [java:30643]
Modules linked in: xt_multiport splxmod(OE) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink scsi_transport_iscsi ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_multipath sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd pcspkr vmw_balloon joydev sg vmw_vmci parport_pc i2c_piix4 parport sch_fq_codel
ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper ata_piix syscopyarea sysfillrect sysimgblt fb_sys_fops ttm libata mptspi drm scsi_transport_spi crct10dif_pclmul crct10dif_common mptscsih crc32c_intel serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: splxmod]
CPU: 1 PID: 30643 Comm: java Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
task: ffffa0e33e47e300 ti: ffffa0e385604000 task.ti: ffffa0e385604000
RIP: 0010:[<ffffffffb9516b82>]  [<ffffffffb9516b82>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffffa0e385607be0  EFLAGS: 00000202
RAX: 00000000000008fd RBX: ffffffffb94e10ae RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000286
RBP: ffffa0e385607c30 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffffa0e53170bd80 R11: 00007f86fce94000 R12: 00000000895b59eb
R13: fffff75308d574b0 R14: 00007f86fce94000 R15: ffffa0e33e47e300
FS:  00007f86fcf91700(0000) GS:ffffa0e53fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8722fcb380 CR3: 00000002340ba000 CR4: 00000000000407e0
Call Trace:
[<ffffffffb947ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb947ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb947ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb9516c9f>] smp_call_function_single+0x5f/0xa0
[<ffffffffb9786825>] ? cpumask_next_and+0x35/0x50
[<ffffffffb951724b>] smp_call_function_many+0x22b/0x270
[<ffffffffb947e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb947e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb95ee666>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb95f012f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb95f0253>] tlb_finish_mmu+0x23/0x30
[<ffffffffb95f371a>] zap_page_range+0xea/0x130
[<ffffffffb95fd35c>] ? do_mmap+0x39c/0x590
[<ffffffffb95edc0d>] SyS_madvise+0x49d/0xac0
[<ffffffffb95ded9b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffb9b99ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb9b99ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb9b99f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb9b99ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 60 18 15 ba 48 89 df e8 17 6e 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #373581
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff98b0779a
Build id: 3.10.0-862.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff98a76e50
Build id: 3.10.0-862.el7.x86_64 		0x110
3
? leave_mm
vmlinux 0xffffffff98a76e50
Build id: 3.10.0-862.el7.x86_64 		0x110
4
? leave_mm
vmlinux 0xffffffff98a76e50
Build id: 3.10.0-862.el7.x86_64 		0x110
5
smp_call_function_single
vmlinux 0xffffffff98b078af
Build id: 3.10.0-862.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff98d4a275
Build id: 3.10.0-862.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff98b07e5b
Build id: 3.10.0-862.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff98a77018
Build id: 3.10.0-862.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff98a77088
Build id: 3.10.0-862.el7.x86_64 		0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffff98bc0497
Build id: 3.10.0-862.el7.x86_64 		0x37
11
tlb_finish_mmu
vmlinux 0xffffffff98bc1c75
Build id: 3.10.0-862.el7.x86_64 		0x55
12
zap_page_range
vmlinux 0xffffffff98bc4a02
Build id: 3.10.0-862.el7.x86_64 		0x142
13
? do_mmap
vmlinux 0xffffffff98bcdf0c
Build id: 3.10.0-862.el7.x86_64 		0x3ac
14
SyS_madvise
vmlinux 0xffffffff98bbfafd
Build id: 3.10.0-862.el7.x86_64 		0x3cd
15
? vm_mmap_pgoff
vmlinux 0xffffffff98bb35eb
Build id: 3.10.0-862.el7.x86_64 		0xeb
16
? system_call_after_swapgs
vmlinux 0xffffffff9911f721
Build id: 3.10.0-862.el7.x86_64 		0xae
17
? system_call_after_swapgs
vmlinux 0xffffffff9911f715
Build id: 3.10.0-862.el7.x86_64 		0xa2
18
system_call_fastpath
vmlinux 0xffffffff9911f7d5
Build id: 3.10.0-862.el7.x86_64 		0x1c
19
? system_call_after_swapgs
vmlinux 0xffffffff9911f721
Build id: 3.10.0-862.el7.x86_64 		0xae 
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [java:55596]
Modules linked in: af_packet_diag netlink_diag tcp_lp nls_utf8 isofs fuse binfmt_misc vsock_diag vsock sctp_diag sctp tcp_diag udp_diag inet_diag unix_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink sunrpc ext4 mbcache jbd2 virtio_balloon sg ppdev parport_pc parport i2c_piix4 pcspkr joydev sch_fq_codel ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_scsi virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci i2c_core virtio_ring floppy virtio dm_mirror
dm_region_hash dm_log dm_mod
CPU: 6 PID: 55596 Comm: java Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff8d5ef723dee0 ti: ffff8d59ceb84000 task.ti: ffff8d59ceb84000
RIP: 0010:[<ffffffff98b0779a>]  [<ffffffff98b0779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8d59ceb87bf0  EFLAGS: 00000202
RAX: 0000000000000060 RBX: ffff8d59ceb87bc0 RCX: 0000000000000020
RDX: 00000000ffffffff RSI: 0000000000000060 RDI: 0000000000000282
RBP: ffff8d59ceb87c38 R08: ffffffff992133e8 R09: ffff8d57b45e3008
R10: 0000000000000008 R11: 00007fd4deedc000 R12: ffff8d5708029000
R13: 00000000f384c646 R14: 00007fd4deedc000 R15: ffff8d5ef723dee0
FS:  00007fd4defd9700(0000) GS:ffff8d5f1df80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000559723373dc8 CR3: 0000000010a58000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff98a76e50>] ? leave_mm+0x110/0x110
[<ffffffff98a76e50>] ? leave_mm+0x110/0x110
[<ffffffff98a76e50>] ? leave_mm+0x110/0x110
[<ffffffff98b078af>] smp_call_function_single+0x5f/0xa0
[<ffffffff98d4a275>] ? cpumask_next_and+0x35/0x50
[<ffffffff98b07e5b>] smp_call_function_many+0x22b/0x270
[<ffffffff98a77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff98a77088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff98bc0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff98bc1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff98bc4a02>] zap_page_range+0x142/0x190
[<ffffffff98bcdf0c>] ? do_mmap+0x3ac/0x4d0
[<ffffffff98bbfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffff98bb35eb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff9911f721>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff9911f715>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffff9911f7d5>] system_call_fastpath+0x1c/0x21
[<ffffffff9911f721>] ? system_call_after_swapgs+0xae/0x146
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 73 99 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #317916
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffabf1109a
Build id: 3.10.0-957.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffffabe7a3d0
Build id: 3.10.0-957.el7.x86_64 		0x110
3
? leave_mm
vmlinux 0xffffffffabe7a3d0
Build id: 3.10.0-957.el7.x86_64 		0x110
4
? leave_mm
vmlinux 0xffffffffabe7a3d0
Build id: 3.10.0-957.el7.x86_64 		0x110
5
smp_call_function_single
vmlinux 0xffffffffabf111af
Build id: 3.10.0-957.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffac175465
Build id: 3.10.0-957.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffabf1175b
Build id: 3.10.0-957.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffabe7a598
Build id: 3.10.0-957.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffabe7a608
Build id: 3.10.0-957.el7.x86_64 		0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffabfe4077
Build id: 3.10.0-957.el7.x86_64 		0x37
11
tlb_finish_mmu
vmlinux 0xffffffffabfe5975
Build id: 3.10.0-957.el7.x86_64 		0x55
12
zap_page_range
vmlinux 0xffffffffabfe87c2
Build id: 3.10.0-957.el7.x86_64 		0x142
13
? do_mmap
vmlinux 0xffffffffabff28f8
Build id: 3.10.0-957.el7.x86_64 		0x378
14
SyS_madvise
vmlinux 0xffffffffabfe368d
Build id: 3.10.0-957.el7.x86_64 		0x49d
15
? vm_mmap_pgoff
vmlinux 0xffffffffabfd612b
Build id: 3.10.0-957.el7.x86_64 		0xeb
16
? system_call_after_swapgs
vmlinux 0xffffffffac574d21
Build id: 3.10.0-957.el7.x86_64 		0xae
17
? system_call_after_swapgs
vmlinux 0xffffffffac574d15
Build id: 3.10.0-957.el7.x86_64 		0xa2
18
system_call_fastpath
vmlinux 0xffffffffac574ddb
Build id: 3.10.0-957.el7.x86_64 		0x22
19
? system_call_after_swapgs
vmlinux 0xffffffffac574d21
Build id: 3.10.0-957.el7.x86_64 		0xae 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [java:22032]
Modules linked in: ipt_REJECT nf_reject_ipv4 xt_comment nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter sunrpc ext4 mbcache jbd2 ppdev sg pcspkr joydev virtio_balloon i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common sr_mod cdrom ata_generic pata_acpi serio_raw bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm floppy libahci ata_piix drm_panel_orientation_quirks e1000 libata virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 22032 Comm: java Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff93733fef4100 ti: ffff937192b20000 task.ti: ffff937192b20000
RIP: 0010:[<ffffffffabf1109a>]  [<ffffffffabf1109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff937192b23bf0  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff937192b23bc0 RCX: 0000000000000830
RDX: 0000000000000004 RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff937192b23c38 R08: ffffffffac613500 R09: 00003ffffffff000
R10: ffff937282700870 R11: 00007f3cf2e05000 R12: ffff937283549540
R13: 0000000082733aaa R14: 00007f3cf2e05000 R15: ffff93733fef4100
FS:  00007f3cf2e42700(0000) GS:ffff9373bfc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4d14099000 CR3: 000000009683c000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffabe7a3d0>] ? leave_mm+0x110/0x110
[<ffffffffabe7a3d0>] ? leave_mm+0x110/0x110
[<ffffffffabe7a3d0>] ? leave_mm+0x110/0x110
[<ffffffffabf111af>] smp_call_function_single+0x5f/0xa0
[<ffffffffac175465>] ? cpumask_next_and+0x35/0x50
[<ffffffffabf1175b>] smp_call_function_many+0x22b/0x270
[<ffffffffabe7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffabe7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffabfe4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffabfe5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffabfe87c2>] zap_page_range+0x142/0x190
[<ffffffffabff28f8>] ? do_mmap+0x378/0x530
[<ffffffffabfe368d>] SyS_madvise+0x49d/0xac0
[<ffffffffabfd612b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffac574d21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffac574d15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffac574ddb>] system_call_fastpath+0x22/0x27
[<ffffffffac574d21>] ? system_call_after_swapgs+0xae/0x146
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 b4 ac 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #251003
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa2b16a32
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffffa2a7ddd0
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffffa2a7ddd0
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffffa2a7ddd0
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffffa2b16b4f
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa2d86a25
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffa2b170fb
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa2a7e258
Build id: 3.10.0-1160.41.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa2a7e2cb
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa2bee676
Build id: 3.10.0-1160.41.1.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffa2bf013f
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffa2bf0263
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x23
13
zap_page_range
vmlinux 0xffffffffa2bf372a
Build id: 3.10.0-1160.41.1.el7.x86_64 		0xea
14
SyS_madvise
vmlinux 0xffffffffa2bedc1d
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x49d
15
? vm_mmap_pgoff
vmlinux 0xffffffffa2bdedab
Build id: 3.10.0-1160.41.1.el7.x86_64 		0xeb
16
? system_call_after_swapgs
vmlinux 0xffffffffa3195ed5
Build id: 3.10.0-1160.41.1.el7.x86_64 		0xa2
17
? system_call_after_swapgs
vmlinux 0xffffffffa3195ec9
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x96
18
system_call_fastpath
vmlinux 0xffffffffa3195f92
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x25
19
? system_call_after_swapgs
vmlinux 0xffffffffa3195ed5
Build id: 3.10.0-1160.41.1.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [qtp2076287037-3:3885]
Modules linked in: tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr parport_pc sg parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper e1000 syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul crct10dif_common crc32c_intel mptspi serio_raw scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks
floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 3885 Comm: qtp2076287037-3 Kdump: loaded Not tainted 3.10.0-1160.41.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff962abe37e300 ti: ffff962a2c8d8000 task.ti: ffff962a2c8d8000
RIP: 0010:[<ffffffffa2b16a32>]  [<ffffffffa2b16a32>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff962a2c8dbbe0  EFLAGS: 00000202
RAX: 00000000000008fd RBX: 0000000000000001 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000286
RBP: ffff962a2c8dbc30 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffff9629647b41b0 R11: 00007f0ab5e3d000 R12: ffff962abe37e300
R13: fffffb31c26072b0 R14: 00007f0ab5e3d000 R15: 0000000100400006
FS:  00007f0ab5e79700(0000) GS:ffff962affc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8768900000 CR3: 00000000a9adc000 CR4: 00000000000007f0
Call Trace:
[<ffffffffa2a7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa2a7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa2a7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa2b16b4f>] smp_call_function_single+0x5f/0xa0
[<ffffffffa2d86a25>] ? cpumask_next_and+0x35/0x50
[<ffffffffa2b170fb>] smp_call_function_many+0x22b/0x270
[<ffffffffa2a7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa2a7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa2bee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa2bf013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa2bf0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffa2bf372a>] zap_page_range+0xea/0x130
[<ffffffffa2bedc1d>] SyS_madvise+0x49d/0xac0
[<ffffffffa2bdedab>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffa3195ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffa3195ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffa3195f92>] system_call_fastpath+0x25/0x2a
[<ffffffffa3195ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 60 15 75 a3 48 89 df e8 67 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #198529
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff8151117a
Build id: 3.10.0-957.1.3.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff8147a3d0
Build id: 3.10.0-957.1.3.el7.x86_64 		0x110
3
? leave_mm
vmlinux 0xffffffff8147a3d0
Build id: 3.10.0-957.1.3.el7.x86_64 		0x110
4
? leave_mm
vmlinux 0xffffffff8147a3d0
Build id: 3.10.0-957.1.3.el7.x86_64 		0x110
5
smp_call_function_single
vmlinux 0xffffffff8151128f
Build id: 3.10.0-957.1.3.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff817756d5
Build id: 3.10.0-957.1.3.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff8151183b
Build id: 3.10.0-957.1.3.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8147a598
Build id: 3.10.0-957.1.3.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8147a608
Build id: 3.10.0-957.1.3.el7.x86_64 		0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff815e42e7
Build id: 3.10.0-957.1.3.el7.x86_64 		0x37
11
tlb_finish_mmu
vmlinux 0xffffffff815e5be5
Build id: 3.10.0-957.1.3.el7.x86_64 		0x55
12
zap_page_range
vmlinux 0xffffffff815e8a32
Build id: 3.10.0-957.1.3.el7.x86_64 		0x142
13
SyS_madvise
vmlinux 0xffffffff815e38fd
Build id: 3.10.0-957.1.3.el7.x86_64 		0x49d
14
? vm_mmap_pgoff
vmlinux 0xffffffff815d639b
Build id: 3.10.0-957.1.3.el7.x86_64 		0xeb
15
? system_call_after_swapgs
vmlinux 0xffffffff81b74d21
Build id: 3.10.0-957.1.3.el7.x86_64 		0xae
16
? system_call_after_swapgs
vmlinux 0xffffffff81b74d15
Build id: 3.10.0-957.1.3.el7.x86_64 		0xa2
17
system_call_fastpath
vmlinux 0xffffffff81b74ddb
Build id: 3.10.0-957.1.3.el7.x86_64 		0x22
18
? system_call_after_swapgs
vmlinux 0xffffffff81b74d21
Build id: 3.10.0-957.1.3.el7.x86_64 		0xae 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [java:14023]
Modules linked in: vsock_diag tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw e1000e ptp pps_core vmwgfx ata_generic pata_acpi drm_kms_helper ahci syscopyarea sysfillrect sysimgblt
libahci fb_sys_fops ttm ata_piix nfit drm vmw_pvscsi libata libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 14023 Comm: java Kdump: loaded Tainted: G             L ------------   3.10.0-957.1.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
task: ffff9dc5e0eb9040 ti: ffff9dc5e4e30000 task.ti: ffff9dc5e4e30000
RIP: 0010:[<ffffffff8151117a>]  [<ffffffff8151117a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9dc5e4e33bf0  EFLAGS: 00000202
RAX: ffff9dc35427f900 RBX: ffff9dc350412b70 RCX: ffff9dc35427f990
RDX: ffff9dc5ffc1b700 RSI: ffff9dc5e4e33bf0 RDI: ffff9dc5e4e33bf0
RBP: ffff9dc5e4e33c38 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffff9dc35064d6c0 R11: 00007f6d456ee000 R12: ffff9dc5e4e33bb0
R13: 0000000000000001 R14: ffff9dc5ffd1ac00 R15: ffff9dc5e0eb90a8
FS:  00007f6d457eb700(0000) GS:ffff9dc5ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4492a6ae00 CR3: 00000003a62e8000 CR4: 00000000000007e0
Call Trace:
[<ffffffff8147a3d0>] ? leave_mm+0x110/0x110
[<ffffffff8147a3d0>] ? leave_mm+0x110/0x110
[<ffffffff8147a3d0>] ? leave_mm+0x110/0x110
[<ffffffff8151128f>] smp_call_function_single+0x5f/0xa0
[<ffffffff817756d5>] ? cpumask_next_and+0x35/0x50
[<ffffffff8151183b>] smp_call_function_many+0x22b/0x270
[<ffffffff8147a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8147a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff815e42e7>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff815e5be5>] tlb_finish_mmu+0x55/0x60
[<ffffffff815e8a32>] zap_page_range+0x142/0x190
[<ffffffff815e38fd>] SyS_madvise+0x49d/0xac0
[<ffffffff815d639b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff81b74d21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff81b74d15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffff81b74ddb>] system_call_fastpath+0x22/0x27
[<ffffffff81b74d21>] ? system_call_after_swapgs+0xae/0x146
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 14 82 48 89 df e8 aa b3 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #183463
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffbad0aa2a
Build id: 3.10.0-862.14.4.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffffbac79c90
Build id: 3.10.0-862.14.4.el7.x86_64 		0x110
3
? leave_mm
vmlinux 0xffffffffbac79c90
Build id: 3.10.0-862.14.4.el7.x86_64 		0x110
4
? leave_mm
vmlinux 0xffffffffbac79c90
Build id: 3.10.0-862.14.4.el7.x86_64 		0x110
5
smp_call_function_single
vmlinux 0xffffffffbad0ab3f
Build id: 3.10.0-862.14.4.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffbaf4edb5
Build id: 3.10.0-862.14.4.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffbad0b0eb
Build id: 3.10.0-862.14.4.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffbac79e58
Build id: 3.10.0-862.14.4.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffbac79ec8
Build id: 3.10.0-862.14.4.el7.x86_64 		0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffffbadc3487
Build id: 3.10.0-862.14.4.el7.x86_64 		0x37
11
tlb_finish_mmu
vmlinux 0xffffffffbadc4d85
Build id: 3.10.0-862.14.4.el7.x86_64 		0x55
12
zap_page_range
vmlinux 0xffffffffbadc7bd2
Build id: 3.10.0-862.14.4.el7.x86_64 		0x142
13
SyS_madvise
vmlinux 0xffffffffbadc2add
Build id: 3.10.0-862.14.4.el7.x86_64 		0x3cd
14
? vm_mmap_pgoff
vmlinux 0xffffffffbadb63ab
Build id: 3.10.0-862.14.4.el7.x86_64 		0xeb
15
? system_call_after_swapgs
vmlinux 0xffffffffbb3256e1
Build id: 3.10.0-862.14.4.el7.x86_64 		0xae
16
? system_call_after_swapgs
vmlinux 0xffffffffbb3256d5
Build id: 3.10.0-862.14.4.el7.x86_64 		0xa2
17
system_call_fastpath
vmlinux 0xffffffffbb32579b
Build id: 3.10.0-862.14.4.el7.x86_64 		0x22
18
? system_call_after_swapgs
vmlinux 0xffffffffbb3256e1
Build id: 3.10.0-862.14.4.el7.x86_64 		0xae 
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [java:41660]
Modules linked in: tcp_lp rfcomm fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel btusb btrtl btbcm btintel aesni_intel bluetooth lrw snd_ens1371 gf128mul glue_helper snd_rawmidi ablk_helper cryptd snd_ac97_codec ppdev ac97_bus snd_seq vmw_balloon snd_seq_device snd_pcm rfkill snd_timer pcspkr snd sg vmw_vmci joydev soundcore i2c_piix4 nfit shpchp libnvdimm parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif
crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul drm crct10dif_common libata crc32c_intel mptspi e1000 scsi_transport_spi serio_raw mptscsih mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 41660 Comm: java Kdump: loaded Tainted: G             L ------------   3.10.0-862.14.4.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffffa14698e7af70 ti: ffffa14673a30000 task.ti: ffffa14673a30000
RIP: 0010:[<ffffffffbad0aa2a>]  [<ffffffffbad0aa2a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffffa14673a33bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffffa14673a33bc0 RCX: 0000000000000001
RDX: ffffffffbb4132b0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffffa14673a33c38 R08: ffffffffbb4132a8 R09: 00003ffffffff000
R10: ffffa1456534ea20 R11: 00007f8822269000 R12: ffffa146bffcc000
R13: ffffffffbada1626 R14: ffffa14673a33c00 R15: 0000000000020200
FS:  00007f8822366700(0000) GS:ffffa146ab700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7df333e000 CR3: 000000009e988000 CR4: 00000000000607e0
Call Trace:
[<ffffffffbac79c90>] ? leave_mm+0x110/0x110
[<ffffffffbac79c90>] ? leave_mm+0x110/0x110
[<ffffffffbac79c90>] ? leave_mm+0x110/0x110
[<ffffffffbad0ab3f>] smp_call_function_single+0x5f/0xa0
[<ffffffffbaf4edb5>] ? cpumask_next_and+0x35/0x50
[<ffffffffbad0b0eb>] smp_call_function_many+0x22b/0x270
[<ffffffffbac79e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbac79ec8>] flush_tlb_mm_range+0x68/0x140
[<ffffffffbadc3487>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffbadc4d85>] tlb_finish_mmu+0x55/0x60
[<ffffffffbadc7bd2>] zap_page_range+0x142/0x190
[<ffffffffbadc2add>] SyS_madvise+0x3cd/0x9c0
[<ffffffffbadb63ab>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffbb3256e1>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffbb3256d5>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffbb32579b>] system_call_fastpath+0x22/0x27
[<ffffffffbb3256e1>] ? system_call_after_swapgs+0xae/0x146
Code: c0 96 01 00 48 89 de 48 03 14 c5 a0 54 93 bb 48 89 df e8 6a a1 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #15668
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb3f16f72
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffffb3e7e730
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffffb3e7e730
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffffb3e7e730
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffffb3f1708f
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb4185005
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffb3f1763b
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb3e7e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb3e7e96b
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x6b
10
zap_page_range
vmlinux 0xffffffffb3ff2ed8
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xd8
11
? do_mmap
vmlinux 0xffffffffb3ffcb28
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x378
12
SyS_madvise
vmlinux 0xffffffffb3fed3ed
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x49d
13
? vm_mmap_pgoff
vmlinux 0xffffffffb3fde63b
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xeb
14
? system_call_after_swapgs
vmlinux 0xffffffffb4592e15
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xa2
15
? system_call_after_swapgs
vmlinux 0xffffffffb4592e09
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x96
16
system_call_fastpath
vmlinux 0xffffffffb4592ed2
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x25
17
? system_call_after_swapgs
vmlinux 0xffffffffb4592e15
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [qtp366590980-21:17425]
Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dm_mirror dm_region_hash dm_log dm_mod sg virtio_balloon joydev pcspkr i2c_piix4 ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi virtio_net net_failover failover virtio_blk bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks
floppy
CPU: 0 PID: 17425 Comm: qtp366590980-21 Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-48-gd9c812dda519-prebuilt.qemu.org 04/01/2014
task: ffff9a94f458e2a0 ti: ffff9a9357924000 task.ti: ffff9a9357924000
RIP: 0010:[<ffffffffb3f16f72>]  [<ffffffffb3f16f72>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9a9357927c30  EFLAGS: 00000202
RAX: 0000000000000002 RBX: ffff9a9357927c00 RCX: 0000000000000830
RDX: 0000000000000002 RSI: 0000000000000002 RDI: 0000000000000286
RBP: ffff9a9357927c80 R08: ffffffffb4613940 R09: 00007fd87e58a000
R10: 0000000000000008 R11: 0000000000000206 R12: ffffffffb3fcdd35
R13: ffff9a9357927ba8 R14: ffff9a9357927c88 R15: ffff9a9357927c88
FS:  00007fd87e5c6700(0000) GS:ffff9a94ffc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fafe5c76000 CR3: 00000000b682c000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb3e7e730>] ? leave_mm+0x120/0x120
[<ffffffffb3e7e730>] ? leave_mm+0x120/0x120
[<ffffffffb3e7e730>] ? leave_mm+0x120/0x120
[<ffffffffb3f1708f>] smp_call_function_single+0x5f/0xa0
[<ffffffffb4185005>] ? cpumask_next_and+0x35/0x50
[<ffffffffb3f1763b>] smp_call_function_many+0x22b/0x270
[<ffffffffb3e7e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb3e7e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb3ff2ed8>] zap_page_range+0xd8/0x150
[<ffffffffb3ffcb28>] ? do_mmap+0x378/0x530
[<ffffffffb3fed3ed>] SyS_madvise+0x49d/0xab0
[<ffffffffb3fde63b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffb4592e15>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb4592e09>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb4592ed2>] system_call_fastpath+0x25/0x2a
[<ffffffffb4592e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 a0 10 b5 b4 48 89 df e8 07 52 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #6753
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9e1163c2
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffff9e07e2b0
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x110
3
? leave_mm
vmlinux 0xffffffff9e07e2b0
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x110
4
? leave_mm
vmlinux 0xffffffff9e07e2b0
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x110
5
smp_call_function_single
vmlinux 0xffffffff9e1164df
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9e381635
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff9e116a8b
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9e07e478
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9e07e4eb
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9e1ec026
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff9e1edaef
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff9e1edc13
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x23
13
zap_page_range
vmlinux 0xffffffff9e1f091a
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xea
14
? do_mmap
vmlinux 0xffffffff9e1fad08
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x378
15
SyS_madvise
vmlinux 0xffffffff9e1eb62d
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x49d
16
? vm_mmap_pgoff
vmlinux 0xffffffff9e1ddfdb
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xeb
17
? system_call_after_swapgs
vmlinux 0xffffffff9e78de21
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xae
18
? system_call_after_swapgs
vmlinux 0xffffffff9e78de15
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xa2
19
system_call_fastpath
vmlinux 0xffffffff9e78dede
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x25
20
? system_call_after_swapgs
vmlinux 0xffffffff9e78de21
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xae 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [qtp1225197672-3:15107]
Modules linked in: nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dm_mirror dm_region_hash dm_log dm_mod ppdev aesni_intel lrw gf128mul sg glue_helper ablk_helper cryptd pcspkr joydev i2c_piix4 virtio_balloon parport_pc parport ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_common sr_mod cdrom virtio_net virtio_console virtio_scsi
ata_generic pata_acpi bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm ata_piix libata drm_panel_orientation_quirks virtio_pci serio_raw virtio_ring virtio floppy
CPU: 2 PID: 15107 Comm: qtp1225197672-3 Tainted: G             L ------------   3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014
task: ffff9707ef4f5230 ti: ffff9706bbafc000 task.ti: ffff9706bbafc000
RIP: 0010:[<ffffffff9e1163c2>]  [<ffffffff9e1163c2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9706bbaffbe0  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff9706bbaffbb0 RCX: 000000000000003c
RDX: 000000000000000f RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff9706bbaffc30 R08: ffffffff9e813540 R09: 00003ffffffff000
R10: ffff970aa2f7a798 R11: 00007fb77eeb1000 R12: 000000007fc1ad00
R13: fffffd88c96e36f0 R14: 00007fb77eeb1000 R15: 000000018040003c
FS:  00007fb77eeed700(0000) GS:ffff97057fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f562fb7c818 CR3: 00000002d669a000 CR4: 00000000000006e0
Call Trace:
[<ffffffff9e07e2b0>] ? leave_mm+0x110/0x110
[<ffffffff9e07e2b0>] ? leave_mm+0x110/0x110
[<ffffffff9e07e2b0>] ? leave_mm+0x110/0x110
[<ffffffff9e1164df>] smp_call_function_single+0x5f/0xa0
[<ffffffff9e381635>] ? cpumask_next_and+0x35/0x50
[<ffffffff9e116a8b>] smp_call_function_many+0x22b/0x270
[<ffffffff9e07e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9e07e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9e1ec026>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9e1edaef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9e1edc13>] tlb_finish_mmu+0x23/0x30
[<ffffffff9e1f091a>] zap_page_range+0xea/0x130
[<ffffffff9e1fad08>] ? do_mmap+0x378/0x530
[<ffffffff9e1eb62d>] SyS_madvise+0x49d/0xac0
[<ffffffff9e1ddfdb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff9e78de21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff9e78de15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffff9e78dede>] system_call_fastpath+0x25/0x2a
[<ffffffff9e78de21>] ? system_call_after_swapgs+0xae/0x146
Code: 89 de 48 03 14 c5 20 e8 d4 9e 48 89 df e8 f7 22 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #2210
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff8c71117a
Build id: 3.10.0-957.1.3.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff8c67a3d0
Build id: 3.10.0-957.1.3.el7.x86_64 		0x110
3
? leave_mm
vmlinux 0xffffffff8c67a3d0
Build id: 3.10.0-957.1.3.el7.x86_64 		0x110
4
? leave_mm
vmlinux 0xffffffff8c67a3d0
Build id: 3.10.0-957.1.3.el7.x86_64 		0x110
5
smp_call_function_single
vmlinux 0xffffffff8c71128f
Build id: 3.10.0-957.1.3.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff8c9756d5
Build id: 3.10.0-957.1.3.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff8c71183b
Build id: 3.10.0-957.1.3.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8c67a598
Build id: 3.10.0-957.1.3.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8c67a608
Build id: 3.10.0-957.1.3.el7.x86_64 		0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff8c7e42e7
Build id: 3.10.0-957.1.3.el7.x86_64 		0x37
11
tlb_finish_mmu
vmlinux 0xffffffff8c7e5be5
Build id: 3.10.0-957.1.3.el7.x86_64 		0x55
12
zap_page_range
vmlinux 0xffffffff8c7e8a32
Build id: 3.10.0-957.1.3.el7.x86_64 		0x142
13
? do_mmap
vmlinux 0xffffffff8c7f2b68
Build id: 3.10.0-957.1.3.el7.x86_64 		0x378
14
SyS_madvise
vmlinux 0xffffffff8c7e38fd
Build id: 3.10.0-957.1.3.el7.x86_64 		0x49d
15
? vm_mmap_pgoff
vmlinux 0xffffffff8c7d639b
Build id: 3.10.0-957.1.3.el7.x86_64 		0xeb
16
? system_call_after_swapgs
vmlinux 0xffffffff8cd74d21
Build id: 3.10.0-957.1.3.el7.x86_64 		0xae
17
? system_call_after_swapgs
vmlinux 0xffffffff8cd74d15
Build id: 3.10.0-957.1.3.el7.x86_64 		0xa2
18
system_call_fastpath
vmlinux 0xffffffff8cd74ddb
Build id: 3.10.0-957.1.3.el7.x86_64 		0x22
19
? system_call_after_swapgs
vmlinux 0xffffffff8cd74d21
Build id: 3.10.0-957.1.3.el7.x86_64 		0xae 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [java:5110]
Modules linked in: nfring(OE) xt_iprange xt_NFQUEUE nfnetlink_log binfmt_misc xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype br_netfilter ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock ppdev sb_edac coretemp vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport
vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata mptspi scsi_transport_spi vmxnet3 mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: nfring]
CPU: 1 PID: 5110 Comm: java Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-957.1.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffffa07a41456180 ti: ffffa07bfb1d0000 task.ti: ffffa07bfb1d0000
RIP: 0010:[<ffffffff8c71117a>]  [<ffffffff8c71117a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffffa07bfb1d3bf0  EFLAGS: 00000202
RAX: ffffa0807fb5fc00 RBX: ffffa07bc1e1c4b0 RCX: ffffa0807fb5fce0
RDX: ffffa0817fc1b700 RSI: ffffa07bfb1d3bf0 RDI: ffffa07bfb1d3bf0
RBP: ffffa07bfb1d3c38 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffffa08060af36c8 R11: 00007f6edf305000 R12: ffffa07a97112500
R13: 00000000edcab7c0 R14: 00007f6edf305000 R15: ffffa07a41456180
FS:  00007f6edf33a700(0000) GS:ffffa0817fc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6a40de802f CR3: 000000031eca6000 CR4: 00000000000607e0
Call Trace:
[<ffffffff8c67a3d0>] ? leave_mm+0x110/0x110
[<ffffffff8c67a3d0>] ? leave_mm+0x110/0x110
[<ffffffff8c67a3d0>] ? leave_mm+0x110/0x110
[<ffffffff8c71128f>] smp_call_function_single+0x5f/0xa0
[<ffffffff8c9756d5>] ? cpumask_next_and+0x35/0x50
[<ffffffff8c71183b>] smp_call_function_many+0x22b/0x270
[<ffffffff8c67a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8c67a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff8c7e42e7>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff8c7e5be5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8c7e8a32>] zap_page_range+0x142/0x190
[<ffffffff8c7f2b68>] ? do_mmap+0x378/0x530
[<ffffffff8c7e38fd>] SyS_madvise+0x49d/0xac0
[<ffffffff8c7d639b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff8cd74d21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff8cd74d15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffff8cd74ddb>] system_call_fastpath+0x22/0x27
[<ffffffff8cd74d21>] ? system_call_after_swapgs+0xae/0x146
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 34 8d 48 89 df e8 aa b3 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00