Info

Permalink
Function
smp_call_function_many
First occurrence
2020-11-13
Last occurrence
2025-08-11
State
NEW
Components
kernel
Quality
-27

Statistics

Operating system Count
CentOS 7 304
Architecture Count
x86_64 304
Related packages Count
kernel 304
 0:3.10.0-1160.59.1.el7 79
 0:3.10.0-1062.12.1.el7 67
 0:3.10.0-1160.el7 28
 0:3.10.0-1160.71.1.el7 22
 0:3.10.0-1062.el7 18
 0:3.10.0-1160.76.1.el7 18
 0:3.10.0-1160.31.1.el7 13
 0:3.10.0-1160.36.2.el7 11
 0:3.10.0-1160.11.1.el7 10
 0:3.10.0-1160.15.2.el7 10
 0:3.10.0-1160.6.1.el7 6
 0:3.10.0-1160.62.1.el7 6
 0:3.10.0-1127.el7 3
 0:3.10.0-1062.4.3.el7 2
 0:3.10.0-1160.49.1.el7 2
 0:3.10.0-1160.25.1.el7 2
 0:3.10.0-1160.53.1.el7 2
 0:3.10.0-1160.66.1.el7 2
 0:3.10.0-1062.9.1.el7 1
 0:3.10.0-1160.24.1.el7 1
 0:3.10.0-1127.18.2.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #545040
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbbf16eea
Build id: 3.10.0-1160.25.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbbe7e868
Build id: 3.10.0-1160.25.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbbe7e8db
Build id: 3.10.0-1160.25.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbbfedf56
Build id: 3.10.0-1160.25.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbbfefa1f
Build id: 3.10.0-1160.25.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbbfefb43
Build id: 3.10.0-1160.25.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffbbff300a
Build id: 3.10.0-1160.25.1.el7.x86_64
0xea
8
? do_sync_write
vmlinux 0xffffffffbc04d313
Build id: 3.10.0-1160.25.1.el7.x86_64
0x93
9
SyS_madvise
vmlinux 0xffffffffbbfed4fd
Build id: 3.10.0-1160.25.1.el7.x86_64
0x49d
10
? vfs_write
vmlinux 0xffffffffbc04dec2
Build id: 3.10.0-1160.25.1.el7.x86_64
0x182
11
? system_call_after_swapgs
vmlinux 0xffffffffbc595ed5
Build id: 3.10.0-1160.25.1.el7.x86_64
0xa2
12
? system_call_after_swapgs
vmlinux 0xffffffffbc595ec9
Build id: 3.10.0-1160.25.1.el7.x86_64
0x96
13
system_call_fastpath
vmlinux 0xffffffffbc595f92
Build id: 3.10.0-1160.25.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffffbc595ed5
Build id: 3.10.0-1160.25.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [iphit_0MsgLog:16658]
Modules linked in: tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink vboxsf(OE) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc vmwgfx snd_intel8x0 ttm snd_ac97_codec drm_kms_helper ac97_bus syscopyarea snd_seq sysfillrect snd_seq_device sysimgblt virtio_net snd_pcm fb_sys_fops joydev net_failover drm ppdev snd_timer failover parport_pc vboxguest(OE) drm_panel_orientation_quirks snd
parport sg i2c_piix4 pcspkr soundcore video ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi ahci serio_raw libahci ata_piix virtio_pci e1000 virtio_ring libata crc32c_intel virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 16658 Comm: iphit_0MsgLog Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff9cff35c42100 ti: ffff9d02aad14000 task.ti: ffff9d02aad14000
RIP: 0010:[<ffffffffbbf16eea>]  [<ffffffffbbf16eea>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9d02aad17c70  EFLAGS: 00010202
RAX: 0000000000000002 RBX: ffffffffbbe7e6a0 RCX: ffff9d031fca0450
RDX: 0000000000000002 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff9d02aad17ca8 R08: ffff9d0079359000 R09: ffffffffbc186549
R10: ffff9d031fcdf160 R11: fffff14285f1e400 R12: ffff9d02aad17c40
R13: ffffffffbc18649f R14: ffff9d02aad17c68 R15: 0000000000000057
FS:  00007f81d93c8700(0000) GS:ffff9d031fcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f81c80012c8 CR3: 000000036b1e4000 CR4: 00000000000206e0
Call Trace:
[<ffffffffbbe7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbbe7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbbfedf56>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffbbfefa1f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbbfefb43>] tlb_finish_mmu+0x23/0x30
[<ffffffffbbff300a>] zap_page_range+0xea/0x130
[<ffffffffbc04d313>] ? do_sync_write+0x93/0xe0
[<ffffffffbbfed4fd>] SyS_madvise+0x49d/0xac0
[<ffffffffbc04dec2>] ? vfs_write+0x182/0x1f0
[<ffffffffbc595ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffbc595ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffbc595f92>] system_call_fastpath+0x25/0x2a
[<ffffffffbc595ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 fe 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 15 b5 bc f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #514839
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb171722a
Build id: 3.10.0-1160.62.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb167e258
Build id: 3.10.0-1160.62.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb167e2cb
Build id: 3.10.0-1160.62.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb17ee666
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb17f012f
Build id: 3.10.0-1160.62.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb17f0253
Build id: 3.10.0-1160.62.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffb17f371a
Build id: 3.10.0-1160.62.1.el7.x86_64
0xea
8
? do_sync_write
vmlinux 0xffffffffb184da03
Build id: 3.10.0-1160.62.1.el7.x86_64
0x93
9
SyS_madvise
vmlinux 0xffffffffb17edc0d
Build id: 3.10.0-1160.62.1.el7.x86_64
0x49d
10
? vfs_write
vmlinux 0xffffffffb184e538
Build id: 3.10.0-1160.62.1.el7.x86_64
0x168
11
? system_call_after_swapgs
vmlinux 0xffffffffb1d99ed5
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa2
12
? system_call_after_swapgs
vmlinux 0xffffffffb1d99ec9
Build id: 3.10.0-1160.62.1.el7.x86_64
0x96
13
system_call_fastpath
vmlinux 0xffffffffb1d99f92
Build id: 3.10.0-1160.62.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffffb1d99ed5
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [IPDL Background:2776]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel nfit libnvdimm vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_vmci joydev pcspkr sg i2c_piix4 parport_pc parport ip_tables xfs
libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ata_piix crct10dif_pclmul crct10dif_common ttm crc32c_intel libata serio_raw drm e1000 mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 2776 Comm: IPDL Background Kdump: loaded Tainted: G             L ------------   3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9f4bb63eb180 ti: ffff9f4b07e00000 task.ti: ffff9f4b07e00000
RIP: 0010:[<ffffffffb171722a>]  [<ffffffffb171722a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9f4b07e03c70  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff9f4b07e03c30 RCX: ffff9f4bbfc9fc10
RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff9f4b07e03ca8 R08: ffff9f4bbb152000 R09: ffffffffb19869d9
R10: ffff9f4bbfc1f160 R11: fffffee982721800 R12: ffff9f4b07e03c40
R13: ffffffffb198692f R14: ffff9f4bbb152c00 R15: 0000000000000001
FS:  00007fc717f7c700(0000) GS:ffff9f4bbfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc6ecf6b000 CR3: 000000009fc9c000 CR4: 00000000000407f0
Call Trace:
[<ffffffffb167e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb167e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb17ee666>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb17f012f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb17f0253>] tlb_finish_mmu+0x23/0x30
[<ffffffffb17f371a>] zap_page_range+0xea/0x130
[<ffffffffb184da03>] ? do_sync_write+0x93/0xe0
[<ffffffffb17edc0d>] SyS_madvise+0x49d/0xac0
[<ffffffffb184e538>] ? vfs_write+0x168/0x1f0
[<ffffffffb1d99ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb1d99ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb1d99f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb1d99ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 7e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 35 b2 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #474099
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb7b1720a
Build id: 3.10.0-1160.66.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb7a7e258
Build id: 3.10.0-1160.66.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb7a7e2cb
Build id: 3.10.0-1160.66.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb7bee676
Build id: 3.10.0-1160.66.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb7bf013f
Build id: 3.10.0-1160.66.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb7bf0263
Build id: 3.10.0-1160.66.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffb7bf372a
Build id: 3.10.0-1160.66.1.el7.x86_64
0xea
8
? __slab_free
vmlinux 0xffffffffb7c267ec
Build id: 3.10.0-1160.66.1.el7.x86_64
0x9c
9
SyS_madvise
vmlinux 0xffffffffb7bedc1d
Build id: 3.10.0-1160.66.1.el7.x86_64
0x49d
10
? system_call_after_swapgs
vmlinux 0xffffffffb8199ed5
Build id: 3.10.0-1160.66.1.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffffb8199ec9
Build id: 3.10.0-1160.66.1.el7.x86_64
0x96
12
system_call_fastpath
vmlinux 0xffffffffb8199f92
Build id: 3.10.0-1160.66.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffffb8199ed5
Build id: 3.10.0-1160.66.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [nessusd:31196]
Modules linked in: cmac arc4 md4 nls_utf8 cifs ccm dns_resolver xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ipmi_devintf ipmi_msghandler dell_rbu ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_mangle iptable_security iptable_raw ip_set ebtable_filter ebtables ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 dcdbas nf_nat nf_conntrack sunrpc nfit libnvdimm snd_intel8x0 snd_ac97_codec iosf_mbi crc32_pclmul ac97_bus snd_seq ppdev ghash_clmulni_intel snd_seq_device snd_pcm aesni_intel joydev lrw gf128mul snd_timer glue_helper ablk_helper
sg cryptd snd parport_pc video parport pcspkr i2c_piix4 soundcore ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common e1000 crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 31196 Comm: nessusd Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.66.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff8a9dca9d2100 ti: ffff8a9dce598000 task.ti: ffff8a9dce598000
RIP: 0010:[<ffffffffb7b1720a>]  [<ffffffffb7b1720a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8a9dce59bc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000c0000b7a7ddd0 RCX: ffff8a9ddfc20320
RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8a9dce59bca8 R08: ffff8a9b36320800 R09: ffffffffb7d869e9
R10: ffff8a9ddfc5f160 R11: ffffdd9dc4e45000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8a9dce59bc20 R15: 0000000000000001
FS:  00007f4ecb3fd700(0000) GS:ffff8a9ddfc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4ec59a6000 CR3: 000000025a39e000 CR4: 00000000000606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb7a7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb7a7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb7bee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb7bf013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb7bf0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffb7bf372a>] zap_page_range+0xea/0x130
[<ffffffffb7c267ec>] ? __slab_free+0x9c/0x290
[<ffffffffb7bedc1d>] SyS_madvise+0x49d/0xac0
[<ffffffffb8199ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb8199ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb8199f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb8199ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 9e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 75 b8 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 6d 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #442755
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9031720a
Build id: 3.10.0-1160.66.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9027e258
Build id: 3.10.0-1160.66.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9027e2cb
Build id: 3.10.0-1160.66.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff903ee676
Build id: 3.10.0-1160.66.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff903f013f
Build id: 3.10.0-1160.66.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff903f0263
Build id: 3.10.0-1160.66.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff903f372a
Build id: 3.10.0-1160.66.1.el7.x86_64
0xea
8
? kmem_cache_free
vmlinux 0xffffffff90427022
Build id: 3.10.0-1160.66.1.el7.x86_64
0x1e2
9
SyS_madvise
vmlinux 0xffffffff903edc1d
Build id: 3.10.0-1160.66.1.el7.x86_64
0x49d
10
? system_call_after_swapgs
vmlinux 0xffffffff90999ed5
Build id: 3.10.0-1160.66.1.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffff90999ec9
Build id: 3.10.0-1160.66.1.el7.x86_64
0x96
12
system_call_fastpath
vmlinux 0xffffffff90999f92
Build id: 3.10.0-1160.66.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffff90999ed5
Build id: 3.10.0-1160.66.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 39s! [Thread (pooled):22073]
Modules linked in: tcp_lp nls_utf8 isofs xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer joydev snd ppdev parport_pc pcspkr sg soundcore vboxguest(OE) i2c_piix4 parport video binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common
ata_generic pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm ata_piix libahci drm crc32c_intel serio_raw libata e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 5 PID: 22073 Comm: Thread (pooled) Tainted: G           OE  ------------   3.10.0-1160.66.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff8913869ce300 ti: ffff89133774c000 task.ti: ffff89133774c000
RIP: 0010:[<ffffffff9031720a>]  [<ffffffff9031720a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff89133774fc70  EFLAGS: 00010202
RAX: 0000000000000000 RBX: ffffffff9027ddd0 RCX: ffff8914e00204a0
RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff89133774fca8 R08: ffff89143afd6000 R09: ffffffff905869e9
R10: ffff8914e015f160 R11: ffffe9b200d4a300 R12: ffff89133774fc40
R13: ffffffff9058693f R14: ffff8914d1a71970 R15: ffff8914d1c45a00
FS:  00007fba900c9700(0000) GS:ffff8914e0140000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00001ae9856fc950 CR3: 0000000035cb6000 CR4: 00000000000206e0
Call Trace:
[<ffffffff9027e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9027e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff903ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff903f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff903f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff903f372a>] zap_page_range+0xea/0x130
[<ffffffff90427022>] ? kmem_cache_free+0x1e2/0x200
[<ffffffff903edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff90999ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff90999ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff90999f92>] system_call_fastpath+0x25/0x2a
[<ffffffff90999ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 9e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 f5 90 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 6d 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #433542
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa01170da
Build id: 3.10.0-1160.59.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa007e258
Build id: 3.10.0-1160.59.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa007e2cb
Build id: 3.10.0-1160.59.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa01ee676
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa01f013f
Build id: 3.10.0-1160.59.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa01f0263
Build id: 3.10.0-1160.59.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffa01f372a
Build id: 3.10.0-1160.59.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffa01edc1d
Build id: 3.10.0-1160.59.1.el7.x86_64
0x49d
9
? pick_next_entity
vmlinux 0xffffffffa00e22d9
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa9
10
? set_next_entity
vmlinux 0xffffffffa00e26ec
Build id: 3.10.0-1160.59.1.el7.x86_64
0x3c
11
? system_call_after_swapgs
vmlinux 0xffffffffa0799ed5
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa2
12
? system_call_after_swapgs
vmlinux 0xffffffffa0799ec9
Build id: 3.10.0-1160.59.1.el7.x86_64
0x96
13
system_call_fastpath
vmlinux 0xffffffffa0799f92
Build id: 3.10.0-1160.59.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffffa0799ed5
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [containerd-shim:2990]
Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter rpcsec_gss_krb5 auth_rpcgss binfmt_misc nfsv4 dns_resolver nfs tun lockd grace fscache overlay(T) nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper vmw_balloon pcspkr ablk_helper
cryptd joydev sg parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c vmwgfx sr_mod cdrom drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod ata_generic pata_acpi crc_t10dif crct10dif_generic drm ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel e1000 serio_raw vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 2990 Comm: containerd-shim Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 10/22/2013
task: ffff8adc4edbb180 ti: ffff8ad95704c000 task.ti: ffff8ad95704c000
RIP: 0010:[<ffffffffa01170da>]  [<ffffffffa01170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8ad95704fc70  EFLAGS: 00000202
RAX: 0000000000000003 RBX: ffffffffa007ddd0 RCX: ffff8adcffda0600
RDX: 0000000000000003 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff8ad95704fca8 R08: ffff8ada36345c00 R09: ffffffffa0386c09
R10: ffff8adcffd1f160 R11: ffffd19d8adafe00 R12: ffff8ad95704fc40
R13: ffffffffa0386b5f R14: 0000000000000200 R15: ffffffffa11dcb80
FS:  000000c000200490(0000) GS:ffff8adcffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f94c478d37c CR3: 0000000384ad4000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa007e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa007e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa01ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa01f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa01f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffa01f372a>] zap_page_range+0xea/0x130
[<ffffffffa01edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffffa00e22d9>] ? pick_next_entity+0xa9/0x190
[<ffffffffa00e26ec>] ? set_next_entity+0x3c/0xe0
[<ffffffffa0799ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffa0799ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffa0799f92>] system_call_fastpath+0x25/0x2a
[<ffffffffa0799ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 d5 a0 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #424650
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8a1170da
Build id: 3.10.0-1160.53.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8a07e258
Build id: 3.10.0-1160.53.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8a07e2cb
Build id: 3.10.0-1160.53.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8a1ee676
Build id: 3.10.0-1160.53.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8a1f013f
Build id: 3.10.0-1160.53.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8a1f0263
Build id: 3.10.0-1160.53.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff8a1f372a
Build id: 3.10.0-1160.53.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff8a1edc1d
Build id: 3.10.0-1160.53.1.el7.x86_64
0x49d
9
? __switch_to
vmlinux 0xffffffff8a02b621
Build id: 3.10.0-1160.53.1.el7.x86_64
0x151
10
? system_call_after_swapgs
vmlinux 0xffffffff8a795ed5
Build id: 3.10.0-1160.53.1.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffff8a795ec9
Build id: 3.10.0-1160.53.1.el7.x86_64
0x96
12
system_call_fastpath
vmlinux 0xffffffff8a795f92
Build id: 3.10.0-1160.53.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffff8a795ed5
Build id: 3.10.0-1160.53.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 33s! [MarkLogic:32758]
Modules linked in: dcdbas vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 sb_edac ppdev iosf_mbi crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport auth_rpcgss sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix e1000 mptspi serio_raw libata nfit scsi_transport_spi mptscsih libnvdimm mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 12 PID: 32758 Comm: MarkLogic Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff892f93704200 ti: ffff892af0bdc000 task.ti: ffff892af0bdc000
RIP: 0010:[<ffffffff8a1170da>]  [<ffffffff8a1170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff892af0bdfc70  EFLAGS: 00000202
RAX: 000000000000000f RBX: ffffffff8a07ddd0 RCX: ffff89371f5e07b8
RDX: 000000000000000f RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff892af0bdfca8 R08: ffff892083169400 R09: ffffffff8a386bf9
R10: ffff89371f51f160 R11: ffffe64cb6163c00 R12: 0000000000000286
R13: ffff892af0bdfc20 R14: ffff8920bfd5bc00 R15: 0000000000000001
FS:  00007f917f019700(0000) GS:ffff89371f500000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f91435f6000 CR3: 0000000f9539e000 CR4: 00000000000407e0
Call Trace:
[<ffffffff8a07e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8a07e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8a1ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8a1f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8a1f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff8a1f372a>] zap_page_range+0xea/0x130
[<ffffffff8a1edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff8a02b621>] ? __switch_to+0x151/0x580
[<ffffffff8a795ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8a795ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff8a795f92>] system_call_fastpath+0x25/0x2a
[<ffffffff8a795ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 d5 8a f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #424568
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffaf11722a
Build id: 3.10.0-1160.62.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffaf07e258
Build id: 3.10.0-1160.62.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffaf07e2cb
Build id: 3.10.0-1160.62.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffaf1ee666
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffaf1f012f
Build id: 3.10.0-1160.62.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffaf1f0253
Build id: 3.10.0-1160.62.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffaf1f371a
Build id: 3.10.0-1160.62.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffaf1edc0d
Build id: 3.10.0-1160.62.1.el7.x86_64
0x49d
9
? put_prev_entity
vmlinux 0xffffffffaf0e64cb
Build id: 3.10.0-1160.62.1.el7.x86_64
0x8b
10
? __switch_to
vmlinux 0xffffffffaf02b621
Build id: 3.10.0-1160.62.1.el7.x86_64
0x151
11
? system_call_after_swapgs
vmlinux 0xffffffffaf799ed5
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa2
12
? system_call_after_swapgs
vmlinux 0xffffffffaf799ec9
Build id: 3.10.0-1160.62.1.el7.x86_64
0x96
13
system_call_fastpath
vmlinux 0xffffffffaf799f92
Build id: 3.10.0-1160.62.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffffaf799ed5
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [StreamT~ns #275:5750]
Modules linked in: tcp_lp nls_utf8 isofs veth vboxsf(OE) xt_addrtype br_netfilter overlay(T) xt_CHECKSUM tun devlink xt_REDIRECT nf_nat_redirect xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_mark ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer ppdev joydev vboxguest(OE) snd parport_pc pcspkr sg
soundcore parport video i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic crct10dif_common pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm libahci drm ata_piix drm_panel_orientation_quirks serio_raw crc32c_intel libata e1000 fuse
CPU: 1 PID: 5750 Comm: StreamT~ns #275 Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff95b3e6029080 ti: ffff95b4969f8000 task.ti: ffff95b4969f8000
RIP: 0010:[<ffffffffaf11722a>]  [<ffffffffaf11722a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff95b4969fbc70  EFLAGS: 00010202
RAX: 0000000000000002 RBX: 0000000000000010 RCX: ffff95b49fd20400
RDX: 0000000000000002 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff95b4969fbca8 R08: ffff95b498491800 R09: ffffffffaf3869d9
R10: ffff95b49fc9f160 R11: ffffdbdb816cbf00 R12: ffffffffffffff10
R13: 0000000000000282 R14: 00000000000000fc R15: 00000000000008fc
FS:  00007ff682afc700(0000) GS:ffff95b49fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6a93687000 CR3: 000000009bc8c000 CR4: 00000000000206e0
Call Trace:
[<ffffffffaf07e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaf07e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaf1ee666>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffaf1f012f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaf1f0253>] tlb_finish_mmu+0x23/0x30
[<ffffffffaf1f371a>] zap_page_range+0xea/0x130
[<ffffffffaf1edc0d>] SyS_madvise+0x49d/0xac0
[<ffffffffaf0e64cb>] ? put_prev_entity+0x8b/0x400
[<ffffffffaf02b621>] ? __switch_to+0x151/0x580
[<ffffffffaf799ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffaf799ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffaf799f92>] system_call_fastpath+0x25/0x2a
[<ffffffffaf799ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 7e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 d5 af f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #369317
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff927170da
Build id: 3.10.0-1160.49.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9267e258
Build id: 3.10.0-1160.49.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9267e2cb
Build id: 3.10.0-1160.49.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff927ee676
Build id: 3.10.0-1160.49.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff927f013f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff927f0263
Build id: 3.10.0-1160.49.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff927f372a
Build id: 3.10.0-1160.49.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff927edc1d
Build id: 3.10.0-1160.49.1.el7.x86_64
0x49d
9
? __check_object_size
vmlinux 0xffffffff9284b13a
Build id: 3.10.0-1160.49.1.el7.x86_64
0x1ca
10
? system_call_after_swapgs
vmlinux 0xffffffff92d95ed5
Build id: 3.10.0-1160.49.1.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffff92d95ec9
Build id: 3.10.0-1160.49.1.el7.x86_64
0x96
12
system_call_fastpath
vmlinux 0xffffffff92d95f92
Build id: 3.10.0-1160.49.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffff92d95ed5
Build id: 3.10.0-1160.49.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [DOM Worker:10243]
Modules linked in: vmfsrtx_export(POE) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr joydev sg vmw_balloon parport_pc parport vmw_vmci i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif
crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit ata_piix libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel mptspi scsi_transport_spi libata mptscsih serio_raw mptbase vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: vmfsrtx_export]
CPU: 3 PID: 10243 Comm: DOM Worker Kdump: loaded Tainted: P        W  OEL ------------   3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff88995ef52100 ti: ffff8899f6a98000 task.ti: ffff8899f6a98000
RIP: 0010:[<ffffffff927170da>]  [<ffffffff927170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8899f6a9bc70  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffffff9267ddd0 RCX: ffffd8c33f644118
RDX: 0000000000000001 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff8899f6a9bca8 R08: ffff887b732b0800 R09: ffffffff92986bf9
R10: ffff8899feedf160 R11: ffffed5a3de4ae00 R12: 0000000000000286
R13: ffff8899f6a9bc20 R14: ffff887b73690000 R15: 0000000000000001
FS:  00007fdfb65ff700(0000) GS:ffff8899feec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002428ffa16c28 CR3: 0000001f666c0000 CR4: 00000000000407e0
Call Trace:
[<ffffffff9267e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9267e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff927ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff927f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff927f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff927f372a>] zap_page_range+0xea/0x130
[<ffffffff927edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff9284b13a>] ? __check_object_size+0x1ca/0x250
[<ffffffff92d95ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff92d95ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff92d95f92>] system_call_fastpath+0x25/0x2a
[<ffffffff92d95ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 35 93 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #366140
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffad516c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffad47e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffad47e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffad5edd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffad5ef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffad5ef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffad5f2dba
Build id: 3.10.0-1160.el7.x86_64
0xea
8
? handle_mm_fault
vmlinux 0xffffffffad5f5e0d
Build id: 3.10.0-1160.el7.x86_64
0xbfd
9
SyS_madvise
vmlinux 0xffffffffad5ed2ad
Build id: 3.10.0-1160.el7.x86_64
0x49d
10
? system_call_after_swapgs
vmlinux 0xffffffffadb93ed5
Build id: 3.10.0-1160.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffffadb93ec9
Build id: 3.10.0-1160.el7.x86_64
0x96
12
system_call_fastpath
vmlinux 0xffffffffadb93f92
Build id: 3.10.0-1160.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffffadb93ed5
Build id: 3.10.0-1160.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [containerd:2600]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc snd_hda_codec_idt snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer ppdev snd sg parport_pc joydev soundcore pcspkr i2c_piix4 parport video ip_tables
xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ahci libahci crc32c_intel libata e1000 serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 2600 Comm: containerd Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff8f89bffdb180 ti: ffff8f89d83c4000 task.ti: ffff8f89d83c4000
RIP: 0010:[<ffffffffad516c0a>]  [<ffffffffad516c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8f89d83c7c70  EFLAGS: 00010202
RAX: 0000000000000000 RBX: ffffffffad47e6a0 RCX: ffff8f89df420400
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff8f89d83c7ca8 R08: ffff8f89b43bd800 R09: ffffffffad785f99
R10: ffff8f89df49f160 R11: ffffe634836a8000 R12: ffff8f89d83c7c40
R13: ffffffffad785eef R14: ffff8f89bfc01500 R15: 0000000000000001
FS:  00007fa19ffff700(0000) GS:ffff8f89df480000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0008e3010 CR3: 00000000d7fec000 CR4: 00000000000006e0
Call Trace:
[<ffffffffad47e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffad47e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffad5edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffad5ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffad5ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffffad5f2dba>] zap_page_range+0xea/0x130
[<ffffffffad5f5e0d>] ? handle_mm_fault+0xbfd/0xfb0
[<ffffffffad5ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffffadb93ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffadb93ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffadb93f92>] system_call_fastpath+0x25/0x2a
[<ffffffffadb93ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 15 ae f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #287408
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9cd16a7e
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffff9cc7e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9cc7e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9cdec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9cdedb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9cdedc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff9cdf098a
Build id: 3.10.0-1062.12.1.el7.x86_64
0xea
8
? do_sync_write
vmlinux 0xffffffff9ce49d83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x93
9
SyS_madvise
vmlinux 0xffffffff9cdeb69d
Build id: 3.10.0-1062.12.1.el7.x86_64
0x49d
10
? system_call_after_swapgs
vmlinux 0xffffffff9d38de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
11
? system_call_after_swapgs
vmlinux 0xffffffff9d38de15
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa2
12
system_call_fastpath
vmlinux 0xffffffff9d38dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffff9d38de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [mysqld:32468]
Modules linked in: binfmt_misc tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ppdev parport_pc parport joydev sg virtio_balloon i2c_piix4 pcspkr ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_scsi virtio_blk cirrus drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata serio_raw virtio_pci virtio_ring floppy virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 32468 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff8f2ba73f20e0 ti: ffff8f2baffd0000 task.ti: ffff8f2baffd0000
RIP: 0010:[<ffffffff9cd16a7e>]  [<ffffffff9cd16a7e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8f2baffd3c70  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 000800009cc7e2b0 RCX: ffff8f2c2cfa04f0
RDX: 0000000000000006 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff8f2baffd3ca8 R08: ffff8f2979c1c400 R09: ffffffff9cf81839
R10: ffff8f2c2cfdf160 R11: ffffcc7fcec2bc00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8f2baffd3c20 R15: 0000000000000001
FS:  00007fe9052cf700(0000) GS:ffff8f2c2cfc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe9052c6a00 CR3: 00000000ba0a6000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Modules linked in: binfmt_misc tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ppdev parport_pc parport joydev sg virtio_balloon i2c_piix4 pcspkr ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_scsi virtio_blk cirrus drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata serio_raw virtio_pci virtio_ring floppy virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 32469 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff8f2ba73f0000 ti: ffff8f2c2e9f0000 task.ti: ffff8f2c2e9f0000
RIP: 0010:[<ffffffff9cd16a7a>]  [<ffffffff9cd16a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8f2c2e9f3c70  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 000800009cc7e2b0 RCX: ffff8f2c2cf9fc10
RDX: 0000000000000006 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff8f2c2e9f3ca8 R08: ffff8f297fdf7000 R09: ffffffff9cf81839
R10: ffff8f2c2ce1f160 R11: ffffcc7fce88fe00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8f2c2e9f3c20 R15: ffffffff9ceb6cd0
FS:  00007fe905351700(0000) GS:ffff8f2c2ce00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe905348a00 CR3: 00000000ba0a6000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9cc7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9cc7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9cdec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9cdedb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9cdedc83>] tlb_finish_mmu+0x23/0x30
[<ffffffff9cdf098a>] zap_page_range+0xea/0x130
[<ffffffff9ce49d83>] ? do_sync_write+0x93/0xe0
[<ffffffff9cdeb69d>] SyS_madvise+0x49d/0xac0
[<ffffffff9d38de21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff9d38de15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffff9d38dede>] system_call_fastpath+0x25/0x2a
[<ffffffff9d38de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 94 9d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #195049
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa3516eea
Build id: 3.10.0-1160.25.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa347e868
Build id: 3.10.0-1160.25.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa347e8db
Build id: 3.10.0-1160.25.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa35edf56
Build id: 3.10.0-1160.25.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa35efa1f
Build id: 3.10.0-1160.25.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa35efb43
Build id: 3.10.0-1160.25.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffa35f300a
Build id: 3.10.0-1160.25.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffa35ed4fd
Build id: 3.10.0-1160.25.1.el7.x86_64
0x49d
9
? system_call_after_swapgs
vmlinux 0xffffffffa3b95ed5
Build id: 3.10.0-1160.25.1.el7.x86_64
0xa2
10
? system_call_after_swapgs
vmlinux 0xffffffffa3b95ec9
Build id: 3.10.0-1160.25.1.el7.x86_64
0x96
11
system_call_fastpath
vmlinux 0xffffffffa3b95f92
Build id: 3.10.0-1160.25.1.el7.x86_64
0x25
12
? system_call_after_swapgs
vmlinux 0xffffffffa3b95ed5
Build id: 3.10.0-1160.25.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [process reaper:9542]
Modules linked in: event_collector_2_1_364486(OE) binfmt_misc xt_physdev br_netfilter ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter talpa_vfshook(OE) talpa_pedconnector(OE) talpa_pedevice(OE) talpa_vcdevice(OE) talpa_core(OE) talpa_linux(OE) talpa_syscallhook(OE) vmw_vsock_vmci_transport vsock crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev ppdev sg vmw_balloon
parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul ata_piix crct10dif_common crc32c_intel nfit libata mptspi scsi_transport_spi libnvdimm serio_raw mptscsih mptbase vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 9542 Comm: process reaper Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff882afde39080 ti: ffff882abb1b4000 task.ti: ffff882abb1b4000
RIP: 0010:[<ffffffffa3516eea>]  [<ffffffffa3516eea>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff882abb1b7c70  EFLAGS: 00000202
RAX: 0000000000000007 RBX: 00080000a347e6a0 RCX: ffff882c317e04d0
RDX: 0000000000000007 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff882abb1b7ca8 R08: ffff882a68be3000 R09: ffffffffa3786549
R10: ffff882c3165f160 R11: ffffca9982ed1800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff882abb1b7c20 R15: 0000000000000001
FS:  00007f22f40bc700(0000) GS:ffff882c31640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f225c00d000 CR3: 00000000b6c42000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa347e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa347e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa35edf56>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa35efa1f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa35efb43>] tlb_finish_mmu+0x23/0x30
[<ffffffffa35f300a>] zap_page_range+0xea/0x130
[<ffffffffa35ed4fd>] SyS_madvise+0x49d/0xac0
[<ffffffffa3b95ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffa3b95ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffa3b95f92>] system_call_fastpath+0x25/0x2a
[<ffffffffa3b95ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 fe 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 15 15 a4 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #177863
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb13150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb127d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb127d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb13ea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb13ebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb13ebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffb13eea5a
Build id: 3.10.0-1062.el7.x86_64
0xea
8
? kmem_cache_free
vmlinux 0xffffffffb1422912
Build id: 3.10.0-1062.el7.x86_64
0x1e2
9
SyS_madvise
vmlinux 0xffffffffb13e976d
Build id: 3.10.0-1062.el7.x86_64
0x49d
10
? system_call_after_swapgs
vmlinux 0xffffffffb198be21
Build id: 3.10.0-1062.el7.x86_64
0xae
11
? system_call_after_swapgs
vmlinux 0xffffffffb198be15
Build id: 3.10.0-1062.el7.x86_64
0xa2
12
system_call_fastpath
vmlinux 0xffffffffb198bede
Build id: 3.10.0-1062.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffffb198be21
Build id: 3.10.0-1062.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 21s! [mysqld:27046]
Modules linked in: binfmt_misc ip_set nfnetlink tcp_diag inet_diag rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc ppdev i2c_piix4 sg virtio_balloon joydev parport_pc parport pcspkr ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi virtio_net virtio_scsi bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci serio_raw virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 27046 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014
task: ffff9619dca05230 ti: ffff961b0cae8000 task.ti: ffff961b0cae8000
RIP: 0010:[<ffffffffb13150ca>]  [<ffffffffb13150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff961b0caebc70  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00080000b127d1d0 RCX: ffff961bbfca04a0
RDX: 0000000000000002 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff961b0caebca8 R08: ffff9618f9c88400 R09: ffffffffb157f629
R10: ffff961bbfddf160 R11: ffffe7d99096f200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff961b0caebc20 R15: 000000000000000e
FS:  00007f0ae5c1d700(0000) GS:ffff961bbfdc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f675bc6c000 CR3: 00000004262d4000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb127d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb127d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb13ea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb13ebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb13ebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffffb13eea5a>] zap_page_range+0xea/0x130
[<ffffffffb1422912>] ? kmem_cache_free+0x1e2/0x200
[<ffffffffb13e976d>] SyS_madvise+0x49d/0xac0
[<ffffffffb198be21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffb198be15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffb198bede>] system_call_fastpath+0x25/0x2a
[<ffffffffb198be21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 f4 b1 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #103011
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff89b1760a
Build id: 3.10.0-1127.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff89a7e8f8
Build id: 3.10.0-1127.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff89a7e96b
Build id: 3.10.0-1127.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff89beddb6
Build id: 3.10.0-1127.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff89bef87f
Build id: 3.10.0-1127.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff89bef9a3
Build id: 3.10.0-1127.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff89bf2e7d
Build id: 3.10.0-1127.el7.x86_64
0xfd
8
SyS_madvise
vmlinux 0xffffffff89bed36d
Build id: 3.10.0-1127.el7.x86_64
0x49d
9
? system_call_after_swapgs
vmlinux 0xffffffff8a192e15
Build id: 3.10.0-1127.el7.x86_64
0xa2
10
? system_call_after_swapgs
vmlinux 0xffffffff8a192e09
Build id: 3.10.0-1127.el7.x86_64
0x96
11
system_call_fastpath
vmlinux 0xffffffff8a192ed2
Build id: 3.10.0-1127.el7.x86_64
0x25
12
? system_call_after_swapgs
vmlinux 0xffffffff8a192e15
Build id: 3.10.0-1127.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [JS Helper:18252]
Modules linked in: tcp_lp vfat fat uas usb_storage fuse binfmt_misc xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ipmi_ssif iTCO_wdt gpio_ich iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev lpc_ich hpilo hpwdt i7core_edac
ipmi_si sg ipmi_devintf ipmi_msghandler pcc_cpufreq acpi_power_meter ip_tables xfs libcrc32c radeon sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel hpsa serio_raw bnx2 drm_panel_orientation_quirks scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 18252 Comm: JS Helper Kdump: loaded Tainted: G          I  L ------------   3.10.0-1127.el7.x86_64 #1
Hardware name: HP ProLiant DL360 G7, BIOS P68 09/30/2010
task: ffff9dd746be1070 ti: ffff9dd73a2f8000 task.ti: ffff9dd73a2f8000
RIP: 0010:[<ffffffff89b1760a>]  [<ffffffff89b1760a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9dd73a2fbc70  EFLAGS: 00000202
RAX: 0000000000000007 RBX: ffffffff89a7e730 RCX: ffff9dda1f6e05d8
RDX: 0000000000000007 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff9dd73a2fbca8 R08: ffff9dd91d990c00 R09: ffffffff89d85099
R10: ffff9dda1f69f160 R11: ffffda2b86852800 R12: 0000000000000286
R13: 0000000000000018 R14: ffff9dd73a2fbbe8 R15: 0000000000000286
FS:  00007f0dc3d00700(0000) GS:ffff9dda1f680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0dc012a000 CR3: 00000001883cc000 CR4: 00000000000207e0
Call Trace:
[<ffffffff89a7e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff89a7e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff89beddb6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff89bef87f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff89bef9a3>] tlb_finish_mmu+0x23/0x30
[<ffffffff89bf2e7d>] zap_page_range+0xfd/0x150
[<ffffffff89bed36d>] SyS_madvise+0x49d/0xab0
[<ffffffff8a192e15>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8a192e09>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff8a192ed2>] system_call_fastpath+0x25/0x2a
[<ffffffff8a192e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 1e 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 0f 75 8a f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 7a c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #15259
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff983161da
Build id: 3.10.0-1062.4.3.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9827e478
Build id: 3.10.0-1062.4.3.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9827e4eb
Build id: 3.10.0-1062.4.3.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff983eb796
Build id: 3.10.0-1062.4.3.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff983ed25f
Build id: 3.10.0-1062.4.3.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff983ed383
Build id: 3.10.0-1062.4.3.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff983f008a
Build id: 3.10.0-1062.4.3.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff983ead9d
Build id: 3.10.0-1062.4.3.el7.x86_64
0x49d
9
? system_call_after_swapgs
vmlinux 0xffffffff9898ce21
Build id: 3.10.0-1062.4.3.el7.x86_64
0xae
10
? system_call_after_swapgs
vmlinux 0xffffffff9898ce15
Build id: 3.10.0-1062.4.3.el7.x86_64
0xa2
11
system_call_fastpath
vmlinux 0xffffffff9898cede
Build id: 3.10.0-1062.4.3.el7.x86_64
0x25
12
? system_call_after_swapgs
vmlinux 0xffffffff9898ce21
Build id: 3.10.0-1062.4.3.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [mysqld:2301]
Modules linked in: nfsd nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc sg ppdev virtio_balloon i2c_piix4 joydev parport_pc parport pcspkr ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi virtio_net virtio_console virtio_blk virtio_scsi bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci virtio_ring floppy virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 9 PID: 2301 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-1062.4.3.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014
task: ffff9e02bb42c1c0 ti: ffff9e020bc28000 task.ti: ffff9e020bc28000
RIP: 0010:[<ffffffff983161da>]  [<ffffffff983161da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9e020bc2bc70  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 000800009827e2b0 RCX: ffff9e0335320540
RDX: 0000000000000004 RSI: 000000000000000c RDI: 0000000000000000
RBP: ffff9e020bc2bca8 R08: ffff9e033308ec00 R09: ffffffff98580ed9
R10: ffff9e033545f160 R11: ffffee09c2d14200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9e020bc2bc20 R15: 0000000000000001
FS:  00007fb940081700(0000) GS:ffff9e0335440000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb9400782a0 CR3: 00000000b2e4e000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9827e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9827e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff983eb796>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff983ed25f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff983ed383>] tlb_finish_mmu+0x23/0x30
[<ffffffff983f008a>] zap_page_range+0xea/0x130
[<ffffffff983ead9d>] SyS_madvise+0x49d/0xac0
[<ffffffff9898ce21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff9898ce15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffff9898cede>] system_call_fastpath+0x25/0x2a
[<ffffffff9898ce21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 0e 57 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 d7 f4 98 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 dd 56 c4 00 eb b7 0f b6 4d cc 4c 89