Complete report #1410419
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffff9affdc7c
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x3cc
|
|
2
|
change_protection
|
vmlinux |
0xffffffff9affe190
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x3a0
|
|
3
|
|
vmlinux |
0xffffffff9aee46dc
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x14c
|
|
4
|
change_prot_numa
|
vmlinux |
0xffffffff9b01aa7b
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x1b
|
|
5
|
task_numa_work
|
vmlinux |
0xffffffff9aee0032
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x212
|
|
6
|
task_work_run
|
vmlinux |
0xffffffff9aec28db
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xbb
|
|
7
|
do_notify_resume
|
vmlinux |
0xffffffff9ae2cc65
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xa5
|
|
8
|
retint_signal
|
vmlinux |
0xffffffff9b58c57c
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x48
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 44s! [node:20883]
Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) nfsv3 nfs fscache nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev vmw_balloon gf128mul glue_helper ablk_helper joydev cryptd pcspkr
sg vmw_vmci parport_pc i2c_piix4 parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm ata_piix libahci nfit libata libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 20883 Comm: node Kdump: loaded Tainted: G ------------ T 3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff988d41390000 ti: ffff988d4e3e4000 task.ti: ffff988d4e3e4000
RIP: 0010:[<ffffffff9affdc7c>] [<ffffffff9affdc7c>] change_pte_range+0x3cc/0x540
RSP: 0000:ffff988d4e3e7d50 EFLAGS: 00010282
RAX: fffff51f13239180 RBX: ffff988c01c1acc0 RCX: 00000000034c8e46
RDX: 80003fcb371b9966 RSI: 00000f96721ab000 RDI: 80003fcb371b9966
RBP: ffff988d4e3e7db0 R08: 8000000000000966 R09: 00003ffffffff000
R10: ffff98acd4ba0798 R11: 00000f9672180000 R12: ffff988d41390698
R13: 00000001713c3180 R14: ffff988c01c00000 R15: ffffffff9ae2b621
FS: 00007fb999f01700(0000) GS:ffff988c01c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc1497893c0 CR3: 000000208cbe8000 CR4: 00000000001607f0
Call Trace:
[<ffffffff9affe190>] change_protection+0x3a0/0x680
[<ffffffff9aee46dc>] ? update_curr+0x14c/0x1e0
[<ffffffff9b01aa7b>] change_prot_numa+0x1b/0x40
[<ffffffff9aee0032>] task_numa_work+0x212/0x370
[<ffffffff9aec28db>] task_work_run+0xbb/0xe0
[<ffffffff9ae2cc65>] do_notify_resume+0xa5/0xc0
[<ffffffff9b58c57c>] retint_signal+0x48/0x8c
Code: e2 e8 d9 02 e7 ff 66 90 e9 fb fe ff ff 66 90 48 8b 7d b8 4c 89 c2 48 89 de 4c 89 45 a8 e8 7d 2a ff ff 48 85 c0 0f 84 e2 fe ff ff <48> 8b 40 08 83 e0 03 48 83 f8 03 0f 84 d1 fe ff ff 4c 8b 45 a8
Complete report #1255313
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffff9820a2bc
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0x3dc
|
|
2
|
change_protection
|
vmlinux |
0xffffffff9820a7d0
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0x3a0
|
|
3
|
|
vmlinux |
0xffffffff98202243
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0xaf3
|
|
4
|
change_prot_numa
|
vmlinux |
0xffffffff982278bb
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0x1b
|
|
5
|
task_numa_work
|
vmlinux |
0xffffffff980e6096
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0x216
|
|
6
|
task_work_run
|
vmlinux |
0xffffffff980c7f6b
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0xbb
|
|
7
|
do_notify_resume
|
vmlinux |
0xffffffff9802dcfd
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0xad
|
|
8
|
retint_signal
|
vmlinux |
0xffffffff987bb5bc
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0x48
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [InactivityMonit:18253]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter nfsv3 nfs_acl nfs lockd grace fscache vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod ahci crc_t10dif crct10dif_generic drm ata_piix libahci
nfit libata libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse
CPU: 2 PID: 18253 Comm: InactivityMonit Kdump: loaded Tainted: G L ------------ 3.10.0-1160.108.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8e3f2b222100 ti: ffff8e488b7c8000 task.ti: ffff8e488b7c8000
RIP: 0010:[<ffffffff9820a2bc>] [<ffffffff9820a2bc>] change_pte_range+0x3dc/0x550
RSP: 0018:ffff8e488b7cbd50 EFLAGS: 00010282
RAX: ffffe2554f589380 RBX: ffffe255903a4830 RCX: 00000000003d624e
RDX: 80003ffc29db1966 RSI: 00000000d6829000 RDI: 80003ffc29db1966
RBP: ffff8e488b7cbdb0 R08: 8000000000000966 R09: 00003ffffffff000
R10: ffff8e4a31f5bd88 R11: 00000000d6800000 R12: 0000000000017b28
R13: ffff8e3f2b222100 R14: 0000000022a0cc08 R15: 00000000ffffffff
FS: 00007fab0cd71700(0000) GS:ffff8e418da80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1943c552b0 CR3: 000000054beee000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff9820a7d0>] change_protection+0x3a0/0x680
[<ffffffff98202243>] ? handle_mm_fault+0xaf3/0x1190
[<ffffffff982278bb>] change_prot_numa+0x1b/0x40
[<ffffffff980e6096>] task_numa_work+0x216/0x370
[<ffffffff980c7f6b>] task_work_run+0xbb/0xe0
[<ffffffff9802dcfd>] do_notify_resume+0xad/0xd0
[<ffffffff987bb5bc>] retint_signal+0x48/0x8c
Code: e9 f3 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 48 8b 7d b8 4c 89 c2 48 89 de 4c 89 45 a8 e8 7d 26 ff ff 48 85 c0 0f 84 d2 fe ff ff <48> 8b 40 08 83 e0 03 48 83 f8 03 0f 84 c1 fe ff ff 4c 8b 45 a8
Complete report #22569
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
vm_normal_page
|
vmlinux |
0xffffffffb17f062f
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x5f
|
|
2
|
change_pte_range
|
vmlinux |
0xffffffffb17fdb13
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x3c3
|
|
3
|
change_protection
|
vmlinux |
0xffffffffb17fe030
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x3a0
|
|
4
|
|
vmlinux |
0xffffffffb16e0628
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x78
|
|
5
|
change_prot_numa
|
vmlinux |
0xffffffffb181a89b
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x1b
|
|
6
|
task_numa_work
|
vmlinux |
0xffffffffb16e0952
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x212
|
|
7
|
task_work_run
|
vmlinux |
0xffffffffb16c31cb
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xbb
|
|
8
|
do_notify_resume
|
vmlinux |
0xffffffffb162cc65
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xa5
|
|
9
|
int_signal
|
vmlinux |
0xffffffffb1d9322f
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x12
|
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [postgres:1930]
crct10dif_common libata
Modules linked in: binfmt_misc vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev ppdev vmw_balloon sg i2c_piix4 parport_pc parport vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi ata_piix scsi_transport_spi crct10dif_pclmul mptscsih crct10dif_common libata crc32c_intel serio_raw mptbase vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 10 PID: 1930 Comm: postgres Kdump: loaded Tainted: G L ------------ 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9d6332dee2a0 ti: ffff9d60244c8000 task.ti: ffff9d60244c8000
RIP: 0010:[<ffffffffb17f062f>] [<ffffffffb17f062f>] vm_normal_page+0x5f/0xb0
RSP: 0018:ffff9d60244cbd40 EFLAGS: 00010246
RAX: 8000000467f5a805 RBX: ffff9d6332dee938 RCX: 0000000000467f5a
RDX: 8000000467f5a805 RSI: 000000000183a000 RDI: 8000000467f5a805
RBP: ffff9d60244cbd40 R08: 8000000000000805 R09: 00003ffffffff000
R10: ffff9d602443ae58 R11: 000000000183a000 R12: ffff9d633fc80000
R13: ffffffffb162b621 R14: ffff9d60244cbcf8 R15: ffff9d60244cbd08
FS: 00007f0297a8a880(0000) GS:ffff9d633fc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9a808b1000 CR3: 00000003244ee000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb17fdb13>] change_pte_range+0x3c3/0x540
[<ffffffffb17fe030>] change_protection+0x3a0/0x680
[<ffffffffb16e0628>] ? __enqueue_entity+0x78/0x80
[<ffffffffb181a89b>] change_prot_numa+0x1b/0x40
[<ffffffffb16e0952>] task_numa_work+0x212/0x370
[<ffffffffb16c31cb>] task_work_run+0xbb/0xe0
[<ffffffffb162cc65>] do_notify_resume+0xa5/0xc0
[<ffffffffb1d9322f>] int_signal+0x12/0x17
Code: 01 00 00 0f 94 c1 49 f7 d0 0f b6 c9 4d 21 c8 48 f7 d9 48 31 c1 4c 21 c1 49 f7 d0 49 21 d0 48 c1 e9 0c 41 f7 c0 00 02 00 00 75 19 <48> 39 0d b2 07 b7 00 72 34 48 89 c8 48 c1 e0 06 48 03 05 aa 55
Complete report #901943
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffffb91fe39c
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x3cc
|
|
2
|
change_protection
|
vmlinux |
0xffffffffb91fe8b0
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x3a0
|
|
3
|
change_prot_numa
|
vmlinux |
0xffffffffb921b1ab
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x1b
|
|
4
|
task_numa_work
|
vmlinux |
0xffffffffb90dff02
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x212
|
|
5
|
task_work_run
|
vmlinux |
0xffffffffb90c299b
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xbb
|
|
6
|
do_notify_resume
|
vmlinux |
0xffffffffb902cc65
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xa5
|
|
7
|
int_signal
|
vmlinux |
0xffffffffb97962ef
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x12
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kube-proxy:7131]
Modules linked in: xt_statistic xt_nat xt_recent ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink xt_addrtype br_netfilter binfmt_misc netfilter_edr(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod sb_edac coretemp ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel joydev aesni_intel sg parport_pc lrw gf128mul glue_helper vmw_vmci ablk_helper parport cryptd i2c_piix4 pcspkr ip_tables xfs libcrc32c sr_mod sd_mod cdrom
crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect ata_piix sysimgblt fb_sys_fops mptspi ttm scsi_transport_spi crct10dif_pclmul drm crct10dif_common mptscsih libata e1000 serio_raw crc32c_intel mptbase vmxnet3 drm_panel_orientation_quirks floppy fuse
CPU: 2 PID: 7131 Comm: kube-proxy Kdump: loaded Tainted: G OEL ------------ T 3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffffa10df65dc200 ti: ffffa11950bc0000 task.ti: ffffa11950bc0000
RIP: 0010:[<ffffffffb91fe39c>] [<ffffffffb91fe39c>] change_pte_range+0x3cc/0x540
RSP: 0018:ffffa11950bc3d50 EFLAGS: 00010286
RAX: ffffeb0370ac7200 RBX: ffffa10d6fc80000 RCX: 0000000000c2b1c8
RDX: 8000000c2b1c8867 RSI: 000000c00053d000 RDI: 8000000c2b1c8867
RBP: ffffa11950bc3db0 R08: 8000000000000867 R09: 00003ffffffff000
R10: ffffa1152dca01b0 R11: 000000c000400000 R12: ffffa11950bc3d18
R13: ffffa11950bc3d28 R14: ffffa10d69c38180 R15: ffffa10ab634b1e8
FS: 000000c00082a090(0000) GS:ffffa10d6fc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f635a282100 CR3: 0000000be8534000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb91fe8b0>] change_protection+0x3a0/0x680
[<ffffffffb921b1ab>] change_prot_numa+0x1b/0x40
[<ffffffffb90dff02>] task_numa_work+0x212/0x370
[<ffffffffb90c299b>] task_work_run+0xbb/0xe0
[<ffffffffb902cc65>] do_notify_resume+0xa5/0xc0
[<ffffffffb97962ef>] int_signal+0x12/0x17
Code: e2 e8 b9 fb e6 ff 66 90 e9 fb fe ff ff 66 90 48 8b 7d b8 4c 89 c2 48 89 de 4c 89 45 a8 e8 7d 2a ff ff 48 85 c0 0f 84 e2 fe ff ff <48> 8b 40 08 83 e0 03 48 83 f8 03 0f 84 d1 fe ff ff 4c 8b 45 a8
Complete report #878032
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffffa83fe37c
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x3cc
|
|
2
|
change_protection
|
vmlinux |
0xffffffffa83fe890
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x3a0
|
|
3
|
change_prot_numa
|
vmlinux |
0xffffffffa841b1ab
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x1b
|
|
4
|
task_numa_work
|
vmlinux |
0xffffffffa82dff52
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x212
|
|
5
|
task_work_run
|
vmlinux |
0xffffffffa82c2acb
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xbb
|
|
6
|
do_notify_resume
|
vmlinux |
0xffffffffa822cc65
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xa5
|
|
7
|
retint_signal
|
vmlinux |
0xffffffffa899057c
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x48
|
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [redisson-netty-:11244]
Modules linked in: ipt_rpfilter xt_set iptable_raw ip_set_hash_ip nf_tables ip_set_hash_net vxlan ip6_udp_tunnel udp_tunnel rbd libceph rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_multiport veth ip6t_MASQUERADE nf_nat_masquerade_ipv6 xt_nat xt_statistic xt_mark xt_comment ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip_set ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack_netlink xt_addrtype br_netfilter falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_16004(E) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter
ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod vmwgfx crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix nfit drm libata crct10dif_pclmul libnvdimm crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 12 PID: 11244 Comm: redisson-netty- Kdump: loaded Tainted: P E ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8f01f8dfc200 ti: ffff8f021a390000 task.ti: ffff8f021a390000
RIP: 0010:[<ffffffffa83fe37c>] [<ffffffffa83fe37c>] change_pte_range+0x3cc/0x540
RSP: 0018:ffff8f021a393d50 EFLAGS: 00010282
RAX: ffffe7e89d04da80 RBX: ffff8f021fd1acc0 RCX: 000000000074136a
RDX: 80003ff8bec95966 RSI: 00000000a01ce000 RDI: 80003ff8bec95966
RBP: ffff8f021a393db0 R08: 8000000000000966 R09: 00003ffffffff000
R10: ffff8f052b60fcb0 R11: 00000000a0000000 R12: ffff8f0a1bde3840
R13: ffff8f021fd1acc0 R14: ffff8f015280ea40 R15: 0000000000000000
FS: 00007f1189f65b38(0000) GS:ffff8f021fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00002fe68 CR3: 00000005039ae000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa83fe890>] change_protection+0x3a0/0x680
[<ffffffffa841b1ab>] change_prot_numa+0x1b/0x40
[<ffffffffa82dff52>] task_numa_work+0x212/0x370
[<ffffffffa82c2acb>] task_work_run+0xbb/0xe0
[<ffffffffa822cc65>] do_notify_resume+0xa5/0xc0
[<ffffffffa899057c>] retint_signal+0x48/0x8c
Code: e2 e8 d9 fb e6 ff 66 90 e9 fb fe ff ff 66 90 48 8b 7d b8 4c 89 c2 48 89 de 4c 89 45 a8 e8 cd 2a ff ff 48 85 c0 0f 84 e2 fe ff ff <48> 8b 40 08 83 e0 03 48 83 f8 03 0f 84 d1 fe ff ff 4c 8b 45 a8
Complete report #851071
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffff957fe39c
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x3cc
|
|
2
|
change_protection
|
vmlinux |
0xffffffff957fe8b0
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x3a0
|
|
3
|
|
vmlinux |
0xffffffff957f665b
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xadb
|
|
4
|
change_prot_numa
|
vmlinux |
0xffffffff9581b1ab
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x1b
|
|
5
|
task_numa_work
|
vmlinux |
0xffffffff956dff02
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x212
|
|
6
|
task_work_run
|
vmlinux |
0xffffffff956c299b
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xbb
|
|
7
|
do_notify_resume
|
vmlinux |
0xffffffff9562cc65
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xa5
|
|
8
|
retint_signal
|
vmlinux |
0xffffffff95d9057c
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x48
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 249s! [DefaultQuartzSc:14668]
Modules linked in: tcp_diag udp_diag inet_diag xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper vmw_balloon cryptd sg pcspkr joydev vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod drm_kms_helper crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci nfit ata_piix libahci drm
libnvdimm crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 14668 Comm: DefaultQuartzSc Tainted: G W L ------------ T 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
task: ffff8aea9d871080 ti: ffff8acc10778000 task.ti: ffff8acc10778000
RIP: 0010:[<ffffffff957fe39c>] [<ffffffff957fe39c>] change_pte_range+0x3cc/0x540
RSP: 0000:ffff8acc1077bd50 EFLAGS: 00010286
RAX: ffffebbb7b7014c0 RBX: ffff8acc04a6ebd0 RCX: 0000000001edc053
RDX: 80003fe123fac966 RSI: 00007fe7981f6000 RDI: 80003fe123fac966
RBP: ffff8acc1077bdb0 R08: 8000000000000966 R09: 00003ffffffff000
R10: ffff8ae87b482000 R11: 00007fe798000000 R12: ffffffff95659577
R13: ffff8acc1077bdb0 R14: ffffffff95986c09 R15: ffff8acc1077bcb8
FS: 00007fe68f8f8700(0000) GS:ffff8adaffa80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe68f8f1370 CR3: 0000001dab6d2000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff957fe8b0>] change_protection+0x3a0/0x680
[<ffffffff957f665b>] ? handle_mm_fault+0xadb/0xfb0
[<ffffffff9581b1ab>] change_prot_numa+0x1b/0x40
[<ffffffff956dff02>] task_numa_work+0x212/0x370
[<ffffffff956c299b>] task_work_run+0xbb/0xe0
[<ffffffff9562cc65>] do_notify_resume+0xa5/0xc0
[<ffffffff95d9057c>] retint_signal+0x48/0x8c
Code: e2 e8 b9 fb e6 ff 66 90 e9 fb fe ff ff 66 90 48 8b 7d b8 4c 89 c2 48 89 de 4c 89 45 a8 e8 7d 2a ff ff 48 85 c0 0f 84 e2 fe ff ff <48> 8b 40 08 83 e0 03 48 83 f8 03 0f 84 d1 fe ff ff 4c 8b 45 a8
Complete report #688433
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffff8fffdbb0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x3d0
|
|
2
|
change_protection
|
vmlinux |
0xffffffff8fffe0c0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x3a0
|
|
3
|
|
vmlinux |
0xffffffff8fedfca8
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x78
|
|
4
|
change_prot_numa
|
vmlinux |
0xffffffff9001a9ab
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x1b
|
|
5
|
task_numa_work
|
vmlinux |
0xffffffff8fedffd2
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x212
|
|
6
|
task_work_run
|
vmlinux |
0xffffffff8fec29ab
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xbb
|
|
7
|
do_notify_resume
|
vmlinux |
0xffffffff8fe2cc65
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xa5
|
|
8
|
retint_signal
|
vmlinux |
0xffffffff9058a57c
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x48
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [virtuoso:2144]
Modules linked in: tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nfsv3 nfs_acl nfs lockd grace fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev joydev vmw_balloon pcspkr sg vmw_vmci parport_pc parport binfmt_misc ip_tables xfs libcrc32c
sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea sysfillrect sysimgblt crc_t10dif fb_sys_fops crct10dif_generic ttm nfit drm libnvdimm ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 2144 Comm: virtuoso Kdump: loaded Tainted: G W ------------ 3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
task: ffff9fff29fa1080 ti: ffff9ffde8bfc000 task.ti: ffff9ffde8bfc000
RIP: 0010:[<ffffffff8fffdbb0>] [<ffffffff8fffdbb0>] change_pte_range+0x3d0/0x540
RSP: 0000:ffff9ffde8bffd50 EFLAGS: 00000286
RAX: ffff9fff4c794051 RBX: 0000000000000001 RCX: 00000000016bc7c8
RDX: 80003fe943837966 RSI: 000000003fdd1000 RDI: 80003fe943837966
RBP: ffff9ffde8bffdb0 R08: 8000000000000966 R09: 00003ffffffff000
R10: ffff9fff36201878 R11: 000000003fc00000 R12: ffff9fff36201878
R13: 000000003fc00000 R14: ffffa0014da00000 R15: ffff9ffde8bffd40
FS: 00007faa9782ed40(0000) GS:ffffa0014da00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fefd11aca88 CR3: 00000008f4838000 CR4: 00000000001607f0
Call Trace:
[<ffffffff8fffe0c0>] change_protection+0x3a0/0x680
[<ffffffff8fedfca8>] ? __enqueue_entity+0x78/0x80
[<ffffffff9001a9ab>] change_prot_numa+0x1b/0x40
[<ffffffff8fedffd2>] task_numa_work+0x212/0x370
[<ffffffff8fec29ab>] task_work_run+0xbb/0xe0
[<ffffffff8fe2cc65>] do_notify_resume+0xa5/0xc0
[<ffffffff9058a57c>] retint_signal+0x48/0x8c
Code: e7 ff 66 90 e9 fb fe ff ff 66 90 48 8b 7d b8 4c 89 c2 48 89 de 4c 89 45 a8 e8 dd 2a ff ff 48 85 c0 0f 84 e2 fe ff ff 48 8b 40 08 <83> e0 03 48 83 f8 03 0f 84 d1 fe ff ff 4c 8b 45 a8 41 81 e0 01
Complete report #655408
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_lock
|
vmlinux |
0xffffffffa458ecec
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0xc
|
|
2
|
|
vmlinux |
0xffffffffa3ffe12b
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0x15b
|
|
3
|
change_protection
|
vmlinux |
0xffffffffa3ffe8b0
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0x3a0
|
|
4
|
|
vmlinux |
0xffffffffa3e7dfe9
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0x109
|
|
5
|
change_prot_numa
|
vmlinux |
0xffffffffa401b1ab
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0x1b
|
|
6
|
task_numa_work
|
vmlinux |
0xffffffffa3ee0032
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0x212
|
|
7
|
task_work_run
|
vmlinux |
0xffffffffa3ec2acb
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0xbb
|
|
8
|
do_notify_resume
|
vmlinux |
0xffffffffa3e2cc65
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0xa5
|
|
9
|
retint_signal
|
vmlinux |
0xffffffffa459057c
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0x48
|
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 21s! [master:25635]
Modules linked in: keyspan ezusb isofs loop binfmt_misc nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache tcp_lp nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg vmw_vmci
i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm ata_piix libahci nfit libnvdimm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 12 PID: 25635 Comm: master Kdump: loaded Tainted: G L ------------ 3.10.0-1160.66.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
task: ffff8fe242e6c200 ti: ffff8fde8dcf0000 task.ti: ffff8fde8dcf0000
RIP: 0010:[<ffffffffa458ecec>] [<ffffffffa458ecec>] _raw_spin_lock+0xc/0x30
RSP: 0018:ffff8fde8dcf3d48 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8fde8dcf3d18 RCX: 0000000ad1968000
RDX: 0000000000000001 RSI: 00003ffffffff000 RDI: fffff23deb465a30
RBP: ffff8fde8dcf3db0 R08: ffff8fd8000005e0 R09: 0000000000000000
R10: 000000000000000a R11: 00007f6da5cbc000 R12: ffff8fe242e6c268
R13: ffff8fe41a73b1e8 R14: ffff8fe43fd1acc0 R15: ffffffffa3ee281c
FS: 00007f6dab3ab880(0000) GS:ffff8fe43fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055a2aec24db0 CR3: 0000000af2886000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffa3ffe12b>] ? change_pte_range+0x15b/0x540
[<ffffffffa3ffe8b0>] change_protection+0x3a0/0x680
[<ffffffffa3e7dfe9>] ? switch_mm_irqs_off+0x109/0x290
[<ffffffffa401b1ab>] change_prot_numa+0x1b/0x40
[<ffffffffa3ee0032>] task_numa_work+0x212/0x370
[<ffffffffa3ec2acb>] task_work_run+0xbb/0xe0
[<ffffffffa3e2cc65>] do_notify_resume+0xa5/0xc0
[<ffffffffa459057c>] retint_signal+0x48/0x8c
Code: 5d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 58 20 ff ff 5d
Complete report #647211
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffff831fb94c
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x3cc
|
|
2
|
change_protection
|
vmlinux |
0xffffffff831fbe56
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x396
|
|
3
|
|
vmlinux |
0xffffffff83780ca0
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x310
|
|
4
|
change_prot_numa
|
vmlinux |
0xffffffff8321864b
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x1b
|
|
5
|
task_numa_work
|
vmlinux |
0xffffffff830e04c2
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x212
|
|
6
|
task_work_run
|
vmlinux |
0xffffffff830c2e5b
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0xbb
|
|
7
|
do_notify_resume
|
vmlinux |
0xffffffff8302cc65
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0xa5
|
|
8
|
int_signal
|
vmlinux |
0xffffffff8378e23b
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x12
|
NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [signal_thread.*:31761]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_14604(E) falcon_lsm_pinned_14203(E) falcon_lsm_pinned_13804(E) falcon_lsm_pinned_13501(E) falcon_lsm_pinned_13109(E) falcon_lsm_pinned_13005(E) falcon_lsm_pinned_12905(E) binfmt_misc cmac arc4 md4 nls_utf8 cifs ccm dns_resolver ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_comment xt_mark iptable_filter iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_netlink nfnetlink iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi overlay(T) vmw_vsock_vmci_transport vsock falcon_lsm_pinned_12803(E) ext4 sb_edac mbcache iosf_mbi jbd2 crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper pcc_cpufreq
cryptd ppdev vmw_balloon pcspkr joydev parport_pc parport vmw_vmci sg i2c_piix4 br_netfilter bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi scsi_transport_spi nfit crct10dif_pclmul libata crct10dif_common crc32c_intel mptscsih libnvdimm serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: falcon_kal]
CPU: 16 PID: 31761 Comm: signal_thread.* Kdump: loaded Tainted: P EL ------------ T 3.10.0-1062.18.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff88f5204f62a0 ti: ffff890383c78000 task.ti: ffff890383c78000
RIP: 0010:[<ffffffff831fb94c>] [<ffffffff831fb94c>] change_pte_range+0x3cc/0x540
RSP: 0018:ffff890383c7bd50 EFLAGS: 00010286
RAX: ffffc4feacaccd00 RBX: ffff89133f21ac80 RCX: 0000000000b2b334
RDX: 80003ff4d4ccb964 RSI: 00007f4128f56000 RDI: 80003ff4d4ccb964
RBP: ffff890383c7bdb0 R08: 8000000000000964 R09: 00003ffffffff000
R10: ffff890894126000 R11: 00007f4128e00000 R12: ffff88f5204f6938
R13: 0000000183c7bcd8 R14: ffff89133f200000 R15: ffffffff8302b621
FS: 00007f41281b6700(0000) GS:ffff89133f200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000006f4ef47 CR3: 00000010661dc000 CR4: 00000000001607e0
Call Trace:
[<ffffffff831fbe56>] change_protection+0x396/0x670
[<ffffffff83780ca0>] ? __schedule+0x310/0x840
[<ffffffff8321864b>] change_prot_numa+0x1b/0x40
[<ffffffff830e04c2>] task_numa_work+0x212/0x370
[<ffffffff830c2e5b>] task_work_run+0xbb/0xe0
[<ffffffff8302cc65>] do_notify_resume+0xa5/0xc0
[<ffffffff8378e23b>] int_signal+0x12/0x17
Code: e2 e8 c9 25 e7 ff 66 90 e9 fb fe ff ff 66 90 48 8b 7d b8 4c 89 c2 48 89 de 4c 89 45 a8 e8 dd 2a ff ff 48 85 c0 0f 84 e2 fe ff ff <48> 8b 40 08 83 e0 03 48 83 f8 03 0f 84 d1 fe ff ff 4c 8b 45 a8
Complete report #529155
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffffa5dfdc7c
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x3cc
|
|
2
|
change_protection
|
vmlinux |
0xffffffffa5dfe190
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x3a0
|
|
3
|
change_prot_numa
|
vmlinux |
0xffffffffa5e1aa7b
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x1b
|
|
4
|
task_numa_work
|
vmlinux |
0xffffffffa5ce0032
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x212
|
|
5
|
task_work_run
|
vmlinux |
0xffffffffa5cc28db
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xbb
|
|
6
|
do_notify_resume
|
vmlinux |
0xffffffffa5c2cc65
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xa5
|
|
7
|
retint_signal
|
vmlinux |
0xffffffffa638c57c
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x48
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [calico-node:6597]
Modules linked in: qax_tq_base(OE) xt_recent ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_statistic ipt_rpfilter xt_multiport xt_set iptable_raw ip_set_hash_ip ip_set_hash_net ip_set vxlan ip6_udp_tunnel udp_tunnel xt_comment xt_mark xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter devlink ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev vmw_balloon glue_helper ablk_helper cryptd joydev pcspkr sg i2c_piix4 vmw_vmci parport_pc
parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci nfit drm libahci ata_piix libnvdimm crct10dif_pclmul libata crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: qax_tq_base]
CPU: 2 PID: 6597 Comm: calico-node Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff92345c13e300 ti: ffff923864f44000 task.ti: ffff923864f44000
RIP: 0010:[<ffffffffa5dfdc7c>] [<ffffffffa5dfdc7c>] change_pte_range+0x3cc/0x540
RSP: 0000:ffff923864f47d50 EFLAGS: 00010282
RAX: ffffe1de55f93d40 RBX: ffff9234afc9acc0 RCX: 000000000057e4f5
RDX: 80003ffa81b0a966 RSI: 00007ffad435b000 RDI: 80003ffa81b0a966
RBP: ffff923864f47db0 R08: 8000000000000966 R09: 00003ffffffff000
R10: ffff9231e9bb11b8 R11: 00007ffad4288000 R12: ffff92345c13e998
R13: 00000001f9b6b180 R14: ffff9234afc80000 R15: ffffffffa5c2b621
FS: 00007ffacaffd700(0000) GS:ffff9234afc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0006f2000 CR3: 00000003e6b24000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa5dfe190>] change_protection+0x3a0/0x680
[<ffffffffa5e1aa7b>] change_prot_numa+0x1b/0x40
[<ffffffffa5ce0032>] task_numa_work+0x212/0x370
[<ffffffffa5cc28db>] task_work_run+0xbb/0xe0
[<ffffffffa5c2cc65>] do_notify_resume+0xa5/0xc0
[<ffffffffa638c57c>] retint_signal+0x48/0x8c
Code: e2 e8 d9 02 e7 ff 66 90 e9 fb fe ff ff 66 90 48 8b 7d b8 4c 89 c2 48 89 de 4c 89 45 a8 e8 7d 2a ff ff 48 85 c0 0f 84 e2 fe ff ff <48> 8b 40 08 83 e0 03 48 83 f8 03 0f 84 d1 fe ff ff 4c 8b 45 a8
Complete report #510774
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffff999fe39c
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x3cc
|
|
2
|
change_protection
|
vmlinux |
0xffffffff999fe8b0
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x3a0
|
|
3
|
|
vmlinux |
0xffffffff9987dfe9
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x109
|
|
4
|
change_prot_numa
|
vmlinux |
0xffffffff99a1b1ab
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x1b
|
|
5
|
task_numa_work
|
vmlinux |
0xffffffff998dff02
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x212
|
|
6
|
task_work_run
|
vmlinux |
0xffffffff998c299b
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0xbb
|
|
7
|
do_notify_resume
|
vmlinux |
0xffffffff9982cc65
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0xa5
|
|
8
|
retint_signal
|
vmlinux |
0xffffffff99f8c57c
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x48
|
NMI watchdog: BUG: soft lockup - CPU#26 stuck for 23s! [vivado:25718]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg i2c_piix4 vmw_balloon vmw_vmci pcspkr joydev parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix mptspi scsi_transport_spi crct10dif_pclmul drm crct10dif_common libata mptscsih crc32c_intel serio_raw
mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 26 PID: 25718 Comm: vivado Kdump: loaded Tainted: G L ------------ 3.10.0-1160.42.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9a72f2673180 ti: ffff9a6ec469c000 task.ti: ffff9a6ec469c000
RIP: 0010:[<ffffffff999fe39c>] [<ffffffff999fe39c>] change_pte_range+0x3cc/0x540
RSP: 0000:ffff9a6ec469fd50 EFLAGS: 00010286
RAX: ffffe6bd9f882500 RBX: ffff9a738f080000 RCX: 00000000017e2094
RDX: 80003fe81df6b966 RSI: 0000000022532000 RDI: 80003fe81df6b966
RBP: ffff9a6ec469fdb0 R08: 8000000000000966 R09: 00003ffffffff000
R10: ffff9a72e8174af8 R11: 0000000022400000 R12: ffff9a6ec469fd18
R13: ffffffff9987dfe9 R14: ffffffff9987dff5 R15: ffffffff9987dfe9
FS: 00007f227e410700(0000) GS:ffff9a738f080000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f94c3ed15d8 CR3: 0000000dfbc2e000 CR4: 00000000001607e0
Call Trace:
[<ffffffff999fe8b0>] change_protection+0x3a0/0x680
[<ffffffff9987dfe9>] ? switch_mm_irqs_off+0x109/0x290
[<ffffffff99a1b1ab>] change_prot_numa+0x1b/0x40
[<ffffffff998dff02>] task_numa_work+0x212/0x370
[<ffffffff998c299b>] task_work_run+0xbb/0xe0
[<ffffffff9982cc65>] do_notify_resume+0xa5/0xc0
[<ffffffff99f8c57c>] retint_signal+0x48/0x8c
Code: e2 e8 b9 fb e6 ff 66 90 e9 fb fe ff ff 66 90 48 8b 7d b8 4c 89 c2 48 89 de 4c 89 45 a8 e8 7d 2a ff ff 48 85 c0 0f 84 e2 fe ff ff <48> 8b 40 08 83 e0 03 48 83 f8 03 0f 84 d1 fe ff ff 4c 8b 45 a8
Complete report #494558
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffff8f9f9e6c
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x3cc
|
|
2
|
change_protection
|
vmlinux |
0xffffffff8f9fa376
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x396
|
|
3
|
change_prot_numa
|
vmlinux |
0xffffffff8fa1665b
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x1b
|
|
4
|
task_numa_work
|
vmlinux |
0xffffffff8f8df1b2
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x212
|
|
5
|
task_work_run
|
vmlinux |
0xffffffff8f8c1c0b
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0xbb
|
|
6
|
do_notify_resume
|
vmlinux |
0xffffffff8f82cc65
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0xa5
|
|
7
|
retint_signal
|
vmlinux |
0xffffffff8ff8357c
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x48
|
NMI watchdog: BUG: soft lockup - CPU#14 stuck for 31s! [dotnet:261092]
Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd parport_pc pcspkr sg joydev parport i2c_piix4 vmw_vmci xfs libcrc32c sr_mod ata_generic cdrom pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea crc_t10dif sysfillrect crct10dif_generic sysimgblt fb_sys_fops ttm ahci drm libahci ata_piix nfit libata crct10dif_pclmul libnvdimm crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy
dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 14 PID: 261092 Comm: dotnet Tainted: G W L ------------ T 3.10.0-1062.1.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff9aab420a3150 ti: ffff9aab75d88000 task.ti: ffff9aab75d88000
RIP: 0010:[<ffffffff8f9f9e6c>] [<ffffffff8f9f9e6c>] change_pte_range+0x3cc/0x540
RSP: 0000:ffff9aab75d8bd50 EFLAGS: 00010282
RAX: fffff75e8b735580 RBX: 0000000000000000 RCX: 00000000002dcd56
RDX: 80003ffd232a9946 RSI: 00007f593c012000 RDI: 80003ffd232a9946
RBP: ffff9aab75d8bdb0 R08: 8000000000000946 R09: 00003ffffffff000
R10: ffff9aae746b5368 R11: 00007f593c000000 R12: fffff75e81ad4bb0
R13: 0010054a00080078 R14: 0000000000000000 R15: 0000000000000001
FS: 00007f5a317fa700(0000) GS:ffff9aae7fd80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5a6cfa2248 CR3: 0000000069538000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8f9fa376>] change_protection+0x396/0x670
[<ffffffff8fa1665b>] change_prot_numa+0x1b/0x40
[<ffffffff8f8df1b2>] task_numa_work+0x212/0x370
[<ffffffff8f8c1c0b>] task_work_run+0xbb/0xe0
[<ffffffff8f82cc65>] do_notify_resume+0xa5/0xc0
[<ffffffff8ff8357c>] retint_signal+0x48/0x8c
Code: e2 e8 c9 2f e7 ff 66 90 e9 fb fe ff ff 66 90 48 8b 7d b8 4c 89 c2 48 89 de 4c 89 45 a8 e8 9d 2a ff ff 48 85 c0 0f 84 e2 fe ff ff <48> 8b 40 08 83 e0 03 48 83 f8 03 0f 84 d1 fe ff ff 4c 8b 45 a8
Complete report #444193
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffffb2bf390c
Build id: 3.10.0-957.el7.x86_64
|
0x3bc
|
|
2
|
change_protection
|
vmlinux |
0xffffffffb2bf3e16
Build id: 3.10.0-957.el7.x86_64
|
0x396
|
|
3
|
|
vmlinux |
0xffffffffb31678fa
Build id: 3.10.0-957.el7.x86_64
|
0x5ea
|
|
4
|
change_prot_numa
|
vmlinux |
0xffffffffb2c0f66b
Build id: 3.10.0-957.el7.x86_64
|
0x1b
|
|
5
|
task_numa_work
|
vmlinux |
0xffffffffb2adba52
Build id: 3.10.0-957.el7.x86_64
|
0x202
|
|
6
|
task_work_run
|
vmlinux |
0xffffffffb2abe79b
Build id: 3.10.0-957.el7.x86_64
|
0xbb
|
|
7
|
do_notify_resume
|
vmlinux |
0xffffffffb2a2bc65
Build id: 3.10.0-957.el7.x86_64
|
0xa5
|
|
8
|
int_signal
|
vmlinux |
0xffffffffb3175124
Build id: 3.10.0-957.el7.x86_64
|
0x12
|
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [packagekitd:15172]
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [packagekitd:15172]
CPU: 7 PID: 15172 Comm: packagekitd Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8f846c006180 ti: ffff8f8438f9c000 task.ti: ffff8f8438f9c000
RIP: 0010:[<ffffffffb2bf390c>] [<ffffffffb2bf390c>] change_pte_range+0x3bc/0x530
RSP: 0018:ffff8f8438f9fd50 EFLAGS: 00010282
RAX: ffffe3a58e585580 RBX: ffff8f846fddab80 RCX: 0000000000396156
RDX: 8000000396156867 RSI: 0000562d4effc000 RDI: 8000000396156867
RBP: ffff8f8438f9fdb0 R08: 8000000000000867 R09: 00003ffffffff000
R10: ffff8f884cb697a0 R11: 0000562d4eed6000 R12: ffff8f846c006818
R13: 00000001b9c80068 R14: ffff8f846fdc0000 R15: ffffffffb2a2a621
FS: 00007f46975658c0(0000) GS:ffff8f846fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffae6ffda08 CR3: 0000000807bca000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffb2bf3e16>] change_protection+0x396/0x680
[<ffffffffb31678fa>] ? __schedule+0x5ea/0x890
[<ffffffffb2c0f66b>] change_prot_numa+0x1b/0x40
[<ffffffffb2adba52>] task_numa_work+0x202/0x360
[<ffffffffb2abe79b>] task_work_run+0xbb/0xe0
[<ffffffffb2a2bc65>] do_notify_resume+0xa5/0xc0
[<ffffffffb3175124>] int_signal+0x12/0x17
Code: e2 e8 79 73 e7 ff 66 90 e9 fb fe ff ff 66 90 48 8b 7d b8 4c 89 c2 48 89 de 4c 89 45 a8 e8 0d 2d ff ff 48 85 c0 0f 84 e2 fe ff ff <48> 8b 40 08 83 e0 03 48 83 f8 03 0f 84 d1 fe ff ff 4c 8b 45 a8
Complete report #292330
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffffa91fd927
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x1a7
|
|
2
|
change_protection
|
vmlinux |
0xffffffffa91fe060
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x3a0
|
|
3
|
change_prot_numa
|
vmlinux |
0xffffffffa921a94b
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x1b
|
|
4
|
task_numa_work
|
vmlinux |
0xffffffffa90dff02
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x212
|
|
5
|
task_work_run
|
vmlinux |
0xffffffffa90c28db
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0xbb
|
|
6
|
do_notify_resume
|
vmlinux |
0xffffffffa902cc65
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0xa5
|
|
7
|
int_signal
|
vmlinux |
0xffffffffa97962ef
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x12
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 47s! [splunkd:18840]
Modules linked in: vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg vmw_vmci parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic drm ata_piix nfit libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 6 PID: 18840 Comm: splunkd Kdump: loaded Tainted: G W L ------------ 3.10.0-1160.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff955931940000 ti: ffff9559343a0000 task.ti: ffff9559343a0000
RIP: 0010:[<ffffffffa91fd927>] [<ffffffffa91fd927>] change_pte_range+0x1a7/0x540
RSP: 0018:ffff9559343a3d50 EFLAGS: 00000287
RAX: 00003ffffffff000 RBX: ffff9559343a3d90 RCX: 00000000001adde4
RDX: 80003ffe5221b946 RSI: 00007efc8d876000 RDI: 80003ffe5221b946
RBP: ffff9559343a3db0 R08: 80003ffe69a42946 R09: 00003ffffffff000
R10: ffff9559344c17a0 R11: 00007efc8d800000 R12: 000000000f1d1000
R13: 0000000000000001 R14: ffff9557e98e6900 R15: ffffffffc03523c4
FS: 00007efcb9bff700(0000) GS:ffff955abfc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007efccd27d000 CR3: 000000026db88000 CR4: 00000000000407e0
Call Trace:
[<ffffffffa91fe060>] change_protection+0x3a0/0x680
[<ffffffffa921a94b>] change_prot_numa+0x1b/0x40
[<ffffffffa90dff02>] task_numa_work+0x212/0x370
[<ffffffffa90c28db>] task_work_run+0xbb/0xe0
[<ffffffffa902cc65>] do_notify_resume+0xa5/0xc0
[<ffffffffa97962ef>] int_signal+0x12/0x17
Code: e8 48 c7 45 c0 00 00 00 00 49 be 00 f0 ff ff ff 3f 00 00 83 e0 01 48 89 45 b0 66 2e 0f 1f 84 00 00 00 00 00 4c 89 f0 4d 8b 04 24 <48> f7 d0 48 0b 05 cf 84 a4 00 4c 21 c0 a9 01 01 00 00 0f 84 49
Complete report #261930
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffff957fe39c
Build id: 3.10.0-1160.41.1.el7.x86_64
|
0x3cc
|
|
2
|
change_protection
|
vmlinux |
0xffffffff957fe8b0
Build id: 3.10.0-1160.41.1.el7.x86_64
|
0x3a0
|
|
3
|
|
vmlinux |
0xffffffff9567dfe9
Build id: 3.10.0-1160.41.1.el7.x86_64
|
0x109
|
|
4
|
change_prot_numa
|
vmlinux |
0xffffffff9581b1ab
Build id: 3.10.0-1160.41.1.el7.x86_64
|
0x1b
|
|
5
|
task_numa_work
|
vmlinux |
0xffffffff956dff02
Build id: 3.10.0-1160.41.1.el7.x86_64
|
0x212
|
|
6
|
task_work_run
|
vmlinux |
0xffffffff956c299b
Build id: 3.10.0-1160.41.1.el7.x86_64
|
0xbb
|
|
7
|
do_notify_resume
|
vmlinux |
0xffffffff9562cc65
Build id: 3.10.0-1160.41.1.el7.x86_64
|
0xa5
|
|
8
|
int_signal
|
vmlinux |
0xffffffff95d962ef
Build id: 3.10.0-1160.41.1.el7.x86_64
|
0x12
|
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [gsd-color:6066]
Modules linked in: tcp_lp cfg80211 rfkill xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul pcspkr ppdev glue_helper ablk_helper cryptd joydev vmw_balloon sg parport_pc parport vmw_vmci
i2c_piix4 sch_fq_codel ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic ahci drm ata_piix libahci nfit libata libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 11 PID: 6066 Comm: gsd-color Kdump: loaded Tainted: G L ------------ T 3.10.0-1160.41.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff946f64fec200 ti: ffff946f5160c000 task.ti: ffff946f5160c000
RIP: 0010:[<ffffffff957fe39c>] [<ffffffff957fe39c>] change_pte_range+0x3cc/0x540
RSP: 0000:ffff946f5160fd50 EFLAGS: 00010286
RAX: ffffed4e53abbbc0 RBX: 0000000000000003 RCX: 00000000004eaeef
RDX: 80000004eaeef867 RSI: 0000000000fa6000 RDI: 80000004eaeef867
RBP: ffff946f5160fdb0 R08: 8000000000000867 R09: 00003ffffffff000
R10: ffff946f5179f0e0 R11: 0000000000e00000 R12: ffff946f5160fd18
R13: ffff946f5160fdb0 R14: ffff9472ae971200 R15: ffff9472ab2b8840
FS: 00007f79e48a0a40(0000) GS:ffff9472bfcc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc46baf49e5 CR3: 00000004d17d6000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff957fe8b0>] change_protection+0x3a0/0x680
[<ffffffff9567dfe9>] ? switch_mm_irqs_off+0x109/0x290
[<ffffffff9581b1ab>] change_prot_numa+0x1b/0x40
[<ffffffff956dff02>] task_numa_work+0x212/0x370
[<ffffffff956c299b>] task_work_run+0xbb/0xe0
[<ffffffff9562cc65>] do_notify_resume+0xa5/0xc0
[<ffffffff95d962ef>] int_signal+0x12/0x17
Code: e2 e8 b9 fb e6 ff 66 90 e9 fb fe ff ff 66 90 48 8b 7d b8 4c 89 c2 48 89 de 4c 89 45 a8 e8 7d 2a ff ff 48 85 c0 0f 84 e2 fe ff ff <48> 8b 40 08 83 e0 03 48 83 f8 03 0f 84 d1 fe ff ff 4c 8b 45 a8
Complete report #16263
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffffa15fb94c
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x3cc
|
|
2
|
change_protection
|
vmlinux |
0xffffffffa15fbe56
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x396
|
|
3
|
|
vmlinux |
0xffffffffa14ca8a9
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x29
|
|
4
|
change_prot_numa
|
vmlinux |
0xffffffffa161864b
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x1b
|
|
5
|
task_numa_work
|
vmlinux |
0xffffffffa14e04c2
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x212
|
|
6
|
task_work_run
|
vmlinux |
0xffffffffa14c2e5b
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0xbb
|
|
7
|
do_notify_resume
|
vmlinux |
0xffffffffa142cc65
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0xa5
|
|
8
|
int_signal
|
vmlinux |
0xffffffffa1b8e23b
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x12
|
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [mysqld:2653]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi ppdev crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul pcspkr glue_helper ablk_helper joydev cryptd vmw_vmci sg parport_pc parport i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix libata drm mptspi crct10dif_pclmul scsi_transport_spi crct10dif_common
crc32c_intel mptscsih mptbase e1000 serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 8 PID: 2653 Comm: mysqld Kdump: loaded Tainted: G L ------------ 3.10.0-1062.18.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
task: ffff935a37898000 ti: ffff935b3b868000 task.ti: ffff935b3b868000
RIP: 0010:[<ffffffffa15fb94c>] [<ffffffffa15fb94c>] change_pte_range+0x3cc/0x540
RSP: 0018:ffff935b3b86bd50 EFLAGS: 00010286
RAX: ffffe91180f4cec0 RBX: ffff935a3b61ac80 RCX: 000000000003d33b
RDX: 80003fffc2cc4966 RSI: 00007f2f30f57000 RDI: 80003fffc2cc4966
RBP: ffff935b3b86bdb0 R08: 8000000000000966 R09: 00003ffffffff000
R10: ffff9359bafa8ca8 R11: 00007f2f30e00000 R12: ffff935a37898698
R13: 000000013b86bcd8 R14: ffff935a3b600000 R15: ffffffffa142b621
FS: 00007f2d9a7de700(0000) GS:ffff935a3b600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe0481d8000 CR3: 00000000ae0d0000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa15fbe56>] change_protection+0x396/0x670
[<ffffffffa14ca8a9>] ? hrtimer_try_to_cancel+0x29/0x120
[<ffffffffa161864b>] change_prot_numa+0x1b/0x40
[<ffffffffa14e04c2>] task_numa_work+0x212/0x370
[<ffffffffa14c2e5b>] task_work_run+0xbb/0xe0
[<ffffffffa142cc65>] do_notify_resume+0xa5/0xc0
[<ffffffffa1b8e23b>] int_signal+0x12/0x17
Code: e2 e8 c9 25 e7 ff 66 90 e9 fb fe ff ff 66 90 48 8b 7d b8 4c 89 c2 48 89 de 4c 89 45 a8 e8 dd 2a ff ff 48 85 c0 0f 84 e2 fe ff ff <48> 8b 40 08 83 e0 03 48 83 f8 03 0f 84 d1 fe ff ff 4c 8b 45 a8
Complete report #163564
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_lock
|
vmlinux |
0xffffffff8b581abc
Build id: 3.10.0-1062.el7.x86_64
|
0xc
|
|
2
|
|
vmlinux |
0xffffffff8aff9bfb
Build id: 3.10.0-1062.el7.x86_64
|
0x15b
|
|
3
|
change_protection
|
vmlinux |
0xffffffff8affa376
Build id: 3.10.0-1062.el7.x86_64
|
0x396
|
|
4
|
change_prot_numa
|
vmlinux |
0xffffffff8b01665b
Build id: 3.10.0-1062.el7.x86_64
|
0x1b
|
|
5
|
task_numa_work
|
vmlinux |
0xffffffff8aedf1b2
Build id: 3.10.0-1062.el7.x86_64
|
0x212
|
|
6
|
task_work_run
|
vmlinux |
0xffffffff8aec1c0b
Build id: 3.10.0-1062.el7.x86_64
|
0xbb
|
|
7
|
do_notify_resume
|
vmlinux |
0xffffffff8ae2cc65
Build id: 3.10.0-1062.el7.x86_64
|
0xa5
|
|
8
|
int_signal
|
vmlinux |
0xffffffff8b58c23b
Build id: 3.10.0-1062.el7.x86_64
|
0x12
|
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 27s! [crond:2028]
Modules linked in: zram ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack libcrc32c ip6table_filter ip6_tables vmw_vsock_vmci_transport vsock iptable_filter ppdev vmw_balloon coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci parport_pc i2c_piix4 parport binfmt_misc sch_fq_codel ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm ata_piix libahci crct10dif_pclmul libata crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 11 PID: 2028 Comm: crond Kdump: loaded Tainted: G L ------------ 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff96496ecf41c0 ti: ffff96496b678000 task.ti: ffff96496b678000
RIP: 0010:[<ffffffff8b581abc>] [<ffffffff8b581abc>] _raw_spin_lock+0xc/0x30
RSP: 0018:ffff96496b67bd48 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff96496b67bd08 RCX: 00000003ae706000
RDX: 0000000000000001 RSI: 00003ffffffff000 RDI: fffffcd98eb9c1b0
RBP: ffff96496b67bdb0 R08: ffff9645c0000008 R09: 0000000000000000
R10: 000000000000bbb5 R11: 00005594ed801000 R12: ffff96496ecf4228
R13: ffff964cffd5acc0 R14: ffffffff8aedf91c R15: ffff96496b67bcc0
FS: 00007f4650346840(0000) GS:ffff964cffd40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1d018c1000 CR3: 00000003a9ffc000 CR4: 00000000000007e0
Call Trace:
[<ffffffff8aff9bfb>] ? change_pte_range+0x15b/0x540
[<ffffffff8affa376>] change_protection+0x396/0x670
[<ffffffff8b01665b>] change_prot_numa+0x1b/0x40
[<ffffffff8aedf1b2>] task_numa_work+0x212/0x370
[<ffffffff8aec1c0b>] task_work_run+0xbb/0xe0
[<ffffffff8ae2cc65>] do_notify_resume+0xa5/0xc0
[<ffffffff8b58c23b>] int_signal+0x12/0x17
Code: 5d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 66 66 66 66 90 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 8a 18 ff ff 5d
Complete report #52824
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffff87bfdb1c
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x3cc
|
|
2
|
change_protection
|
vmlinux |
0xffffffff87bfe030
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x3a0
|
|
3
|
|
vmlinux |
0xffffffff88185850
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x310
|
|
4
|
change_prot_numa
|
vmlinux |
0xffffffff87c1a89b
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x1b
|
|
5
|
task_numa_work
|
vmlinux |
0xffffffff87ae0952
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x212
|
|
6
|
task_work_run
|
vmlinux |
0xffffffff87ac31cb
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xbb
|
|
7
|
do_notify_resume
|
vmlinux |
0xffffffff87a2cc65
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xa5
|
|
8
|
retint_signal
|
vmlinux |
0xffffffff8818957c
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x48
|
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 44s! [nxnode.bin:3733]
Modules linked in: tcp_lp fuse nfsv3 nfs_acl rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock ppdev vmw_balloon sg joydev iosf_mbi crc32_pclmul ghash_clmulni_intel parport_pc aesni_intel parport lrw vmw_vmci i2c_piix4 gf128mul glue_helper ablk_helper cryptd pcspkr auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit ahci ata_piix
libnvdimm libahci drm libata crct10dif_pclmul crct10dif_common crc32c_intel vmxnet3 serio_raw vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 12 PID: 3733 Comm: nxnode.bin Kdump: loaded Tainted: G L ------------ 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018
task: ffff9c5ce10b41c0 ti: ffff9c38f719c000 task.ti: ffff9c38f719c000
RIP: 0010:[<ffffffff87bfdb1c>] [<ffffffff87bfdb1c>] change_pte_range+0x3cc/0x540
RSP: 0000:ffff9c38f719fd50 EFLAGS: 00010286
RAX: ffffe8588b60efc0 RBX: ffff9c66a9b08000 RCX: 00000000042d83bf
RDX: 80000042d83bf865 RSI: 0000000000f68000 RDI: 80000042d83bf865
RBP: ffff9c38f719fdb0 R08: 8000000000000865 R09: 00003ffffffff000
R10: ffff9c5d8821d6c8 R11: 0000000000e00000 R12: 0000000060129fd2
R13: ffff9c66a9b00020 R14: ffff9c39fbe01210 R15: ffff9c3d8bc294d0
FS: 00007f0a1e736900(0000) GS:ffff9c41edd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000002cdbf90 CR3: 0000003fe4982000 CR4: 00000000007607e0
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 44s! [gam_server:30072]
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff87bfe030>] change_protection+0x3a0/0x680
[<ffffffff88185850>] ? __schedule+0x310/0x840
[<ffffffff87c1a89b>] change_prot_numa+0x1b/0x40
[<ffffffff87ae0952>] task_numa_work+0x212/0x370
[<ffffffff87ac31cb>] task_work_run+0xbb/0xe0
[<ffffffff87a2cc65>] do_notify_resume+0xa5/0xc0
[<ffffffff8818957c>] retint_signal+0x48/0x8c
Complete report #6747
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffffa67f504c
Build id: 3.10.0-957.27.2.el7.x86_64
|
0x3bc
|
|
2
|
change_protection
|
vmlinux |
0xffffffffa67f5556
Build id: 3.10.0-957.27.2.el7.x86_64
|
0x396
|
|
3
|
change_prot_numa
|
vmlinux |
0xffffffffa6810e2b
Build id: 3.10.0-957.27.2.el7.x86_64
|
0x1b
|
|
4
|
task_numa_work
|
vmlinux |
0xffffffffa66dcdc2
Build id: 3.10.0-957.27.2.el7.x86_64
|
0x202
|
|
5
|
task_work_run
|
vmlinux |
0xffffffffa66bf9eb
Build id: 3.10.0-957.27.2.el7.x86_64
|
0xbb
|
|
6
|
do_notify_resume
|
vmlinux |
0xffffffffa662bc65
Build id: 3.10.0-957.27.2.el7.x86_64
|
0xa5
|
NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [java:29034]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp ppdev iosf_mbi crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper mptsas syscopyarea sysfillrect sysimgblt fb_sys_fops scsi_transport_sas ttm ata_piix mptscsih crct10dif_pclmul crct10dif_common drm libata crc32c_intel serio_raw mptbase vmxnet3
drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 15 PID: 29034 Comm: java Kdump: loaded Tainted: G L ------------ 3.10.0-957.27.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8b9725f4a080 ti: ffff8b986ab40000 task.ti: ffff8b986ab40000
RIP: 0010:[<ffffffffa67f504c>] [<ffffffffa67f504c>] change_pte_range+0x3bc/0x530
RSP: 0018:ffff8b986ab43d50 EFLAGS: 00010286
RAX: ffffd84f05251c00 RBX: 0000000000000010 RCX: 0000000000149470
RDX: 80003ffeb6b8f966 RSI: 00007fa1a8f0d000 RDI: 80003ffeb6b8f966
RBP: ffff8b986ab43db0 R08: 8000000000000966 R09: 00003ffffffff000
R10: ffff8b97706c6288 R11: 00007fa1a8e00000 R12: ffffffffffffff10
R13: 0000000000000000 R14: 0000000000000010 R15: 0000000000000001
FS: 00007fa1e22e2700(0000) GS:ffff8b997fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa22e2ae3a8 CR3: 00000001e9786000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa67f5556>] change_protection+0x396/0x680
[<ffffffffa6810e2b>] change_prot_numa+0x1b/0x40
[<ffffffffa66dcdc2>] task_numa_work+0x202/0x360
[<ffffffffa66bf9eb>] task_work_run+0xbb/0xe0
[<ffffffffa662bc65>] do_notify_resume+0xa5/0xc0
Complete report #6063
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffff9dff3f1c
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x3bc
|
|
2
|
change_protection
|
vmlinux |
0xffffffff9dff4426
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x396
|
|
3
|
change_prot_numa
|
vmlinux |
0xffffffff9e00fcfb
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x1b
|
|
4
|
task_numa_work
|
vmlinux |
0xffffffff9dedbce2
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x202
|
|
5
|
task_work_run
|
vmlinux |
0xffffffff9debe90b
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xbb
|
|
6
|
do_notify_resume
|
vmlinux |
0xffffffff9de2bc65
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xa5
|
|
7
|
retint_signal
|
vmlinux |
0xffffffff9e56c56c
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x48
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [python:16463]
Modules linked in: xt_owner tcp_diag inet_diag ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 ip6table_filter nf_log_ipv4 nf_log_common nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter vfat fat isofs ext4 mbcache jbd2 loop vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw vmw_balloon gf128mul glue_helper ablk_helper cryptd vmw_vmci pcspkr joydev sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix mptspi drm scsi_transport_spi crct10dif_pclmul libata mptscsih crct10dif_common crc32c_intel serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 16463 Comm: python Kdump: loaded Tainted: G L ------------ 3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff982db5271040 ti: ffff982d8427c000 task.ti: ffff982d8427c000
RIP: 0010:[<ffffffff9dff3f1c>] [<ffffffff9dff3f1c>] change_pte_range+0x3bc/0x530
RSP: 0018:ffff982d8427fd50 EFLAGS: 00010282
RAX: ffffe7d950470d40 RBX: ffffffff9deda115 RCX: 0000000000411c35
RDX: 8000000411c35867 RSI: 00002ae1c7c6c000 RDI: 8000000411c35867
RBP: ffff982d8427fdb0 R08: 8000000000000867 R09: 00003ffffffff000
R10: ffff9829b5a7ba28 R11: 00002ae1c7c6b000 R12: ffff982d8427fd08
R13: ffff982db52710a8 R14: ffff982db52710a8 R15: ffff9829b7c1ac00
FS: 00002ae1c69c0880(0000) GS:ffff9829b7c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb42f936304 CR3: 0000000410012000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9dff4426>] change_protection+0x396/0x680
[<ffffffff9e00fcfb>] change_prot_numa+0x1b/0x40
[<ffffffff9dedbce2>] task_numa_work+0x202/0x360
[<ffffffff9debe90b>] task_work_run+0xbb/0xe0
[<ffffffff9de2bc65>] do_notify_resume+0xa5/0xc0
[<ffffffff9e56c56c>] retint_signal+0x48/0x8c
Code: e2 e8 a9 6d e7 ff 66 90 e9 fb fe ff ff 66 90 48 8b 7d b8 4c 89 c2 48 89 de 4c 89 45 a8 e8 0d 2d ff ff 48 85 c0 0f 84 e2 fe ff ff <48> 8b 40 08 83 e0 03 48 83 f8 03 0f 84 d1 fe ff ff 4c 8b 45 a8
Complete report #2333
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
change_pte_range
|
vmlinux |
0xffffffffb0dfb94c
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x3cc
|
|
2
|
change_protection
|
vmlinux |
0xffffffffb0dfbe56
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x396
|
|
3
|
change_prot_numa
|
vmlinux |
0xffffffffb0e1864b
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x1b
|
|
4
|
task_numa_work
|
vmlinux |
0xffffffffb0ce04c2
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x212
|
|
5
|
task_work_run
|
vmlinux |
0xffffffffb0cc2e5b
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0xbb
|
|
6
|
do_notify_resume
|
vmlinux |
0xffffffffb0c2cc65
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0xa5
|
|
7
|
int_signal
|
vmlinux |
0xffffffffb138e23b
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x12
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [systemd:1]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac coretemp ppdev iosf_mbi crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc parport vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi crct10dif_pclmul crct10dif_common scsi_transport_spi
crc32c_intel mptscsih e1000 mptbase serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1 Comm: systemd Kdump: loaded Tainted: G L ------------ 3.10.0-1062.18.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
task: ffff95517e628000 ti: ffff954ef93d0000 task.ti: ffff954ef93d0000
RIP: 0010:[<ffffffffb0dfb94c>] [<ffffffffb0dfb94c>] change_pte_range+0x3cc/0x540
RSP: 0018:ffff954ef93d3d50 EFLAGS: 00010286
RAX: ffffed3bc25a3e00 RBX: 0000000000000003 RCX: 00000000000968f8
RDX: 80000000968f8867 RSI: 00005619fed38000 RDI: 80000000968f8867
RBP: ffff954ef93d3db0 R08: 8000000000000867 R09: 00003ffffffff000
R10: ffff954df570a948 R11: 00005619fec46000 R12: ffff95517e628698
R13: ffff954ef93d3db0 R14: ffff954efb280000 R15: ffffffffb0c2b621
FS: 00007f3c9a798940(0000) GS:ffff954efb280000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000f120e77b000 CR3: 0000000035c9e000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb0dfbe56>] change_protection+0x396/0x670
[<ffffffffb0e1864b>] change_prot_numa+0x1b/0x40
[<ffffffffb0ce04c2>] task_numa_work+0x212/0x370
[<ffffffffb0cc2e5b>] task_work_run+0xbb/0xe0
[<ffffffffb0c2cc65>] do_notify_resume+0xa5/0xc0
[<ffffffffb138e23b>] int_signal+0x12/0x17
Code: e2 e8 c9 25 e7 ff 66 90 e9 fb fe ff ff 66 90 48 8b 7d b8 4c 89 c2 48 89 de 4c 89 45 a8 e8 dd 2a ff ff 48 85 c0 0f 84 e2 fe ff ff <48> 8b 40 08 83 e0 03 48 83 f8 03 0f 84 d1 fe ff ff 4c 8b 45 a8