Complete report #1054403
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
fsnotify
|
vmlinux |
0xffffffff8286182a
Build id: 3.10.0-862.el7.x86_64
|
0x20a
|
|
2
|
|
vmlinux |
0xffffffff8281a1f3
Build id: 3.10.0-862.el7.x86_64
|
0x93
|
|
3
|
vfs_write
|
vmlinux |
0xffffffff8281ad78
Build id: 3.10.0-862.el7.x86_64
|
0x168
|
|
4
|
SyS_write
|
vmlinux |
0xffffffff8281baff
Build id: 3.10.0-862.el7.x86_64
|
0x7f
|
|
5
|
system_call_fastpath
|
vmlinux |
0xffffffff82d1f7d5
Build id: 3.10.0-862.el7.x86_64
|
0x1c
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [ds_am:2585]
Modules linked in: veth redirfs(OE) nf_conntrack_netlink nfnetlink br_netfilter tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_CHECKSUM iptable_mangle overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci shpchp i2c_piix4 nfit libnvdimm parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm
ahci sd_mod ata_piix libahci crc_t10dif crct10dif_generic libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 i2c_core vmw_pvscsi
CPU: 6 PID: 2585 Comm: ds_am Kdump: loaded Tainted: P OE ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff963bc43a0000 ti: ffff963b9a278000 task.ti: ffff963b9a278000
RIP: 0010:[<ffffffff8286182a>] [<ffffffff8286182a>] fsnotify+0x20a/0x510
RSP: 0018:ffff963b9a27bdf0 EFLAGS: 00000202
RAX: ffff963bdef95c00 RBX: ffff963b9a27be08 RCX: 0000000000000002
RDX: ffff963bdef95c00 RSI: 0000000000000000 RDI: ffffffff837e7a40
RBP: ffff963b9a27bec0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 00000000648e91d8
R13: ffffffff82d12062 R14: 0000427bce871773 R15: 0000000000000000
FS: 00007f2200021700(0000) GS:ffff963bffd80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f135b7cf000 CR3: 00000005c92d0000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8281a1f3>] ? do_sync_write+0x93/0xe0
[<ffffffff8281ad78>] vfs_write+0x168/0x1f0
[<ffffffff8281baff>] SyS_write+0x7f/0xf0
[<ffffffff82d1f7d5>] system_call_fastpath+0x1c/0x21
Code: d2 0f 95 c0 20 c2 41 89 d4 0f 85 82 01 00 00 48 85 db 41 0f 94 c4 4d 85 ed 0f 94 c1 84 c9 0f 85 f8 02 00 00 8b 4d a0 85 c9 74 26 <48> 85 db 74 0d f6 43 44 01 75 07 c7 43 40 00 00 00 00 4d 85 ed
Complete report #518541
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
fsnotify
|
vmlinux |
0xffffffff89c6186b
Build id: 3.10.0-862.el7.x86_64
|
0x24b
|
|
2
|
|
vmlinux |
0xffffffff89c1a1f3
Build id: 3.10.0-862.el7.x86_64
|
0x93
|
|
3
|
vfs_write
|
vmlinux |
0xffffffff89c1ad78
Build id: 3.10.0-862.el7.x86_64
|
0x168
|
|
4
|
SyS_write
|
vmlinux |
0xffffffff89c1baff
Build id: 3.10.0-862.el7.x86_64
|
0x7f
|
|
5
|
symev_write
|
symev_custom_3_10_0_862_el7_x86_64 |
0xffffffffc05814e6
Build id: 3.10.0-862.el7.x86_64
|
0x46
|
|
6
|
system_call_fastpath
|
vmlinux |
0xffffffff8a11f7d5
Build id: 3.10.0-862.el7.x86_64
|
0x1c
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [java:13950]
Modules linked in: veth vxlan ip6_udp_tunnel udp_tunnel xt_set ip_set_hash_ipportnet ip_set_hash_ipport ip_set_bitmap_port ip_set_hash_ipportip ip_set dummy ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter symap_custom_3_10_0_862_el7_x86_64(POE) devlink symev_custom_3_10_0_862_el7_x86_64(OE) nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack overlay(T) vmw_vsock_vmci_transport vsock sunrpc xfs libcrc32c ppdev iosf_mbi crc32_pclmul vmw_balloon ghash_clmulni_intel
aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 nfit libnvdimm parport_pc shpchp parport ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 i2c_core vmw_pvscsi floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 13950 Comm: java Kdump: loaded Tainted: P OE ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff96a8cde75ee0 ti: ffff96a7d8c90000 task.ti: ffff96a7d8c90000
RIP: 0010:[<ffffffff89c6186b>] [<ffffffff89c6186b>] fsnotify+0x24b/0x510
RSP: 0018:ffff96a7d8c93dc0 EFLAGS: 00000246
RAX: ffff96b6296706f0 RBX: 0000000000000001 RCX: 0000000000000002
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff8abe7a40
RBP: ffff96a7d8c93e90 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff96a7d8c93e10
R13: ffff96a7d8c93dc8 R14: ffff96a7d8c93dd8 R15: 0000000000000001
FS: 00007f58a4365700(0000) GS:ffff96b7bf2c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000412512 CR3: 0000000cd4abe000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff89c1a1f3>] ? do_sync_write+0x93/0xe0
[<ffffffff89c1ad78>] vfs_write+0x168/0x1f0
[<ffffffff89c1baff>] SyS_write+0x7f/0xf0
[<ffffffffc05814e6>] symev_write+0x46/0x90 [symev_custom_3_10_0_862_el7_x86_64]
[<ffffffff8a11f7d5>] system_call_fastpath+0x1c/0x21
Code: 0f 41 f6 45 44 01 75 08 41 c7 45 40 00 00 00 00 48 85 db 0f 84 97 01 00 00 44 8b 73 40 4c 8b 5b 08 41 f7 d6 44 23 33 44 23 75 a4 <4d> 85 ed 0f 84 8c 01 00 00 45 8b 65 40 4d 8b 5d 08 41 f7 d4 45
Complete report #242262
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
fsnotify
|
vmlinux |
0xffffffffae0617d7
Build id: 3.10.0-862.el7.x86_64
|
0x1b7
|
|
2
|
|
vmlinux |
0xffffffffae01a1f3
Build id: 3.10.0-862.el7.x86_64
|
0x93
|
|
3
|
vfs_write
|
vmlinux |
0xffffffffae01ad78
Build id: 3.10.0-862.el7.x86_64
|
0x168
|
|
4
|
SyS_write
|
vmlinux |
0xffffffffae01baff
Build id: 3.10.0-862.el7.x86_64
|
0x7f
|
NMI watchdog: BUG: soft lockup - CPU#17 stuck for 23s! [java:20417]
Modules linked in: veth vxlan ip6_udp_tunnel udp_tunnel xt_set ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_bitmap_port ip_set_hash_ipport ip_set dummy ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc symap_custom_3_10_0_862_el7_x86_64(POE) ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink nf_conntrack_ipv4 nf_defrag_ipv4 symev_custom_3_10_0_862_el7_x86_64(OE) ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack overlay(T) vmw_vsock_vmci_transport vsock sunrpc xfs libcrc32c ppdev vmw_balloon joydev pcspkr sg nfit vmw_vmci shpchp
i2c_piix4 libnvdimm parport_pc parport ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm ata_piix libahci libata serio_raw vmxnet3 vmw_pvscsi i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 17 PID: 20417 Comm: java Kdump: loaded Tainted: P OEL ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8e4dee4faf70 ti: ffff8e493cc60000 task.ti: ffff8e493cc60000
RIP: 0010:[<ffffffffae0617d7>] [<ffffffffae0617d7>] fsnotify+0x1b7/0x510
RSP: 0018:ffff8e493cc63dc0 EFLAGS: 00000202
RAX: ffff8e4accbf8150 RBX: 0000000000000001 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffaefe7a40
RBP: ffff8e493cc63e90 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8e493cc63e10
R13: ffff8e493cc63dc8 R14: ffff8e493cc63dd8 R15: 0000000000000001
FS: 00007f6cb64a2700(0000) GS:ffff8e5c7f340000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000c682f050 CR3: 0000001e9a3f8000 CR4: 00000000000007e0
Call Trace:
[<ffffffffae01a1f3>] ? do_sync_write+0x93/0xe0
[<ffffffffae01ad78>] vfs_write+0x168/0x1f0
[<ffffffffae01baff>] SyS_write+0x7f/0xf0
Complete report #233531
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
fsnotify
|
vmlinux |
0xffffffff9d4619d9
Build id: 3.10.0-862.el7.x86_64
|
0x3b9
|
|
2
|
|
vmlinux |
0xffffffff9d41a1f3
Build id: 3.10.0-862.el7.x86_64
|
0x93
|
|
3
|
vfs_write
|
vmlinux |
0xffffffff9d41ad78
Build id: 3.10.0-862.el7.x86_64
|
0x168
|
|
4
|
SyS_write
|
vmlinux |
0xffffffff9d41baff
Build id: 3.10.0-862.el7.x86_64
|
0x7f
|
|
5
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff9d91f721
Build id: 3.10.0-862.el7.x86_64
|
0xae
|
NMI watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [java:32539]
Modules linked in: veth vxlan ip6_udp_tunnel udp_tunnel xt_set ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipportip ip_set_hash_ipport ip_set dummy ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter symap_custom_3_10_0_862_el7_x86_64(POE) devlink nf_conntrack_ipv4 nf_defrag_ipv4 symev_custom_3_10_0_862_el7_x86_64(OE) ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack overlay(T) vmw_vsock_vmci_transport vsock sunrpc xfs libcrc32c ppdev vmw_balloon pcspkr joydev sg vmw_vmci nfit i2c_piix4
shpchp libnvdimm parport_pc parport ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm ata_piix libahci libata vmxnet3 serio_raw i2c_core vmw_pvscsi floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 20 PID: 32539 Comm: java Kdump: loaded Tainted: P OEL ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff93530c96eeb0 ti: ffff9362a7e38000 task.ti: ffff9362a7e38000
RIP: 0010:[<ffffffff9d4619d9>] [<ffffffff9d4619d9>] fsnotify+0x3b9/0x510
RSP: 0018:ffff9362a7e3bdc0 EFLAGS: 00000246
RAX: ffff9362b63b9900 RBX: 0000000000000001 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff9e3e7a40
RBP: ffff9362a7e3be90 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff9362a7e3be10
R13: ffff9362a7e3bdc8 R14: ffff9362a7e3bdd8 R15: 0000000000000001
FS: 00007f2da9b3c700(0000) GS:ffff9362bf400000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000c2f3a4a4 CR3: 0000000c4a618000 CR4: 00000000000007e0
Call Trace:
[<ffffffff9d41a1f3>] ? do_sync_write+0x93/0xe0
[<ffffffff9d41ad78>] vfs_write+0x168/0x1f0
[<ffffffff9d41baff>] SyS_write+0x7f/0xf0
[<ffffffff9d91f721>] ? system_call_after_swapgs+0xae/0x146
Complete report #163539
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
fsnotify
|
vmlinux |
0xffffffffb6861867
Build id: 3.10.0-862.el7.x86_64
|
0x247
|
|
2
|
|
vmlinux |
0xffffffffb681a1f3
Build id: 3.10.0-862.el7.x86_64
|
0x93
|
|
3
|
vfs_write
|
vmlinux |
0xffffffffb681ad78
Build id: 3.10.0-862.el7.x86_64
|
0x168
|
|
4
|
SyS_write
|
vmlinux |
0xffffffffb681baff
Build id: 3.10.0-862.el7.x86_64
|
0x7f
|
|
5
|
system_call_fastpath
|
vmlinux |
0xffffffffb6d1f7d5
Build id: 3.10.0-862.el7.x86_64
|
0x1c
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [java:13454]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache veth vxlan ip6_udp_tunnel udp_tunnel xt_statistic xt_nat xt_recent xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc kvm_ivshmem(OE) ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) nls_utf8 isofs sunrpc iosf_mbi ppdev crc32_pclmul sg ghash_clmulni_intel aesni_intel joydev virtio_balloon lrw gf128mul glue_helper ablk_helper cryptd parport_pc i2c_piix4 parport shpchp pcspkr ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console
virtio_net virtio_blk pv_channel(OE) cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel virtio_pci i2c_core virtio_ring serio_raw virtio floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 13454 Comm: java Kdump: loaded Tainted: G OE ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: OpenStack Foundation OpenStack Nova, BIOS rel-1.10.2-0-g5f4c7b1-20181220_000000-szxrtosci10000 04/01/2014
task: ffff8985916adee0 ti: ffff89859b76c000 task.ti: ffff89859b76c000
RIP: 0010:[<ffffffffb6861867>] [<ffffffffb6861867>] fsnotify+0x247/0x510
RSP: 0018:ffff89859b76fdf0 EFLAGS: 00000246
RAX: ffff897ff5d59978 RBX: ffff89859b76fe08 RCX: 0000000000000002
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffb77e7a40
RBP: ffff89859b76fec0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 000000004529a449
R13: ffff89859b76fd78 R14: 000000010001fdc0 R15: ffff8984592180c0
FS: 00007f1786698b10(0000) GS:ffff8987df380000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff6068b5160 CR3: 00000000a77f8000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb681a1f3>] ? do_sync_write+0x93/0xe0
[<ffffffffb681ad78>] vfs_write+0x168/0x1f0
[<ffffffffb681baff>] SyS_write+0x7f/0xf0
[<ffffffffb6d1f7d5>] system_call_fastpath+0x1c/0x21
Code: 4d 85 ed 74 0f 41 f6 45 44 01 75 08 41 c7 45 40 00 00 00 00 48 85 db 0f 84 97 01 00 00 44 8b 73 40 4c 8b 5b 08 41 f7 d6 44 23 33 <44> 23 75 a4 4d 85 ed 0f 84 8c 01 00 00 45 8b 65 40 4d 8b 5d 08