Complete report #7841
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff87b1761a
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff87a7e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xb8
|
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 21s! [in:imjournal:1663]
Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper sg ablk_helper cryptd ppdev vmw_balloon pcspkr joydev parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix mptspi nfit scsi_transport_spi libata mptscsih serio_raw libnvdimm mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 9 PID: 1663 Comm: in:imjournal Tainted: G L ------------ 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/19/2018
task: ffff8b04a7bcd230 ti: ffff8b013888c000 task.ti: ffff8b013888c000
RIP: 0010:[<ffffffff87b1761a>] [<ffffffff87b1761a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b013888fcb0 EFLAGS: 00000202
RAX: 0000000000000005 RBX: 0008000087a7e730 RCX: ffff8b04ad760740
RDX: 0000000000000005 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8b013888fce8 R08: ffff8b01f9ccf800 R09: ffffffff87d851b9
R10: ffff8b04ad85f160 R11: ffffdd4210af9e00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8b013888fc60 R15: 00007f584b7ef000
FS: 00007f5846e57700(0000) GS:ffff8b04ad840000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f584b7ef000 CR3: 000000042978e000 CR4: 00000000001607e0
Call Trace:
[<ffffffff87a7e8f8>] native_flush_tlb_others+0xb8/0xc0
Complete report #753371
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8d31519a
Build id: 3.10.0-1062.4.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8d27d478
Build id: 3.10.0-1062.4.1.el7.x86_64
|
0xb8
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [BESClient:3057]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_15309(E) devlink binfmt_misc falcon_lsm_pinned_15003(E) ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper snd_hda_codec_generic ablk_helper cryptd joydev snd_hda_intel virtio_balloon snd_hda_codec
sg snd_hda_core snd_hwdep parport_pc snd_seq parport snd_seq_device snd_pcm snd_timer snd pcspkr soundcore i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel virtio_pci floppy virtio_ring serio_raw virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: falcon_kal]
CPU: 1 PID: 3057 Comm: BESClient Kdump: loaded Tainted: P EL ------------ 3.10.0-1062.4.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff889b03e01070 ti: ffff889af1600000 task.ti: ffff889af1600000
RIP: 0010:[<ffffffff8d31519a>] [<ffffffff8d31519a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff889af1603cb0 EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000800008d27d2b0 RCX: ffff889b1f0e0320
RDX: 0000000000000003 RSI: 0000000000000018 RDI: 0000000000000000
RBP: ffff889af1603ce8 R08: ffff886cf6c07400 R09: ffffffff8d57fe99
R10: ffff889b1f05f160 R11: ffffe84e3d52da00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff889af1603c60 R15: 00007f0f60051000
FS: 00007f0f60043740(0000) GS:ffff889b1f040000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000062592001 CR3: 0000002f31690000 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8d27d478>] native_flush_tlb_others+0xb8/0xc0
Complete report #313159
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8691173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8687a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [java:30937]
Modules linked in: kbase(OE) syshook_linux(OE) binfmt_misc udp_diag tcp_diag inet_diag fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel virtio_balloon aesni_intel parport_pc lrw parport gf128mul glue_helper i2c_piix4 ablk_helper cryptd joydev pcspkr ip_tables ext4 mbcache jbd2 virtio_console
virtio_blk virtio_net ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm floppy ata_piix drm crct10dif_pclmul crct10dif_common crc32c_intel libata serio_raw drm_panel_orientation_quirks virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod [last unloaded: kbase]
CPU: 15 PID: 30937 Comm: java Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: Fedora Project OpenStack Nova, BIOS 0.5.1 01/01/2011
task: ffff9b5f8ef34100 ti: ffff9b50c3474000 task.ti: ffff9b50c3474000
RIP: 0010:[<ffffffff8691173a>] [<ffffffff8691173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9b50c3477cb8 EFLAGS: 00000202
RAX: 0000000000000008 RBX: 00000000000000fc RCX: ffff9b5fbf420618
RDX: 0000000000000008 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff9b50c3477cf0 R08: ffff9b51294f6800 R09: ffffffff86b755c9
R10: ffff9b5fbf5df160 R11: fffffa0bfea4ac00 R12: 0000000000000296
R13: ffff9b50c3477c68 R14: 0000000000000001 R15: fffffa0bc629f840
FS: 00007f490aff1700(0000) GS:ffff9b5fbf5c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4b89f4c000 CR3: 0000000ff6970000 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8687a598>] native_flush_tlb_others+0xb8/0xc0
Complete report #196537
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810d6d62
Build id: 3.10.0-229.14.1.el7.x86_64
|
0x202
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8105fc78
Build id: 3.10.0-229.14.1.el7.x86_64
|
0xb8
|
BUG: soft lockup - CPU#11 stuck for 22s! [ftdc:4230]
dahdi: Detected time shift.
INFO: rcu_sched detected stalls on CPUs/tasks: { 5 15} (detected by 2, t=575832 jiffies, g=15349850, c=15349849, q=0)
Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink bluetooth rfkill tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables binfmt_misc dahdi(OF) crc_ccitt netatop(OF) xt_CHECKSUM iptable_mangle xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_nat_ipv4 nf_nat nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_recent xt_multiport vmw_vsock_vmci_transport vsock xt_mark xt_conntrack nf_conntrack iptable_filter coretemp ext4 mbcache jbd2 snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus crc32_pclmul snd_seq ghash_clmulni_intel snd_seq_device snd_pcm ppdev aesni_intel lrw gf128mul snd_timer glue_helper vmw_balloon ablk_helper ahci cryptd snd soundcore libahci pcspkr sg i2c_piix4 vmw_vmci shpchp parport_pc parport nfsd
auth_rpcgss nfs_acl lockd sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix mptspi serio_raw scsi_transport_spi libata mptscsih i2c_core e1000 mptbase dm_mirror dm_region_hash dm_log dm_mod
CPU: 11 PID: 4230 Comm: ftdc Tainted: GF O-------------- 3.10.0-229.14.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff880317f471c0 ti: ffff8802db5a0000 task.ti: ffff8802db5a0000
RIP: 0010:[<ffffffff810d6d62>] [<ffffffff810d6d62>] smp_call_function_many+0x202/0x260
sending NMI to all CPUs:
RSP: 0018:ffff8802db5a3d10 EFLAGS: 00000202
RAX: 0000000000000007 RBX: 000800008105fab0 RCX: ffff8803326f7660
RDX: 0000000000000007 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8802db5a3d48 R08: ffff88032d704800 R09: ffff8803327764a0
R10: ffffea0002e7ca00 R11: ffffffff812d4779 R12: 0000000000000282
R13: 0000000000000018 R14: ffff8802db5a3c80 R15: 0000000000000282
FS: 00007fdc8f364700(0000) GS:ffff880332760000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fdc9bb4a000 CR3: 0000000311921000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000100000000 ffff880326508988 ffff880326508640 00007fdc9bb4a000
00007fdc9bb4b000 ffff880326508988 ffff8800b90e5b00 ffff8802db5a3d98
ffffffff8105fc78 ffff880326508640 00007fdc9bb4a000 00007fdc9bb4b000
Call Trace:
[<ffffffff8105fc78>] native_flush_tlb_others+0xb8/0xc0
Complete report #1784
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb9d175ca
Build id: 3.10.0-1127.13.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb9c7e8f8
Build id: 3.10.0-1127.13.1.el7.x86_64
|
0xb8
|
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [pool:54779]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper vmw_balloon ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic vmwgfx sr_mod cdrom ata_generic drm_kms_helper pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common mptspi scsi_transport_spi mptscsih
e1000 mptbase crc32c_intel ata_piix libata serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 8 PID: 54779 Comm: pool Kdump: loaded Tainted: G W L ------------ 3.10.0-1127.13.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
task: ffff8b5fcaf28000 ti: ffff8b5fd0098000 task.ti: ffff8b5fd0098000
RIP: 0010:[<ffffffffb9d175ca>] [<ffffffffb9d175ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b5fd009bcb0 EFLAGS: 00000202
RAX: 0000000000000006 RBX: ffffffffb9c7e730 RCX: ffffb9163ef828b8
RDX: 0000000000000006 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff8b5fd009bce8 R08: ffff8b62a7adf400 R09: ffffffffb9f85179
R10: ffff8b62ee81f160 R11: ffffe1d5444a3a00 R12: 0000000000000286
R13: ffff8b5fd009bc60 R14: 00007fcac1910000 R15: 00007fcac190f000
FS: 00007fcaaa7fc700(0000) GS:ffff8b62ee800000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007faa95068000 CR3: 0000000005eb4000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb9c7e8f8>] native_flush_tlb_others+0xb8/0xc0