Info

Permalink
Function
run_timer_softirq
First occurrence
2020-09-23
Last occurrence
2026-03-14
State
NEW
Components
kernel
Quality
-2

Graphs

Statistics

Operating system Count
CentOS 7 3,375
CentOS 8 2
Architecture Count
x86_64 3,377
Related packages Count
kernel 3,362
 0:3.10.0-1062.4.1.el7 500
 0:3.10.0-957.el7 370
 0:3.10.0-1160.el7 312
 0:3.10.0-862.el7 189
 0:3.10.0-327.el7 184
 0:3.10.0-1127.10.1.el7 118
 0:3.10.0-1062.el7 98
 0:3.10.0-1160.76.1.el7 92
 0:3.10.0-1160.45.1.el7 70
 0:3.10.0-693.el7 65
 0:3.10.0-1160.119.1.el7 60
 0:3.10.0-1160.71.1.el7 57
 0:3.10.0-1062.12.1.el7 55
 0:3.10.0-957.21.3.el7 53
 0:3.10.0-1160.88.1.el7 48
 0:3.10.0-862.14.4.el7 47
 0:3.10.0-1160.25.1.el7 46
 0:3.10.0-1160.90.1.el7 44
 0:3.10.0-1127.el7 41
 0:3.10.0-1160.21.1.el7 40
 0:3.10.0-1160.53.1.el7 37
 0:3.10.0-1127.19.1.el7 35
 0:3.10.0-1160.108.1.el7 34
 0:3.10.0-1160.59.1.el7 34
 0:3.10.0-1160.36.2.el7 29
 0:3.10.0-1160.15.2.el7 28
 0:3.10.0-1160.6.1.el7 28
 0:3.10.0-1062.18.1.el7 27
 0:3.10.0-1160.11.1.el7 26
 0:3.10.0-1160.80.1.el7 25
 0:3.10.0-862.3.2.el7 25
 0:3.10.0-1062.1.1.el7 23
 0:3.10.0-1160.92.1.el7 23
 0:3.10.0-1160.62.1.el7 22
 0:3.10.0-1160.49.1.el7 21
 0:3.10.0-1160.81.1.el7 21
 0:3.10.0-957.5.1.el7 21
 0:3.10.0-1062.9.1.el7 20
 0:3.10.0-1127.18.2.el7 20
 0:3.10.0-1160.83.1.el7 20
 0:3.10.0-1062.4.3.el7 19
 0:3.10.0-1160.99.1.el7 19
 0:3.10.0-862.3.3.el7 19
 0:3.10.0-693.5.2.el7 18
 0:3.10.0-1160.105.1.el7 17
 0:3.10.0-1160.118.1.el7 17
 0:3.10.0-1160.95.1.el7 16
 0:3.10.0-1127.13.1.el7 15
 0:3.10.0-1160.102.1.el7 15
 0:3.10.0-1160.31.1.el7 15
 0:3.10.0-123.20.1.el7 14
 0:3.10.0-514.10.2.el7 14
 0:3.10.0-514.el7 13
 0:3.10.0-693.2.2.el7 13
 0:3.10.0-957.1.3.el7 11
 0:3.10.0-1160.66.1.el7 10
 0:3.10.0-327.13.1.el7 10
 0:3.10.0-957.12.2.el7 10
 0:3.10.0-693.17.1.el7 9
 0:3.10.0-1160.41.1.el7 8
 0:3.10.0-1160.114.2.el7 7
 0:3.10.0-514.16.1.el7 7
 0:3.10.0-862.11.6.el7 7
 0:3.10.0-1160.42.2.el7 6
 0:3.10.0-957.27.2.el7 6
 0:3.10.0-327.18.2.el7 5
 0:3.10.0-1127.8.2.el7 4
 0:3.10.0-1160.2.2.el7 4
 0:3.10.0-957.10.1.el7 4
 0:3.10.0-514.26.2.el7 3
 0:3.10.0-693.11.1.el7 3
 0:3.10.0-327.10.1.el7 2
 0:3.10.0-693.21.1.el7 2
 0:3.10.0-862.2.3.el7 2
 0:3.10.0-957.12.1.el7 2
 0:3.10.0-957.21.2.el7 2
 0:3.10.0-1062.1.2.el7 1
 0:3.10.0-1160.24.1.el7 1
 0:3.10.0-327.28.3.el7 1
 0:3.10.0-327.36.1.el7 1
 0:3.10.0-514.6.1.el7 1
 0:3.10.0-327.28.2.el7 1
kernel-lt 15
 0:4.4.241-1.el7.elrepo 9
 0:4.4.60-1.el7.elrepo 4
 0:4.4.245-1.el7.elrepo 2

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #9595
Frame # Function Binary Source or offset Line
1
run_timer_softirq
vmlinux 0xffffffff95caa4b2
Build id: 3.10.0-957.21.3.el7.x86_64
0x1d2
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [freeswitch:13800]
Modules linked in: fuse tcp_diag inet_diag xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd vmw_balloon pcspkr sg joydev vmw_vmci i2c_piix4 parport_pc parport
ip_tables xfs libcrc32c vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper ata_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm pata_acpi drm nfit mptspi scsi_transport_spi e1000 mptscsih ata_piix libnvdimm libata crct10dif_pclmul crct10dif_common mptbase crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 13800 Comm: freeswitch Kdump: loaded Tainted: G        W    L ------------   3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff94ccce62d140 ti: ffff94ccc47c0000 task.ti: ffff94ccc47c0000
RIP: 0010:[<ffffffff95caa4b2>]  [<ffffffff95caa4b2>] run_timer_softirq+0x1d2/0x300
RSP: 0000:ffff94cd3fc03ea8  EFLAGS: 00000297
RAX: 0000000284c98a77 RBX: 000000001cf2f8b0 RCX: ffff94cd3fc03ed0
RDX: 0000000284c98a76 RSI: ffff94cd3fc03ed0 RDI: ffffffff96ce3c00
RBP: ffff94cd3fc03f10 R08: ffff94cd3fc1a420 R09: ffffffff96ce3c03
R10: 0000000000000000 R11: ffff94cd3fc03de8 R12: ffff94cd3fc03e18
R13: ffffffff96376df2 R14: ffff94cd3fc03f10 R15: ffffffff96ce3c00
FS:  00007fd0040aa700(0000) GS:ffff94cd3fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005599522c4847 CR3: 00000000bb7da000 CR4: 00000000003607f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
Complete report #2384
Frame # Function Binary Source or offset Line
1
run_timer_softirq
vmlinux 0xffffffff8c6aa3ab
Build id: 3.10.0-957.1.3.el7.x86_64
0x23b
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [kworker/0:3:26219]
Modules linked in: nfring(OE) xt_iprange xt_NFQUEUE nfnetlink_log binfmt_misc xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype br_netfilter ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock ppdev sb_edac coretemp vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport
vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata mptspi scsi_transport_spi vmxnet3 mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: nfring]
CPU: 0 PID: 26219 Comm: kworker/0:3 Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-957.1.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
Workqueue: xfs-data/dm-0 xfs_end_io [xfs]
task: ffffa0813ab59040 ti: ffffa07a55564000 task.ti: ffffa07a55564000
RIP: 0010:[<ffffffff8c6aa3ab>]  [<ffffffff8c6aa3ab>] run_timer_softirq+0x23b/0x300
RSP: 0000:ffffa0817fc03ea8  EFLAGS: 00000246
RAX: dead000000000200 RBX: ffffa0817fc03e60 RCX: ffffa07d715d6110
RDX: ffffa0817fc03ed0 RSI: ffffa0817fc03ed0 RDI: ffffffff8d6e3c00
RBP: ffffa0817fc03f10 R08: ffffffff8d6e4328 R09: ffffffff8d6e3c00
R10: 0000000000000000 R11: ffffa0817fc03de8 R12: ffffa0817fc03e18
R13: ffffffff8cd75df2 R14: ffffa0817fc03f10 R15: ffffffff8d6e3c00
FS:  0000000000000000(0000) GS:ffffa0817fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f35eaccc1a0 CR3: 00000007f7210000 CR4: 00000000000607f0
Call Trace:
Complete report #1019789
Frame # Function Binary Source or offset Line
1
run_timer_softirq
vmlinux 0xffffffffb7cb31eb
Build id: 3.10.0-1160.108.1.el7.x86_64
0x24b
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 32s! [swapper/1:0]
Modules linked in: xt_owner xt_REDIRECT nf_nat_redirect xt_multiport ipt_rpfilter iptable_raw ip_set_hash_ip ip_set_hash_net ipip tunnel4 ip_tunnel rpcsec_gss_krb5 veth auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_set ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipportip ip_set_hash_ipport dummy nf_tables ip_set ip_vs_sh ip_vs_wrr ip_vs_rr ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_filter ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6_tables xt_mark xt_comment iptable_mangle xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_addrtype iptable_filter scini(POE) ip_vs nf_conntrack binfmt_misc vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon
sg parport_pc joydev parport pcspkr vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix mptspi scsi_transport_spi mptscsih drm libata crc32c_intel serio_raw mptbase vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod br_netfilter bridge stp llc overlay(T)
CPU: 1 PID: 0 Comm: swapper/1 Tainted: P        W  OEL ------------ T 3.10.0-1160.108.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffffa05df8e3a100 ti: ffffa05df8e84000 task.ti: ffffa05df8e84000
RIP: 0010:[<ffffffffb7cb31eb>]  [<ffffffffb7cb31eb>] run_timer_softirq+0x24b/0x340
RSP: 0018:ffffa05dfbc83ea8  EFLAGS: 00000246
RAX: dead000000000200 RBX: 00000001c070e2fa RCX: ffffa05df606bff0
RDX: ffffa05dfbc83ed0 RSI: ffffa05dfbc83ed0 RDI: ffffa05dfbc93940
RBP: ffffa05dfbc83f10 R08: ffffa05dfbc93968 R09: ffffa05dfbc93940
R10: 000000000000be62 R11: 0000000000000400 R12: ffffa05dfbc83e18
R13: ffffffffb83c63f2 R14: ffffa05dfbc83f10 R15: ffffa05dfbc93940
FS:  0000000000000000(0000) GS:ffffa05dfbc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc806bed1c0 CR3: 00000001ebf5a000 CR4: 00000000000007e0
Call Trace:
Complete report #1188
Frame # Function Binary Source or offset Line
1
run_timer_softirq
vmlinux 0xffffffffb9caec4e
Build id: 3.10.0-1127.13.1.el7.x86_64
0x23e
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 33s! [gsd-color:8283]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper vmw_balloon ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic vmwgfx sr_mod cdrom ata_generic drm_kms_helper pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common mptspi scsi_transport_spi mptscsih
e1000 mptbase crc32c_intel ata_piix libata serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 8283 Comm: gsd-color Kdump: loaded Tainted: G             L ------------   3.10.0-1127.13.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
task: ffff8b60291dd230 ti: ffff8b6057ab4000 task.ti: ffff8b6057ab4000
RIP: 0010:[<ffffffffb9caec4e>]  [<ffffffffb9caec4e>] run_timer_softirq+0x23e/0x300
RSP: 0000:ffff8b62ee6c3ea8  EFLAGS: 00000246
RAX: dead000000000200 RBX: 0000000339c6fd90 RCX: ffff8b62e8758548
RDX: ffff8b62ee6c3ed0 RSI: ffff8b62ee6c3ed0 RDI: ffff8b62ee6d3940
RBP: ffff8b62ee6c3f10 R08: ffff8b62ee6d4168 R09: ffff8b62ee6d3940
R10: 000000000000ba7e R11: 0000000000000005 R12: ffff8b62ee6c3e18
R13: ffffffffba393efa R14: ffff8b62ee6c3f10 R15: ffff8b62ee6d3940
FS:  00007f90032cba40(0000) GS:ffff8b62ee6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9003303000 CR3: 0000000012a42000 CR4: 00000000001607e0
Call Trace:
Complete report #610858
Frame # Function Binary Source or offset Line
1
run_timer_softirq
vmlinux 0xffffffff8108ddc5
Build id: 3.10.0-327.28.2.el7.x86_64
0x225
BUG: soft lockup - CPU#0 stuck for 24s! [swapper/0:0]
Modules linked in: zram(C) vmw_vsock_vmci_transport vsock ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables iptable_filter coretemp crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon pcspkr sg vmxnet3 vmw_vmci i2c_piix4 shpchp parport_pc parport sch_fq_codel binfmt_misc ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common serio_raw drm_kms_helper ttm ahci libahci vmw_pvscsi drm i2c_core ata_piix libata floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C   L ------------   3.10.0-327.28.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffffffff81951440 ti: ffffffff8193c000 task.ti: ffffffff8193c000
RIP: 0010:[<ffffffff8108ddc5>]  [<ffffffff8108ddc5>] run_timer_softirq+0x225/0x340
RSP: 0018:ffff88017fc03e68  EFLAGS: 00000212
RAX: dead000000200200 RBX: 000000008158853c RCX: ffff8800b9470348
RDX: ffff88017fc03e90 RSI: ffff88017fc03e90 RDI: ffff8800b9470348
RBP: ffff88017fc03ed0 R08: ffffffff81daefe8 R09: ffffffff81daecc0
R10: ffff880175f64368 R11: ffff88017795f000 R12: ffff88017fc03dd8
R13: ffffffff8164789d R14: ffff88017fc03ed0 R15: ffffffff81daecc0
FS:  0000000000000000(0000) GS:ffff88017fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fcdcd393cdc CR3: 00000001149ce000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
 ffffffff81db08e8 ffffffff81db04e8 ffffffff81db00e8 ffffffff81dafce8
 ffff8800b9470000 ffff88017fc03e90 ffff88017fc03e90 c9e2483820edf5ae
 0000000000000001 0000000000000001 ffffffff8193c000 ffffffff81943088
Call Trace:
Complete report #528480
Frame # Function Binary Source or offset Line
1
run_timer_softirq
vmlinux 0xffffffff9f6ae52e
Build id: 3.10.0-1160.76.1.el7.x86_64
0x24e
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [C2 CompilerThre:14448]
Modules linked in: veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c br_netfilter bridge stp llc falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_14203(E) ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit ata_piix
libnvdimm drm crct10dif_pclmul crct10dif_common mptspi scsi_transport_spi libata crc32c_intel serio_raw mptscsih vmxnet3 mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 14448 Comm: C2 CompilerThre Kdump: loaded Tainted: P            E  ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9bd0ab40b180 ti: ffff9bd0a0fb0000 task.ti: ffff9bd0a0fb0000
RIP: 0010:[<ffffffff9f6ae52e>]  [<ffffffff9f6ae52e>] run_timer_softirq+0x24e/0x340
RSP: 0018:ffff9bd17fd03ea8  EFLAGS: 00000246
RAX: dead000000000200 RBX: 0000000077d747c5 RCX: ffff9bcdf5febbd0
RDX: ffff9bcdf5febbd0 RSI: ffff9bd17fd03ed0 RDI: ffff9bd17fd13940
RBP: ffff9bd17fd03f10 R08: ffff9bcdf5febbd0 R09: ffff9bd17fd13940
R10: 00000000f5feb601 R11: ffff9bcdf5feb8c0 R12: ffff9bd17fd03e18
R13: ffffffff9fd9afba R14: ffff9bd17fd03f10 R15: ffff9bd17fd13940
FS:  00007f21696fe700(0000) GS:ffff9bd17fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f81ee0921e0 CR3: 0000000371da4000 CR4: 00000000000407e0
Call Trace:
Complete report #418885
Frame # Function Binary Source or offset Line
1
run_timer_softirq
vmlinux 0xffffffff84eae3bb
Build id: 3.10.0-1160.59.1.el7.x86_64
0x24b
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 54s! [C2 CompilerThre:3388]
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 54s! [tor:28094]
Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter rpcsec_gss_krb5 auth_rpcgss tun nfsv4 dns_resolver binfmt_misc nfs lockd grace fscache overlay(T) nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon
pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c vmwgfx sr_mod cdrom drm_kms_helper ata_generic pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops sd_mod ttm crc_t10dif crct10dif_generic drm ata_piix crct10dif_pclmul crct10dif_common e1000 libata crc32c_intel serio_raw vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 28094 Comm: tor Kdump: loaded Tainted: G    B        L ------------ T 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 10/22/2013
task: ffff884ea4d91080 ti: ffff884b24f7c000 task.ti: ffff884b24f7c000
RIP: 0010:[<ffffffff84eae3bb>]  [<ffffffff84eae3bb>] run_timer_softirq+0x24b/0x340
RSP: 0000:ffff884f3fd83ea8  EFLAGS: 00000246
RAX: dead000000000200 RBX: ffffffff85a070c8 RCX: ffff884e367d49d0
RDX: ffff884f3fd83ed0 RSI: ffff884f3fd83ed0 RDI: ffff884f3fd93940
RBP: ffff884f3fd83f10 R08: ffff884f3fd93be8 R09: ffff884f3fd93940
R10: ffff884b762a2800 R11: 0000000000000005 R12: ffff884f3fd83e18
R13: ffffffff8559afba R14: ffff884f3fd83f10 R15: ffff884f3fd93940
FS:  00007fca99a81d48(0000) GS:ffff884f3fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000786292000 CR3: 00000001c4cb0000 CR4: 00000000000007e0
Call Trace:
Complete report #143696
Frame # Function Binary Source or offset Line
1
run_timer_softirq
vmlinux 0xffffffff8109b56a
Build id: 3.10.0-693.17.1.el7.x86_64
0x1ea
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 55s! [swapper/0:0]
Modules linked in: fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vboxsf(OE) joydev ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev vboxvideo(OE) aesni_intel ttm lrw gf128mul glue_helper ablk_helper cryptd drm_kms_helper syscopyarea sysfillrect sysimgblt vboxguest(OE) fb_sys_fops drm sg e1000 pcspkr i2c_piix4 i2c_core parport_pc parport
video nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi mptspi scsi_transport_spi mptscsih ata_piix libata mptbase crct10dif_pclmul crct10dif_common crc32c_intel serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W  OEL ------------   3.10.0-693.17.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffffffff81a02480 ti: ffffffff819ec000 task.ti: ffffffff819ec000
RIP: 0010:[<ffffffff8109b56a>]  [<ffffffff8109b56a>] run_timer_softirq+0x1ea/0x310
RSP: 0018:ffff88041fc03e68  EFLAGS: 00000246
RAX: 0000000000000001 RBX: 000000001fc03e20 RCX: ffff88041fc03e90
RDX: 00000001054c5792 RSI: ffff88041fc03e90 RDI: ffffffff81e99180
RBP: ffff88041fc03ed0 R08: ffffffff81e99c08 R09: ffffffff81e99183
R10: 0000000000000002 R11: ffff88041fc03da8 R12: ffff88041fc03dd8
R13: ffffffff816b9d32 R14: ffff88041fc03ed0 R15: ffffffff81e99180
FS:  0000000000000000(0000) GS:ffff88041fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007faf01b93000 CR3: 00000003faab2000 CR4: 00000000000606f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
Complete report #23012
Frame # Function Binary Source or offset Line
1
run_timer_softirq
vmlinux 0xffffffff9a6a3b28
Build id: 3.10.0-862.el7.x86_64
0x38
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [watchdog/2:17]
Modules linked in: rpcsec_gss_krb5 vhost_net vhost macvtap macvlan tun fuse xt_multiport ext4 mbcache jbd2 devlink nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nct6775 hwmon_vid vfat fat edac_mce_amd kvm_amd snd_hda_codec_realtek snd_hda_codec_generic kvm snd_hda_codec_hdmi irqbypass crc32_pclmul snd_hda_intel ghash_clmulni_intel snd_hda_codec aesni_intel lrw
gf128mul glue_helper snd_hda_core ablk_helper cryptd snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer pcspkr snd ccp soundcore i2c_piix4 sg gpio_amdpt i2c_designware_platform pinctrl_amd i2c_designware_core wmi shpchp acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic amdkfd amd_iommu_v2 radeon crct10dif_pclmul crct10dif_common crc32c_intel i2c_algo_bit drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm r8169 mii libahci libata i2c_core rr272x_1x(POE) dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 17 Comm: watchdog/2 Kdump: loaded Tainted: P           OE  ------------   3.10.0-862.el7.x86_64 #1
Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./AB350 Pro4, BIOS P2.50 04/20/2017
task: ffff98c7fc8feeb0 ti: ffff98c7fc924000 task.ti: ffff98c7fc924000
RIP: 0010:[<ffffffff9a6a3b28>]  [<ffffffff9a6a3b28>] run_timer_softirq+0x38/0x310
RSP: 0000:ffff98c896683e88  EFLAGS: 00000202
RAX: 00000000fffd9eda RBX: ffff98c896683e18 RCX: 0000000000000100
RDX: ffff98c896680000 RSI: ffff98c896683eb0 RDI: ffffffff9b2070c8
RBP: ffff98c896683ef0 R08: 0000000000000084 R09: ffff98c7fc927ea0
R10: 0000000000000000 R11: 0000000000000000 R12: ffff98c896683df8
R13: ffffffff9ad207f2 R14: ffff98c896683ef0 R15: ffff98c7fc958000
FS:  00007fce49603700(0000) GS:ffff98c896680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa42393e524 CR3: 00000000c5fb6000 CR4: 00000000003407e0
Call Trace: