Info

Permalink
Function
__warn
First occurrence
2020-09-23
Last occurrence
2024-02-20
State
NEW
Components
kernel
Quality
-17

Graphs

Statistics

Operating system Count
CentOS 7 640
CentOS 8 7
Architecture Count
x86_64 647
Related packages Count
kernel 647
 0:3.10.0-1062.9.1.el7 64
 0:3.10.0-1160.21.1.el7 49
 0:3.10.0-1160.el7 46
 0:3.10.0-1160.53.1.el7 41
 0:3.10.0-1127.19.1.el7 40
 0:3.10.0-1160.31.1.el7 40
 0:3.10.0-1160.15.2.el7 35
 0:3.10.0-1160.11.1.el7 29
 0:3.10.0-1062.el7 26
 0:3.10.0-1160.25.1.el7 24
 0:3.10.0-1160.76.1.el7 20
 0:3.10.0-957.27.2.el7 20
 0:3.10.0-693.el7 16
 0:3.10.0-1160.36.2.el7 15
 0:3.10.0-1160.62.1.el7 14
 0:3.10.0-1062.1.1.el7 13
 0:3.10.0-1062.12.1.el7 13
 0:3.10.0-1160.45.1.el7 12
 0:3.10.0-1160.71.1.el7 12
 0:3.10.0-1160.95.1.el7 12
 0:3.10.0-1160.6.1.el7 11
 0:3.10.0-1160.49.1.el7 9
 0:3.10.0-1160.80.1.el7 8
 0:3.10.0-1127.el7 7
 0:3.10.0-1160.59.1.el7 6
 0:3.10.0-957.el7 6
 0:3.10.0-1127.18.2.el7 6
 0:3.10.0-1127.8.2.el7 5
 0:3.10.0-1160.41.1.el7 5
 0:3.10.0-1160.99.1.el7 5
 0:3.10.0-1160.81.1.el7 5
 0:3.10.0-1160.83.1.el7 5
 0:3.10.0-1160.92.1.el7 4
 0:3.10.0-862.14.4.el7 4
 0:3.10.0-1160.24.1.el7 3
 0:3.10.0-1160.105.1.el7 2
 0:3.10.0-1160.90.1.el7 2
 0:3.10.0-957.21.3.el7 2
 0:3.10.0-1160.102.1.el7 2
 0:3.10.0-1160.88.1.el7 2
 0:3.10.0-1160.66.1.el7 1
 0:3.10.0-957.10.1.el7 1
 0:3.10.0-1160.42.2.el7 1
 0:3.10.0-1160.114.2.el7 1
 0:3.10.0-862.3.3.el7 1
 0:3.10.0-862.el7 1
 0:3.10.0-1160.2.2.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #820678
Frame # Function Binary Source or offset Line
1
? update_curr
vmlinux 0xffffffffb98e00bc
Build id: 3.10.0-957.el7.x86_64
0x14c
2
net_rx_action
vmlinux 0xffffffffb9e39e9f
Build id: 3.10.0-957.el7.x86_64
0x26f
3
__do_softirq
vmlinux 0xffffffffb98a0f05
Build id: 3.10.0-957.el7.x86_64
0xf5
4
run_ksoftirqd
vmlinux 0xffffffffb98a10c8
Build id: 3.10.0-957.el7.x86_64
0x38
5
smpboot_thread_fn
vmlinux 0xffffffffb98ca5c4
Build id: 3.10.0-957.el7.x86_64
0x144
6
? lg_double_unlock
vmlinux 0xffffffffb98ca480
Build id: 3.10.0-957.el7.x86_64
0x40
7
kthread
vmlinux 0xffffffffb98c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
8
? insert_kthread_work
vmlinux 0xffffffffb98c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
9
ret_from_fork_nospec_begin
vmlinux 0xffffffffb9f74c37
Build id: 3.10.0-957.el7.x86_64
0x21
10
? insert_kthread_work
vmlinux 0xffffffffb98c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 53s! [kworker/7:1:22358]
[<ffffffffb98e00bc>] ? update_curr+0x14c/0x1e0
[<ffffffffb9e39e9f>] net_rx_action+0x26f/0x390
[<ffffffffb98a0f05>] __do_softirq+0xf5/0x280
[<ffffffffb98a10c8>] run_ksoftirqd+0x38/0x50
[<ffffffffb98ca5c4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffb98ca480>] ? lg_double_unlock+0x40/0x40
[<ffffffffb98c1c31>] kthread+0xd1/0xe0
[<ffffffffb98c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb9f74c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb98c1b60>] ? insert_kthread_work+0x40/0x40
Complete report #295642
Frame # Function Binary Source or offset Line
1
run_ksoftirqd
vmlinux 0xffffffff9c6a4d58
Build id: 3.10.0-1160.2.2.el7.x86_64
0x38
2
smpboot_thread_fn
vmlinux 0xffffffff9c6ce674
Build id: 3.10.0-1160.2.2.el7.x86_64
0x144
3
? lg_double_unlock
vmlinux 0xffffffff9c6ce530
Build id: 3.10.0-1160.2.2.el7.x86_64
0x40
4
kthread
vmlinux 0xffffffff9c6c5c21
Build id: 3.10.0-1160.2.2.el7.x86_64
0xd1
5
? insert_kthread_work
vmlinux 0xffffffff9c6c5b50
Build id: 3.10.0-1160.2.2.el7.x86_64
0x40
6
ret_from_fork_nospec_begin
vmlinux 0xffffffff9cd93df7
Build id: 3.10.0-1160.2.2.el7.x86_64
0x21
7
? insert_kthread_work
vmlinux 0xffffffff9c6c5b50
Build id: 3.10.0-1160.2.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [NetstatCollecto:20851]
[<ffffffff9c6a4d58>] run_ksoftirqd+0x38/0x50
[<ffffffff9c6ce674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff9c6ce530>] ? lg_double_unlock+0x40/0x40
[<ffffffff9c6c5c21>] kthread+0xd1/0xe0
[<ffffffff9c6c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff9cd93df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9c6c5b50>] ? insert_kthread_work+0x40/0x40
Complete report #6504
Frame # Function Binary Source or offset Line
1
dump_stack
vmlinux 0xffffffff97d7ffa5
Build id: 3.10.0-1127.19.1.el7.x86_64
0x19
2
__warn
vmlinux 0xffffffff9769bd18
Build id: 3.10.0-1127.19.1.el7.x86_64
0xd8
3
warn_slowpath_null
vmlinux 0xffffffff9769be5d
Build id: 3.10.0-1127.19.1.el7.x86_64
0x1d
4
net_rx_action
vmlinux 0xffffffff97c54693
Build id: 3.10.0-1127.19.1.el7.x86_64
0x343
5
__do_softirq
vmlinux 0xffffffff976a5695
Build id: 3.10.0-1127.19.1.el7.x86_64
0xf5
6
run_ksoftirqd
vmlinux 0xffffffff976a5858
Build id: 3.10.0-1127.19.1.el7.x86_64
0x38
7
smpboot_thread_fn
vmlinux 0xffffffff976cf0e4
Build id: 3.10.0-1127.19.1.el7.x86_64
0x144
8
? lg_double_unlock
vmlinux 0xffffffff976cefa0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
9
kthread
vmlinux 0xffffffff976c6691
Build id: 3.10.0-1127.19.1.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffff976c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffff97d92d1d
Build id: 3.10.0-1127.19.1.el7.x86_64
0x7
12
? insert_kthread_work
vmlinux 0xffffffff976c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
WARNING: CPU: 0 PID: 6 at net/core/dev.c:5464 net_rx_action+0x343/0x390
Modules linked in: nls_utf8 isofs uas usb_storage tcp_lp rfcomm fuse cmac xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink binfmt_misc ipmi_devintf ipmi_msghandler drbg ansi_cprng ccm ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) bnep sunrpc squashfs loop arc4 vfat fat hid_multitouch dell_wmi intel_pmc_core iTCO_wdt iTCO_vendor_support intel_powerclamp
ath10k_pci coretemp uvcvideo btusb intel_rapl btrtl btbcm btintel kvm_intel ath10k_core bluetooth videobuf2_vmalloc videobuf2_memops videobuf2_core kvm dell_laptop videodev joydev ath snd_hda_codec_hdmi dell_smbios mac80211 dell_wmi_descriptor snd_soc_skl irqbypass dcdbas snd_soc_skl_ipc dell_led crc32_pclmul snd_hda_ext_core ghash_clmulni_intel snd_soc_sst_dsp snd_soc_sst_ipc aesni_intel cfg80211 snd_soc_acpi lrw snd_soc_core snd_hda_codec_realtek gf128mul glue_helper ablk_helper cryptd snd_hda_codec_generic snd_compress rfkill pcspkr snd_hda_intel sg idma64 i2c_designware_platform virt_dma i2c_designware_core snd_hda_codec wmi snd_hda_core snd_hwdep snd_seq pinctrl_sunrisepoint pinctrl_intel snd_seq_device intel_hid tpm_crb int3400_thermal snd_pcm sparse_keymap acpi_pad snd_timer snd
soundcore mei_me mei i2c_i801 ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic rtsx_usb_sdmmc mmc_core rtsx_usb i915 i2c_algo_bit iosf_mbi drm_kms_helper syscopyarea sysfillrect ahci sysimgblt fb_sys_fops libahci drm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw r8169 drm_panel_orientation_quirks video i2c_hid dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G           OE  ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: Dell Inc. Vostro 3581/0KVNNP, BIOS 1.7.0 02/14/2020
Call Trace:
[<ffffffff97d7ffa5>] dump_stack+0x19/0x1b
[<ffffffff9769bd18>] __warn+0xd8/0x100
[<ffffffff9769be5d>] warn_slowpath_null+0x1d/0x20
[<ffffffff97c54693>] net_rx_action+0x343/0x390
[<ffffffff976a5695>] __do_softirq+0xf5/0x280
[<ffffffff976a5858>] run_ksoftirqd+0x38/0x50
[<ffffffff976cf0e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff976cefa0>] ? lg_double_unlock+0x40/0x40
[<ffffffff976c6691>] kthread+0xd1/0xe0
[<ffffffff976c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff97d92d1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff976c65c0>] ? insert_kthread_work+0x40/0x40
Complete report #713843
Frame # Function Binary Source or offset Line
1
__do_softirq
vmlinux 0xffffffff988a957d
Build id: 3.10.0-1160.83.1.el7.x86_64
0xed
2
run_ksoftirqd
vmlinux 0xffffffff988a9758
Build id: 3.10.0-1160.83.1.el7.x86_64
0x38
3
smpboot_thread_fn
vmlinux 0xffffffff988d42ec
Build id: 3.10.0-1160.83.1.el7.x86_64
0x14c
4
? lg_double_unlock
vmlinux 0xffffffff988d41a0
Build id: 3.10.0-1160.83.1.el7.x86_64
0x40
5
kthread
vmlinux 0xffffffff988cb511
Build id: 3.10.0-1160.83.1.el7.x86_64
0xd1
6
? insert_kthread_work
vmlinux 0xffffffff988cb440
Build id: 3.10.0-1160.83.1.el7.x86_64
0x40
7
ret_from_fork_nospec_begin
vmlinux 0xffffffff98fc51dd
Build id: 3.10.0-1160.83.1.el7.x86_64
0x7
8
? insert_kthread_work
vmlinux 0xffffffff988cb440
Build id: 3.10.0-1160.83.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ksoftirqd/1:14]
Modules linked in: tcp_lp rfcomm xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set bridge stp ebtable_filter llc ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport bnep vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ppdev crc32_pclmul snd_ens1371 ghash_clmulni_intel snd_rawmidi btusb snd_ac97_codec btrtl aesni_intel btbcm btintel ac97_bus bluetooth lrw snd_seq vmw_balloon gf128mul
glue_helper ablk_helper snd_seq_device cryptd snd_pcm joydev pcspkr snd_timer snd rfkill soundcore vmw_vmci sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ata_piix ttm e1000 libata drm mptspi crct10dif_pclmul nfit crct10dif_common scsi_transport_spi crc32c_intel mptscsih mptbase libnvdimm serio_raw drm_panel_orientation_quirks fuse
CPU: 1 PID: 14 Comm: ksoftirqd/1 Kdump: loaded Not tainted 3.10.0-1160.83.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff94d4f6248000 ti: ffff94d4f6250000 task.ti: ffff94d4f6250000
RIP: 0010:[<ffffffff988a957d>]  [<ffffffff988a957d>] __do_softirq+0xed/0x290
RSP: 0000:ffff94d4f6253e18  EFLAGS: 00000206
RAX: ffffffff988f04f0 RBX: ffffffff98e7fffd RCX: 0000000000000100
RDX: ffff94d4f9640000 RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffff94d4f6253e78 R08: 0000000000000000 R09: 0000000000000001
R10: 0000000000000008 R11: 0000000000000000 R12: ffffffff994070d8
R13: ffff94d4f46ccb70 R14: 0000000000000000 R15: ffffffff994070d8
FS:  0000000000000000(0000) GS:ffff94d4f9640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000108004012000 CR3: 000000008d460000 CR4: 0000000000360fe0
Call Trace:
[<ffffffff988a9758>] run_ksoftirqd+0x38/0x60
[<ffffffff988d42ec>] smpboot_thread_fn+0x14c/0x1b0
[<ffffffff988d41a0>] ? lg_double_unlock+0x40/0x40
[<ffffffff988cb511>] kthread+0xd1/0xe0
[<ffffffff988cb440>] ? insert_kthread_work+0x40/0x40
[<ffffffff98fc51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff988cb440>] ? insert_kthread_work+0x40/0x40
Code: 70 40 99 49 c1 fd 03 8b 80 44 c0 ff ff 44 89 eb 89 45 d4 44 89 e8 48 89 45 b8 65 ff 04 85 c8 5b 01 00 0f 1f 44 00 00 49 8b 04 24 <4c> 89 e7 e8 bb 46 30 00 0f 1f 44 00 00 48 8b 45 c8 8b 4d d4 3b 
Complete report #664072
Frame # Function Binary Source or offset Line
1
run_timer_softirq
vmlinux 0xffffffffa08b317f
Build id: 3.10.0-1160.88.1.el7.x86_64
0x1df
2
__do_softirq
vmlinux 0xffffffffa08a9595
Build id: 3.10.0-1160.88.1.el7.x86_64
0xf5
3
run_ksoftirqd
vmlinux 0xffffffffa08a9768
Build id: 3.10.0-1160.88.1.el7.x86_64
0x38
4
smpboot_thread_fn
vmlinux 0xffffffffa08d43fc
Build id: 3.10.0-1160.88.1.el7.x86_64
0x14c
5
? lg_double_unlock
vmlinux 0xffffffffa08d42b0
Build id: 3.10.0-1160.88.1.el7.x86_64
0x40
6
kthread
vmlinux 0xffffffffa08cb621
Build id: 3.10.0-1160.88.1.el7.x86_64
0xd1
7
? insert_kthread_work
vmlinux 0xffffffffa08cb550
Build id: 3.10.0-1160.88.1.el7.x86_64
0x40
8
ret_from_fork_nospec_begin
vmlinux 0xffffffffa0fc51dd
Build id: 3.10.0-1160.88.1.el7.x86_64
0x7
9
? insert_kthread_work
vmlinux 0xffffffffa08cb550
Build id: 3.10.0-1160.88.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [ksoftirqd/1:14]
rr640l:[0 1 2] config_device path_id :1 
rr640l:[0 1 3] start device soft reset.
Modules linked in: dm_crypt drbg ansi_cprng xts gf128mul tun nfnetlink_queue nfnetlink_log bluetooth rfkill xt_nat iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c iptable_filter dm_mirror dm_region_hash dm_log dm_mod amd64_edac_mod edac_mce_amd kvm_amd kvm dcdbas irqbypass pcspkr sg k10temp i2c_nforce2 acpi_cpufreq ip_tables ext4 mbcache jbd2 raid1 sd_mod crc_t10dif crct10dif_generic crct10dif_common sr_mod cdrom radeon i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_generic pata_acpi drm tg3 serio_raw sata_nv libata ptp pps_core rr640l(POE) drm_panel_orientation_quirks
CPU: 1 PID: 14 Comm: ksoftirqd/1 Kdump: loaded Tainted: P           OE  ------------   3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge T105 /0P957K, BIOS 1.3.2 08/20/2008
task: ffff88b57aeea100 ti: ffff88b57aef8000 task.ti: ffff88b57aef8000
RIP: 0010:[<ffffffffa08b317f>]  [<ffffffffa08b317f>] run_timer_softirq+0x1df/0x340
RSP: 0018:ffff88b57aefbda0  EFLAGS: 00000297
RAX: 0000000140ba787d RBX: ffff88b62fc93940 RCX: ffff88b57aefbdc8
RDX: 0000000140ba787c RSI: ffff88b57aefbdc8 RDI: ffff88b62fc93940
RBP: ffff88b57aefbe08 R08: ffff88b5ba761540 R09: 0000000180190000
R10: 00000000ba761901 R11: ffff88b5ba761540 R12: ffff88b4cf230f08
R13: 0000000000000246 R14: 000000017aefbd48 R15: 0000000000000000
FS:  00007f7102872880(0000) GS:ffff88b62fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000056476574fa60 CR3: 0000000224b96000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa08a9595>] __do_softirq+0xf5/0x290
[<ffffffffa08a9768>] run_ksoftirqd+0x38/0x60
[<ffffffffa08d43fc>] smpboot_thread_fn+0x14c/0x1b0
[<ffffffffa08d42b0>] ? lg_double_unlock+0x40/0x40
[<ffffffffa08cb621>] kthread+0xd1/0xe0
[<ffffffffa08cb550>] ? insert_kthread_work+0x40/0x40
[<ffffffffa0fc51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffa08cb550>] ? insert_kthread_work+0x40/0x40
Code: 20 00 00 00 0f 85 ea fe ff ff 48 8b 05 9b 9e cb 00 48 89 43 10 48 c7 43 08 00 00 00 00 48 89 df c6 07 00 66 66 66 90 fb 66 66 90 <66> 66 90 48 8b 45 d0 65 48 33 04 25 28 00 00 00 0f 85 40 01 00 
Complete report #658527
Frame # Function Binary Source or offset Line
1
net_rx_action
vmlinux 0xffffffff96bf152f
Build id: 3.10.0-862.14.4.el7.x86_64
0x26f
2
__do_softirq
vmlinux 0xffffffff9669dba5
Build id: 3.10.0-862.14.4.el7.x86_64
0xf5
3
run_ksoftirqd
vmlinux 0xffffffff9669dd68
Build id: 3.10.0-862.14.4.el7.x86_64
0x38
4
smpboot_thread_fn
vmlinux 0xffffffff966c6774
Build id: 3.10.0-862.14.4.el7.x86_64
0x144
5
? lg_double_unlock
vmlinux 0xffffffff966c6630
Build id: 3.10.0-862.14.4.el7.x86_64
0x40
6
kthread
vmlinux 0xffffffff966bdf21
Build id: 3.10.0-862.14.4.el7.x86_64
0xd1
7
? insert_kthread_work
vmlinux 0xffffffff966bde50
Build id: 3.10.0-862.14.4.el7.x86_64
0x40
8
ret_from_fork_nospec_begin
vmlinux 0xffffffff96d255f7
Build id: 3.10.0-862.14.4.el7.x86_64
0x21
9
? insert_kthread_work
vmlinux 0xffffffff966bde50
Build id: 3.10.0-862.14.4.el7.x86_64
0x40
WARNING: CPU: 14 PID: 0 at net/core/dev.c:2573 skb_warn_bad_offload+0xcd/0xda
vmxnet3: caps=(0x000000018011cba9, 0x00003001801bcba9) len=4396 data_len=2888 gso_size=1448 gso_type=1 ip_summed=1
vmsecmod(OE)
[<ffffffff96bf152f>] net_rx_action+0x26f/0x390
[<ffffffff9669dba5>] __do_softirq+0xf5/0x280
Modules linked in: vmsec_nfq(OE) tcp_lp
fuse
binfmt_misc
nfnetlink_queue
udp_diag
[<ffffffff9669dd68>] run_ksoftirqd+0x38/0x50
nf_conntrack_ipv6 nf_defrag_ipv6
[<ffffffff966c6774>] smpboot_thread_fn+0x144/0x1a0
tcp_diag inet_diag
nf_conntrack_ipv4 nf_defrag_ipv4
[<ffffffff966c6630>] ? lg_double_unlock+0x40/0x40
fuse nfnetlink_queue
nf_conntrack vmw_vsock_vmci_transport
[<ffffffff966bdf21>] kthread+0xd1/0xe0
nfnetlink
vsock sunrpc
vmsecmod(OE)
sb_edac
nf_conntrack_ipv6
[<ffffffff966bde50>] ? insert_kthread_work+0x40/0x40
iosf_mbi
nfnetlink
crc32_pclmul
nf_defrag_ipv6
[<ffffffff96d255f7>] ret_from_fork_nospec_begin+0x21/0x21
ghash_clmulni_intel aesni_intel
[<ffffffff966bde50>] ? insert_kthread_work+0x40/0x40
nf_conntrack_ipv4 nf_defrag_ipv4
vmsecmod(OE)
lrw
Complete report #372176
Frame # Function Binary Source or offset Line
1
run_timer_softirq
vmlinux 0xffffffffa3cab5fe
Build id: 3.10.0-957.27.2.el7.x86_64
0x23e
2
__do_softirq
vmlinux 0xffffffffa3ca2155
Build id: 3.10.0-957.27.2.el7.x86_64
0xf5
3
run_ksoftirqd
vmlinux 0xffffffffa3ca2318
Build id: 3.10.0-957.27.2.el7.x86_64
0x38
4
smpboot_thread_fn
vmlinux 0xffffffffa3ccb814
Build id: 3.10.0-957.27.2.el7.x86_64
0x144
5
? lg_double_unlock
vmlinux 0xffffffffa3ccb6d0
Build id: 3.10.0-957.27.2.el7.x86_64
0x40
6
kthread
vmlinux 0xffffffffa3cc2e81
Build id: 3.10.0-957.27.2.el7.x86_64
0xd1
7
? insert_kthread_work
vmlinux 0xffffffffa3cc2db0
Build id: 3.10.0-957.27.2.el7.x86_64
0x40
8
ret_from_fork_nospec_begin
vmlinux 0xffffffffa4376c37
Build id: 3.10.0-957.27.2.el7.x86_64
0x21
9
? insert_kthread_work
vmlinux 0xffffffffa3cc2db0
Build id: 3.10.0-957.27.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 32s! [ksoftirqd/0:3]
Modules linked in: ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 ip6table_filter nf_log_ipv4 nf_log_common nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon pcspkr joydev parport_pc sg parport i2c_piix4 vmw_vmci tcp_htcp ip_tables xfs libcrc32c vmwgfx drm_kms_helper sd_mod sr_mod syscopyarea cdrom sysfillrect crc_t10dif sysimgblt fb_sys_fops crct10dif_generic ata_generic pata_acpi
crct10dif_pclmul ttm ata_piix crct10dif_common drm crc32c_intel serio_raw e1000 vmw_pvscsi libata drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: talpa_syscallhook]
CPU: 0 PID: 3 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W  OEL ------------   3.10.0-957.27.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffffa0faba312080 ti: ffffa0faba338000 task.ti: ffffa0faba338000
RIP: 0010:[<ffffffffa3cab5fe>]  [<ffffffffa3cab5fe>] run_timer_softirq+0x23e/0x300
RSP: 0000:ffffa0faba33bda0  EFLAGS: 00000246
RAX: dead000000000200 RBX: ffffa0faba33bd58 RCX: ffffffffa49182d0
RDX: ffffa0faba33bdc8 RSI: ffffa0faba33bdc8 RDI: ffffffffa4ce3c00
RBP: ffffa0faba33be08 R08: ffffffffa4ce4028 R09: ffffffffa4ce3c00
R10: 0000000000000001 R11: ffffef6c01abd800 R12: ffffa0fd6aec0548
R13: ffffffffa3cb8e5c R14: ffffa0faba33bd48 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffa0fd7fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fabfa7271f0 CR3: 00000000367c6000 CR4: 00000000000407f0
Call Trace:
[<ffffffffa3ca2155>] __do_softirq+0xf5/0x280
[<ffffffffa3ca2318>] run_ksoftirqd+0x38/0x50
[<ffffffffa3ccb814>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa3ccb6d0>] ? lg_double_unlock+0x40/0x40
[<ffffffffa3cc2e81>] kthread+0xd1/0xe0
[<ffffffffa3cc2db0>] ? insert_kthread_work+0x40/0x40
[<ffffffffa4376c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa3cc2db0>] ? insert_kthread_work+0x40/0x40
Code: 24 08 49 83 c4 10 4c 89 fe e8 bf d0 2d 00 49 8b 04 24 48 85 c0 75 e6 e9 ef fe ff ff 90 c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 8b 55 b8 4c 89 ff 4c 89 ee e8 63 db ff ff 48 89 df e8 6b 10 
Complete report #327315
Frame # Function Binary Source or offset Line
1
run_timer_softirq
vmlinux 0xffffffff8109981e
Build id: 3.10.0-693.el7.x86_64
0x21e
2
__do_softirq
vmlinux 0xffffffff81090b3f
Build id: 3.10.0-693.el7.x86_64
0xef
3
run_ksoftirqd
vmlinux 0xffffffff81090d08
Build id: 3.10.0-693.el7.x86_64
0x38
4
smpboot_thread_fn
vmlinux 0xffffffff810b909f
Build id: 3.10.0-693.el7.x86_64
0x12f
5
? lg_double_unlock
vmlinux 0xffffffff810b8f70
Build id: 3.10.0-693.el7.x86_64
0x40
6
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
7
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
8
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
9
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 33s! [ksoftirqd/1:13]
Modules linked in: vmw_vsock_vmci_transport vsock xfs libcrc32c sb_edac edac_core coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon sg pcspkr joydev i2c_piix4 shpchp vmw_vmci parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea crct10dif_pclmul sysfillrect sysimgblt crct10dif_common fb_sys_fops mptspi crc32c_intel ttm serio_raw scsi_transport_spi mptscsih ata_piix drm vmxnet3 mptbase libata i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 13 Comm: ksoftirqd/1 Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff88017d606eb0 ti: ffff88017d684000 task.ti: ffff88017d684000
RIP: 0010:[<ffffffff8109981e>]  [<ffffffff8109981e>] run_timer_softirq+0x21e/0x310
RSP: 0000:ffff88017d687da0  EFLAGS: 00000246
RAX: dead000000000200 RBX: ffff88017d687d58 RCX: ffffffff81adf5d0
RDX: ffff88017d687dc8 RSI: ffff88017d687dc8 RDI: ffff88017d6a0000
RBP: ffff88017d687e08 R08: ffff88017d6a0428 R09: ffff88017d6a0000
R10: 0000000006c00000 R11: 0000000000000400 R12: ffffffff810cd794
R13: ffffffff810a66fc R14: ffff88017d687d48 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8801bfc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fc6728bd448 CR3: 00000001a2fdb000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
 ffff88017d6a1c28 ffff88017d6a1828 ffff88017d6a1428 ffff88017d6a1028
 ffffffff81adf440 ffff88017d687dc8 ffff88017d687dc8 00000000b19eee2e
 0000000000000001 0000000000000001 ffffffff81a27440 ffffffff819eb0c8
Call Trace:
 [<ffffffff81090b3f>] __do_softirq+0xef/0x280
 [<ffffffff81090d08>] run_ksoftirqd+0x38/0x50
 [<ffffffff810b909f>] smpboot_thread_fn+0x12f/0x180
 [<ffffffff810b8f70>] ? lg_double_unlock+0x40/0x40
 [<ffffffff810b098f>] kthread+0xcf/0xe0
 [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
 [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
 [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 28 00 00 00 0f 85 08 01 00 00 48 83 c4 40 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 40 00 c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 8b 55 b8 4c 89 ff 4c 89 ee e8 b3 da ff ff 48 89 df e8 fb 21 
Complete report #256366
Frame # Function Binary Source or offset Line
1
__do_softirq
vmlinux 0xffffffff81090ae7
Build id: 3.10.0-693.el7.x86_64
0x97
2
run_ksoftirqd
vmlinux 0xffffffff81090d08
Build id: 3.10.0-693.el7.x86_64
0x38
3
smpboot_thread_fn
vmlinux 0xffffffff810b909f
Build id: 3.10.0-693.el7.x86_64
0x12f
4
? lg_double_unlock
vmlinux 0xffffffff810b8f70
Build id: 3.10.0-693.el7.x86_64
0x40
5
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
6
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
7
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
8
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 28s! [ksoftirqd/2:18]
Modules linked in: nfsv3 nfs fscache vmw_vsock_vmci_transport vsock coretemp ppdev vmw_balloon pcspkr joydev sg parport_pc parport shpchp vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif crct10dif_generic cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi scsi_transport_spi mptscsih ata_piix crc32c_intel libata serio_raw mptbase vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 18 Comm: ksoftirqd/2 Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff88017a2dbf40 ti: ffff88017a2f8000 task.ti: ffff88017a2f8000
RIP: 0010:[<ffffffff81090ae7>]  [<ffffffff81090ae7>] __do_softirq+0x97/0x280
RSP: 0000:ffff88017a2fbe18  EFLAGS: 00000206
RAX: ffff88017a2fbfd8 RBX: 0000000000000001 RCX: 0000000000000002
RDX: 000000018eca0b17 RSI: ffff88017a2dbf40 RDI: ffff88017a2dbf40
RBP: ffff88017a2fbe78 R08: ffff88017a2f8000 R09: 0000000000000000
R10: 000000000000bca7 R11: 0000000000000001 R12: ffff88043fd16cc0
R13: ffff88043fd16d40 R14: ffff88017a2dbfa8 R15: ffff88043fd16d40
FS:  0000000000000000(0000) GS:ffff88043fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f95995613f8 CR3: 000000042b3d3000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff88017a2fbfd8 0000000a04208040 000000018eca0b19 ffff88017a2dbf40
ffff88017a2fbfd8 ffff88017a2fbfd8 ffff880100000002 ffff88017a2dbf40
ffff88017fc02950 ffffffff81a27440 0000000000000000 0000000000000000
Call Trace:
[<ffffffff81090d08>] run_ksoftirqd+0x38/0x50
[<ffffffff810b909f>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f70>] ? lg_double_unlock+0x40/0x40
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 45 ac 0a 00 00 00 89 4d d0 48 89 45 c0 48 89 45 c8 48 89 45 a0 0f 1f 80 00 00 00 00 65 c7 05 5d 35 f8 7e 00 00 00 00 fb 66 66 90 <66> 66 90 49 c7 c6 c0 b0 9e 81 eb 0e 0f 1f 44 00 00 49 83 c6 08 
Complete report #226600
Frame # Function Binary Source or offset Line
1
dump_stack
vmlinux 0xffffffff816a3d91
Build id: 3.10.0-693.el7.x86_64
0x19
2
__warn
vmlinux 0xffffffff810879c8
Build id: 3.10.0-693.el7.x86_64
0xd8
3
warn_slowpath_null
vmlinux 0xffffffff81087b0d
Build id: 3.10.0-693.el7.x86_64
0x1d
4
net_rx_action
vmlinux 0xffffffff81587b0e
Build id: 3.10.0-693.el7.x86_64
0x2de
5
__do_softirq
vmlinux 0xffffffff81090b3f
Build id: 3.10.0-693.el7.x86_64
0xef
6
run_ksoftirqd
vmlinux 0xffffffff81090d08
Build id: 3.10.0-693.el7.x86_64
0x38
7
smpboot_thread_fn
vmlinux 0xffffffff810b909f
Build id: 3.10.0-693.el7.x86_64
0x12f
8
? lg_double_unlock
vmlinux 0xffffffff810b8f70
Build id: 3.10.0-693.el7.x86_64
0x40
9
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
10
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
11
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
12
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
WARNING: CPU: 2 PID: 18 at net/core/dev.c:5000 net_rx_action+0x2de/0x380
Modules linked in: tcp_lp cmac rfcomm fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ipmi_devintf ipmi_msghandler drbg ansi_cprng ccm arc4 ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep intel_powerclamp coretemp intel_rapl ath10k_pci iosf_mbi ath10k_core snd_hda_codec_hdmi kvm_intel kvm ath snd_hda_codec_realtek snd_hda_codec_generic mac80211 irqbypass crc32_pclmul ghash_clmulni_intel snd_hda_intel
snd_hda_codec aesni_intel msi_wmi sparse_keymap lrw iTCO_wdt gf128mul btusb iTCO_vendor_support glue_helper ablk_helper snd_hda_core cryptd btrtl btbcm uvcvideo snd_hwdep btintel snd_seq bluetooth videobuf2_vmalloc cfg80211 videobuf2_memops snd_seq_device videobuf2_core snd_pcm videodev snd_timer pcspkr joydev snd sg mei_me mei rfkill i2c_i801 soundcore shpchp acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c rtsx_usb_sdmmc mmc_core rtsx_usb sd_mod crc_t10dif crct10dif_generic nouveau i915 ttm i2c_algo_bit drm_kms_helper ahci libahci syscopyarea sysfillrect sysimgblt libata fb_sys_fops mxm_wmi drm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw alx mdio i2c_hid wmi video i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 18 Comm: ksoftirqd/2 Not tainted 3.10.0-693.el7.x86_64 #1
Hardware name: Micro-Star International Co., Ltd. GS63 7RD/MS-16K4, BIOS E16K4IMS.10C 06/22/2017
0000000000000000 000000007566322c ffff88017cfc3d38 ffffffff816a3d91
ffff88017cfc3d78 ffffffff810879c8 0000138800000001 ffff88028ea97c80
0000000000000000 0000000000000040 0000000000000051 0000000000000001
Call Trace:
[<ffffffff816a3d91>] dump_stack+0x19/0x1b
[<ffffffff810879c8>] __warn+0xd8/0x100
[<ffffffff81087b0d>] warn_slowpath_null+0x1d/0x20
[<ffffffff81587b0e>] net_rx_action+0x2de/0x380
[<ffffffff81090b3f>] __do_softirq+0xef/0x280
[<ffffffff81090d08>] run_ksoftirqd+0x38/0x50
[<ffffffff810b909f>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f70>] ? lg_double_unlock+0x40/0x40
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Complete report #225385
Frame # Function Binary Source or offset Line
1
run_timer_softirq
vmlinux 0xffffffffa60ae39e
Build id: 3.10.0-1160.36.2.el7.x86_64
0x24e
2
__do_softirq
vmlinux 0xffffffffa60a4bd5
Build id: 3.10.0-1160.36.2.el7.x86_64
0xf5
3
run_ksoftirqd
vmlinux 0xffffffffa60a4d98
Build id: 3.10.0-1160.36.2.el7.x86_64
0x38
4
smpboot_thread_fn
vmlinux 0xffffffffa60ce894
Build id: 3.10.0-1160.36.2.el7.x86_64
0x144
5
? lg_double_unlock
vmlinux 0xffffffffa60ce750
Build id: 3.10.0-1160.36.2.el7.x86_64
0x40
6
kthread
vmlinux 0xffffffffa60c5e41
Build id: 3.10.0-1160.36.2.el7.x86_64
0xd1
7
? insert_kthread_work
vmlinux 0xffffffffa60c5d70
Build id: 3.10.0-1160.36.2.el7.x86_64
0x40
8
ret_from_fork_nospec_begin
vmlinux 0xffffffffa6795ddd
Build id: 3.10.0-1160.36.2.el7.x86_64
0x7
9
? insert_kthread_work
vmlinux 0xffffffffa60c5d70
Build id: 3.10.0-1160.36.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 17408s! [ksoftirqd/5:34]
Modules linked in: nf_tables vxlan ip6_udp_tunnel udp_tunnel xt_statistic ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6t_MASQUERADE nf_nat_masquerade_ipv6 xt_comment xt_mark veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc vfat fat joydev crc32_pclmul ghash_clmulni_intel aesni_intel sg hv_utils
lrw gf128mul ptp glue_helper pps_core ablk_helper hv_balloon pcspkr cryptd ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic hv_storvsc serio_raw scsi_transport_fc hv_netvsc scsi_tgt hyperv_fb hyperv_keyboard hid_hyperv hv_vmbus crct10dif_pclmul crct10dif_common crc32c_intel dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 34 Comm: ksoftirqd/5 Kdump: loaded Tainted: G        W      ------------ T 3.10.0-1160.36.2.el7.x86_64 #1
Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.0 11/01/2019
task: ffffa01a75c3d280 ti: ffffa01a75c5c000 task.ti: ffffa01a75c5c000
RIP: 0010:[<ffffffffa60ae39e>]  [<ffffffffa60ae39e>] run_timer_softirq+0x24e/0x340
RSP: 0018:ffffa01a75c5fda0  EFLAGS: 00000246
RAX: dead000000000200 RBX: ffffa01a75c5fd58 RCX: ffffa019f7ae1ab0
RDX: ffffa01ce06f1270 RSI: ffffa01a75c5fdc8 RDI: ffffa01d07d53940
RBP: ffffa01a75c5fe08 R08: ffffa01ce06f1270 R09: ffffa01d07d53940
R10: ffffa01c43c34808 R11: ffffa01a75dfd000 R12: ffffffffa663ae68
R13: ffffffffa60bbdbc R14: ffffa01a75c5fd48 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffa01d07d40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001bca018 CR3: 00000003dd848000 CR4: 00000000003606e0
Call Trace:
[<ffffffffa60a4bd5>] __do_softirq+0xf5/0x280
[<ffffffffa60a4d98>] run_ksoftirqd+0x38/0x50
[<ffffffffa60ce894>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa60ce750>] ? lg_double_unlock+0x40/0x40
[<ffffffffa60c5e41>] kthread+0xd1/0xe0
[<ffffffffa60c5d70>] ? insert_kthread_work+0x40/0x40
[<ffffffffa6795ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffa60c5d70>] ? insert_kthread_work+0x40/0x40
Code: 24 08 49 83 c4 10 4c 89 fe e8 7f 9b 2e 00 49 8b 04 24 48 85 c0 75 e6 e9 ef fe ff ff 90 c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 8b 55 b8 4c 89 ff 4c 89 ee e8 b3 d9 ff ff 48 89 df e8 eb d5 
Complete report #199255
Frame # Function Binary Source or offset Line
1
run_timer_softirq
vmlinux 0xffffffffa88ae2fe
Build id: 3.10.0-1160.25.1.el7.x86_64
0x24e
2
__do_softirq
vmlinux 0xffffffffa88a4b35
Build id: 3.10.0-1160.25.1.el7.x86_64
0xf5
3
run_ksoftirqd
vmlinux 0xffffffffa88a4cf8
Build id: 3.10.0-1160.25.1.el7.x86_64
0x38
4
smpboot_thread_fn
vmlinux 0xffffffffa88ce7f4
Build id: 3.10.0-1160.25.1.el7.x86_64
0x144
5
? lg_double_unlock
vmlinux 0xffffffffa88ce6b0
Build id: 3.10.0-1160.25.1.el7.x86_64
0x40
6
kthread
vmlinux 0xffffffffa88c5da1
Build id: 3.10.0-1160.25.1.el7.x86_64
0xd1
7
? insert_kthread_work
vmlinux 0xffffffffa88c5cd0
Build id: 3.10.0-1160.25.1.el7.x86_64
0x40
8
ret_from_fork_nospec_begin
vmlinux 0xffffffffa8f95df7
Build id: 3.10.0-1160.25.1.el7.x86_64
0x21
9
? insert_kthread_work
vmlinux 0xffffffffa88c5cd0
Build id: 3.10.0-1160.25.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 30s! [ksoftirqd/1:14]
Modules linked in: event_collector_2_3_560374(OE) tcp_lp nls_utf8 isofs rfcomm xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter tun bnep vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw vmw_balloon gf128mul glue_helper ablk_helper cryptd btusb btrtl btbcm btintel uvcvideo bluetooth joydev pcspkr videobuf2_vmalloc
videobuf2_memops videobuf2_core videodev sg rfkill vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmwgfx drm_kms_helper e1000 syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi mptscsih mptbase drm ata_piix nfit drm_panel_orientation_quirks libata libnvdimm dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: event_collector_2_3_560374]
CPU: 1 PID: 14 Comm: ksoftirqd/1 Kdump: loaded Tainted: G           OE  ------------   3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff99d130bb8000 ti: ffff99d130bb4000 task.ti: ffff99d130bb4000
RIP: 0010:[<ffffffffa88ae2fe>]  [<ffffffffa88ae2fe>] run_timer_softirq+0x24e/0x340
RSP: 0018:ffff99d130bb7da0  EFLAGS: 00000246
RAX: dead000000000200 RBX: ffff99d130bb7d58 RCX: ffff99d1377e6348
RDX: ffff99d130bb7dc8 RSI: ffff99d130bb7dc8 RDI: ffff99d1f5653940
RBP: ffff99d130bb7e08 R08: ffff99d1f5653b68 R09: ffff99d1f5653940
R10: 0000000000000000 R11: 0000000000000293 R12: ffffffffa8d10369
R13: ffff99d130bb7d48 R14: ffff99d1f6cf18c8 R15: ffffffffa8d103c0
FS:  0000000000000000(0000) GS:ffff99d1f5640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1b66644000 CR3: 00000001e9b18000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa88a4b35>] __do_softirq+0xf5/0x280
[<ffffffffa88a4cf8>] run_ksoftirqd+0x38/0x50
[<ffffffffa88ce7f4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa88ce6b0>] ? lg_double_unlock+0x40/0x40
[<ffffffffa88c5da1>] kthread+0xd1/0xe0
[<ffffffffa88c5cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffffa8f95df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa88c5cd0>] ? insert_kthread_work+0x40/0x40
Code: 24 08 49 83 c4 10 4c 89 fe e8 bf 95 2e 00 49 8b 04 24 48 85 c0 75 e6 e9 ef fe ff ff 90 c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 8b 55 b8 4c 89 ff 4c 89 ee e8 b3 d9 ff ff 48 89 df e8 3b d4 
Complete report #194292
Frame # Function Binary Source or offset Line
1
dump_stack
vmlinux 0xffffffff903835a9
Build id: 3.10.0-1160.31.1.el7.x86_64
0x19
2
__warn
vmlinux 0xffffffff8fc9b248
Build id: 3.10.0-1160.31.1.el7.x86_64
0xd8
3
warn_slowpath_null
vmlinux 0xffffffff8fc9b38d
Build id: 3.10.0-1160.31.1.el7.x86_64
0x1d
4
net_rx_action
vmlinux 0xffffffff902572a3
Build id: 3.10.0-1160.31.1.el7.x86_64
0x343
5
__do_softirq
vmlinux 0xffffffff8fca4bc5
Build id: 3.10.0-1160.31.1.el7.x86_64
0xf5
6
run_ksoftirqd
vmlinux 0xffffffff8fca4d88
Build id: 3.10.0-1160.31.1.el7.x86_64
0x38
7
smpboot_thread_fn
vmlinux 0xffffffff8fcce884
Build id: 3.10.0-1160.31.1.el7.x86_64
0x144
8
? lg_double_unlock
vmlinux 0xffffffff8fcce740
Build id: 3.10.0-1160.31.1.el7.x86_64
0x40
9
kthread
vmlinux 0xffffffff8fcc5e31
Build id: 3.10.0-1160.31.1.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffff8fcc5d60
Build id: 3.10.0-1160.31.1.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffff90395df7
Build id: 3.10.0-1160.31.1.el7.x86_64
0x21
12
? insert_kthread_work
vmlinux 0xffffffff8fcc5d60
Build id: 3.10.0-1160.31.1.el7.x86_64
0x40
WARNING: CPU: 0 PID: 6 at net/core/dev.c:5464 net_rx_action+0x343/0x390
Modules linked in: tcp_lp rfcomm cmac xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 devlink tun drbg ansi_cprng ccm ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter vboxnetadp(OE) vboxnetflt(OE) ip6_tables iptable_filter arc4 vboxdrv(OE) bnep sunrpc snd_hda_codec_hdmi vfat fat dell_led snd_soc_skl snd_soc_skl_ipc snd_hda_ext_core snd_soc_sst_dsp snd_soc_sst_ipc intel_pmc_core snd_soc_acpi snd_hda_codec_realtek snd_soc_core ath10k_pci snd_hda_codec_generic
intel_powerclamp coretemp squashfs hid_multitouch ath10k_core snd_compress intel_rapl kvm_intel snd_hda_intel loop iTCO_wdt iTCO_vendor_support kvm ath snd_hda_codec mac80211 snd_hda_core dell_laptop snd_hwdep irqbypass snd_seq crc32_pclmul btusb ghash_clmulni_intel btrtl snd_seq_device btbcm btintel snd_pcm bluetooth aesni_intel uvcvideo cfg80211 lrw videobuf2_vmalloc gf128mul videobuf2_memops glue_helper videobuf2_core ablk_helper cryptd videodev joydev dell_wmi snd_timer dell_smbios snd dcdbas dell_wmi_descriptor rfkill soundcore i2c_i801 pcspkr sg mei_me mei i2c_designware_platform i2c_designware_core idma64 virt_dma pinctrl_sunrisepoint pinctrl_intel intel_hid sparse_keymap int3400_thermal acpi_pad ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic rtsx_usb_sdmmc mmc_core
rtsx_usb i915 nouveau ttm i2c_algo_bit iosf_mbi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci drm libahci mxm_wmi libata crct10dif_pclmul crct10dif_common crc32c_intel r8169 serio_raw drm_panel_orientation_quirks wmi video i2c_hid dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W  OE  ------------   3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: Dell Inc. Inspiron 7572/082HPJ, BIOS 1.2.1 05/31/2019
Call Trace:
[<ffffffff903835a9>] dump_stack+0x19/0x1b
[<ffffffff8fc9b248>] __warn+0xd8/0x100
[<ffffffff8fc9b38d>] warn_slowpath_null+0x1d/0x20
[<ffffffff902572a3>] net_rx_action+0x343/0x390
[<ffffffff8fca4bc5>] __do_softirq+0xf5/0x280
[<ffffffff8fca4d88>] run_ksoftirqd+0x38/0x50
[<ffffffff8fcce884>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff8fcce740>] ? lg_double_unlock+0x40/0x40
[<ffffffff8fcc5e31>] kthread+0xd1/0xe0
[<ffffffff8fcc5d60>] ? insert_kthread_work+0x40/0x40
[<ffffffff90395df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8fcc5d60>] ? insert_kthread_work+0x40/0x40
Complete report #20746
Frame # Function Binary Source or offset Line
1
dump_stack
vmlinux 0xffffffff8ab7ffa5
Build id: 3.10.0-1127.19.1.el7.x86_64
0x19
2
__warn
vmlinux 0xffffffff8a49bd18
Build id: 3.10.0-1127.19.1.el7.x86_64
0xd8
3
warn_slowpath_null
vmlinux 0xffffffff8a49be5d
Build id: 3.10.0-1127.19.1.el7.x86_64
0x1d
4
net_rx_action
vmlinux 0xffffffff8aa54693
Build id: 3.10.0-1127.19.1.el7.x86_64
0x343
5
__do_softirq
vmlinux 0xffffffff8a4a5695
Build id: 3.10.0-1127.19.1.el7.x86_64
0xf5
6
run_ksoftirqd
vmlinux 0xffffffff8a4a5858
Build id: 3.10.0-1127.19.1.el7.x86_64
0x38
7
smpboot_thread_fn
vmlinux 0xffffffff8a4cf0e4
Build id: 3.10.0-1127.19.1.el7.x86_64
0x144
8
? lg_double_unlock
vmlinux 0xffffffff8a4cefa0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
9
kthread
vmlinux 0xffffffff8a4c6691
Build id: 3.10.0-1127.19.1.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffff8a4c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffff8ab92d37
Build id: 3.10.0-1127.19.1.el7.x86_64
0x21
12
? insert_kthread_work
vmlinux 0xffffffff8a4c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
WARNING: CPU: 0 PID: 6 at net/core/dev.c:5464 net_rx_action+0x343/0x390
Modules linked in: hidp uinput tcp_lp rfcomm fuse cmac xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 devlink tun drbg ansi_cprng ccm ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep sunrpc vfat fat arc4 intel_pmc_core intel_powerclamp coretemp intel_rapl snd_soc_skl kvm_intel snd_soc_skl_ipc snd_hda_codec_hdmi snd_hda_ext_core snd_soc_sst_dsp snd_soc_sst_ipc kvm irqbypass crc32_pclmul ghash_clmulni_intel
aesni_intel lrw gf128mul glue_helper dell_led snd_hda_codec_realtek snd_hda_codec_generic snd_soc_acpi snd_soc_core snd_compress snd_hda_intel snd_hda_codec uvcvideo snd_hda_core ath10k_pci snd_hwdep snd_seq ath10k_core snd_seq_device snd_pcm ath snd_timer snd btusb mac80211 soundcore videobuf2_vmalloc iTCO_wdt videobuf2_memops videobuf2_core btrtl iTCO_vendor_support videodev ablk_helper cryptd dell_laptop dell_wmi dell_smbios dcdbas btbcm cfg80211 btintel hid_multitouch i2c_i801 bluetooth intel_hid dell_wmi_descriptor rfkill i2c_designware_platform i2c_designware_core pinctrl_sunrisepoint pinctrl_intel tpm_crb mei_me mei idma64 virt_dma pcspkr joydev sg acpi_pad int3400_thermal sparse_keymap ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic rtsx_usb_sdmmc mmc_core rtsx_usb
i915 nouveau ttm i2c_algo_bit iosf_mbi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mxm_wmi ahci drm libahci libata crct10dif_pclmul crct10dif_common crc32c_intel r8169 serio_raw drm_panel_orientation_quirks wmi video i2c_hid dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 6 Comm: ksoftirqd/0 Tainted: G        W      ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: Dell Inc. Inspiron 7472/082HPJ, BIOS 1.1.6 06/14/2018
Call Trace:
[<ffffffff8ab7ffa5>] dump_stack+0x19/0x1b
[<ffffffff8a49bd18>] __warn+0xd8/0x100
[<ffffffff8a49be5d>] warn_slowpath_null+0x1d/0x20
[<ffffffff8aa54693>] net_rx_action+0x343/0x390
[<ffffffff8a4a5695>] __do_softirq+0xf5/0x280
[<ffffffff8a4a5858>] run_ksoftirqd+0x38/0x50
[<ffffffff8a4cf0e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff8a4cefa0>] ? lg_double_unlock+0x40/0x40
[<ffffffff8a4c6691>] kthread+0xd1/0xe0
[<ffffffff8a4c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff8ab92d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8a4c65c0>] ? insert_kthread_work+0x40/0x40
Complete report #17944
Frame # Function Binary Source or offset Line
1
dump_stack
vmlinux 0xffffffffb9381400
Build id: 3.10.0-1160.6.1.el7.x86_64
0x19
2
__warn
vmlinux 0xffffffffb8c9b228
Build id: 3.10.0-1160.6.1.el7.x86_64
0xd8
3
warn_slowpath_null
vmlinux 0xffffffffb8c9b36d
Build id: 3.10.0-1160.6.1.el7.x86_64
0x1d
4
net_rx_action
vmlinux 0xffffffffb92556f3
Build id: 3.10.0-1160.6.1.el7.x86_64
0x343
5
__do_softirq
vmlinux 0xffffffffb8ca4b95
Build id: 3.10.0-1160.6.1.el7.x86_64
0xf5
6
run_ksoftirqd
vmlinux 0xffffffffb8ca4d58
Build id: 3.10.0-1160.6.1.el7.x86_64
0x38
7
smpboot_thread_fn
vmlinux 0xffffffffb8cce674
Build id: 3.10.0-1160.6.1.el7.x86_64
0x144
8
? lg_double_unlock
vmlinux 0xffffffffb8cce530
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
9
kthread
vmlinux 0xffffffffb8cc5c21
Build id: 3.10.0-1160.6.1.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffffb8cc5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffffb9394ddd
Build id: 3.10.0-1160.6.1.el7.x86_64
0x7
12
? insert_kthread_work
vmlinux 0xffffffffb8cc5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
WARNING: CPU: 1 PID: 14 at net/core/dev.c:5464 net_rx_action+0x343/0x390
Modules linked in: cdc_acm cdc_ether usbnet r8152 mii nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink vmnet(OE) ppdev parport_pc parport fuse vmw_vsock_vmci_transport vsock vmw_vmci vmmon(OE) overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) sunrpc snd_usb_audio snd_usbmidi_lib snd_rawmidi ext4 mbcache
jbd2 joydev hid_multitouch iTCO_wdt iTCO_vendor_support mei_wdt dell_rbtn ath10k_pci vfat ath10k_core fat snd_hda_codec_hdmi intel_pmc_core dell_laptop ath mac80211 intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel dell_led kvm snd_hda_codec_realtek snd_hda_codec_generic irqbypass dell_wmi snd_hda_intel cfg80211 dell_smbios pcspkr dcdbas intel_wmi_thunderbolt dell_wmi_descriptor snd_hda_codec snd_hda_core snd_seq i2c_i801 rfkill snd_hwdep rtsx_pci_ms snd_seq_device memstick snd_pcm mei_me mei snd_timer snd i2c_designware_platform idma64 i2c_designware_core virt_dma soundcore dell_smo8800 pinctrl_cannonlake pinctrl_intel acpi_pad tpm_crb intel_hid int3400_thermal sparse_keymap binfmt_misc ip_tables xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) mxm_wmi rtsx_pci_sdmmc
mmc_core drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm e1000e crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel serio_raw nvme ptp rtsx_pci drm_panel_orientation_quirks pps_core ipmi_devintf nvme_core ipmi_msghandler wmi video i2c_hid dm_mirror dm_region_hash dm_log authenc cmac zlib wp512 twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common tea sha512_ssse3 sha512_generic serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic seed salsa20_generic salsa20_x86_64 rmd320 rmd256 rmd160 rmd128 michael_mic md4 khazad ghash_generic gcm fcrypt drbg dm_crypt dm_mod des_generic deflate cts crypto_null crct10dif_generic crct10dif_common crc32c_intel ccm cast6_avx_x86_64 cast6_generic cast_common camellia_generic camellia_aesni_avx2
camellia_aesni_avx_x86_64 ablk_helper cryptd camellia_x86_64 xts lrw gf128mul glue_helper blowfish_generic blowfish_x86_64 blowfish_common arc4 ansi_cprng [last unloaded: tcrypt]
CPU: 1 PID: 14 Comm: ksoftirqd/1 Tainted: P        W  OE  ------------ T 3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: Dell Inc. Precision 7730/05W5TJ, BIOS 1.10.1 07/29/2019
Call Trace:
[<ffffffffb9381400>] dump_stack+0x19/0x1b
[<ffffffffb8c9b228>] __warn+0xd8/0x100
[<ffffffffb8c9b36d>] warn_slowpath_null+0x1d/0x20
[<ffffffffb92556f3>] net_rx_action+0x343/0x390
[<ffffffffb8ca4b95>] __do_softirq+0xf5/0x280
[<ffffffffb8ca4d58>] run_ksoftirqd+0x38/0x50
[<ffffffffb8cce674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffb8cce530>] ? lg_double_unlock+0x40/0x40
[<ffffffffb8cc5c21>] kthread+0xd1/0xe0
[<ffffffffb8cc5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffb9394ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb8cc5b50>] ? insert_kthread_work+0x40/0x40
Complete report #9907
Frame # Function Binary Source or offset Line
1
run_timer_softirq
vmlinux 0xffffffff94caebe2
Build id: 3.10.0-1127.8.2.el7.x86_64
0x1d2
2
__do_softirq
vmlinux 0xffffffff94ca5695
Build id: 3.10.0-1127.8.2.el7.x86_64
0xf5
3
run_ksoftirqd
vmlinux 0xffffffff94ca5858
Build id: 3.10.0-1127.8.2.el7.x86_64
0x38
4
smpboot_thread_fn
vmlinux 0xffffffff94ccf0e4
Build id: 3.10.0-1127.8.2.el7.x86_64
0x144
5
? lg_double_unlock
vmlinux 0xffffffff94ccefa0
Build id: 3.10.0-1127.8.2.el7.x86_64
0x40
6
kthread
vmlinux 0xffffffff94cc6691
Build id: 3.10.0-1127.8.2.el7.x86_64
0xd1
7
? insert_kthread_work
vmlinux 0xffffffff94cc65c0
Build id: 3.10.0-1127.8.2.el7.x86_64
0x40
8
ret_from_fork_nospec_begin
vmlinux 0xffffffff95392d37
Build id: 3.10.0-1127.8.2.el7.x86_64
0x21
9
? insert_kthread_work
vmlinux 0xffffffff94cc65c0
Build id: 3.10.0-1127.8.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 24s! [ksoftirqd/8:49]
Modules linked in: veth dummy vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch xt_nat xt_statistic ipt_REJECT nf_reject_ipv4 xt_NFLOG xt_physdev xt_state nfnetlink_log ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs ip_set_hash_ip xt_set ip_set nf_tables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c
sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci crct10dif_pclmul crct10dif_common crc32c_intel ata_piix libahci drm libata nfit serio_raw vmxnet3 libnvdimm vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 8 PID: 49 Comm: ksoftirqd/8 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1127.8.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9699d2dd0000 ti: ffff9699d2dd8000 task.ti: ffff9699d2dd8000
RIP: 0010:[<ffffffff94caebe2>]  [<ffffffff94caebe2>] run_timer_softirq+0x1d2/0x300
RSP: 0000:ffff9699d2ddbda0  EFLAGS: 00000287
RAX: 00000001426f2720 RBX: ffff9699d2ddbd58 RCX: ffff9699d2ddbdc8
RDX: 00000001426f271f RSI: ffff9699d2ddbdc8 RDI: ffff96b83d813940
RBP: ffff9699d2ddbe08 R08: ffff96b83d814108 R09: ffffffffc084d2a8
R10: ffffc51d3c002c10 R11: ffffcfb36ec9eb00 R12: 0000000000000000
R13: ffff96b4327af5c0 R14: 00040bfaeea36e00 R15: ffffffff94ce4eac
FS:  0000000000000000(0000) GS:ffff96b83d800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000002527bee5000 CR3: 0000001f1cfbe000 CR4: 00000000001607e0
Call Trace:
[<ffffffff94ca5695>] __do_softirq+0xf5/0x280
[<ffffffff94ca5858>] run_ksoftirqd+0x38/0x50
[<ffffffff94ccf0e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff94ccefa0>] ? lg_double_unlock+0x40/0x40
[<ffffffff94cc6691>] kthread+0xd1/0xe0
[<ffffffff94cc65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff95392d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff94cc65c0>] ? insert_kthread_work+0x40/0x40
Code: 00 0f 85 ea fe ff ff 48 8b 05 3b d4 cb 00 48 89 43 10 48 c7 43 08 00 00 00 00 48 89 df c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 8b 45 d0 65 48 33 04 25 28 00 00 00 0f 85 0e 01 00 00 48 83 
Complete report #3399
Frame # Function Binary Source or offset Line
1
dump_stack
vmlinux 0xffffffffab97ffa5
Build id: 3.10.0-1127.18.2.el7.x86_64
0x19
2
__warn
vmlinux 0xffffffffab29bd18
Build id: 3.10.0-1127.18.2.el7.x86_64
0xd8
3
warn_slowpath_null
vmlinux 0xffffffffab29be5d
Build id: 3.10.0-1127.18.2.el7.x86_64
0x1d
4
net_rx_action
vmlinux 0xffffffffab854693
Build id: 3.10.0-1127.18.2.el7.x86_64
0x343
5
__do_softirq
vmlinux 0xffffffffab2a5695
Build id: 3.10.0-1127.18.2.el7.x86_64
0xf5
6
run_ksoftirqd
vmlinux 0xffffffffab2a5858
Build id: 3.10.0-1127.18.2.el7.x86_64
0x38
7
smpboot_thread_fn
vmlinux 0xffffffffab2cf0e4
Build id: 3.10.0-1127.18.2.el7.x86_64
0x144
8
? lg_double_unlock
vmlinux 0xffffffffab2cefa0
Build id: 3.10.0-1127.18.2.el7.x86_64
0x40
9
kthread
vmlinux 0xffffffffab2c6691
Build id: 3.10.0-1127.18.2.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffffab2c65c0
Build id: 3.10.0-1127.18.2.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffffab992d1d
Build id: 3.10.0-1127.18.2.el7.x86_64
0x7
12
? insert_kthread_work
vmlinux 0xffffffffab2c65c0
Build id: 3.10.0-1127.18.2.el7.x86_64
0x40
WARNING: CPU: 10 PID: 59 at net/core/dev.c:5464 net_rx_action+0x343/0x390
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 devlink tun vmnet(OE) ppdev parport_pc parport fuse vmw_vsock_vmci_transport vsock vmw_vmci vmmon(OE) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw vboxnetadp(OE) vboxnetflt(OE) nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vboxdrv(OE) sunrpc sr_mod cdrom sg uas cdc_ether usbnet snd_usb_audio usb_storage snd_usbmidi_lib snd_rawmidi r8152 mii joydev hid_multitouch squashfs loop dell_rbtn
iTCO_wdt iTCO_vendor_support mei_wdt dell_laptop vfat fat snd_hda_codec_hdmi intel_pmc_core ath10k_pci ath10k_core intel_powerclamp coretemp dell_led intel_rapl iosf_mbi ath kvm_intel snd_hda_codec_realtek mac80211 snd_hda_codec_generic kvm cfg80211 irqbypass snd_hda_intel dell_wmi pcspkr dell_smbios dcdbas snd_hda_codec dell_wmi_descriptor intel_wmi_thunderbolt snd_hda_core i2c_i801 rtsx_pci_ms snd_hwdep rfkill memstick snd_seq snd_seq_device snd_pcm snd_timer mei_me snd mei i2c_designware_platform soundcore i2c_designware_core idma64 virt_dma pinctrl_cannonlake dell_smo8800 pinctrl_intel tpm_crb intel_hid sparse_keymap int3400_thermal acpi_pad binfmt_misc ip_tables xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) rtsx_pci_sdmmc mmc_core mxm_wmi drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops drm e1000e crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ptp aesni_intel pps_core serio_raw nvme rtsx_pci drm_panel_orientation_quirks ipmi_devintf nvme_core ipmi_msghandler wmi video i2c_hid dm_mirror dm_region_hash dm_log authenc cmac zlib wp512 twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common tea sha512_ssse3 sha512_generic serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic seed salsa20_generic salsa20_x86_64 rmd320 rmd256 rmd160 rmd128 michael_mic md4 khazad ghash_generic gcm fcrypt drbg dm_crypt dm_mod des_generic deflate cts crypto_null crct10dif_generic crct10dif_common crc32c_intel ccm cast6_avx_x86_64 cast6_generic cast_common camellia_generic camellia_aesni_avx2 camellia_aesni_avx_x86_64
ablk_helper cryptd camellia_x86_64 xts lrw gf128mul glue_helper blowfish_generic blowfish_x86_64 blowfish_common arc4 ansi_cprng [last unloaded: tcrypt]
CPU: 10 PID: 59 Comm: ksoftirqd/10 Tainted: P           OE  ------------   3.10.0-1127.18.2.el7.x86_64 #1
Hardware name: Dell Inc. Precision 7730/05W5TJ, BIOS 1.10.1 07/29/2019
Call Trace:
[<ffffffffab97ffa5>] dump_stack+0x19/0x1b
[<ffffffffab29bd18>] __warn+0xd8/0x100
[<ffffffffab29be5d>] warn_slowpath_null+0x1d/0x20
[<ffffffffab854693>] net_rx_action+0x343/0x390
[<ffffffffab2a5695>] __do_softirq+0xf5/0x280
[<ffffffffab2a5858>] run_ksoftirqd+0x38/0x50
[<ffffffffab2cf0e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffab2cefa0>] ? lg_double_unlock+0x40/0x40
[<ffffffffab2c6691>] kthread+0xd1/0xe0
[<ffffffffab2c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffab992d1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffab2c65c0>] ? insert_kthread_work+0x40/0x40
Complete report #945
Frame # Function Binary Source or offset Line
1
dump_stack
vmlinux 0xffffffffb7d64147
Build id: 3.10.0-957.27.2.el7.x86_64
0x19
2
__warn
vmlinux 0xffffffffb7698848
Build id: 3.10.0-957.27.2.el7.x86_64
0xd8
3
warn_slowpath_null
vmlinux 0xffffffffb769898d
Build id: 3.10.0-957.27.2.el7.x86_64
0x1d
4
net_rx_action
vmlinux 0xffffffffb7c3bff3
Build id: 3.10.0-957.27.2.el7.x86_64
0x343
5
__do_softirq
vmlinux 0xffffffffb76a2155
Build id: 3.10.0-957.27.2.el7.x86_64
0xf5
6
run_ksoftirqd
vmlinux 0xffffffffb76a2318
Build id: 3.10.0-957.27.2.el7.x86_64
0x38
7
smpboot_thread_fn
vmlinux 0xffffffffb76cb814
Build id: 3.10.0-957.27.2.el7.x86_64
0x144
8
? lg_double_unlock
vmlinux 0xffffffffb76cb6d0
Build id: 3.10.0-957.27.2.el7.x86_64
0x40
9
kthread
vmlinux 0xffffffffb76c2e81
Build id: 3.10.0-957.27.2.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffffb76c2db0
Build id: 3.10.0-957.27.2.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffffb7d76c1d
Build id: 3.10.0-957.27.2.el7.x86_64
0x7
12
? insert_kthread_work
vmlinux 0xffffffffb76c2db0
Build id: 3.10.0-957.27.2.el7.x86_64
0x40
WARNING: CPU: 0 PID: 3 at net/core/dev.c:5333 net_rx_action+0x343/0x390
Modules linked in: rfcomm fuse xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ccm ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) bnep sunrpc ext4 mbcache jbd2 arc4 snd_hda_codec_hdmi iTCO_wdt mei_wdt iTCO_vendor_support ppdev intel_wmi_thunderbolt snd_hda_codec_realtek snd_hda_codec_generic vfat fat ath10k_pci ath10k_core snd_hda_intel intel_pmc_core intel_powerclamp
coretemp snd_hda_codec ath intel_rapl mac80211 kvm_intel btusb snd_hda_core btrtl btbcm kvm btintel snd_hwdep snd_seq bluetooth cfg80211 snd_seq_device irqbypass snd_pcm joydev rfkill snd_timer snd mei_me sg mei soundcore i2c_i801 pcspkr parport_pc parport tpm_crb wmi acpi_pad ip_tables xfs libcrc32c dm_crypt drbg ansi_cprng sd_mod crc_t10dif crct10dif_generic i915 i2c_algo_bit iosf_mbi crct10dif_pclmul crct10dif_common crc32_pclmul drm_kms_helper crc32c_intel ghash_clmulni_intel ahci syscopyarea sysfillrect sysimgblt fb_sys_fops libahci e1000e drm aesni_intel libata glue_helper lrw gf128mul ablk_helper nvme cryptd nvme_core ptp serio_raw pps_core drm_panel_orientation_quirks video i2c_hid dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G               ------------ T 3.10.0-957.27.2.el7.x86_64 #1
Hardware name: LENOVO 10MRCTO1WW/3111, BIOS M1AKT36A 04/20/2018
Call Trace:
[<ffffffffb7d64147>] dump_stack+0x19/0x1b
[<ffffffffb7698848>] __warn+0xd8/0x100
[<ffffffffb769898d>] warn_slowpath_null+0x1d/0x20
[<ffffffffb7c3bff3>] net_rx_action+0x343/0x390
[<ffffffffb76a2155>] __do_softirq+0xf5/0x280
[<ffffffffb76a2318>] run_ksoftirqd+0x38/0x50
[<ffffffffb76cb814>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffb76cb6d0>] ? lg_double_unlock+0x40/0x40
[<ffffffffb76c2e81>] kthread+0xd1/0xe0
[<ffffffffb76c2db0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb7d76c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb76c2db0>] ? insert_kthread_work+0x40/0x40