Info

Permalink
Function
e1000_alloc_rx_buffers
First occurrence
2020-09-29
Last occurrence
2026-07-25
State
NEW
Components
kernel
Quality
-26

Statistics

Operating system Count
CentOS 7 681
Architecture Count
x86_64 681
Related packages Count
kernel 681
 0:3.10.0-1160.41.1.el7 193
 0:3.10.0-1160.el7 64
 0:3.10.0-957.el7 54
 0:3.10.0-1062.el7 52
 0:3.10.0-693.el7 47
 0:3.10.0-862.el7 46
 0:3.10.0-1127.el7 27
 0:3.10.0-1160.45.1.el7 23
 0:3.10.0-1160.118.1.el7 19
 0:3.10.0-1160.80.1.el7 19
 0:3.10.0-1160.62.1.el7 13
 0:3.10.0-1160.59.1.el7 10
 0:3.10.0-1160.102.1.el7 9
 0:3.10.0-1160.119.1.el7 9
 0:3.10.0-1160.21.1.el7 8
 0:3.10.0-1160.92.1.el7 8
 0:3.10.0-1160.49.1.el7 8
 0:3.10.0-1062.1.2.el7 5
 0:3.10.0-1160.11.1.el7 4
 0:3.10.0-1160.6.1.el7 4
 0:3.10.0-1160.76.1.el7 4
 0:3.10.0-1160.83.1.el7 4
 0:3.10.0-1160.88.1.el7 4
 0:3.10.0-1127.19.1.el7 4
 0:3.10.0-957.27.2.el7 4
 0:3.10.0-1160.31.1.el7 4
 0:3.10.0-1160.53.1.el7 3
 0:3.10.0-1062.9.1.el7 3
 0:3.10.0-1127.18.2.el7 3
 0:3.10.0-1062.12.1.el7 3
 0:3.10.0-1160.25.1.el7 2
 0:3.10.0-1160.81.1.el7 2
 0:3.10.0-1160.15.2.el7 2
 0:3.10.0-1160.36.2.el7 2
 0:3.10.0-1160.90.1.el7 2
 0:3.10.0-1127.8.2.el7 1
 0:3.10.0-1160.42.2.el7 1
 0:3.10.0-862.14.4.el7 1
 0:3.10.0-957.10.1.el7 1
 0:3.10.0-693.21.1.el7 1
 0:3.10.0-1062.18.1.el7 1
 0:3.10.0-957.5.1.el7 1
 0:3.10.0-1160.71.1.el7 1
 0:3.10.0-1160.99.1.el7 1
 0:3.10.0-957.12.1.el7 1
 0:3.10.0-1160.108.1.el7 1
 0:3.10.0-862.9.1.el7 1
 0:3.10.0-1160.66.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #1420426
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc008cc2e
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc008c612
Build id: 3.10.0-1160.76.1.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc008d100
Build id: 3.10.0-1160.76.1.el7.x86_64
0x280
4
? usb_hcd_poll_rh_status
vmlinux 0xffffffffa2d10ab0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x180
5
net_rx_action
vmlinux 0xffffffffa2e5a1bf
Build id: 3.10.0-1160.76.1.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffffa28a4d85
Build id: 3.10.0-1160.76.1.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffffa28a4f48
Build id: 3.10.0-1160.76.1.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffffa28ce9e4
Build id: 3.10.0-1160.76.1.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffffa28ce8a0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffffa28c5f91
Build id: 3.10.0-1160.76.1.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffffa28c5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffffa2f99ddd
Build id: 3.10.0-1160.76.1.el7.x86_64
0x7
13
? insert_kthread_work
vmlinux 0xffffffffa28c5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 43s! [ksoftirqd/4:29]
Modules linked in: nbd(E) ch osst st mhvtl(OE) tcp_lp nls_utf8 isofs target_core_mod xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg parport_pc parport vmw_vmci i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm crc32c_intel ata_piix
nfit libata serio_raw libnvdimm mptspi scsi_transport_spi e1000 mptscsih drm_panel_orientation_quirks mptbase dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: sfs]
CPU: 4 PID: 29 Comm: ksoftirqd/4 Tainted: G        W  OEL ------------   3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff91ef93729080 ti: ffff91ef93730000 task.ti: ffff91ef93730000
RIP: 0010:[<ffffffffc008cc2e>]  [<ffffffffc008cc2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff91ef93733bd8  EFLAGS: 00000286
RAX: ffffb930007c2818 RBX: ffffffffa2e59819 RCX: 00000000000005f2
RDX: 000000000000002a RSI: ffff91ef74c43580 RDI: ffff91ef91ab68c0
RBP: ffff91ef93733c30 R08: 000000000001f120 R09: ffffffffa2e416e7
R10: ffff91ef93f1f120 R11: ffffdf7300a76200 R12: ffffffffa356a000
R13: ffff91ef91ab7238 R14: ffff91ef69d89d00 R15: 00000000d152b9fd
FS:  0000000000000000(0000) GS:ffff91ef93f00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055831f878da0 CR3: 00000000505a0000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc008c612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc008d100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffa2d10ab0>] ? usb_hcd_poll_rh_status+0x180/0x180
[<ffffffffa2e5a1bf>] net_rx_action+0x26f/0x390
[<ffffffffa28a4d85>] __do_softirq+0xf5/0x280
[<ffffffffa28a4f48>] run_ksoftirqd+0x38/0x50
[<ffffffffa28ce9e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa28ce8a0>] ? lg_double_unlock+0x40/0x40
[<ffffffffa28c5f91>] kthread+0xd1/0xe0
[<ffffffffa28c5ec0>] ? insert_kthread_work+0x40/0x40
[<ffffffffa2f99ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffa28c5ec0>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #647905
Frame # Function Binary Source or offset Line
1
e1000_clean_rx_irq
e1000 0xffffffffc02d82b9
Build id: 3.10.0-862.el7.x86_64
0x129
2
e1000_clean
e1000 0xffffffffc02d90eb
Build id: 3.10.0-862.el7.x86_64
0x27b
3
? nf_ct_delete
nf_conntrack 0xffffffffc08ae04d
Build id: 3.10.0-862.el7.x86_64
0x18d
4
? usb_hcd_poll_rh_status
vmlinux 0xffffffffaf6c4db0
Build id: 3.10.0-862.el7.x86_64
0x180
5
net_rx_action
vmlinux 0xffffffffaf7ec3af
Build id: 3.10.0-862.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffffaf29a945
Build id: 3.10.0-862.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffffaf29ab08
Build id: 3.10.0-862.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffffaf2c35e4
Build id: 3.10.0-862.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffffaf2c34a0
Build id: 3.10.0-862.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffffaf2bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffffaf2bad60
Build id: 3.10.0-862.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffffaf91f637
Build id: 3.10.0-862.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffffaf2bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [ksoftirqd/0:3]
<EOI> 
Modules linked in:
rfcomm fuse vsock_diag sctp_diag sctp tcp_diag udp_diag inet_diag unix_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink bnep vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 crc32_pclmul snd_rawmidi snd_ac97_codec ghash_clmulni_intel ac97_bus snd_seq aesni_intel snd_seq_device btusb ppdev btrtl btbcm btintel lrw gf128mul glue_helper ablk_helper cryptd bluetooth snd_pcm snd_timer sg joydev snd pcspkr rfkill nfit soundcore vmw_vmci i2c_piix4 libnvdimm shpchp parport_pc parport ip_tables xfs
libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel e1000 mptspi serio_raw scsi_transport_spi mptscsih mptbase i2c_core
CPU: 0 PID: 3 Comm: ksoftirqd/0 Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9ba4c6b09fa0 ti: ffff9ba4c6b98000 task.ti: ffff9ba4c6b98000
RIP: 0010:[<ffffffffc02d82b9>]  [<ffffffffc02d82b9>] e1000_clean_rx_irq+0x129/0x510 [e1000]
RSP: 0000:ffff9ba4c6b9bc40  EFLAGS: 00000283
RAX: ffff9ba4d579c440 RBX: ffffffffaf7ebd08 RCX: 0000000000000000
RDX: ffff9ba4c6b9bd54 RSI: 0000000000000007 RDI: 0000000000000000
RBP: ffff9ba4c6b9bcd8 R08: ffff9ba4cb332100 R09: 000000018040003f
R10: 0000000000000001 R11: fffff497822ccc00 R12: ffff9ba4dadee8c0
R13: ffff9ba4cd2654d0 R14: ffff9ba4dadeeb70 R15: ffff9ba4cb332100
FS:  0000000000000000(0000) GS:ffff9ba4dae00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f331cf9d000 CR3: 000000009bd34000 CR4: 00000000003607f0
Call Trace:
[<ffffffffc02d90eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffffc08ae04d>] ? nf_ct_delete+0x18d/0x270 [nf_conntrack]
[<ffffffffaf6c4db0>] ? usb_hcd_poll_rh_status+0x180/0x180
[<ffffffffaf7ec3af>] net_rx_action+0x26f/0x390
[<ffffffffaf29a945>] __do_softirq+0xf5/0x280
[<ffffffffaf29ab08>] run_ksoftirqd+0x38/0x50
[<ffffffffaf2c35e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffaf2c34a0>] ? lg_double_unlock+0x40/0x40
[<ffffffffaf2bae31>] kthread+0xd1/0xe0
[<ffffffffaf2bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffaf91f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffaf2bad60>] ? insert_kthread_work+0x40/0x40
Code: 00 83 c0 01 89 02 41 0f b6 45 0c 45 0f b7 75 08 88 45 cf 48 8b 45 d0 41 0f b7 de 3b 1d 89 99 01 00 48 8b 00 48 89 45 b0 0f 18 08 <0f> 87 c1 01 00 00 48 8b 7d a0 ba 20 00 00 00 89 de e8 91 26 50 
Complete report #566844
Frame # Function Binary Source or offset Line
1
e1000_clean
e1000 0xffffffffc02d9220
Build id: 3.10.0-1160.el7.x86_64
0x3a0
2
? usb_hcd_poll_rh_status
vmlinux 0xffffffffb990fd90
Build id: 3.10.0-1160.el7.x86_64
0x180
3
net_rx_action
vmlinux 0xffffffffb9a555bf
Build id: 3.10.0-1160.el7.x86_64
0x26f
4
__do_softirq
vmlinux 0xffffffffb94a4b95
Build id: 3.10.0-1160.el7.x86_64
0xf5
5
run_ksoftirqd
vmlinux 0xffffffffb94a4d58
Build id: 3.10.0-1160.el7.x86_64
0x38
6
smpboot_thread_fn
vmlinux 0xffffffffb94ce674
Build id: 3.10.0-1160.el7.x86_64
0x144
7
? lg_double_unlock
vmlinux 0xffffffffb94ce530
Build id: 3.10.0-1160.el7.x86_64
0x40
8
kthread
vmlinux 0xffffffffb94c5c21
Build id: 3.10.0-1160.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffffb94c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffffb9b93df7
Build id: 3.10.0-1160.el7.x86_64
0x21
11
? insert_kthread_work
vmlinux 0xffffffffb94c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 84s! [kworker/1:2:11058]
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [ksoftirqd/2:19]
Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi iosf_mbi snd_ac97_codec crc32_pclmul ghash_clmulni_intel ac97_bus aesni_intel lrw gf128mul snd_seq snd_seq_device snd_pcm ppdev glue_helper ablk_helper cryptd pcspkr joydev snd_timer vmw_balloon snd soundcore sg parport_pc parport vmw_vmci i2c_piix4 ip_tables
xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea ata_piix sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ttm e1000 crc32c_intel drm serio_raw libata mptspi nfit scsi_transport_spi mptscsih libnvdimm mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 19 Comm: ksoftirqd/2 Tainted: G             L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff92963fb15280 ti: ffff92963fb34000 task.ti: ffff92963fb34000
RIP: 0010:[<ffffffffc02d9220>]  [<ffffffffc02d9220>] e1000_clean+0x3a0/0x8e0 [e1000]
RSP: 0018:ffff92963fb37ce8  EFLAGS: 00000246
RAX: 000000000000009d RBX: ffff9296380c4000 RCX: 0000000000000000
RDX: ffffa27100600000 RSI: 0000000000000000 RDI: ffff9296380c4b70
RBP: ffff92963fb37d88 R08: 0000000000000016 R09: 0000000000000001
R10: ffff92963c69f120 R11: ffffea920044ed00 R12: ffff9296398a2098
R13: ffff9296380c48c0 R14: ffff9296398a2000 R15: ffffffffc02d8599
FS:  0000000000000000(0000) GS:ffff92963c680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8d5753c000 CR3: 0000000037340000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb990fd90>] ? usb_hcd_poll_rh_status+0x180/0x180
[<ffffffffb9a555bf>] net_rx_action+0x26f/0x390
[<ffffffffb94a4b95>] __do_softirq+0xf5/0x280
[<ffffffffb94a4d58>] run_ksoftirqd+0x38/0x50
[<ffffffffb94ce674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffb94ce530>] ? lg_double_unlock+0x40/0x40
[<ffffffffb94c5c21>] kthread+0xd1/0xe0
[<ffffffffb94c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffb9b93df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb94c5b50>] ? insert_kthread_work+0x40/0x40
Code: 31 f6 48 89 df e8 21 c0 77 f9 48 8b 83 08 05 00 00 a8 04 75 20 48 8b 7d a0 b8 9d 00 00 00 48 8b 97 20 01 00 00 89 82 d0 00 00 00 <48> 8b 87 20 01 00 00 8b 40 08 8b 45 cc 48 8b 55 d0 65 48 33 14 
Complete report #1191962
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0115c2e
Build id: 3.10.0-1127.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc0115612
Build id: 3.10.0-1127.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc0116100
Build id: 3.10.0-1127.el7.x86_64
0x280
4
? nf_ct_delete
nf_conntrack 0xffffffffc05ed02d
Build id: 3.10.0-1127.el7.x86_64
0x18d
5
? mod_timer
vmlinux 0xffffffff90aaf5ec
Build id: 3.10.0-1127.el7.x86_64
0x10c
6
net_rx_action
vmlinux 0xffffffff910545bf
Build id: 3.10.0-1127.el7.x86_64
0x26f
7
__do_softirq
vmlinux 0xffffffff90aa5695
Build id: 3.10.0-1127.el7.x86_64
0xf5
8
run_ksoftirqd
vmlinux 0xffffffff90aa5858
Build id: 3.10.0-1127.el7.x86_64
0x38
9
smpboot_thread_fn
vmlinux 0xffffffff90acf0e4
Build id: 3.10.0-1127.el7.x86_64
0x144
10
? lg_double_unlock
vmlinux 0xffffffff90acefa0
Build id: 3.10.0-1127.el7.x86_64
0x40
11
kthread
vmlinux 0xffffffff90ac6691
Build id: 3.10.0-1127.el7.x86_64
0xd1
12
? insert_kthread_work
vmlinux 0xffffffff90ac65c0
Build id: 3.10.0-1127.el7.x86_64
0x40
13
ret_from_fork_nospec_begin
vmlinux 0xffffffff91192d1d
Build id: 3.10.0-1127.el7.x86_64
0x7
14
? insert_kthread_work
vmlinux 0xffffffff90ac65c0
Build id: 3.10.0-1127.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [ksoftirqd/1:14]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables cmac arc4 tun md4 nls_utf8 cifs ccm dns_resolver xt_conntrack iptable_filter xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd crc32_pclmul ghash_clmulni_intel aesni_intel soundcore sg lrw ppdev gf128mul glue_helper ablk_helper cryptd pcspkr joydev vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper nfit syscopyarea
sysfillrect libnvdimm sysimgblt fb_sys_fops ttm ata_piix libata drm crct10dif_pclmul crct10dif_common mptspi crc32c_intel scsi_transport_spi e1000 mptscsih serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 14 Comm: ksoftirqd/1 Kdump: loaded Tainted: G             L ------------   3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9079b7130000 ti: ffff9079b712c000 task.ti: ffff9079b712c000
RIP: 0010:[<ffffffffc0115c2e>]  [<ffffffffc0115c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff9079b712fbd8  EFLAGS: 00000286
RAX: ffffb524431c2818 RBX: ffffffff91053c19 RCX: ffffffff90db0490
RDX: 0000000000000090 RSI: 00000000929b5ac0 RDI: 00000000000005f2
RBP: ffff9079b712fc30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff9078d29b5ac0 R11: fffff26d13964f00 R12: 0000000000000608
R13: ffff907d680a8000 R14: ffff907d627a7d80 R15: ffff907d680a9238
FS:  0000000000000000(0000) GS:ffff907d69640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0d51a3a000 CR3: 0000000527e42000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc0115612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc0116100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffc05ed02d>] ? nf_ct_delete+0x18d/0x270 [nf_conntrack]
[<ffffffff90aaf5ec>] ? mod_timer+0x10c/0x230
[<ffffffff910545bf>] net_rx_action+0x26f/0x390
[<ffffffff90aa5695>] __do_softirq+0xf5/0x280
[<ffffffff90aa5858>] run_ksoftirqd+0x38/0x50
[<ffffffff90acf0e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff90acefa0>] ? lg_double_unlock+0x40/0x40
[<ffffffff90ac6691>] kthread+0xd1/0xe0
[<ffffffff90ac65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff91192d1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff90ac65c0>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #1140507
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0186c2e
Build id: 3.10.0-1127.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc0186612
Build id: 3.10.0-1127.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc0187100
Build id: 3.10.0-1127.el7.x86_64
0x280
4
? tcp_init_xmit_timers
vmlinux 0xffffffffbcec0770
Build id: 3.10.0-1127.el7.x86_64
0x30
5
net_rx_action
vmlinux 0xffffffffbce545bf
Build id: 3.10.0-1127.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffffbc8a5695
Build id: 3.10.0-1127.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffffbc8a5858
Build id: 3.10.0-1127.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffffbc8cf0e4
Build id: 3.10.0-1127.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffffbc8cefa0
Build id: 3.10.0-1127.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffffbc8c6691
Build id: 3.10.0-1127.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffffbc8c65c0
Build id: 3.10.0-1127.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffffbcf92d24
Build id: 3.10.0-1127.el7.x86_64
0xe
13
? insert_kthread_work
vmlinux 0xffffffffbc8c65c0
Build id: 3.10.0-1127.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [ksoftirqd/0:6]
Modules linked in: veth nfnetlink_queue nfnetlink_log xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock bnep sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm btusb btrtl crc32_pclmul btbcm btintel bluetooth ghash_clmulni_intel aesni_intel snd_timer lrw gf128mul snd rfkill glue_helper soundcore ablk_helper cryptd ppdev sg pcspkr joydev vmw_balloon vmw_vmci
i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix libata mptspi drm scsi_transport_spi nfit mptscsih libnvdimm mptbase e1000 crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm_panel_orientation_quirks
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffffa134fa7ad230 ti: ffffa134fa0e0000 task.ti: ffffa134fa0e0000
RIP: 0010:[<ffffffffc0186c2e>]  [<ffffffffc0186c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0000:ffffa134fa0e3bd8  EFLAGS: 00000286
RAX: ffffc2b001d02818 RBX: 00000000000005f2 RCX: ffffffffbcbb0490
RDX: 000000000000004f RSI: 00000000bd6b9000 RDI: 00000000000005f2
RBP: ffffa134fa0e3c30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffffa135a73e69c0 R11: ffffa134f9da6000 R12: 00000002273e69c0
R13: 000000026cc99238 R14: ffffffffbcbb078b R15: ffffa134fa0e3b90
FS:  0000000000000000(0000) GS:ffffa1376e600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00003df03027d000 CR3: 0000000079410000 CR4: 00000000003407f0
Call Trace:
[<ffffffffc0186612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc0187100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffbcec0770>] ? tcp_init_xmit_timers+0x30/0x30
[<ffffffffbce545bf>] net_rx_action+0x26f/0x390
[<ffffffffbc8a5695>] __do_softirq+0xf5/0x280
[<ffffffffbc8a5858>] run_ksoftirqd+0x38/0x50
[<ffffffffbc8cf0e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffbc8cefa0>] ? lg_double_unlock+0x40/0x40
[<ffffffffbc8c6691>] kthread+0xd1/0xe0
[<ffffffffbc8c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffbcf92d24>] ret_from_fork_nospec_begin+0xe/0x21
[<ffffffffbc8c65c0>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #969364
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc00d8c1e
Build id: 3.10.0-862.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc00d8602
Build id: 3.10.0-862.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc00d90eb
Build id: 3.10.0-862.el7.x86_64
0x27b
4
? __netif_receive_skb
vmlinux 0xffffffffa8febd08
Build id: 3.10.0-862.el7.x86_64
0x18
5
? netif_receive_skb_internal
vmlinux 0xffffffffa8febd90
Build id: 3.10.0-862.el7.x86_64
0x40
6
net_rx_action
vmlinux 0xffffffffa8fec3af
Build id: 3.10.0-862.el7.x86_64
0x26f
7
__do_softirq
vmlinux 0xffffffffa8a9a945
Build id: 3.10.0-862.el7.x86_64
0xf5
8
run_ksoftirqd
vmlinux 0xffffffffa8a9ab08
Build id: 3.10.0-862.el7.x86_64
0x38
9
smpboot_thread_fn
vmlinux 0xffffffffa8ac35e4
Build id: 3.10.0-862.el7.x86_64
0x144
10
? lg_double_unlock
vmlinux 0xffffffffa8ac34a0
Build id: 3.10.0-862.el7.x86_64
0x40
11
kthread
vmlinux 0xffffffffa8abae31
Build id: 3.10.0-862.el7.x86_64
0xd1
12
? insert_kthread_work
vmlinux 0xffffffffa8abad60
Build id: 3.10.0-862.el7.x86_64
0x40
13
ret_from_fork_nospec_begin
vmlinux 0xffffffffa911f637
Build id: 3.10.0-862.el7.x86_64
0x21
14
? insert_kthread_work
vmlinux 0xffffffffa8abad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [ksoftirqd/0:3]
Modules linked in: binfmt_misc fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg snd_timer snd soundcore
vmw_balloon nfit pcspkr joydev libnvdimm parport_pc parport i2c_piix4 shpchp vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul libata crct10dif_common crc32c_intel mptspi scsi_transport_spi mptscsih e1000 serio_raw mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8b67f5b69fa0 ti: ffff8b67f5b8c000 task.ti: ffff8b67f5b8c000
RIP: 0010:[<ffffffffc00d8c1e>]  [<ffffffffc00d8c1e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff8b67f5b8fbd8  EFLAGS: 00000286
RAX: ffff99ef03202818 RBX: 00000000000005f2 RCX: ffffffffa8d73b20
RDX: 0000000000000016 RSI: 00000000be884000 RDI: 00000000000005f2
RBP: ffff8b67f5b8fc30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff8b68e7e83540 R11: ffffe7a1083e9100 R12: 0000000267e83540
R13: 00000002905e7238 R14: ffffffffa8d73e1b R15: ffff8b67f5b8fb90
FS:  0000000000000000(0000) GS:ffff8b6d90600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006dee20 CR3: 0000000677e52000 CR4: 00000000007607f0
PKRU: 00000000
Call Trace:
[<ffffffffc00d8602>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc00d90eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffffa8febd08>] ? __netif_receive_skb+0x18/0x60
[<ffffffffa8febd90>] ? netif_receive_skb_internal+0x40/0xc0
[<ffffffffa8fec3af>] net_rx_action+0x26f/0x390
[<ffffffffa8a9a945>] __do_softirq+0xf5/0x280
[<ffffffffa8a9ab08>] run_ksoftirqd+0x38/0x50
[<ffffffffa8ac35e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa8ac34a0>] ? lg_double_unlock+0x40/0x40
[<ffffffffa8abae31>] kthread+0xd1/0xe0
[<ffffffffa8abad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa911f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa8abad60>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #919263
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc01d6c1e
Build id: 3.10.0-862.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc01d6602
Build id: 3.10.0-862.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc01d70eb
Build id: 3.10.0-862.el7.x86_64
0x27b
4
? nf_ct_delete
nf_conntrack 0xffffffffc06b504d
Build id: 3.10.0-862.el7.x86_64
0x18d
5
? __queue_work
vmlinux 0xffffffffbb8b0cbc
Build id: 3.10.0-862.el7.x86_64
0x13c
6
net_rx_action
vmlinux 0xffffffffbbdec3af
Build id: 3.10.0-862.el7.x86_64
0x26f
7
__do_softirq
vmlinux 0xffffffffbb89a945
Build id: 3.10.0-862.el7.x86_64
0xf5
8
run_ksoftirqd
vmlinux 0xffffffffbb89ab08
Build id: 3.10.0-862.el7.x86_64
0x38
9
smpboot_thread_fn
vmlinux 0xffffffffbb8c35e4
Build id: 3.10.0-862.el7.x86_64
0x144
10
? lg_double_unlock
vmlinux 0xffffffffbb8c34a0
Build id: 3.10.0-862.el7.x86_64
0x40
11
kthread
vmlinux 0xffffffffbb8bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
12
? insert_kthread_work
vmlinux 0xffffffffbb8bad60
Build id: 3.10.0-862.el7.x86_64
0x40
13
ret_from_fork_nospec_begin
vmlinux 0xffffffffbbf1f637
Build id: 3.10.0-862.el7.x86_64
0x21
14
? insert_kthread_work
vmlinux 0xffffffffbb8bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 32s! [ksoftirqd/0:3]
Modules linked in: binfmt_misc fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus crc32_pclmul ghash_clmulni_intel snd_seq aesni_intel lrw gf128mul snd_seq_device snd_pcm glue_helper snd_timer snd ablk_helper cryptd pcspkr ppdev soundcore
nfit vmw_balloon joydev sg libnvdimm parport_pc parport i2c_piix4 vmw_vmci shpchp ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata mptspi serio_raw scsi_transport_spi mptscsih e1000 mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9de7f5b69fa0 ti: ffff9de7f5b8c000 task.ti: ffff9de7f5b8c000
RIP: 0010:[<ffffffffc01d6c1e>]  [<ffffffffc01d6c1e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff9de7f5b8fbd8  EFLAGS: 00000286
RAX: ffffaca6831c2818 RBX: ffffffffbbdeb9f9 RCX: ffffffffbbb73b20
RDX: 0000000000000080 RSI: 000000004871c440 RDI: 00000000000005f2
RBP: ffff9de7f5b8fc30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff9de6c871c440 R11: ffffe2a3dc1ddd00 R12: ffffffffbc54d7a0
R13: ffff9dea75e1d238 R14: ffff9ded87775800 R15: 0000000052e81c0a
FS:  0000000000000000(0000) GS:ffff9ded90600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbf937df000 CR3: 0000000039cc6000 CR4: 00000000007607f0
PKRU: 00000000
Call Trace:
[<ffffffffc01d6602>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc01d70eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffffc06b504d>] ? nf_ct_delete+0x18d/0x270 [nf_conntrack]
[<ffffffffbb8b0cbc>] ? __queue_work+0x13c/0x3e0
[<ffffffffbbdec3af>] net_rx_action+0x26f/0x390
[<ffffffffbb89a945>] __do_softirq+0xf5/0x280
[<ffffffffbb89ab08>] run_ksoftirqd+0x38/0x50
[<ffffffffbb8c35e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffbb8c34a0>] ? lg_double_unlock+0x40/0x40
[<ffffffffbb8bae31>] kthread+0xd1/0xe0
[<ffffffffbb8bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffbbf1f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffbb8bad60>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #902175
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc00c8c2e
Build id: 3.10.0-1160.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc00c8612
Build id: 3.10.0-1160.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc00c9100
Build id: 3.10.0-1160.el7.x86_64
0x280
4
? __enqueue_entity
vmlinux 0xffffffff904dfb48
Build id: 3.10.0-1160.el7.x86_64
0x78
5
net_rx_action
vmlinux 0xffffffff90a555bf
Build id: 3.10.0-1160.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffff904a4b95
Build id: 3.10.0-1160.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffff904a4d58
Build id: 3.10.0-1160.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffff904ce674
Build id: 3.10.0-1160.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffff904ce530
Build id: 3.10.0-1160.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffff904c5c21
Build id: 3.10.0-1160.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffff904c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffff90b93df7
Build id: 3.10.0-1160.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffff904c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 27s! [ksoftirqd/4:29]
Modules linked in: vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon pcspkr sg joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix mptspi libata scsi_transport_spi mptscsih serio_raw nfit e1000 mptbase libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 4 PID: 29 Comm: ksoftirqd/4 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff97d34d7d9080 ti: ffff97d34d7e8000 task.ti: ffff97d34d7e8000
RIP: 0010:[<ffffffffc00c8c2e>]  [<ffffffffc00c8c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff97d34d7ebbd8  EFLAGS: 00000286
RAX: ffffb883831c2818 RBX: 00000000000005f2 RCX: ffffffff907b1370
RDX: 00000000000000b3 RSI: 00000000bfb13800 RDI: 00000000000005f2
RBP: ffff97d34d7ebc30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff97d379bb2640 R11: ffffee7405e6e500 R12: 0000000179bb2640
R13: 0000000255fef238 R14: ffffffff907b166b R15: ffff97d34d7ebb90
FS:  0000000000000000(0000) GS:ffff97d8a3700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb0e5ef8c00 CR3: 000000042ea10000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffffc00c8612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc00c9100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff904dfb48>] ? __enqueue_entity+0x78/0x80
[<ffffffff90a555bf>] net_rx_action+0x26f/0x390
[<ffffffff904a4b95>] __do_softirq+0xf5/0x280
[<ffffffff904a4d58>] run_ksoftirqd+0x38/0x50
[<ffffffff904ce674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff904ce530>] ? lg_double_unlock+0x40/0x40
[<ffffffff904c5c21>] kthread+0xd1/0xe0
[<ffffffff904c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff90b93df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff904c5b50>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #875575
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc02d3c1e
Build id: 3.10.0-862.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc02d3602
Build id: 3.10.0-862.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc02d40eb
Build id: 3.10.0-862.el7.x86_64
0x27b
4
? update_curr
vmlinux 0xffffffffa3ed816c
Build id: 3.10.0-862.el7.x86_64
0x14c
5
? account_entity_dequeue
vmlinux 0xffffffffa3ed4ade
Build id: 3.10.0-862.el7.x86_64
0xae
6
net_rx_action
vmlinux 0xffffffffa43ec3af
Build id: 3.10.0-862.el7.x86_64
0x26f
7
__do_softirq
vmlinux 0xffffffffa3e9a945
Build id: 3.10.0-862.el7.x86_64
0xf5
8
run_ksoftirqd
vmlinux 0xffffffffa3e9ab08
Build id: 3.10.0-862.el7.x86_64
0x38
9
smpboot_thread_fn
vmlinux 0xffffffffa3ec35e4
Build id: 3.10.0-862.el7.x86_64
0x144
10
? lg_double_unlock
vmlinux 0xffffffffa3ec34a0
Build id: 3.10.0-862.el7.x86_64
0x40
11
kthread
vmlinux 0xffffffffa3ebae31
Build id: 3.10.0-862.el7.x86_64
0xd1
12
? insert_kthread_work
vmlinux 0xffffffffa3ebad60
Build id: 3.10.0-862.el7.x86_64
0x40
13
ret_from_fork_nospec_begin
vmlinux 0xffffffffa451f637
Build id: 3.10.0-862.el7.x86_64
0x21
14
? insert_kthread_work
vmlinux 0xffffffffa3ebad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [ksoftirqd/0:3]
Modules linked in: binfmt_misc fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus crc32_pclmul ghash_clmulni_intel aesni_intel snd_seq lrw gf128mul glue_helper snd_seq_device sg snd_pcm snd_timer ablk_helper snd joydev cryptd ppdev
soundcore vmw_balloon nfit pcspkr libnvdimm parport_pc parport i2c_piix4 shpchp vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel mptspi ata_piix scsi_transport_spi mptscsih e1000 serio_raw libata mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9c97f5b69fa0 ti: ffff9c97f5b8c000 task.ti: ffff9c97f5b8c000
RIP: 0010:[<ffffffffc02d3c1e>]  [<ffffffffc02d3c1e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff9c97f5b8fbd8  EFLAGS: 00000286
RAX: ffffa91dc31c2818 RBX: 00000000000005f2 RCX: ffffffffa4173b20
RDX: 0000000000000035 RSI: 00000000bc852000 RDI: 00000000000005f2
RBP: ffff9c97f5b8fc30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff9c9cf5fa5ac0 R11: ffffefcf4faad200 R12: 0000000675fa5ac0
R13: 000000028d635238 R14: ffffffffa4173e1b R15: ffff9c97f5b8fb90
FS:  0000000000000000(0000) GS:ffff9c9d91a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc456e75000 CR3: 0000000708360000 CR4: 00000000007607f0
DR0: 00007f93a2f70660 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
PKRU: 00000000
Call Trace:
[<ffffffffc02d3602>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc02d40eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffffa3ed816c>] ? update_curr+0x14c/0x1e0
[<ffffffffa3ed4ade>] ? account_entity_dequeue+0xae/0xd0
[<ffffffffa43ec3af>] net_rx_action+0x26f/0x390
[<ffffffffa3e9a945>] __do_softirq+0xf5/0x280
[<ffffffffa3e9ab08>] run_ksoftirqd+0x38/0x50
[<ffffffffa3ec35e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa3ec34a0>] ? lg_double_unlock+0x40/0x40
[<ffffffffa3ebae31>] kthread+0xd1/0xe0
[<ffffffffa3ebad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa451f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa3ebad60>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #862938
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc00f9c2e
Build id: 3.10.0-1160.66.1.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc00f9599
Build id: 3.10.0-1160.66.1.el7.x86_64
0x3f9
3
e1000_clean
e1000 0xffffffffc00fa100
Build id: 3.10.0-1160.66.1.el7.x86_64
0x280
4
? __queue_work
vmlinux 0xffffffff88cbbf0c
Build id: 3.10.0-1160.66.1.el7.x86_64
0x13c
5
? mod_timer
vmlinux 0xffffffff88caeefc
Build id: 3.10.0-1160.66.1.el7.x86_64
0x10c
6
net_rx_action
vmlinux 0xffffffff8925a1bf
Build id: 3.10.0-1160.66.1.el7.x86_64
0x26f
7
__do_softirq
vmlinux 0xffffffff88ca4d85
Build id: 3.10.0-1160.66.1.el7.x86_64
0xf5
8
run_ksoftirqd
vmlinux 0xffffffff88ca4f48
Build id: 3.10.0-1160.66.1.el7.x86_64
0x38
9
smpboot_thread_fn
vmlinux 0xffffffff88cce9e4
Build id: 3.10.0-1160.66.1.el7.x86_64
0x144
10
? lg_double_unlock
vmlinux 0xffffffff88cce8a0
Build id: 3.10.0-1160.66.1.el7.x86_64
0x40
11
kthread
vmlinux 0xffffffff88cc5f91
Build id: 3.10.0-1160.66.1.el7.x86_64
0xd1
12
? insert_kthread_work
vmlinux 0xffffffff88cc5ec0
Build id: 3.10.0-1160.66.1.el7.x86_64
0x40
13
ret_from_fork_nospec_begin
vmlinux 0xffffffff89399ddd
Build id: 3.10.0-1160.66.1.el7.x86_64
0x7
14
? insert_kthread_work
vmlinux 0xffffffff88cc5ec0
Build id: 3.10.0-1160.66.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [ksoftirqd/3:24]
Modules linked in: nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle xt_conntrack ipt_REJECT nf_reject_ipv4 bridge stp llc ebtable_filter ebtables devlink ip6t_MASQUERADE nf_nat_masquerade_ipv6 ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_mark tun nf_tables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter overlay(T) bnep vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper vmw_balloon cryptd btusb btrtl btbcm btintel bluetooth pcspkr joydev rfkill ahci libahci sg vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables
xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea mptspi ata_piix crct10dif_pclmul sysfillrect sysimgblt fb_sys_fops ttm crct10dif_common nfit scsi_transport_spi drm serio_raw crc32c_intel libnvdimm e1000 mptscsih libata mptbase drm_panel_orientation_quirks fuse
CPU: 3 PID: 24 Comm: ksoftirqd/3 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.66.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9c9aba463180 ti: ffff9c9aba494000 task.ti: ffff9c9aba494000
RIP: 0010:[<ffffffffc00f9c2e>]  [<ffffffffc00f9c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff9c9aba497bd8  EFLAGS: 00000286
RAX: ffffa845c10c2818 RBX: ffffffff89259819 RCX: ffffffff88fb1dc0
RDX: 0000000000000058 RSI: 00000000668269c0 RDI: 00000000000005f2
RBP: ffff9c9aba497c30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff9c99a68269c0 R11: 0000000000000001 R12: ffffffff89969f60
R13: ffff9c9b74bed238 R14: ffff9c9ae8cdae00 R15: 00000000402b1102
FS:  0000000000000000(0000) GS:ffff9c9b756c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fee62b46d70 CR3: 00000000b6142000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc00f9599>] e1000_clean_rx_irq+0x3f9/0x510 [e1000]
[<ffffffffc00fa100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff88cbbf0c>] ? __queue_work+0x13c/0x3f0
[<ffffffff88caeefc>] ? mod_timer+0x10c/0x230
[<ffffffff8925a1bf>] net_rx_action+0x26f/0x390
[<ffffffff88ca4d85>] __do_softirq+0xf5/0x280
[<ffffffff88ca4f48>] run_ksoftirqd+0x38/0x50
[<ffffffff88cce9e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff88cce8a0>] ? lg_double_unlock+0x40/0x40
[<ffffffff88cc5f91>] kthread+0xd1/0xe0
[<ffffffff88cc5ec0>] ? insert_kthread_work+0x40/0x40
[<ffffffff89399ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff88cc5ec0>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #846626
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0470c2e
Build id: 3.10.0-1160.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc0470612
Build id: 3.10.0-1160.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc0471100
Build id: 3.10.0-1160.el7.x86_64
0x280
4
? __enqueue_entity
vmlinux 0xffffffffbd6dfb48
Build id: 3.10.0-1160.el7.x86_64
0x78
5
net_rx_action
vmlinux 0xffffffffbdc555bf
Build id: 3.10.0-1160.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffffbd6a4b95
Build id: 3.10.0-1160.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffffbd6a4d58
Build id: 3.10.0-1160.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffffbd6ce674
Build id: 3.10.0-1160.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffffbd6ce530
Build id: 3.10.0-1160.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffffbd6c5c21
Build id: 3.10.0-1160.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffffbd6c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffffbdd93ddd
Build id: 3.10.0-1160.el7.x86_64
0x7
13
? insert_kthread_work
vmlinux 0xffffffffbd6c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 63s! [ksoftirqd/0:6]
Modules linked in: cmac arc4 md4 nls_utf8 cifs ccm dns_resolver ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc kvm_intel nfit libnvdimm kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr vmw_balloon sg ppdev joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul drm crct10dif_common crc32c_intel ata_piix libata e1000 serio_raw mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffffa0f396fb5280 ti: ffffa0f3973f4000 task.ti: ffffa0f3973f4000
RIP: 0010:[<ffffffffc0470c2e>]  [<ffffffffc0470c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0000:ffffa0f3973f7bd8  EFLAGS: 00000286
RAX: ffffb76b03502818 RBX: ffffffffbdc54c19 RCX: 00000000000005f2
RDX: 0000000000000039 RSI: ffffa0f7e55f7240 RDI: ffffa0f3a1fac8c0
RBP: ffffa0f3973f7c30 R08: 00000000472f0664 R09: 0000000000000b68
R10: 0000000000e82d5c R11: 0000000000000191 R12: ffffffffbe369ac0
R13: ffffa0f3a1fad238 R14: ffffa0f6feb21f00 R15: 0000000037e8fb72
FS:  0000000000000000(0000) GS:ffffa0f7e7600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7e93b1e000 CR3: 000000000cfa8000 CR4: 00000000003607f0
Call Trace:
[<ffffffffc0470612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc0471100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffbd6dfb48>] ? __enqueue_entity+0x78/0x80
[<ffffffffbdc555bf>] net_rx_action+0x26f/0x390
[<ffffffffbd6a4b95>] __do_softirq+0xf5/0x280
[<ffffffffbd6a4d58>] run_ksoftirqd+0x38/0x50
[<ffffffffbd6ce674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffbd6ce530>] ? lg_double_unlock+0x40/0x40
[<ffffffffbd6c5c21>] kthread+0xd1/0xe0
[<ffffffffbd6c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffbdd93ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffbd6c5b50>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #840131
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc00fdc1e
Build id: 3.10.0-862.9.1.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc00fd589
Build id: 3.10.0-862.9.1.el7.x86_64
0x3f9
3
e1000_clean
e1000 0xffffffffc00fe0eb
Build id: 3.10.0-862.9.1.el7.x86_64
0x27b
4
? update_curr
vmlinux 0xffffffff87ad895c
Build id: 3.10.0-862.9.1.el7.x86_64
0x14c
5
? account_entity_dequeue
vmlinux 0xffffffff87ad52ce
Build id: 3.10.0-862.9.1.el7.x86_64
0xae
6
? __schedule
vmlinux 0xffffffff88113bf4
Build id: 3.10.0-862.9.1.el7.x86_64
0x614
7
net_rx_action
vmlinux 0xffffffff87fed46f
Build id: 3.10.0-862.9.1.el7.x86_64
0x26f
8
__do_softirq
vmlinux 0xffffffff87a9b085
Build id: 3.10.0-862.9.1.el7.x86_64
0xf5
9
run_ksoftirqd
vmlinux 0xffffffff87a9b248
Build id: 3.10.0-862.9.1.el7.x86_64
0x38
10
smpboot_thread_fn
vmlinux 0xffffffff87ac3dd4
Build id: 3.10.0-862.9.1.el7.x86_64
0x144
11
? lg_double_unlock
vmlinux 0xffffffff87ac3c90
Build id: 3.10.0-862.9.1.el7.x86_64
0x40
12
kthread
vmlinux 0xffffffff87abb621
Build id: 3.10.0-862.9.1.el7.x86_64
0xd1
13
? insert_kthread_work
vmlinux 0xffffffff87abb550
Build id: 3.10.0-862.9.1.el7.x86_64
0x40
14
ret_from_fork_nospec_begin
vmlinux 0xffffffff881205f7
Build id: 3.10.0-862.9.1.el7.x86_64
0x21
15
? insert_kthread_work
vmlinux 0xffffffff87abb550
Build id: 3.10.0-862.9.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ksoftirqd/1:14]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac snd_ens1371 snd_rawmidi iosf_mbi crc32_pclmul snd_ac97_codec ghash_clmulni_intel aesni_intel lrw ac97_bus snd_seq gf128mul glue_helper ablk_helper snd_seq_device cryptd snd_pcm snd_timer ppdev snd vmw_balloon soundcore nfit joydev libnvdimm sg pcspkr parport_pc parport vmw_vmci i2c_piix4 shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod sr_mod
cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm serio_raw ata_piix mptspi scsi_transport_spi libata e1000 mptscsih mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 14 Comm: ksoftirqd/1 Kdump: loaded Tainted: G        W    L ------------   3.10.0-862.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffffa004f4e2bf40 ti: ffffa004f4e48000 task.ti: ffffa004f4e48000
RIP: 0010:[<ffffffffc00fdc1e>]  [<ffffffffc00fdc1e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffffa004f4e4bbd8  EFLAGS: 00000286
RAX: ffffc33fc3482818 RBX: ffffffff87fecab9 RCX: 00000000000005f2
RDX: 0000000000000050 RSI: ffffa00aee7e8800 RDI: ffffa00af280e8c0
RBP: ffffa004f4e4bc30 R08: 00000000405b5b41 R09: 000000000000a9b0
R10: 000000000000a9b0 R11: ffffa00a83742e00 R12: ffffffff8874d840
R13: ffffa00af280f238 R14: ffffa008f3709000 R15: 00000000d4a02a1a
FS:  0000000000000000(0000) GS:ffffa00af3040000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000001587f9958 CR3: 000000033ab96000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc00fd589>] e1000_clean_rx_irq+0x3f9/0x510 [e1000]
[<ffffffffc00fe0eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffff87ad895c>] ? update_curr+0x14c/0x1e0
[<ffffffff87ad52ce>] ? account_entity_dequeue+0xae/0xd0
[<ffffffff88113bf4>] ? __schedule+0x614/0xa20
[<ffffffff87fed46f>] net_rx_action+0x26f/0x390
[<ffffffff87a9b085>] __do_softirq+0xf5/0x280
[<ffffffff87a9b248>] run_ksoftirqd+0x38/0x50
[<ffffffff87ac3dd4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff87ac3c90>] ? lg_double_unlock+0x40/0x40
[<ffffffff87abb621>] kthread+0xd1/0xe0
[<ffffffff87abb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff881205f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff87abb550>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #822640
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc01adc1e
Build id: 3.10.0-862.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc01ad602
Build id: 3.10.0-862.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc01ae0eb
Build id: 3.10.0-862.el7.x86_64
0x27b
4
? update_curr
vmlinux 0xffffffff8d6d816c
Build id: 3.10.0-862.el7.x86_64
0x14c
5
? account_entity_dequeue
vmlinux 0xffffffff8d6d4ade
Build id: 3.10.0-862.el7.x86_64
0xae
6
? __schedule
vmlinux 0xffffffff8dd12b14
Build id: 3.10.0-862.el7.x86_64
0x614
7
net_rx_action
vmlinux 0xffffffff8dbec3af
Build id: 3.10.0-862.el7.x86_64
0x26f
8
__do_softirq
vmlinux 0xffffffff8d69a945
Build id: 3.10.0-862.el7.x86_64
0xf5
9
run_ksoftirqd
vmlinux 0xffffffff8d69ab08
Build id: 3.10.0-862.el7.x86_64
0x38
10
smpboot_thread_fn
vmlinux 0xffffffff8d6c35e4
Build id: 3.10.0-862.el7.x86_64
0x144
11
? lg_double_unlock
vmlinux 0xffffffff8d6c34a0
Build id: 3.10.0-862.el7.x86_64
0x40
12
kthread
vmlinux 0xffffffff8d6bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
13
? insert_kthread_work
vmlinux 0xffffffff8d6bad60
Build id: 3.10.0-862.el7.x86_64
0x40
14
ret_from_fork_nospec_begin
vmlinux 0xffffffff8dd1f637
Build id: 3.10.0-862.el7.x86_64
0x21
15
? insert_kthread_work
vmlinux 0xffffffff8d6bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ksoftirqd/1:14]
Modules linked in: tcp_lp nls_utf8 isofs fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ext4 crc32_pclmul mbcache jbd2 snd_ens1371 ghash_clmulni_intel snd_rawmidi ppdev snd_ac97_codec ac97_bus aesni_intel snd_seq lrw gf128mul snd_seq_device glue_helper ablk_helper cryptd snd_pcm pcspkr joydev snd_timer nfit libnvdimm snd soundcore sg i2c_piix4 vmw_vmci shpchp parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm crc32c_intel ata_piix e1000 serio_raw libata mptspi scsi_transport_spi mptscsih i2c_core mptbase dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 14 Comm: ksoftirqd/1 Kdump: loaded Not tainted 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9bd016ff3f40 ti: ffff9bd01680c000 task.ti: ffff9bd01680c000
RIP: 0010:[<ffffffffc01adc1e>]  [<ffffffffc01adc1e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0000:ffff9bd01680fbd8  EFLAGS: 00010286
RAX: ffffc08a40982818 RBX: ffff9bd01620cba8 RCX: ffffffff8d973b20
RDX: 00000000000000b1 RSI: 000000009cb880c0 RDI: 00000000000005f2
RBP: ffff9bd01680fc30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff9bcf9cb880c0 R11: fffffead82524600 R12: 0000000000000000
R13: 000000000000004e R14: ffff9bd01620cba8 R15: ffffffff8dc5d46b
FS:  0000000000000000(0000) GS:ffff9bd018e40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6106aeb000 CR3: 000000008d1bc000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc01ad602>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc01ae0eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffff8d6d816c>] ? update_curr+0x14c/0x1e0
[<ffffffff8d6d4ade>] ? account_entity_dequeue+0xae/0xd0
[<ffffffff8dd12b14>] ? __schedule+0x614/0xa20
[<ffffffff8dbec3af>] net_rx_action+0x26f/0x390
[<ffffffff8d69a945>] __do_softirq+0xf5/0x280
[<ffffffff8d69ab08>] run_ksoftirqd+0x38/0x50
[<ffffffff8d6c35e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff8d6c34a0>] ? lg_double_unlock+0x40/0x40
[<ffffffff8d6bae31>] kthread+0xd1/0xe0
[<ffffffff8d6bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff8dd1f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8d6bad60>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #708445
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0505c0e
Build id: 3.10.0-957.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc05055f2
Build id: 3.10.0-957.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc05060dd
Build id: 3.10.0-957.el7.x86_64
0x27d
4
? nf_ct_delete
nf_conntrack 0xffffffffc09d306d
Build id: 3.10.0-957.el7.x86_64
0x18d
5
net_rx_action
vmlinux 0xffffffff9ea39e9f
Build id: 3.10.0-957.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffff9e4a0f05
Build id: 3.10.0-957.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffff9e4a10c8
Build id: 3.10.0-957.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffff9e4ca5c4
Build id: 3.10.0-957.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffff9e4ca480
Build id: 3.10.0-957.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffff9e4c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffff9e4c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffff9eb74c37
Build id: 3.10.0-957.el7.x86_64
0x21
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [ksoftirqd/4:29]
Modules linked in: binfmt_misc rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac coretemp iosf_mbi ppdev crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi aesni_intel snd_ac97_codec lrw gf128mul glue_helper vmw_balloon ablk_helper cryptd ac97_bus snd_seq snd_seq_device snd_pcm snd_timer pcspkr joydev sg snd soundcore parport_pc parport
vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm e1000 mptspi libata scsi_transport_spi mptscsih mptbase nfit drm_panel_orientation_quirks libnvdimm floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 29 Comm: ksoftirqd/4 Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8df634161040 ti: ffff8df63416c000 task.ti: ffff8df63416c000
RIP: 0010:[<ffffffffc0505c0e>]  [<ffffffffc0505c0e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff8df63416fbd8  EFLAGS: 00000286
RAX: ffffa63c031c2818 RBX: 00000000000005f2 RCX: ffffffff9e7a01d0
RDX: 00000000000000b3 RSI: 00000000be482800 RDI: 00000000000005f2
RBP: ffff8df63416fc30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff8dfbb8f7cbc0 R11: ffff8df634b16000 R12: 00000006f8f7cbc0
R13: 00000002dde73238 R14: ffffffff9e7a04cb R15: ffff8df63416fb90
FS:  0000000000000000(0000) GS:ffff8dfcdd700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5ec3974cc5 CR3: 00000005c0e5a000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc05055f2>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc05060dd>] e1000_clean+0x27d/0x8e0 [e1000]
[<ffffffffc09d306d>] ? nf_ct_delete+0x18d/0x270 [nf_conntrack]
[<ffffffff9ea39e9f>] net_rx_action+0x26f/0x390
[<ffffffff9e4a0f05>] __do_softirq+0xf5/0x280
[<ffffffff9e4a10c8>] run_ksoftirqd+0x38/0x50
[<ffffffff9e4ca5c4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff9e4ca480>] ? lg_double_unlock+0x40/0x40
[<ffffffff9e4c1c31>] kthread+0xd1/0xe0
[<ffffffff9e4c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff9eb74c37>] ret_from_fork_nospec_begin+0x21/0x21
Complete report #704459
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc00adc0e
Build id: 3.10.0-957.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc00ad5f2
Build id: 3.10.0-957.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc00ae0dd
Build id: 3.10.0-957.el7.x86_64
0x27d
4
? __queue_work
vmlinux 0xffffffff8d6b7c0c
Build id: 3.10.0-957.el7.x86_64
0x13c
5
net_rx_action
vmlinux 0xffffffff8dc39e9f
Build id: 3.10.0-957.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffff8d6a0f05
Build id: 3.10.0-957.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffff8d6a10c8
Build id: 3.10.0-957.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffff8d6ca5c4
Build id: 3.10.0-957.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffff8d6ca480
Build id: 3.10.0-957.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffff8d6c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffff8d6c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffff8dd74c37
Build id: 3.10.0-957.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffff8d6c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ksoftirqd/0:3]
Modules linked in: binfmt_misc veth xt_nat xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock fuse vmhgfs(OE) overlay(T) sunrpc snd_seq_midi snd_seq_midi_event sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel snd_ens1371 lrw snd_rawmidi gf128mul ppdev glue_helper snd_ac97_codec ablk_helper cryptd ac97_bus vmw_balloon snd_seq snd_seq_device snd_pcm snd_timer snd joydev soundcore pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata mptspi scsi_transport_spi mptscsih e1000 mptbase nfit libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3 Comm: ksoftirqd/0 Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8ce6fa35a080 ti: ffff8ce6f9c58000 task.ti: ffff8ce6f9c58000
RIP: 0010:[<ffffffffc00adc0e>]  [<ffffffffc00adc0e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff8ce6f9c5bbd8  EFLAGS: 00000286
RAX: ffff981301942818 RBX: ffffffff8dc394f9 RCX: 00000000000005f2
RDX: 00000000000000b5 RSI: ffff8ce5b5287180 RDI: ffff8ce9ac8868c0
RBP: ffff8ce6f9c5bc30 R08: ffffffff8de1dae0 R09: ffff8ce6f9cc5140
R10: 0000000000000001 R11: 0000000000000400 R12: ffffffff8e363f20
R13: ffff8ce9ac887238 R14: ffff8ce7d0fb7d00 R15: 0000000051496a20
FS:  0000000000000000(0000) GS:ffff8ce9ad600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000007a1f29000 CR3: 00000000b6e16000 CR4: 00000000003607f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc00ad5f2>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc00ae0dd>] e1000_clean+0x27d/0x8e0 [e1000]
[<ffffffff8d6b7c0c>] ? __queue_work+0x13c/0x3e0
[<ffffffff8dc39e9f>] net_rx_action+0x26f/0x390
[<ffffffff8d6a0f05>] __do_softirq+0xf5/0x280
[<ffffffff8d6a10c8>] run_ksoftirqd+0x38/0x50
[<ffffffff8d6ca5c4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff8d6ca480>] ? lg_double_unlock+0x40/0x40
[<ffffffff8d6c1c31>] kthread+0xd1/0xe0
[<ffffffff8d6c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff8dd74c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8d6c1b60>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #673256
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc03d8c2e
Build id: 3.10.0-1160.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc03d8612
Build id: 3.10.0-1160.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc03d9100
Build id: 3.10.0-1160.el7.x86_64
0x280
4
net_rx_action
vmlinux 0xffffffffb54555bf
Build id: 3.10.0-1160.el7.x86_64
0x26f
5
__do_softirq
vmlinux 0xffffffffb4ea4b95
Build id: 3.10.0-1160.el7.x86_64
0xf5
6
run_ksoftirqd
vmlinux 0xffffffffb4ea4d58
Build id: 3.10.0-1160.el7.x86_64
0x38
7
smpboot_thread_fn
vmlinux 0xffffffffb4ece674
Build id: 3.10.0-1160.el7.x86_64
0x144
8
? lg_double_unlock
vmlinux 0xffffffffb4ece530
Build id: 3.10.0-1160.el7.x86_64
0x40
9
kthread
vmlinux 0xffffffffb4ec5c21
Build id: 3.10.0-1160.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffffb4ec5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffffb5593df7
Build id: 3.10.0-1160.el7.x86_64
0x21
12
? insert_kthread_work
vmlinux 0xffffffffb4ec5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 28s! [ksoftirqd/1:14]
Modules linked in: macvlan br_netfilter overlay(T) tcp_lp tun devlink bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi kvm_intel snd_ens1371 kvm snd_rawmidi irqbypass crc32_pclmul snd_ac97_codec ac97_bus ghash_clmulni_intel ppdev snd_seq cryptd vmw_balloon pcspkr joydev snd_seq_device snd_pcm snd_timer snd soundcore sg parport_pc parport vmw_vmci i2c_piix4 xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops nfit ttm libnvdimm drm crct10dif_pclmul crct10dif_common ata_piix crc32c_intel libata serio_raw e1000 mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ip_tables]
CPU: 1 PID: 14 Comm: ksoftirqd/1 Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffffa0ce397a0000 ti: ffffa0ce3979c000 task.ti: ffffa0ce3979c000
RIP: 0010:[<ffffffffc03d8c2e>]  [<ffffffffc03d8c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffffa0ce3979fbd8  EFLAGS: 00000286
RAX: ffffabd800782818 RBX: ffffffffb5454c19 RCX: 0000000000000000
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [pulseaudio:9487]
RDX: 00000000000000f7 RSI: ffffa0ce3fb5e098 RDI: 00000000000005f2
RBP: ffffa0ce3979fc30 R08: 00000000000005f2 R09: 0000000070811d31
R10: ffffa0ce30811740 R11: fffff15d80d7f800 R12: ffffffffb5b69ac0
R13: ffffa0ce35911238 R14: ffffa0cdf5fe1600 R15: 00000000dcb014e0
FS:  0000000000000000(0000) GS:ffffa0ce3b640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007efd91ae9000 CR3: 000000003a0ac000 CR4: 00000000000607e0
Call Trace:
[<ffffffffc03d8612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc03d9100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffb54555bf>] net_rx_action+0x26f/0x390
[<ffffffffb4ea4b95>] __do_softirq+0xf5/0x280
[<ffffffffb4ea4d58>] run_ksoftirqd+0x38/0x50
[<ffffffffb4ece674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffb4ece530>] ? lg_double_unlock+0x40/0x40
[<ffffffffb4ec5c21>] kthread+0xd1/0xe0
[<ffffffffb4ec5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffb5593df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb4ec5b50>] ? insert_kthread_work+0x40/0x40
Complete report #585630
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc02c9c2e
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc02c9612
Build id: 3.10.0-1160.76.1.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc02ca100
Build id: 3.10.0-1160.76.1.el7.x86_64
0x280
4
? mod_timer
vmlinux 0xffffffffa68aeefc
Build id: 3.10.0-1160.76.1.el7.x86_64
0x10c
5
net_rx_action
vmlinux 0xffffffffa6e5a1bf
Build id: 3.10.0-1160.76.1.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffffa68a4d85
Build id: 3.10.0-1160.76.1.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffffa68a4f48
Build id: 3.10.0-1160.76.1.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffffa68ce9e4
Build id: 3.10.0-1160.76.1.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffffa68ce8a0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffffa68c5f91
Build id: 3.10.0-1160.76.1.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffffa68c5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffffa6f99df7
Build id: 3.10.0-1160.76.1.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffffa68c5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ksoftirqd/0:6]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event sb_edac snd_ens1371 snd_rawmidi btusb iosf_mbi btrtl btbcm snd_ac97_codec ac97_bus btintel snd_seq bluetooth crc32_pclmul ghash_clmulni_intel aesni_intel snd_seq_device vmw_balloon
lrw gf128mul rfkill ppdev glue_helper pcspkr ablk_helper cryptd snd_pcm snd_timer joydev sg snd soundcore parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ttm crc32c_intel nfit ata_piix drm libnvdimm mptspi scsi_transport_spi libata serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks synosnap(OE) fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff95dc2c4d5280 ti: ffff95dc2c594000 task.ti: ffff95dc2c594000
RIP: 0010:[<ffffffffc02c9c2e>]  [<ffffffffc02c9c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff95dc2c597bd8  EFLAGS: 00000286
RAX: ffffa5a0031c2818 RBX: ffffffffa6e59819 RCX: 00000000000005f2
RDX: 0000000000000032 RSI: ffff95e2dea6c440 RDI: ffff95e2dda348c0
RBP: ffff95dc2c597c30 R08: 000000000001f120 R09: ffffffffa6e416e7
R10: ffff95e2dd61f120 R11: ffffd66660520900 R12: ffffffffa7569f60
R13: ffff95e2dda35238 R14: ffff95e2d4824f00 R15: 00000000ff6c4963
FS:  0000000000000000(0000) GS:ffff95e2dd600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000002054c30 CR3: 000000081c832000 CR4: 00000000003607f0
Call Trace:
[<ffffffffc02c9612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc02ca100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffa68aeefc>] ? mod_timer+0x10c/0x230
[<ffffffffa6e5a1bf>] net_rx_action+0x26f/0x390
[<ffffffffa68a4d85>] __do_softirq+0xf5/0x280
[<ffffffffa68a4f48>] run_ksoftirqd+0x38/0x50
[<ffffffffa68ce9e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa68ce8a0>] ? lg_double_unlock+0x40/0x40
[<ffffffffa68c5f91>] kthread+0xd1/0xe0
[<ffffffffa68c5ec0>] ? insert_kthread_work+0x40/0x40
[<ffffffffa6f99df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa68c5ec0>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #554408
Frame # Function Binary Source or offset Line
1
e1000_clean
e1000 0xffffffffc021cec2
Build id: 3.10.0-1160.45.1.el7.x86_64
0x42
2
? e1000_alloc_rx_buffers
e1000 0xffffffffc021ce80
Build id: 3.10.0-1160.45.1.el7.x86_64
0x610
3
net_rx_action
vmlinux 0xffffffff91e571cf
Build id: 3.10.0-1160.45.1.el7.x86_64
0x26f
4
__do_softirq
vmlinux 0xffffffff918a4bf5
Build id: 3.10.0-1160.45.1.el7.x86_64
0xf5
5
run_ksoftirqd
vmlinux 0xffffffff918a4db8
Build id: 3.10.0-1160.45.1.el7.x86_64
0x38
6
smpboot_thread_fn
vmlinux 0xffffffff918ce8b4
Build id: 3.10.0-1160.45.1.el7.x86_64
0x144
7
? lg_double_unlock
vmlinux 0xffffffff918ce770
Build id: 3.10.0-1160.45.1.el7.x86_64
0x40
8
kthread
vmlinux 0xffffffff918c5e61
Build id: 3.10.0-1160.45.1.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffff918c5d90
Build id: 3.10.0-1160.45.1.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffff91f95df7
Build id: 3.10.0-1160.45.1.el7.x86_64
0x21
11
? insert_kthread_work
vmlinux 0xffffffff918c5d90
Build id: 3.10.0-1160.45.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 30s! [ksoftirqd/0:6]
Modules linked in: ppp_deflate bsd_comp ppp_async crc_ccitt ppp_generic slhc tcp_lp nls_utf8 isofs vboxsf(OE) nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter binfmt_misc overlay(T) snd_intel8x0 snd_ac97_codec ac97_bus joydev snd_seq ppdev snd_seq_device vboxguest(OE) snd_pcm iosf_mbi sg snd_timer pcspkr snd i2c_piix4 soundcore parport_pc video parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod cdrom crct10dif_generic crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci ata_piix libahci drm libata crc32c_intel serio_raw e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff9eb10544d280 ti: ffff9eb105478000 task.ti: ffff9eb105478000
RIP: 0010:[<ffffffffc021cec2>]  [<ffffffffc021cec2>] e1000_clean+0x42/0x8e0 [e1000]
RSP: 0018:ffff9eb10547bce8  EFLAGS: 00010246
RAX: ffff9eb15a5ba000 RBX: ffff9eb10547bd08 RCX: dead000000000200
RDX: ffff9eb15a5ba8c0 RSI: 0000000000000040 RDI: ffff9eb15a5bab70
RBP: ffff9eb10547bd88 R08: ffff9eb15a5bab70 R09: 0000000000000001
R10: 0000000000000004 R11: 0000000000000005 R12: ffff9eb10547bd08
R13: ffffffff9248e9e8 R14: 0000000000000000 R15: ffff9eb10548fe68
FS:  0000000000000000(0000) GS:ffff9eb15f200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb6c1cd3d60 CR3: 00000000a6abe000 CR4: 00000000000206f0
Call Trace:
[<ffffffffc021ce80>] ? e1000_alloc_rx_buffers+0x610/0x610 [e1000]
[<ffffffff91e571cf>] net_rx_action+0x26f/0x390
[<ffffffff918a4bf5>] __do_softirq+0xf5/0x280
[<ffffffff918a4db8>] run_ksoftirqd+0x38/0x50
[<ffffffff918ce8b4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff918ce770>] ? lg_double_unlock+0x40/0x40
[<ffffffff918c5e61>] kthread+0xd1/0xe0
[<ffffffff918c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffff91f95df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff918c5d90>] ? insert_kthread_work+0x40/0x40
Code: 8b 77 90 48 8b 87 10 01 00 00 65 48 8b 14 25 28 00 00 00 48 89 55 d0 31 d2 48 8d 97 50 fd ff ff 48 89 7d a0 c7 45 cc 00 00 00 00 <45> 8b 7e 1c 48 89 45 98 4d 8b 4e 20 48 89 55 90 48 89 fa 89 75 
Complete report #449996
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc035ac2e
Build id: 3.10.0-1160.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc035a612
Build id: 3.10.0-1160.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc035b100
Build id: 3.10.0-1160.el7.x86_64
0x280
4
? __queue_work
vmlinux 0xffffffff982bbb9c
Build id: 3.10.0-1160.el7.x86_64
0x13c
5
net_rx_action
vmlinux 0xffffffff988555bf
Build id: 3.10.0-1160.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffff982a4b95
Build id: 3.10.0-1160.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffff982a4d58
Build id: 3.10.0-1160.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffff982ce674
Build id: 3.10.0-1160.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffff982ce530
Build id: 3.10.0-1160.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffff982c5c21
Build id: 3.10.0-1160.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffff982c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffff98993df7
Build id: 3.10.0-1160.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffff982c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [ksoftirqd/5:34]
Modules linked in: binfmt_misc tcp_lp nls_utf8 isofs xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc
parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi crct10dif_pclmul crct10dif_common scsi_transport_spi mptscsih crc32c_intel e1000 serio_raw mptbase drm_panel_orientation_quirks floppy fuse
CPU: 5 PID: 34 Comm: ksoftirqd/5 Kdump: loaded Tainted: G        W      ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 06/02/2011
task: ffff91b339be1080 ti: ffff91b339be8000 task.ti: ffff91b339be8000
RIP: 0010:[<ffffffffc035ac2e>]  [<ffffffffc035ac2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff91b339bebbd8  EFLAGS: 00000286
RAX: ffffa2d380542818 RBX: ffffffff98854c19 RCX: 00000000000005f2
RDX: 00000000000000f7 RSI: ffff91b2f480e140 RDI: ffff91b337ff88c0
RBP: ffff91b339bebc30 R08: 000000000001f120 R09: ffffffff9883c6f7
R10: ffff91b33f75f120 R11: ffffda4f41c87880 R12: ffffffff98f69a20
R13: ffff91b337ff9238 R14: ffff91b3321e3b00 R15: 0000000054b402da
FS:  0000000000000000(0000) GS:ffff91b33f740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005563feb7b530 CR3: 000000004a406000 CR4: 00000000000407e0
Call Trace:
[<ffffffffc035a612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc035b100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff982bbb9c>] ? __queue_work+0x13c/0x3f0
[<ffffffff988555bf>] net_rx_action+0x26f/0x390
[<ffffffff982a4b95>] __do_softirq+0xf5/0x280
[<ffffffff982a4d58>] run_ksoftirqd+0x38/0x50
[<ffffffff982ce674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff982ce530>] ? lg_double_unlock+0x40/0x40
[<ffffffff982c5c21>] kthread+0xd1/0xe0
[<ffffffff982c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff98993df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff982c5b50>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #403704
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0249c2e
Build id: 3.10.0-1062.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc0249612
Build id: 3.10.0-1062.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc024a100
Build id: 3.10.0-1062.el7.x86_64
0x280
4
? nf_ct_delete
nf_conntrack 0xffffffffc080b02d
Build id: 3.10.0-1062.el7.x86_64
0x18d
5
net_rx_action
vmlinux 0xffffffffb244e57f
Build id: 3.10.0-1062.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffffb1ea41e5
Build id: 3.10.0-1062.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffffb1ea43a8
Build id: 3.10.0-1062.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffffb1ecdb04
Build id: 3.10.0-1062.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffffb1ecd9c0
Build id: 3.10.0-1062.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffffb1ec50d1
Build id: 3.10.0-1062.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffffb1ec5000
Build id: 3.10.0-1062.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffffb258bd37
Build id: 3.10.0-1062.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffffb1ec5000
Build id: 3.10.0-1062.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ksoftirqd/2:19]
Modules linked in: binfmt_misc vsock_diag tcp_diag udp_diag inet_diag unix_diag veth tcp_lp nls_utf8 isofs fuse xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 snd_seq_midi snd_seq_midi_event ppdev vmw_balloon crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus aesni_intel snd_seq lrw gf128mul glue_helper ablk_helper cryptd snd_seq_device snd_pcm pcspkr snd_timer snd joydev sg soundcore vmw_vmci i2c_piix4 parport_pc
parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ttm crc32c_intel drm nfit libnvdimm ata_piix serio_raw e1000 mptspi scsi_transport_spi mptscsih libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 19 Comm: ksoftirqd/2 Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff916f740d5230 ti: ffff916f740f8000 task.ti: ffff916f740f8000
RIP: 0010:[<ffffffffc0249c2e>]  [<ffffffffc0249c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff916f740fbbd8  EFLAGS: 00000286
RAX: ffffb32143202818 RBX: ffffffffb244dbd9 RCX: 00000000000005f2
RDX: 0000000000000002 RSI: ffff9176180146c0 RDI: ffff91761ce708c0
RBP: ffff916f740fbc30 R08: 000000000001f120 R09: ffffffffb24356e7
R10: ffff91761d69f120 R11: ffffdbb888e9f600 R12: ffffffffb2b65e20
R13: ffff91761ce71238 R14: ffff91703a7dba00 R15: 000000004941e446
FS:  0000000000000000(0000) GS:ffff91761d680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f84ac8d7140 CR3: 0000000035726000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc0249612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc024a100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffc080b02d>] ? nf_ct_delete+0x18d/0x270 [nf_conntrack]
[<ffffffffb244e57f>] net_rx_action+0x26f/0x390
[<ffffffffb1ea41e5>] __do_softirq+0xf5/0x280
[<ffffffffb1ea43a8>] run_ksoftirqd+0x38/0x50
[<ffffffffb1ecdb04>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffb1ecd9c0>] ? lg_double_unlock+0x40/0x40
[<ffffffffb1ec50d1>] kthread+0xd1/0xe0
[<ffffffffb1ec5000>] ? insert_kthread_work+0x40/0x40
[<ffffffffb258bd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb1ec5000>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #399262
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0289c2e
Build id: 3.10.0-1160.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc0289599
Build id: 3.10.0-1160.el7.x86_64
0x3f9
3
e1000_clean
e1000 0xffffffffc028a100
Build id: 3.10.0-1160.el7.x86_64
0x280
4
? __schedule
vmlinux 0xffffffffa2386e82
Build id: 3.10.0-1160.el7.x86_64
0x5a2
5
? update_curr
vmlinux 0xffffffffa1ce43cc
Build id: 3.10.0-1160.el7.x86_64
0x14c
6
? __schedule
vmlinux 0xffffffffa2386e82
Build id: 3.10.0-1160.el7.x86_64
0x5a2
7
net_rx_action
vmlinux 0xffffffffa22555bf
Build id: 3.10.0-1160.el7.x86_64
0x26f
8
__do_softirq
vmlinux 0xffffffffa1ca4b95
Build id: 3.10.0-1160.el7.x86_64
0xf5
9
run_ksoftirqd
vmlinux 0xffffffffa1ca4d58
Build id: 3.10.0-1160.el7.x86_64
0x38
10
smpboot_thread_fn
vmlinux 0xffffffffa1cce674
Build id: 3.10.0-1160.el7.x86_64
0x144
11
? lg_double_unlock
vmlinux 0xffffffffa1cce530
Build id: 3.10.0-1160.el7.x86_64
0x40
12
kthread
vmlinux 0xffffffffa1cc5c21
Build id: 3.10.0-1160.el7.x86_64
0xd1
13
? insert_kthread_work
vmlinux 0xffffffffa1cc5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
14
ret_from_fork_nospec_begin
vmlinux 0xffffffffa2393ddd
Build id: 3.10.0-1160.el7.x86_64
0x7
15
? insert_kthread_work
vmlinux 0xffffffffa1cc5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 24s! [ksoftirqd/4:29]
Modules linked in: tcp_lp nls_utf8 isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm crc32_pclmul ghash_clmulni_intel aesni_intel snd_timer lrw gf128mul snd soundcore glue_helper ablk_helper sg cryptd ppdev joydev pcspkr vmw_balloon parport_pc parport i2c_piix4 vmw_vmci ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm crc32c_intel ahci serio_raw libahci ata_piix libata nfit mptspi libnvdimm e1000 scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 4 PID: 29 Comm: ksoftirqd/4 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff959784f81080 ti: ffff959784f90000 task.ti: ffff959784f90000
RIP: 0010:[<ffffffffc0289c2e>]  [<ffffffffc0289c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff959784f93bd8  EFLAGS: 00000286
RAX: ffffaecd80a82818 RBX: ffffffffa2254c19 RCX: 0000000000000000
RDX: 00000000000000e5 RSI: ffff9597bac2a098 RDI: 00000000000005f2
RBP: ffff959784f93c30 R08: 00000000000005f2 R09: 0000000027832c31
R10: ffff959727832640 R11: 0000000000000001 R12: ffffffffa2969ac0
R13: ffff959781bbd238 R14: ffff959753a58b00 R15: 000000006f529c77
FS:  0000000000000000(0000) GS:ffff9597ba700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f21d455e280 CR3: 00000000ba0dc000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffffc0289599>] e1000_clean_rx_irq+0x3f9/0x510 [e1000]
[<ffffffffc028a100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffa2386e82>] ? __schedule+0x5a2/0x860
[<ffffffffa1ce43cc>] ? update_curr+0x14c/0x1e0
[<ffffffffa2386e82>] ? __schedule+0x5a2/0x860
[<ffffffffa22555bf>] net_rx_action+0x26f/0x390
[<ffffffffa1ca4b95>] __do_softirq+0xf5/0x280
[<ffffffffa1ca4d58>] run_ksoftirqd+0x38/0x50
[<ffffffffa1cce674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa1cce530>] ? lg_double_unlock+0x40/0x40
[<ffffffffa1cc5c21>] kthread+0xd1/0xe0
[<ffffffffa1cc5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffa2393ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffa1cc5b50>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #371438
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0229c2e
Build id: 3.10.0-1160.41.1.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc0229612
Build id: 3.10.0-1160.41.1.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc022a100
Build id: 3.10.0-1160.41.1.el7.x86_64
0x280
4
net_rx_action
vmlinux 0xffffffffb8c571cf
Build id: 3.10.0-1160.41.1.el7.x86_64
0x26f
5
__do_softirq
vmlinux 0xffffffffb86a4bf5
Build id: 3.10.0-1160.41.1.el7.x86_64
0xf5
6
run_ksoftirqd
vmlinux 0xffffffffb86a4db8
Build id: 3.10.0-1160.41.1.el7.x86_64
0x38
7
smpboot_thread_fn
vmlinux 0xffffffffb86ce8b4
Build id: 3.10.0-1160.41.1.el7.x86_64
0x144
8
? lg_double_unlock
vmlinux 0xffffffffb86ce770
Build id: 3.10.0-1160.41.1.el7.x86_64
0x40
9
kthread
vmlinux 0xffffffffb86c5e61
Build id: 3.10.0-1160.41.1.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffffb86c5d90
Build id: 3.10.0-1160.41.1.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffffb8d95df7
Build id: 3.10.0-1160.41.1.el7.x86_64
0x21
12
? insert_kthread_work
vmlinux 0xffffffffb86c5d90
Build id: 3.10.0-1160.41.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ksoftirqd/1:14]
Modules linked in: lp veeamsnap(OE) tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 iosf_mbi crc32_pclmul snd_rawmidi snd_ac97_codec ghash_clmulni_intel aesni_intel ac97_bus lrw snd_seq gf128mul glue_helper ablk_helper snd_seq_device snd_pcm
ppdev cryptd pcspkr snd_timer joydev snd soundcore vmw_balloon sg parport_pc parport i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx nfit libnvdimm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm crc32c_intel ata_piix serio_raw mptspi scsi_transport_spi libata e1000 mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 1 PID: 14 Comm: ksoftirqd/1 Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.41.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff944bbb438000 ti: ffff944bbb434000 task.ti: ffff944bbb434000
RIP: 0010:[<ffffffffc0229c2e>]  [<ffffffffc0229c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff944bbb437bd8  EFLAGS: 00000286
RAX: ffffaeeb40f82818 RBX: ffffffffb8c56829 RCX: ffffffffb89b1fb0
RDX: 0000000000000039 RSI: 000000007a31d340 RDI: 00000000000005f2
RBP: ffff944bbb437c30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff944aba31d340 R11: ffffd9e385cabe00 R12: ffffffffb9369c60
R13: ffff944bf863f238 R14: ffff944bb2afb000 R15: 00000000ae5b41b0
FS:  0000000000000000(0000) GS:ffff944bf7640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7677054560 CR3: 00000001b5130000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc0229612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc022a100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffb8c571cf>] net_rx_action+0x26f/0x390
[<ffffffffb86a4bf5>] __do_softirq+0xf5/0x280
[<ffffffffb86a4db8>] run_ksoftirqd+0x38/0x50
[<ffffffffb86ce8b4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffb86ce770>] ? lg_double_unlock+0x40/0x40
[<ffffffffb86c5e61>] kthread+0xd1/0xe0
[<ffffffffb86c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffb8d95df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb86c5d90>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #360285
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0153c2e
Build id: 3.10.0-1160.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc0153612
Build id: 3.10.0-1160.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc0154100
Build id: 3.10.0-1160.el7.x86_64
0x280
4
? __schedule
vmlinux 0xffffffff85986e82
Build id: 3.10.0-1160.el7.x86_64
0x5a2
5
net_rx_action
vmlinux 0xffffffff858555bf
Build id: 3.10.0-1160.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffff852a4b95
Build id: 3.10.0-1160.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffff852a4d58
Build id: 3.10.0-1160.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffff852ce674
Build id: 3.10.0-1160.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffff852ce530
Build id: 3.10.0-1160.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffff852c5c21
Build id: 3.10.0-1160.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffff852c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffff85993df7
Build id: 3.10.0-1160.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffff852c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ksoftirqd/0:6]
Modules linked in: tcp_lp rfcomm xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi iosf_mbi snd_ac97_codec crc32_pclmul ghash_clmulni_intel ac97_bus aesni_intel snd_seq ppdev lrw btusb gf128mul glue_helper snd_seq_device btrtl snd_pcm btbcm ablk_helper
btintel cryptd bluetooth snd_timer vmw_balloon snd pcspkr joydev soundcore sg rfkill i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw mptspi scsi_transport_spi mptscsih drm ata_piix mptbase e1000 libata nfit libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Not tainted 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9a84f73b5280 ti: ffff9a84f77f4000 task.ti: ffff9a84f77f4000
RIP: 0010:[<ffffffffc0153c2e>]  [<ffffffffc0153c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff9a84f77f7bd8  EFLAGS: 00000286
RAX: ffffa70600382818 RBX: 0000000000000780 RCX: 0000000000000000
RDX: 00000000000000e7 RSI: ffff9a84f6c09098 RDI: 00000000000005f2
RBP: ffff9a84f77f7c30 R08: 00000000000005f2 R09: 00000000082b42b1
R10: ffff9a84882b3cc0 R11: ffffd01701d69f00 R12: 000000008c002157
R13: 000000000000004e R14: ffff9a84fffd9800 R15: ffff9a84fffdb000
FS:  0000000000000000(0000) GS:ffff9a84fb600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055e61a865000 CR3: 0000000071ac8000 CR4: 00000000001607f0
Call Trace:
[<ffffffffc0153612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc0154100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff85986e82>] ? __schedule+0x5a2/0x860
[<ffffffff858555bf>] net_rx_action+0x26f/0x390
[<ffffffff852a4b95>] __do_softirq+0xf5/0x280
[<ffffffff852a4d58>] run_ksoftirqd+0x38/0x50
[<ffffffff852ce674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff852ce530>] ? lg_double_unlock+0x40/0x40
[<ffffffff852c5c21>] kthread+0xd1/0xe0
[<ffffffff852c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff85993df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff852c5b50>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #310134
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc015fc0e
Build id: 3.10.0-957.12.1.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc015f5f2
Build id: 3.10.0-957.12.1.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc01600dd
Build id: 3.10.0-957.12.1.el7.x86_64
0x27d
4
? update_curr
vmlinux 0xffffffffb7ce015c
Build id: 3.10.0-957.12.1.el7.x86_64
0x14c
5
? __schedule
vmlinux 0xffffffffb8368b5a
Build id: 3.10.0-957.12.1.el7.x86_64
0x5ea
6
net_rx_action
vmlinux 0xffffffffb823af1f
Build id: 3.10.0-957.12.1.el7.x86_64
0x26f
7
__do_softirq
vmlinux 0xffffffffb7ca0f45
Build id: 3.10.0-957.12.1.el7.x86_64
0xf5
8
run_ksoftirqd
vmlinux 0xffffffffb7ca1108
Build id: 3.10.0-957.12.1.el7.x86_64
0x38
9
smpboot_thread_fn
vmlinux 0xffffffffb7cca604
Build id: 3.10.0-957.12.1.el7.x86_64
0x144
10
? lg_double_unlock
vmlinux 0xffffffffb7cca4c0
Build id: 3.10.0-957.12.1.el7.x86_64
0x40
11
kthread
vmlinux 0xffffffffb7cc1c71
Build id: 3.10.0-957.12.1.el7.x86_64
0xd1
12
? insert_kthread_work
vmlinux 0xffffffffb7cc1ba0
Build id: 3.10.0-957.12.1.el7.x86_64
0x40
13
ret_from_fork_nospec_begin
vmlinux 0xffffffffb8375bdd
Build id: 3.10.0-957.12.1.el7.x86_64
0x7
14
? insert_kthread_work
vmlinux 0xffffffffb7cc1ba0
Build id: 3.10.0-957.12.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 39s! [ksoftirqd/0:3]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink binfmt_misc ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi snd_ens1371 kvm_intel kvm snd_rawmidi snd_ac97_codec irqbypass ac97_bus ppdev crc32_pclmul snd_seq ghash_clmulni_intel snd_seq_device snd_pcm aesni_intel
lrw gf128mul snd_timer glue_helper vmw_balloon ablk_helper cryptd snd soundcore pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm mptspi ata_piix serio_raw scsi_transport_spi e1000 drm_panel_orientation_quirks mptscsih mptbase libata nfit libnvdimm
CPU: 0 PID: 3 Comm: ksoftirqd/0 Kdump: loaded Tainted: G             L ------------   3.10.0-957.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9fcb3cf72080 ti: ffff9fcb3c800000 task.ti: ffff9fcb3c800000
RIP: 0010:[<ffffffffc015fc0e>]  [<ffffffffc015fc0e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff9fcb3c803bd8  EFLAGS: 00000286
RAX: ffffb71800d02818 RBX: 00000000000005f2 RCX: ffffffffb7fa0870
RDX: 000000000000004c RSI: 00000000be4cf800 RDI: 00000000000005f2
RBP: ffff9fcb3c803c30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff9fcbb1edadc0 R11: ffffdd44817e7100 R12: 00000001f1edadc0
R13: 00000002f55d9238 R14: ffffffffb7fa0b6b R15: ffff9fcb3c803b90
FS:  0000000000000000(0000) GS:ffff9fcbf5600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6b784c7380 CR3: 0000000223a86000 CR4: 00000000003607f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc015f5f2>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc01600dd>] e1000_clean+0x27d/0x8e0 [e1000]
[<ffffffffb7ce015c>] ? update_curr+0x14c/0x1e0
[<ffffffffb8368b5a>] ? __schedule+0x5ea/0x890
[<ffffffffb823af1f>] net_rx_action+0x26f/0x390
[<ffffffffb7ca0f45>] __do_softirq+0xf5/0x280
[<ffffffffb7ca1108>] run_ksoftirqd+0x38/0x50
[<ffffffffb7cca604>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffb7cca4c0>] ? lg_double_unlock+0x40/0x40
[<ffffffffb7cc1c71>] kthread+0xd1/0xe0
[<ffffffffb7cc1ba0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb8375bdd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb7cc1ba0>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #309307
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0303c2e
Build id: 3.10.0-1127.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc0303612
Build id: 3.10.0-1127.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc0304100
Build id: 3.10.0-1127.el7.x86_64
0x280
4
? nf_ct_delete
nf_conntrack 0xffffffffc079702d
Build id: 3.10.0-1127.el7.x86_64
0x18d
5
? tcp_init_xmit_timers
vmlinux 0xffffffff8dac0770
Build id: 3.10.0-1127.el7.x86_64
0x30
6
net_rx_action
vmlinux 0xffffffff8da545bf
Build id: 3.10.0-1127.el7.x86_64
0x26f
7
__do_softirq
vmlinux 0xffffffff8d4a5695
Build id: 3.10.0-1127.el7.x86_64
0xf5
8
run_ksoftirqd
vmlinux 0xffffffff8d4a5858
Build id: 3.10.0-1127.el7.x86_64
0x38
9
smpboot_thread_fn
vmlinux 0xffffffff8d4cf0e4
Build id: 3.10.0-1127.el7.x86_64
0x144
10
? lg_double_unlock
vmlinux 0xffffffff8d4cefa0
Build id: 3.10.0-1127.el7.x86_64
0x40
11
kthread
vmlinux 0xffffffff8d4c6691
Build id: 3.10.0-1127.el7.x86_64
0xd1
12
? insert_kthread_work
vmlinux 0xffffffff8d4c65c0
Build id: 3.10.0-1127.el7.x86_64
0x40
13
ret_from_fork_nospec_begin
vmlinux 0xffffffff8db92d1d
Build id: 3.10.0-1127.el7.x86_64
0x7
14
? insert_kthread_work
vmlinux 0xffffffff8d4c65c0
Build id: 3.10.0-1127.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 50s! [ksoftirqd/0:6]
Modules linked in: tcp_lp nls_utf8 isofs fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec crc32_pclmul ghash_clmulni_intel ac97_bus snd_seq aesni_intel snd_seq_device lrw ppdev gf128mul snd_pcm glue_helper ablk_helper vmw_balloon cryptd snd_timer pcspkr snd sg joydev soundcore parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm nfit ata_piix drm crct10dif_pclmul libnvdimm crct10dif_common libata crc32c_intel mptspi e1000 scsi_transport_spi serio_raw mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G             L ------------   3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9ffbba635230 ti: ffff9ffbba6f4000 task.ti: ffff9ffbba6f4000
RIP: 0010:[<ffffffffc0303c2e>]  [<ffffffffc0303c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0000:ffff9ffbba6f7bd8  EFLAGS: 00000286
RAX: ffffb81440842818 RBX: ffffffff8da53c19 RCX: 0000000000000000
RDX: 0000000000000045 RSI: ffff9ffbb9482098 RDI: 00000000000005f2
RBP: ffff9ffbba6f7c30 R08: 00000000000005f2 R09: 000000005391ca31
R10: ffff9ffb9391c440 R11: ffffdb1d80109b00 R12: ffffffff8e169660
R13: ffff9ffbb81e3238 R14: ffff9ffb4426c200 R15: 0000000003cf5748
FS:  0000000000000000(0000) GS:ffff9ffbbb600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe96e00e330 CR3: 00000000647d6000 CR4: 00000000007607f0
PKRU: 00000000
Call Trace:
[<ffffffffc0303612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc0304100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffc079702d>] ? nf_ct_delete+0x18d/0x270 [nf_conntrack]
[<ffffffff8dac0770>] ? tcp_init_xmit_timers+0x30/0x30
[<ffffffff8da545bf>] net_rx_action+0x26f/0x390
[<ffffffff8d4a5695>] __do_softirq+0xf5/0x280
[<ffffffff8d4a5858>] run_ksoftirqd+0x38/0x50
[<ffffffff8d4cf0e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff8d4cefa0>] ? lg_double_unlock+0x40/0x40
[<ffffffff8d4c6691>] kthread+0xd1/0xe0
[<ffffffff8d4c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff8db92d1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff8d4c65c0>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #287039
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc02b4c2e
Build id: 3.10.0-1160.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc02b4612
Build id: 3.10.0-1160.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc02b5100
Build id: 3.10.0-1160.el7.x86_64
0x280
4
? update_curr
vmlinux 0xffffffff8cce43cc
Build id: 3.10.0-1160.el7.x86_64
0x14c
5
net_rx_action
vmlinux 0xffffffff8d2555bf
Build id: 3.10.0-1160.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffff8cca4b95
Build id: 3.10.0-1160.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffff8cca4d58
Build id: 3.10.0-1160.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffff8ccce674
Build id: 3.10.0-1160.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffff8ccce530
Build id: 3.10.0-1160.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffff8ccc5c21
Build id: 3.10.0-1160.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffff8ccc5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffff8d393df7
Build id: 3.10.0-1160.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffff8ccc5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 29s! [ksoftirqd/0:6]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype tun br_netfilter devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus iosf_mbi snd_seq crc32_pclmul snd_seq_device ghash_clmulni_intel
snd_pcm ppdev aesni_intel lrw snd_timer gf128mul glue_helper vmw_balloon ablk_helper cryptd snd soundcore joydev pcspkr vmw_vmci sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix mptspi nfit scsi_transport_spi libata libnvdimm serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff95c5773b5280 ti: ffff95c5777f4000 task.ti: ffff95c5777f4000
RIP: 0010:[<ffffffffc02b4c2e>]  [<ffffffffc02b4c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff95c5777f7bd8  EFLAGS: 00000286
RAX: ffffbe4940402818 RBX: ffffffff8d254c19 RCX: 00000000000005f2
RDX: 00000000000000c5 RSI: ffff95c5359c4e00 RDI: ffff95c5769be8c0
RBP: ffff95c5777f7c30 R08: ffff95c5777f7950 R09: ffffffff8ce63e50
R10: ffff95c57b61f120 R11: fffff934c039f500 R12: 0000000000000608
R13: ffff95c5769be000 R14: ffff95c568138580 R15: ffff95c5769bf238
FS:  0000000000000000(0000) GS:ffff95c57b600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000020fc97fcf000 CR3: 000000006420e000 CR4: 00000000003607f0
Call Trace:
[<ffffffffc02b4612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc02b5100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff8cce43cc>] ? update_curr+0x14c/0x1e0
[<ffffffff8d2555bf>] net_rx_action+0x26f/0x390
[<ffffffff8cca4b95>] __do_softirq+0xf5/0x280
[<ffffffff8cca4d58>] run_ksoftirqd+0x38/0x50
[<ffffffff8ccce674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff8ccce530>] ? lg_double_unlock+0x40/0x40
[<ffffffff8ccc5c21>] kthread+0xd1/0xe0
[<ffffffff8ccc5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff8d393df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8ccc5b50>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #274303
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0281c2e
Build id: 3.10.0-1160.45.1.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc0281612
Build id: 3.10.0-1160.45.1.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc0282100
Build id: 3.10.0-1160.45.1.el7.x86_64
0x280
4
? sched_clock_cpu
vmlinux 0xffffffff9f4de305
Build id: 3.10.0-1160.45.1.el7.x86_64
0x85
5
net_rx_action
vmlinux 0xffffffff9fa571cf
Build id: 3.10.0-1160.45.1.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffff9f4a4bf5
Build id: 3.10.0-1160.45.1.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffff9f4a4db8
Build id: 3.10.0-1160.45.1.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffff9f4ce8b4
Build id: 3.10.0-1160.45.1.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffff9f4ce770
Build id: 3.10.0-1160.45.1.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffff9f4c5e61
Build id: 3.10.0-1160.45.1.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffff9f4c5d90
Build id: 3.10.0-1160.45.1.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffff9fb95df7
Build id: 3.10.0-1160.45.1.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffff9f4c5d90
Build id: 3.10.0-1160.45.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 63s! [ksoftirqd/0:6]
Modules linked in: binfmt_misc tun xt_hl ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device aesni_intel snd_pcm lrw gf128mul snd_timer snd glue_helper ablk_helper cryptd ppdev soundcore vmw_balloon pcspkr joydev vmw_vmci sg i2c_piix4 parport_pc parport
ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea crct10dif_pclmul sysfillrect crct10dif_common sysimgblt crc32c_intel fb_sys_fops ttm drm ata_piix nfit libata libnvdimm mptspi scsi_transport_spi serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Tainted: G             L ------------   3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffffa07ff24b5280 ti: ffffa07ff256c000 task.ti: ffffa07ff256c000
RIP: 0010:[<ffffffffc0281c2e>]  [<ffffffffc0281c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffffa07ff256fbd8  EFLAGS: 00000286
RAX: ffffb7f9c3482818 RBX: ffffffff9fa56829 RCX: 00000000000005f2
RDX: 00000000000000a1 RSI: ffffa083a4b259c0 RDI: ffffa083a805e8c0
RBP: ffffa07ff256fc30 R08: 000000000001f120 R09: ffffffff9fa3e6f7
R10: ffffa083a961f120 R11: ffffea284fd99000 R12: ffffffffa0169bc0
R13: ffffa083a805f238 R14: ffffa08276640000 R15: 000000007bd4f945
FS:  0000000000000000(0000) GS:ffffa083a9600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f94906e7008 CR3: 0000000501ab8000 CR4: 00000000000007f0
Call Trace:
[<ffffffffc0281612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc0282100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff9f4de305>] ? sched_clock_cpu+0x85/0xc0
[<ffffffff9fa571cf>] net_rx_action+0x26f/0x390
[<ffffffff9f4a4bf5>] __do_softirq+0xf5/0x280
[<ffffffff9f4a4db8>] run_ksoftirqd+0x38/0x50
[<ffffffff9f4ce8b4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff9f4ce770>] ? lg_double_unlock+0x40/0x40
[<ffffffff9f4c5e61>] kthread+0xd1/0xe0
[<ffffffff9f4c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffff9fb95df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9f4c5d90>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #240347
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0416c1e
Build id: 3.10.0-862.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc0416602
Build id: 3.10.0-862.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc04170eb
Build id: 3.10.0-862.el7.x86_64
0x27b
4
? tcp_init_xmit_timers
vmlinux 0xffffffffbbe53000
Build id: 3.10.0-862.el7.x86_64
0x30
5
net_rx_action
vmlinux 0xffffffffbbdec3af
Build id: 3.10.0-862.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffffbb89a945
Build id: 3.10.0-862.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffffbb89ab08
Build id: 3.10.0-862.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffffbb8c35e4
Build id: 3.10.0-862.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffffbb8c34a0
Build id: 3.10.0-862.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffffbb8bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffffbb8bad60
Build id: 3.10.0-862.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffffbbf1f637
Build id: 3.10.0-862.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffffbb8bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 30s! [ksoftirqd/0:3]
Modules linked in: rpcsec_gss_krb5 tcp_lp fuse veth xt_set xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net ip_set nf_conntrack_netlink tcp_diag nfnetlink inet_diag xt_statistic xt_nat ip_vs_sh ip_vs_wrr ip_vs_rr ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark xt_addrtype xt_CHECKSUM iptable_mangle br_netfilter ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge ip_vs stp llc nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink cfg80211 rfkill vmw_vsock_vmci_transport vsock overlay(T) snd_seq_midi snd_seq_midi_event iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw snd_ens1371 gf128mul
vmw_balloon glue_helper snd_rawmidi ablk_helper cryptd snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm joydev pcspkr snd_timer nfit snd sg vmw_vmci soundcore libnvdimm i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix mptspi scsi_transport_spi libata e1000 mptscsih serio_raw mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3 Comm: ksoftirqd/0 Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff9b92f6469fa0 ti: ffff9b92f648c000 task.ti: ffff9b92f648c000
RIP: 0010:[<ffffffffc0416c1e>]  [<ffffffffc0416c1e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff9b92f648fbd8  EFLAGS: 00000286
RAX: ffffa79983482818 RBX: 00000000000005f2 RCX: ffffffffbbb73b20
RDX: 000000000000009c RSI: 00000000bdd8d800 RDI: 00000000000005f2
RBP: ffff9b92f648fc30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff9b9560ec3cc0 R11: fffff2e09297af00 R12: 00000003e0ec3cc0
R13: 00000002a4d6b238 R14: ffffffffbbb73e1b R15: ffff9b92f648fb90
FS:  0000000000000000(0000) GS:ffff9b97a5600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0eb12aeb00 CR3: 0000000414e12000 CR4: 00000000003607f0
Call Trace:
[<ffffffffc0416602>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc04170eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffffbbe53000>] ? tcp_init_xmit_timers+0x30/0x30
[<ffffffffbbdec3af>] net_rx_action+0x26f/0x390
[<ffffffffbb89a945>] __do_softirq+0xf5/0x280
[<ffffffffbb89ab08>] run_ksoftirqd+0x38/0x50
[<ffffffffbb8c35e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffbb8c34a0>] ? lg_double_unlock+0x40/0x40
[<ffffffffbb8bae31>] kthread+0xd1/0xe0
[<ffffffffbb8bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffbbf1f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffbb8bad60>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #229940
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc04acc1e
Build id: 3.10.0-862.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc04ac602
Build id: 3.10.0-862.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc04ad0eb
Build id: 3.10.0-862.el7.x86_64
0x27b
4
? update_curr
vmlinux 0xffffffff9fad816c
Build id: 3.10.0-862.el7.x86_64
0x14c
5
? account_entity_dequeue
vmlinux 0xffffffff9fad4ade
Build id: 3.10.0-862.el7.x86_64
0xae
6
net_rx_action
vmlinux 0xffffffff9ffec3af
Build id: 3.10.0-862.el7.x86_64
0x26f
7
__do_softirq
vmlinux 0xffffffff9fa9a945
Build id: 3.10.0-862.el7.x86_64
0xf5
8
run_ksoftirqd
vmlinux 0xffffffff9fa9ab08
Build id: 3.10.0-862.el7.x86_64
0x38
9
smpboot_thread_fn
vmlinux 0xffffffff9fac35e4
Build id: 3.10.0-862.el7.x86_64
0x144
10
? lg_double_unlock
vmlinux 0xffffffff9fac34a0
Build id: 3.10.0-862.el7.x86_64
0x40
11
kthread
vmlinux 0xffffffff9fabae31
Build id: 3.10.0-862.el7.x86_64
0xd1
12
? insert_kthread_work
vmlinux 0xffffffff9fabad60
Build id: 3.10.0-862.el7.x86_64
0x40
13
ret_from_fork_nospec_begin
vmlinux 0xffffffffa011f637
Build id: 3.10.0-862.el7.x86_64
0x21
14
? insert_kthread_work
vmlinux 0xffffffff9fabad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 45s! [ksoftirqd/0:3]
Modules linked in: esn_cfs(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm coretemp snd_timer snd soundcore iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_balloon parport_pc parport shpchp vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul
syscopyarea crct10dif_common sysfillrect sysimgblt fb_sys_fops crc32c_intel ttm ata_piix drm mptspi serio_raw scsi_transport_spi mptscsih e1000 mptbase libata i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W  OEL ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2012
task: ffff8870bb5a1fa0 ti: ffff8870bb600000 task.ti: ffff8870bb600000
RIP: 0010:[<ffffffffc04acc1e>]  [<ffffffffc04acc1e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff8870bb603bd8  EFLAGS: 00000286
RAX: ffffb426003c2818 RBX: ffffffff9ffeb9f9 RCX: 0000000000000000
RDX: 00000000000000aa RSI: ffff8870bac84098 RDI: 00000000000005f2
RBP: ffff8870bb603c30 R08: 00000000000005f2 R09: 00000000217ed931
R10: ffff8870617ed340 R11: ffffdcbb01493900 R12: ffffffffa074d7a0
R13: ffff8870b9f71238 R14: ffff8870924e5200 R15: 000000003e9018cd
FS:  0000000000000000(0000) GS:ffff8870bc600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0390001226 CR3: 000000007a274000 CR4: 00000000001607f0
Call Trace:
[<ffffffffc04ac602>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc04ad0eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffff9fad816c>] ? update_curr+0x14c/0x1e0
[<ffffffff9fad4ade>] ? account_entity_dequeue+0xae/0xd0
[<ffffffff9ffec3af>] net_rx_action+0x26f/0x390
[<ffffffff9fa9a945>] __do_softirq+0xf5/0x280
[<ffffffff9fa9ab08>] run_ksoftirqd+0x38/0x50
[<ffffffff9fac35e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff9fac34a0>] ? lg_double_unlock+0x40/0x40
[<ffffffff9fabae31>] kthread+0xd1/0xe0
[<ffffffff9fabad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa011f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9fabad60>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #191277
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc014dc1e
Build id: 3.10.0-862.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc014d602
Build id: 3.10.0-862.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc014e0eb
Build id: 3.10.0-862.el7.x86_64
0x27b
4
? nf_ct_delete
nf_conntrack 0xffffffffc071604d
Build id: 3.10.0-862.el7.x86_64
0x18d
5
net_rx_action
vmlinux 0xffffffff819ec3af
Build id: 3.10.0-862.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffff8149a945
Build id: 3.10.0-862.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffff8149ab08
Build id: 3.10.0-862.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffff814c35e4
Build id: 3.10.0-862.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffff814c34a0
Build id: 3.10.0-862.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffff814bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffff814bad60
Build id: 3.10.0-862.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffff81b1f637
Build id: 3.10.0-862.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffff814bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 80s! [ksoftirqd/2:19]
Modules linked in: xt_set xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net ip_set veth nf_conntrack_netlink nfnetlink xt_nat xt_statistic ip_vs_sh ip_vs_wrr ip_vs_rr rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip_vs ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul snd_ens1371 snd_rawmidi ghash_clmulni_intel snd_ac97_codec ac97_bus aesni_intel snd_seq lrw snd_seq_device
ppdev gf128mul snd_pcm glue_helper ablk_helper vmw_balloon cryptd pcspkr joydev snd_timer snd nfit soundcore sg vmw_vmci libnvdimm i2c_piix4 parport_pc parport shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul mptspi crct10dif_common drm scsi_transport_spi mptscsih crc32c_intel serio_raw mptbase libata e1000 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 19 Comm: ksoftirqd/2 Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff9eab3baf8fd0 ti: ffff9eab3bb04000 task.ti: ffff9eab3bb04000
RIP: 0010:[<ffffffffc014dc1e>]  [<ffffffffc014dc1e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff9eab3bb07bd8  EFLAGS: 00000286
RAX: ffffc08401c42818 RBX: 00000000000005f2 RCX: ffffffff81773b20
RDX: 000000000000001e RSI: 00000000be325000 RDI: 00000000000005f2
RBP: ffff9eab3bb07c30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff9eabfa6fb540 R11: ffffee5508e9bc00 R12: 000000023a6fb540
R13: 000000023b64b238 R14: ffffffff81773e1b R15: ffff9eab3bb07b90
FS:  0000000000000000(0000) GS:ffff9eac1cc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00304b000 CR3: 00000001dbf26000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc014d602>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc014e0eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffffc071604d>] ? nf_ct_delete+0x18d/0x270 [nf_conntrack]
[<ffffffff819ec3af>] net_rx_action+0x26f/0x390
[<ffffffff8149a945>] __do_softirq+0xf5/0x280
[<ffffffff8149ab08>] run_ksoftirqd+0x38/0x50
[<ffffffff814c35e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff814c34a0>] ? lg_double_unlock+0x40/0x40
[<ffffffff814bae31>] kthread+0xd1/0xe0
[<ffffffff814bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff81b1f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff814bad60>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #184041
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0294c2e
Build id: 3.10.0-1160.31.1.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc0294612
Build id: 3.10.0-1160.31.1.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc0295100
Build id: 3.10.0-1160.31.1.el7.x86_64
0x280
4
? __wake_up_common_lock
vmlinux 0xffffffff8c6d31c1
Build id: 3.10.0-1160.31.1.el7.x86_64
0x91
5
net_rx_action
vmlinux 0xffffffff8cc571cf
Build id: 3.10.0-1160.31.1.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffff8c6a4bc5
Build id: 3.10.0-1160.31.1.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffff8c6a4d88
Build id: 3.10.0-1160.31.1.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffff8c6ce884
Build id: 3.10.0-1160.31.1.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffff8c6ce740
Build id: 3.10.0-1160.31.1.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffff8c6c5e31
Build id: 3.10.0-1160.31.1.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffff8c6c5d60
Build id: 3.10.0-1160.31.1.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffff8cd95df7
Build id: 3.10.0-1160.31.1.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffff8c6c5d60
Build id: 3.10.0-1160.31.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 32s! [ksoftirqd/0:6]
Modules linked in: binfmt_misc tun xt_hl ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev vmw_balloon crc32_pclmul snd_ens1371 ghash_clmulni_intel snd_rawmidi snd_ac97_codec aesni_intel ac97_bus lrw gf128mul glue_helper snd_seq ablk_helper cryptd snd_seq_device snd_pcm snd_timer joydev pcspkr snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport
ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea crct10dif_pclmul sysfillrect crct10dif_common sysimgblt fb_sys_fops crc32c_intel ttm drm ata_piix nfit mptspi libata serio_raw libnvdimm scsi_transport_spi mptscsih e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Tainted: G             L ------------   3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff9c96ec4b5280 ti: ffff9c96ec56c000 task.ti: ffff9c96ec56c000
RIP: 0010:[<ffffffffc0294c2e>]  [<ffffffffc0294c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0000:ffff9c96ec56fbd8  EFLAGS: 00000286
RAX: ffffb672831c2818 RBX: 0000000000000780 RCX: ffffffff8c9b1a80
RDX: 0000000000000052 RSI: 00000000b6f21740 RDI: 00000000000005f2
RBP: ffff9c96ec56fc30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff9c9636f21740 R11: ffffffff8d315e40 R12: 00000000dde35796
R13: ffff9c9d9c15f238 R14: ffff9c98eb6f8f00 R15: 00000000dde35796
FS:  0000000000000000(0000) GS:ffff9c9d9d600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f449a827000 CR3: 0000000817ffe000 CR4: 00000000000007f0
Call Trace:
[<ffffffffc0294612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc0295100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff8c6d31c1>] ? __wake_up_common_lock+0x91/0xc0
[<ffffffff8cc571cf>] net_rx_action+0x26f/0x390
[<ffffffff8c6a4bc5>] __do_softirq+0xf5/0x280
[<ffffffff8c6a4d88>] run_ksoftirqd+0x38/0x50
[<ffffffff8c6ce884>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff8c6ce740>] ? lg_double_unlock+0x40/0x40
[<ffffffff8c6c5e31>] kthread+0xd1/0xe0
[<ffffffff8c6c5d60>] ? insert_kthread_work+0x40/0x40
[<ffffffff8cd95df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8c6c5d60>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #133537
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc03b9c2e
Build id: 3.10.0-1062.9.1.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc03b9612
Build id: 3.10.0-1062.9.1.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc03ba100
Build id: 3.10.0-1062.9.1.el7.x86_64
0x280
4
? __wake_up_common_lock
vmlinux 0xffffffffb3ad38c1
Build id: 3.10.0-1062.9.1.el7.x86_64
0x91
5
net_rx_action
vmlinux 0xffffffffb405057f
Build id: 3.10.0-1062.9.1.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffffb3aa5305
Build id: 3.10.0-1062.9.1.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffffb3aa54c8
Build id: 3.10.0-1062.9.1.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffffb3acec24
Build id: 3.10.0-1062.9.1.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffffb3aceae0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffffb3ac61f1
Build id: 3.10.0-1062.9.1.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffffb3ac6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffffb418dd37
Build id: 3.10.0-1062.9.1.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffffb3ac6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [ksoftirqd/0:6]
Modules linked in: xt_nat veth xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock overlay(T) sunrpc snd_seq_midi snd_seq_midi_event sb_edac coretemp iosf_mbi snd_ens1371 crc32_pclmul ppdev snd_rawmidi snd_ac97_codec ghash_clmulni_intel ac97_bus snd_seq snd_seq_device aesni_intel vmw_balloon snd_pcm lrw gf128mul glue_helper ablk_helper cryptd snd_timer snd soundcore sg pcspkr joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom vmwgfx ata_generic drm_kms_helper
syscopyarea pata_acpi sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm mptspi scsi_transport_spi mptscsih e1000 serio_raw ata_piix mptbase libata drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
task: ffff8b353d78d230 ti: ffff8b353d070000 task.ti: ffff8b353d070000
RIP: 0010:[<ffffffffc03b9c2e>]  [<ffffffffc03b9c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff8b353d073bd8  EFLAGS: 00000286
RAX: ffffba3780ec2818 RBX: ffffffffb404fbd9 RCX: ffffffffb3dacaa0
RDX: 00000000000000e0 RSI: 000000003a6a1ec0 RDI: 00000000000005f2
RBP: ffff8b353d073c30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff8b33fa6a1ec0 R11: ffffffffb407bd30 R12: ffffffffb4766ea0
R13: ffff8b357b459238 R14: ffff8b356d68b000 R15: 000000001aa7027a
FS:  0000000000000000(0000) GS:ffff8b357c400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2acf5f2000 CR3: 00000001ae788000 CR4: 00000000001607f0
Call Trace:
[<ffffffffc03b9612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc03ba100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffb3ad38c1>] ? __wake_up_common_lock+0x91/0xc0
[<ffffffffb405057f>] net_rx_action+0x26f/0x390
[<ffffffffb3aa5305>] __do_softirq+0xf5/0x280
[<ffffffffb3aa54c8>] run_ksoftirqd+0x38/0x50
[<ffffffffb3acec24>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffb3aceae0>] ? lg_double_unlock+0x40/0x40
[<ffffffffb3ac61f1>] kthread+0xd1/0xe0
[<ffffffffb3ac6120>] ? insert_kthread_work+0x40/0x40
[<ffffffffb418dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb3ac6120>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #39675
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc03c7c1e
Build id: 3.10.0-862.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc03c7589
Build id: 3.10.0-862.el7.x86_64
0x3f9
3
e1000_clean
e1000 0xffffffffc03c80eb
Build id: 3.10.0-862.el7.x86_64
0x27b
4
? try_to_wake_up
vmlinux 0xffffffff93acebfc
Build id: 3.10.0-862.el7.x86_64
0x18c
5
net_rx_action
vmlinux 0xffffffff93fec3af
Build id: 3.10.0-862.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffff93a9a945
Build id: 3.10.0-862.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffff93a9ab08
Build id: 3.10.0-862.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffff93ac35e4
Build id: 3.10.0-862.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffff93ac34a0
Build id: 3.10.0-862.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffff93abae31
Build id: 3.10.0-862.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffff93abad60
Build id: 3.10.0-862.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffff9411f637
Build id: 3.10.0-862.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffff93abad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ksoftirqd/0:3]
<EOI> 
Modules linked in:
xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper snd_ens1371 cryptd snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd ppdev nfit soundcore joydev pcspkr libnvdimm sg vmw_balloon parport_pc parport shpchp i2c_piix4 vmw_vmci ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi scsi_transport_spi crct10dif_pclmul crct10dif_common libata crc32c_intel mptscsih e1000 serio_raw mptbase i2c_core
CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff99ea75f19fa0 ti: ffff99ea75f9c000 task.ti: ffff99ea75f9c000
RIP: 0010:[<ffffffffc03c7c1e>]  [<ffffffffc03c7c1e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff99ea75f9fbd8  EFLAGS: 00000286
RAX: ffffbdc680d02818 RBX: ffffffff93feb9f9 RCX: 00000000000005f2
RDX: 000000000000000b RSI: ffff99e935f83180 RDI: ffff99ea74e868c0
RBP: ffff99ea75f9fc30 R08: 00000000c0a80167 R09: ffff99e935e80200
R10: 0000000000020420 R11: 0000000000000001 R12: ffffffff9474d840
R13: ffff99ea74e87238 R14: ffff99ea6f949400 R15: 0000000034771587
FS:  0000000000000000(0000) GS:ffff99ea78600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbb41c7a000 CR3: 00000000b87d4000 CR4: 00000000003607f0
Call Trace:
[<ffffffffc03c7589>] e1000_clean_rx_irq+0x3f9/0x510 [e1000]
[<ffffffffc03c80eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffff93acebfc>] ? try_to_wake_up+0x18c/0x350
[<ffffffff93fec3af>] net_rx_action+0x26f/0x390
[<ffffffff93a9a945>] __do_softirq+0xf5/0x280
[<ffffffff93a9ab08>] run_ksoftirqd+0x38/0x50
[<ffffffff93ac35e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff93ac34a0>] ? lg_double_unlock+0x40/0x40
[<ffffffff93abae31>] kthread+0xd1/0xe0
[<ffffffff93abad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff9411f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff93abad60>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #19476
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc017fc0e
Build id: 3.10.0-957.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc017f5f2
Build id: 3.10.0-957.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc01800dd
Build id: 3.10.0-957.el7.x86_64
0x27d
4
? __queue_work
vmlinux 0xffffffffac8b7c0c
Build id: 3.10.0-957.el7.x86_64
0x13c
5
net_rx_action
vmlinux 0xfffffffface39e9f
Build id: 3.10.0-957.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffffac8a0f05
Build id: 3.10.0-957.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffffac8a10c8
Build id: 3.10.0-957.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffffac8ca5c4
Build id: 3.10.0-957.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffffac8ca480
Build id: 3.10.0-957.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffffac8c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffffac8c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffffacf74c37
Build id: 3.10.0-957.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffffac8c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 28s! [ksoftirqd/4:29]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic crct10dif_pclmul crct10dif_common crc32c_intel pata_acpi vmwgfx serio_raw drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm e1000 ata_piix mptspi scsi_transport_spi
mptscsih libata mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 29 Comm: ksoftirqd/4 Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff9db6b7269040 ti: ffff9db6b7274000 task.ti: ffff9db6b7274000
RIP: 0010:[<ffffffffc017fc0e>]  [<ffffffffc017fc0e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff9db6b7277bd8  EFLAGS: 00000286
RAX: ffffb64380a02818 RBX: fffffffface394f9 RCX: ffffffffacba01d0
RDX: 00000000000000c5 RSI: 000000007afdc440 RDI: 00000000000005f2
RBP: ffff9db6b7277c30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff9db5fafdc440 R11: ffffdd2682d5a900 R12: 0000000000000608
R13: ffff9db6b6584000 R14: ffff9db6b1457d80 R15: ffff9db6b6585238
FS:  0000000000000000(0000) GS:ffff9db6b9700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006dee1c CR3: 000000012b810000 CR4: 00000000000607e0
Call Trace:
[<ffffffffc017f5f2>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc01800dd>] e1000_clean+0x27d/0x8e0 [e1000]
[<ffffffffac8b7c0c>] ? __queue_work+0x13c/0x3e0
[<fffffffface39e9f>] net_rx_action+0x26f/0x390
[<ffffffffac8a0f05>] __do_softirq+0xf5/0x280
[<ffffffffac8a10c8>] run_ksoftirqd+0x38/0x50
[<ffffffffac8ca5c4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffac8ca480>] ? lg_double_unlock+0x40/0x40
[<ffffffffac8c1c31>] kthread+0xd1/0xe0
[<ffffffffac8c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffacf74c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffac8c1b60>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #17295
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc016fc2e
Build id: 3.10.0-1127.18.2.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc016f612
Build id: 3.10.0-1127.18.2.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc0170100
Build id: 3.10.0-1127.18.2.el7.x86_64
0x280
4
? mod_timer
vmlinux 0xffffffff882af5ec
Build id: 3.10.0-1127.18.2.el7.x86_64
0x10c
5
net_rx_action
vmlinux 0xffffffff888545bf
Build id: 3.10.0-1127.18.2.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffff882a5695
Build id: 3.10.0-1127.18.2.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffff882a5858
Build id: 3.10.0-1127.18.2.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffff882cf0e4
Build id: 3.10.0-1127.18.2.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffff882cefa0
Build id: 3.10.0-1127.18.2.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffff882c6691
Build id: 3.10.0-1127.18.2.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffff882c65c0
Build id: 3.10.0-1127.18.2.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffff88992d37
Build id: 3.10.0-1127.18.2.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffff882c65c0
Build id: 3.10.0-1127.18.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 21s! [ksoftirqd/4:29]
Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device iosf_mbi crc32_pclmul snd_pcm ghash_clmulni_intel snd_timer aesni_intel lrw snd gf128mul soundcore
glue_helper pcspkr ablk_helper ppdev joydev sg cryptd vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul crct10dif_common drm libata crc32c_intel nfit mptspi scsi_transport_spi libnvdimm serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks
CPU: 4 PID: 29 Comm: ksoftirqd/4 Kdump: loaded Tainted: G             L ------------   3.10.0-1127.18.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffffa01aba919070 ti: ffffa01aba924000 task.ti: ffffa01aba924000
RIP: 0010:[<ffffffffc016fc2e>]  [<ffffffffc016fc2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffffa01aba927bd8  EFLAGS: 00000286
RAX: ffffb04f41942818 RBX: 00000000000005f2 RCX: ffffffff885b0550
RDX: 00000000000000f5 RSI: 00000000bf723800 RDI: 00000000000005f2
RBP: ffffa01aba927c30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffffa01b071d8840 R11: fffff3664caeed00 R12: 00000001c71d8840
R13: 000000020000004e R14: ffffffff885b084b R15: ffffa01aba927b90
FS:  0000000000000000(0000) GS:ffffa01c71700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4843b78000 CR3: 0000000092122000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc016f612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc0170100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff882af5ec>] ? mod_timer+0x10c/0x230
[<ffffffff888545bf>] net_rx_action+0x26f/0x390
[<ffffffff882a5695>] __do_softirq+0xf5/0x280
[<ffffffff882a5858>] run_ksoftirqd+0x38/0x50
[<ffffffff882cf0e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff882cefa0>] ? lg_double_unlock+0x40/0x40
[<ffffffff882c6691>] kthread+0xd1/0xe0
[<ffffffff882c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff88992d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff882c65c0>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #13495
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0053a26
Build id: 3.10.0-693.el7.x86_64
0x366
2
e1000_clean_rx_irq
e1000 0xffffffffc0053625
Build id: 3.10.0-693.el7.x86_64
0x455
3
e1000_clean
e1000 0xffffffffc0053ed8
Build id: 3.10.0-693.el7.x86_64
0x278
4
net_rx_action
vmlinux 0xffffffff8158799d
Build id: 3.10.0-693.el7.x86_64
0x16d
5
__do_softirq
vmlinux 0xffffffff81090b3f
Build id: 3.10.0-693.el7.x86_64
0xef
6
run_ksoftirqd
vmlinux 0xffffffff81090d08
Build id: 3.10.0-693.el7.x86_64
0x38
7
smpboot_thread_fn
vmlinux 0xffffffff810b909f
Build id: 3.10.0-693.el7.x86_64
0x12f
8
? lg_double_unlock
vmlinux 0xffffffff810b8f70
Build id: 3.10.0-693.el7.x86_64
0x40
9
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
10
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
11
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
12
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 44s! [ksoftirqd/2:18]
Modules linked in: nf_conntrack_netlink devlink xt_CHECKSUM xt_addrtype ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter veth xt_nat br_netfilter overlay(T) tun ip_set nfnetlink bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack vmw_vsock_vmci_transport vsock sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev nfit vmw_balloon pcspkr joydev sg libnvdimm parport_pc parport
vmw_vmci i2c_piix4 shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx mptspi scsi_transport_spi mptscsih drm_kms_helper mptbase crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect ata_piix sysimgblt fb_sys_fops ttm libata drm serio_raw e1000 i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: iptable_filter]
CPU: 2 PID: 18 Comm: ksoftirqd/2 Tainted: G        W    L ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff8801370d8000 ti: ffff8801370e0000 task.ti: ffff8801370e0000
RIP: 0010:[<ffffffffc0053a26>]  [<ffffffffc0053a26>] e1000_alloc_rx_buffers+0x366/0x5a0 [e1000]
RSP: 0018:ffff8801370e3bd8  EFLAGS: 00000286
RAX: ffffc90001142818 RBX: ffffffff81586f22 RCX: 00000000000005f2
RDX: 00000000000000ce RSI: ffff88003499d180 RDI: ffff880136a168c0
RBP: ffff8801370e3c30 R08: 0000000000019c40 R09: ffffffff815710d7
R10: ffff880139699c40 R11: ffffea0002d12800 R12: 0000000000000008
R13: ffffffff81b25798 R14: ffff8800b44a0a00 R15: 000000001d17c5fc
FS:  0000000000000000(0000) GS:ffff880139680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd4087e7000 CR3: 00000000b7d76000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
000005f2370e3c08 00000000000005f1 ffff8800b44a0a00 ffff880136a16b70
ffff880136877000 ffff880136877098 0000000000000000 ffff880136a168c0
ffff8800b0c2acf0 ffff8800b44a0a00 ffff8800b0c2ad00 ffff8801370e3cd8
Call Trace:
[<ffffffffc0053625>] e1000_clean_rx_irq+0x455/0x4f0 [e1000]
[<ffffffffc0053ed8>] e1000_clean+0x278/0x8d0 [e1000]
[<ffffffff8158799d>] net_rx_action+0x16d/0x380
[<ffffffff81090b3f>] __do_softirq+0xef/0x280
[<ffffffff81090d08>] run_ksoftirqd+0x38/0x50
[<ffffffff810b909f>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f70>] ? lg_double_unlock+0x40/0x40
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 00 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 1f 02 00 00 0f ae f8 41 0f b7 45 36 49 03 87 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #11945
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc00a7a26
Build id: 3.10.0-693.el7.x86_64
0x366
2
e1000_clean_rx_irq
e1000 0xffffffffc00a7625
Build id: 3.10.0-693.el7.x86_64
0x455
3
e1000_clean
e1000 0xffffffffc00a7ed8
Build id: 3.10.0-693.el7.x86_64
0x278
4
net_rx_action
vmlinux 0xffffffff8158799d
Build id: 3.10.0-693.el7.x86_64
0x16d
5
__do_softirq
vmlinux 0xffffffff81090b3f
Build id: 3.10.0-693.el7.x86_64
0xef
6
run_ksoftirqd
vmlinux 0xffffffff81090d08
Build id: 3.10.0-693.el7.x86_64
0x38
7
smpboot_thread_fn
vmlinux 0xffffffff810b909f
Build id: 3.10.0-693.el7.x86_64
0x12f
8
? lg_double_unlock
vmlinux 0xffffffff810b8f70
Build id: 3.10.0-693.el7.x86_64
0x40
9
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
10
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
11
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
12
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ksoftirqd/2:18]
Modules linked in: tcp_lp fuse binfmt_misc vmw_vsock_vmci_transport vsock iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd vmw_balloon joydev pcspkr i2c_piix4 sg vmw_vmci shpchp nfit libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common drm_kms_helper mptspi scsi_transport_spi crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix serio_raw drm mptscsih mptbase e1000 libata i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 18 Comm: ksoftirqd/2 Not tainted 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff8801370d8000 ti: ffff8801370e0000 task.ti: ffff8801370e0000
RIP: 0010:[<ffffffffc00a7a26>]  [<ffffffffc00a7a26>] e1000_alloc_rx_buffers+0x366/0x5a0 [e1000]
RSP: 0018:ffff8801370e3bd8  EFLAGS: 00000286
RAX: ffffc900006c2818 RBX: ffffffff81586f22 RCX: 00000000000005f2
RDX: 0000000000000029 RSI: ffff880035cd1f00 RDI: ffff8801350868c0
RBP: ffff8801370e3c30 R08: 0000000000019c40 R09: ffffffff815710d7
R10: ffff880139699c40 R11: ffffea000260b800 R12: 0000000000000608
R13: ffffffff81b257f8 R14: ffff8800982e1900 R15: 00000000f8134d37
FS:  0000000000000000(0000) GS:ffff880139680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f130ffda880 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
000005f2370e3c08 00000000000005f1 ffff8800982e1900 ffff880135086b70
ffff88013687d000 ffff88013687d098 0000000000000000 ffff8801350868c0
ffff8800ba62a2a0 ffff8800982e1900 ffff8800ba62a2b0 ffff8801370e3cd8
Call Trace:
[<ffffffffc00a7625>] e1000_clean_rx_irq+0x455/0x4f0 [e1000]
[<ffffffffc00a7ed8>] e1000_clean+0x278/0x8d0 [e1000]
[<ffffffff8158799d>] net_rx_action+0x16d/0x380
[<ffffffff81090b3f>] __do_softirq+0xef/0x280
[<ffffffff81090d08>] run_ksoftirqd+0x38/0x50
[<ffffffff810b909f>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f70>] ? lg_double_unlock+0x40/0x40
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 00 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 1f 02 00 00 0f ae f8 41 0f b7 45 36 49 03 87 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #9445
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc041dc2e
Build id: 3.10.0-1062.18.1.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc041d612
Build id: 3.10.0-1062.18.1.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc041e100
Build id: 3.10.0-1062.18.1.el7.x86_64
0x280
4
? update_curr
vmlinux 0xffffffffbb4e4b0c
Build id: 3.10.0-1062.18.1.el7.x86_64
0x14c
5
net_rx_action
vmlinux 0xffffffffbba5057f
Build id: 3.10.0-1062.18.1.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffffbb4a5435
Build id: 3.10.0-1062.18.1.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffffbb4a55f8
Build id: 3.10.0-1062.18.1.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffffbb4ced54
Build id: 3.10.0-1062.18.1.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffffbb4cec10
Build id: 3.10.0-1062.18.1.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffffbb4c6321
Build id: 3.10.0-1062.18.1.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffffbb4c6250
Build id: 3.10.0-1062.18.1.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffffbbb8dd37
Build id: 3.10.0-1062.18.1.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffffbb4c6250
Build id: 3.10.0-1062.18.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ksoftirqd/0:6]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ppdev vmw_balloon ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq nfit snd_seq_device libnvdimm snd_pcm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr snd_timer snd sg vmw_vmci soundcore i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi crct10dif_pclmul e1000 scsi_transport_spi crct10dif_common mptscsih mptbase crc32c_intel serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.18.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff95653cf0d230 ti: ffff95653cff0000 task.ti: ffff95653cff0000
RIP: 0010:[<ffffffffc041dc2e>]  [<ffffffffc041dc2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff95653cff3bd8  EFLAGS: 00000286
RAX: ffffbcfd80cc2818 RBX: ffffffffbba4fbd9 RCX: 00000000000005f2
RDX: 00000000000000e5 RSI: ffff9563f55ab340 RDI: ffff9565f63528c0
RBP: ffff95653cff3c30 R08: 000000000001f120 R09: ffffffffbba376e7
R10: ffff9565f661f120 R11: ffffee5d42dabb00 R12: 0000000000000608
R13: ffff9565f6352000 R14: ffff9565f5a06580 R15: ffff9565f6353238
FS:  0000000000000000(0000) GS:ffff9565f6600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd29bc5f3bc CR3: 0000000232e16000 CR4: 00000000001607f0
Call Trace:
[<ffffffffc041d612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc041e100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffbb4e4b0c>] ? update_curr+0x14c/0x1e0
[<ffffffffbba5057f>] net_rx_action+0x26f/0x390
[<ffffffffbb4a5435>] __do_softirq+0xf5/0x280
[<ffffffffbb4a55f8>] run_ksoftirqd+0x38/0x50
[<ffffffffbb4ced54>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffbb4cec10>] ? lg_double_unlock+0x40/0x40
[<ffffffffbb4c6321>] kthread+0xd1/0xe0
[<ffffffffbb4c6250>] ? insert_kthread_work+0x40/0x40
[<ffffffffbbb8dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffbb4c6250>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #3864
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0317c2e
Build id: 3.10.0-1062.1.2.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc0317612
Build id: 3.10.0-1062.1.2.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc0318100
Build id: 3.10.0-1062.1.2.el7.x86_64
0x280
4
net_rx_action
vmlinux 0xffffffff8d24e57f
Build id: 3.10.0-1062.1.2.el7.x86_64
0x26f
5
__do_softirq
vmlinux 0xffffffff8cca41e5
Build id: 3.10.0-1062.1.2.el7.x86_64
0xf5
6
run_ksoftirqd
vmlinux 0xffffffff8cca43a8
Build id: 3.10.0-1062.1.2.el7.x86_64
0x38
7
smpboot_thread_fn
vmlinux 0xffffffff8cccdb04
Build id: 3.10.0-1062.1.2.el7.x86_64
0x144
8
? lg_double_unlock
vmlinux 0xffffffff8cccd9c0
Build id: 3.10.0-1062.1.2.el7.x86_64
0x40
9
kthread
vmlinux 0xffffffff8ccc50d1
Build id: 3.10.0-1062.1.2.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffff8ccc5000
Build id: 3.10.0-1062.1.2.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffff8d38cd37
Build id: 3.10.0-1062.1.2.el7.x86_64
0x21
12
? insert_kthread_work
vmlinux 0xffffffff8ccc5000
Build id: 3.10.0-1062.1.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ksoftirqd/2:19]
Modules linked in: tcp_lp nls_utf8 isofs fuse xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle overlay(T) ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter binfmt_misc vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq ppdev coretemp snd_seq_device snd_pcm vmw_balloon snd_timer snd iosf_mbi soundcore crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper sg pcspkr joydev cryptd parport_pc parport vmw_vmci i2c_piix4 sch_fq_codel ip_tables xfs libcrc32c
sd_mod crc_t10dif crct10dif_generic sr_mod cdrom vmwgfx ata_generic pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix scsi_transport_spi nfit mptscsih crct10dif_pclmul mptbase drm e1000 libnvdimm crct10dif_common libata serio_raw crc32c_intel drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]
CPU: 2 PID: 19 Comm: ksoftirqd/2 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.1.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9ed53c415230 ti: ffff9ed53c438000 task.ti: ffff9ed53c438000
RIP: 0010:[<ffffffffc0317c2e>]  [<ffffffffc0317c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0000:ffff9ed53c43bbd8  EFLAGS: 00000286
RAX: ffffb30a40d02818 RBX: 00000000000005f2 RCX: ffffffff8cfaa8e0
RDX: 000000000000008c RSI: 00000000bfb84800 RDI: 00000000000005f2
RBP: ffff9ed53c43bc30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff9ed568f8cbc0 R11: ffffd7658697f000 R12: 00000001a8f8cbc0
R13: 000000023c7c1238 R14: ffffffff8cfaabdb R15: ffff9ed53c43bb90
FS:  0000000000000000(0000) GS:ffff9ed5f5680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055a968ad5000 CR3: 00000001ae388000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc0317612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc0318100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff8d24e57f>] net_rx_action+0x26f/0x390
[<ffffffff8cca41e5>] __do_softirq+0xf5/0x280
[<ffffffff8cca43a8>] run_ksoftirqd+0x38/0x50
[<ffffffff8cccdb04>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff8cccd9c0>] ? lg_double_unlock+0x40/0x40
[<ffffffff8ccc50d1>] kthread+0xd1/0xe0
[<ffffffff8ccc5000>] ? insert_kthread_work+0x40/0x40
[<ffffffff8d38cd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8ccc5000>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #766814
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0460c0e
Build id: 3.10.0-957.27.2.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc04605f2
Build id: 3.10.0-957.27.2.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc04610dd
Build id: 3.10.0-957.27.2.el7.x86_64
0x27d
4
? nf_ct_delete
nf_conntrack 0xffffffffc096101d
Build id: 3.10.0-957.27.2.el7.x86_64
0x18d
5
? neigh_probe
vmlinux 0xffffffffa3a48c8f
Build id: 3.10.0-957.27.2.el7.x86_64
0x5f
6
net_rx_action
vmlinux 0xffffffffa3a3bf1f
Build id: 3.10.0-957.27.2.el7.x86_64
0x26f
7
__do_softirq
vmlinux 0xffffffffa34a2155
Build id: 3.10.0-957.27.2.el7.x86_64
0xf5
8
run_ksoftirqd
vmlinux 0xffffffffa34a2318
Build id: 3.10.0-957.27.2.el7.x86_64
0x38
9
smpboot_thread_fn
vmlinux 0xffffffffa34cb814
Build id: 3.10.0-957.27.2.el7.x86_64
0x144
10
? lg_double_unlock
vmlinux 0xffffffffa34cb6d0
Build id: 3.10.0-957.27.2.el7.x86_64
0x40
11
kthread
vmlinux 0xffffffffa34c2e81
Build id: 3.10.0-957.27.2.el7.x86_64
0xd1
12
? insert_kthread_work
vmlinux 0xffffffffa34c2db0
Build id: 3.10.0-957.27.2.el7.x86_64
0x40
13
ret_from_fork_nospec_begin
vmlinux 0xffffffffa3b76c37
Build id: 3.10.0-957.27.2.el7.x86_64
0x21
14
? insert_kthread_work
vmlinux 0xffffffffa34c2db0
Build id: 3.10.0-957.27.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ksoftirqd/1:14]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables nfnetlink_queue nfnetlink_log iptable_filter nfnetlink bluetooth rfkill vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ppdev ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c vmwgfx sd_mod sr_mod crc_t10dif cdrom crct10dif_generic drm_kms_helper ata_generic pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix mptspi
libata e1000 serio_raw scsi_transport_spi nfit mptscsih libnvdimm mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 14 Comm: ksoftirqd/1 Kdump: loaded Tainted: G             L ------------   3.10.0-957.27.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff99c8f7220000 ti: ffff99c8f7228000 task.ti: ffff99c8f7228000
RIP: 0010:[<ffffffffc0460c0e>]  [<ffffffffc0460c0e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0000:ffff99c8f722bbd8  EFLAGS: 00010286
RAX: ffffbb41810c2818 RBX: 00000000000005f2 RCX: ffffffffa37a2100
RDX: 00000000000000f9 RSI: 00000000bd449000 RDI: 00000000000005f2
RBP: ffff99c8f722bc30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff99c8f5a25ac0 R11: fffffe9602694e00 R12: 0000000135a25ac0
R13: 000000020000004e R14: ffffffffa37a23fb R15: ffff99c8f722bb90
FS:  0000000000000000(0000) GS:ffff99c8f9640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1602cbd000 CR3: 0000000134d92000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc04605f2>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc04610dd>] e1000_clean+0x27d/0x8e0 [e1000]
[<ffffffffc096101d>] ? nf_ct_delete+0x18d/0x270 [nf_conntrack]
[<ffffffffa3a48c8f>] ? neigh_probe+0x5f/0x70
[<ffffffffa3a3bf1f>] net_rx_action+0x26f/0x390
[<ffffffffa34a2155>] __do_softirq+0xf5/0x280
[<ffffffffa34a2318>] run_ksoftirqd+0x38/0x50
[<ffffffffa34cb814>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa34cb6d0>] ? lg_double_unlock+0x40/0x40
[<ffffffffa34c2e81>] kthread+0xd1/0xe0
[<ffffffffa34c2db0>] ? insert_kthread_work+0x40/0x40
[<ffffffffa3b76c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa34c2db0>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #473522
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc00f9c2e
Build id: 3.10.0-1160.31.1.el7.x86_64
0x3be
2
e1000_clean_rx_irq
e1000 0xffffffffc00f9612
Build id: 3.10.0-1160.31.1.el7.x86_64
0x472
3
e1000_clean
e1000 0xffffffffc00fa100
Build id: 3.10.0-1160.31.1.el7.x86_64
0x280
4
? neigh_probe
vmlinux 0xffffffffb42641cf
Build id: 3.10.0-1160.31.1.el7.x86_64
0x5f
5
net_rx_action
vmlinux 0xffffffffb42571cf
Build id: 3.10.0-1160.31.1.el7.x86_64
0x26f
6
__do_softirq
vmlinux 0xffffffffb3ca4bc5
Build id: 3.10.0-1160.31.1.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffffb3ca4d88
Build id: 3.10.0-1160.31.1.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffffb3cce884
Build id: 3.10.0-1160.31.1.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffffb3cce740
Build id: 3.10.0-1160.31.1.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffffb3cc5e31
Build id: 3.10.0-1160.31.1.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffffb3cc5d60
Build id: 3.10.0-1160.31.1.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffffb4395df7
Build id: 3.10.0-1160.31.1.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffffb3cc5d60
Build id: 3.10.0-1160.31.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [ksoftirqd/2:19]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common nfit libnvdimm drm crc32c_intel ata_piix libata serio_raw e1000 mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 19 Comm: ksoftirqd/2 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9e977a43d280 ti: ffff9e977a458000 task.ti: ffff9e977a458000
RIP: 0010:[<ffffffffc00f9c2e>]  [<ffffffffc00f9c2e>] e1000_alloc_rx_buffers+0x3be/0x610 [e1000]
RSP: 0018:ffff9e977a45bbd8  EFLAGS: 00000286
RAX: ffffc3f500d02818 RBX: ffffffffb4256829 RCX: 00000000000005f2
RDX: 0000000000000018 RSI: ffff9e9789deff00 RDI: ffff9e98345b88c0
RBP: ffff9e977a45bc30 R08: ffff9e977a45b950 R09: ffffffffb3e64130
R10: ffff9e983569f120 R11: fffff61e472a7600 R12: 0000000000000608
R13: ffff9e98345b8000 R14: ffff9e9635633d80 R15: ffff9e98345b9238
FS:  0000000000000000(0000) GS:ffff9e9835680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f574f71a100 CR3: 0000000230f3e000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc00f9612>] e1000_clean_rx_irq+0x472/0x510 [e1000]
[<ffffffffc00fa100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffb42641cf>] ? neigh_probe+0x5f/0x70
[<ffffffffb42571cf>] net_rx_action+0x26f/0x390
[<ffffffffb3ca4bc5>] __do_softirq+0xf5/0x280
[<ffffffffb3ca4d88>] run_ksoftirqd+0x38/0x50
[<ffffffffb3cce884>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffb3cce740>] ? lg_double_unlock+0x40/0x40
[<ffffffffb3cc5e31>] kthread+0xd1/0xe0
[<ffffffffb3cc5d60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb4395df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb3cc5d60>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 f1 01 00 00 0f ae f8 41 0f b7 45 36 49 03 86 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45 
Complete report #405672
Frame # Function Binary Source or offset Line
1
e1000_clean
e1000 0xffffffffc016f22a
Build id: 3.10.0-1160.59.1.el7.x86_64
0x3aa
2
? neigh_probe
vmlinux 0xffffffffb6e671cf
Build id: 3.10.0-1160.59.1.el7.x86_64
0x5f
3
net_rx_action
vmlinux 0xffffffffb6e5a1cf
Build id: 3.10.0-1160.59.1.el7.x86_64
0x26f
4
__do_softirq
vmlinux 0xffffffffb68a4bf5
Build id: 3.10.0-1160.59.1.el7.x86_64
0xf5
5
run_ksoftirqd
vmlinux 0xffffffffb68a4db8
Build id: 3.10.0-1160.59.1.el7.x86_64
0x38
6
smpboot_thread_fn
vmlinux 0xffffffffb68ce8b4
Build id: 3.10.0-1160.59.1.el7.x86_64
0x144
7
? lg_double_unlock
vmlinux 0xffffffffb68ce770
Build id: 3.10.0-1160.59.1.el7.x86_64
0x40
8
kthread
vmlinux 0xffffffffb68c5e61
Build id: 3.10.0-1160.59.1.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffffb68c5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffffb6f99df7
Build id: 3.10.0-1160.59.1.el7.x86_64
0x21
11
? insert_kthread_work
vmlinux 0xffffffffb68c5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ksoftirqd/0:6]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 btusb snd_rawmidi btrtl snd_ac97_codec btbcm sb_edac btintel ac97_bus snd_seq iosf_mbi crc32_pclmul
ghash_clmulni_intel snd_seq_device bluetooth snd_pcm aesni_intel rfkill ppdev lrw gf128mul snd_timer glue_helper snd vmw_balloon ablk_helper cryptd soundcore sg joydev pcspkr parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect mptspi sysimgblt scsi_transport_spi fb_sys_fops nfit ata_piix crct10dif_pclmul ttm crct10dif_common drm libnvdimm e1000 crc32c_intel mptscsih serio_raw libata mptbase drm_panel_orientation_quirks fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff9fabf07b5280 ti: ffff9fabf0094000 task.ti: ffff9fabf0094000
RIP: 0010:[<ffffffffc016f22a>]  [<ffffffffc016f22a>] e1000_clean+0x3aa/0x8e0 [e1000]
RSP: 0000:ffff9fabf0097ce8  EFLAGS: 00000246
RAX: 000000000000cb83 RBX: ffff9fb0bbf76000 RCX: 0000000000000000
RDX: ffffb851835c0000 RSI: 0000000000000000 RDI: ffff9fb0bbf76b70
RBP: ffff9fabf0097d88 R08: 0000000000000005 R09: 0000000000000001
R10: ffff9fb0bd01f120 R11: ffffe93f0600cb00 R12: ffff9fb0bde7a098
R13: ffff9fb0bbf768c0 R14: ffff9fb0bde7a000 R15: ffffffffc016e599
FS:  0000000000000000(0000) GS:ffff9fb0bd000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd127476d80 CR3: 000000014dc10000 CR4: 00000000001607f0
Call Trace:
[<ffffffffb6e671cf>] ? neigh_probe+0x5f/0x70
[<ffffffffb6e5a1cf>] net_rx_action+0x26f/0x390
[<ffffffffb68a4bf5>] __do_softirq+0xf5/0x280
[<ffffffffb68a4db8>] run_ksoftirqd+0x38/0x50
[<ffffffffb68ce8b4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffb68ce770>] ? lg_double_unlock+0x40/0x40
[<ffffffffb68c5e61>] kthread+0xd1/0xe0
[<ffffffffb68c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffb6f99df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb68c5d90>] ? insert_kthread_work+0x40/0x40
Code: 48 8b 83 08 05 00 00 a8 04 75 20 48 8b 7d a0 b8 9d 00 00 00 48 8b 97 20 01 00 00 89 82 d0 00 00 00 48 8b 87 20 01 00 00 8b 40 08 <8b> 45 cc 48 8b 55 d0 65 48 33 14 25 28 00 00 00 0f 85 1d 02 00 
Complete report #354335
Frame # Function Binary Source or offset Line
1
e1000_clean
e1000 0xffffffffc01e520b
Build id: 3.10.0-862.el7.x86_64
0x39b
2
? nf_ct_delete
nf_conntrack 0xffffffffc071604d
Build id: 3.10.0-862.el7.x86_64
0x18d
3
? neigh_probe
vmlinux 0xffffffffa6ff824f
Build id: 3.10.0-862.el7.x86_64
0x5f
4
net_rx_action
vmlinux 0xffffffffa6fec3af
Build id: 3.10.0-862.el7.x86_64
0x26f
5
__do_softirq
vmlinux 0xffffffffa6a9a945
Build id: 3.10.0-862.el7.x86_64
0xf5
6
run_ksoftirqd
vmlinux 0xffffffffa6a9ab08
Build id: 3.10.0-862.el7.x86_64
0x38
7
smpboot_thread_fn
vmlinux 0xffffffffa6ac35e4
Build id: 3.10.0-862.el7.x86_64
0x144
8
? lg_double_unlock
vmlinux 0xffffffffa6ac34a0
Build id: 3.10.0-862.el7.x86_64
0x40
9
kthread
vmlinux 0xffffffffa6abae31
Build id: 3.10.0-862.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffffa6abad60
Build id: 3.10.0-862.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffffa711f61d
Build id: 3.10.0-862.el7.x86_64
0x7
12
? insert_kthread_work
vmlinux 0xffffffffa6abad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 24s! [ksoftirqd/4:29]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg snd_ens1371 snd_rawmidi snd_ac97_codec ppdev vmw_balloon ac97_bus joydev pcspkr nfit snd_seq snd_seq_device snd_pcm snd_timer snd soundcore libnvdimm parport_pc parport vmw_vmci shpchp i2c_piix4 ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic
pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata mptspi scsi_transport_spi e1000 mptscsih serio_raw mptbase i2c_core
CPU: 4 PID: 29 Comm: ksoftirqd/4 Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8b66461d2f70 ti: ffff8b6646208000 task.ti: ffff8b6646208000
RIP: 0010:[<ffffffffc01e520b>]  [<ffffffffc01e520b>] e1000_clean+0x39b/0x8e0 [e1000]
RSP: 0000:ffff8b664620bce8  EFLAGS: 00000246
RAX: 000000000000009d RBX: ffff8b667af6e000 RCX: 0000000000000000
RDX: ffffa50c406c0000 RSI: 0000000000000000 RDI: ffff8b667af6eb70
RBP: ffff8b664620bd88 R08: 000000000000003a R09: 0000000000000000
R10: ffff8b664e73adc0 R11: 0000000000000001 R12: ffff8b667b96d098
R13: ffff8b667af6e8c0 R14: ffff8b667b96d000 R15: ffffffffc01e4589
FS:  0000000000000000(0000) GS:ffff8b667a700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fdbe0856000 CR3: 00000000aeb5e000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc071604d>] ? nf_ct_delete+0x18d/0x270 [nf_conntrack]
[<ffffffffa6ff824f>] ? neigh_probe+0x5f/0x70
[<ffffffffa6fec3af>] net_rx_action+0x26f/0x390
[<ffffffffa6a9a945>] __do_softirq+0xf5/0x280
[<ffffffffa6a9ab08>] run_ksoftirqd+0x38/0x50
[<ffffffffa6ac35e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa6ac34a0>] ? lg_double_unlock+0x40/0x40
[<ffffffffa6abae31>] kthread+0xd1/0xe0
[<ffffffffa6abad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa711f61d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffa6abad60>] ? insert_kthread_work+0x40/0x40
Code: 31 f6 48 89 df e8 26 6e e0 e6 48 8b 83 08 05 00 00 a8 04 75 20 48 8b 7d a0 b8 9d 00 00 00 48 8b 97 20 01 00 00 89 82 d0 00 00 00 <48> 8b 87 20 01 00 00 8b 40 08 8b 45 cc 48 8b 55 d0 65 48 33 14 
Complete report #8827
Frame # Function Binary Source or offset Line
1
e1000_alloc_rx_buffers
e1000 0xffffffffc0095a36
Build id: 3.10.0-693.21.1.el7.x86_64
0x366
2
e1000_clean_rx_irq
e1000 0xffffffffc009563a
Build id: 3.10.0-693.21.1.el7.x86_64
0x46a
3
e1000_clean
e1000 0xffffffffc0095eeb
Build id: 3.10.0-693.21.1.el7.x86_64
0x27b
4
? neigh_probe
vmlinux 0xffffffff8159ce0a
Build id: 3.10.0-693.21.1.el7.x86_64
0x5a
5
net_rx_action
vmlinux 0xffffffff81590cc3
Build id: 3.10.0-693.21.1.el7.x86_64
0x173
6
__do_softirq
vmlinux 0xffffffff81094035
Build id: 3.10.0-693.21.1.el7.x86_64
0xf5
7
run_ksoftirqd
vmlinux 0xffffffff810941f8
Build id: 3.10.0-693.21.1.el7.x86_64
0x38
8
smpboot_thread_fn
vmlinux 0xffffffff810bc774
Build id: 3.10.0-693.21.1.el7.x86_64
0x144
9
? lg_double_unlock
vmlinux 0xffffffff810bc630
Build id: 3.10.0-693.21.1.el7.x86_64
0x40
10
kthread
vmlinux 0xffffffff810b4031
Build id: 3.10.0-693.21.1.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffff810b3f60
Build id: 3.10.0-693.21.1.el7.x86_64
0x40
12
ret_from_fork
vmlinux 0xffffffff816c0577
Build id: 3.10.0-693.21.1.el7.x86_64
0x77
13
? insert_kthread_work
vmlinux 0xffffffff810b3f60
Build id: 3.10.0-693.21.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [ksoftirqd/2:18]
Modules linked in: nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi aesni_intel snd_ac97_codec lrw gf128mul glue_helper ablk_helper ac97_bus cryptd snd_seq snd_seq_device snd_pcm ppdev snd_timer vmw_balloon snd sg joydev
soundcore pcspkr nfit vmw_vmci i2c_piix4 shpchp libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif vmwgfx crct10dif_generic drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_generic pata_acpi drm mptspi ata_piix scsi_transport_spi serio_raw mptscsih libata e1000 mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 18 Comm: ksoftirqd/2 Tainted: G             L ------------   3.10.0-693.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff880179868000 ti: ffff880179870000 task.ti: ffff880179870000
RIP: 0010:[<ffffffffc0095a36>]  [<ffffffffc0095a36>] e1000_alloc_rx_buffers+0x366/0x5a0 [e1000]
RSP: 0018:ffff880179873bd8  EFLAGS: 00000286
RAX: ffffc90001982818 RBX: ffffffff8159007a RCX: ffff88042f5fc000
RDX: 00000000000000e6 RSI: 00000000ad9e80c0 RDI: 00000000000005f2
RBP: ffff880179873c30 R08: 00000000000005f2 R09: 0000000000000000
R10: ffff8800ad9e80c0 R11: ffffea00103c9800 R12: 000000000000fd8f
R13: ffffffff81b36b28 R14: ffff88040f262100 R15: ffffffff8157bffa
FS:  0000000000000000(0000) GS:ffff88042d680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f07a2f59000 CR3: 00000004293be000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc009563a>] e1000_clean_rx_irq+0x46a/0x500 [e1000]
[<ffffffffc0095eeb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffff8159ce0a>] ? neigh_probe+0x5a/0x70
[<ffffffff81590cc3>] net_rx_action+0x173/0x380
[<ffffffff81094035>] __do_softirq+0xf5/0x280
[<ffffffff810941f8>] run_ksoftirqd+0x38/0x50
[<ffffffff810bc774>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff810bc630>] ? lg_double_unlock+0x40/0x40
[<ffffffff810b4031>] kthread+0xd1/0xe0
[<ffffffff810b3f60>] ? insert_kthread_work+0x40/0x40
[<ffffffff816c0577>] ret_from_fork+0x77/0xb0
[<ffffffff810b3f60>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 41 3b 5d 18 74 20 85 db 41 89 5d 18 8d 53 ff 0f 84 1f 02 00 00 0f ae f8 41 0f b7 45 36 49 03 87 d0 03 00 00 89 10 <48> 83 c4 30 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 48 8b 45