Info

Permalink
Function
memcpy
First occurrence
2020-09-30
Last occurrence
2026-07-06
State
NEW
Components
kernel
Quality
-34

Statistics

Operating system Count
CentOS 7 43
Architecture Count
x86_64 43
Related packages Count
kernel 43
 0:3.10.0-1160.11.1.el7 6
 0:3.10.0-957.10.1.el7 6
 0:3.10.0-957.el7 3
 0:3.10.0-1160.62.1.el7 3
 0:3.10.0-1160.81.1.el7 3
 0:3.10.0-862.6.3.el7 2
 0:3.10.0-1160.el7 2
 0:3.10.0-862.el7 2
 0:3.10.0-1160.76.1.el7 2
 0:3.10.0-1160.90.1.el7 2
 0:3.10.0-957.21.3.el7 1
 0:3.10.0-1062.4.3.el7 1
 0:3.10.0-1160.6.1.el7 1
 0:3.10.0-514.el7 1
 0:3.10.0-514.10.2.el7 1
 0:3.10.0-1062.12.1.el7 1
 0:3.10.0-1062.el7 1
 0:3.10.0-1160.45.1.el7 1
 0:3.10.0-1127.8.2.el7 1
 0:3.10.0-1127.19.1.el7 1
 0:3.10.0-1160.80.1.el7 1
 0:3.10.0-862.9.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #1417446
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffff857adcd8
Build id: 3.10.0-1160.90.1.el7.x86_64
0x68
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffff857c919b
Build id: 3.10.0-1160.90.1.el7.x86_64
0x7b
3
swiotlb_sync_single
vmlinux 0xffffffff857c91ed
Build id: 3.10.0-1160.90.1.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffff857c920c
Build id: 3.10.0-1160.90.1.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc02873ad
Build id: 3.10.0-1160.90.1.el7.x86_64
0x19d
6
e1000_clean
e1000 0xffffffffc0288180
Build id: 3.10.0-1160.90.1.el7.x86_64
0x280
7
? nf_ct_delete
nf_conntrack 0xffffffffc065a0cd
Build id: 3.10.0-1160.90.1.el7.x86_64
0x18d
8
? tcp_init_xmit_timers
vmlinux 0xffffffff85aef170
Build id: 3.10.0-1160.90.1.el7.x86_64
0x30
9
net_rx_action
vmlinux 0xffffffff85a80037
Build id: 3.10.0-1160.90.1.el7.x86_64
0x277
10
__do_softirq
vmlinux 0xffffffff854a9595
Build id: 3.10.0-1160.90.1.el7.x86_64
0xf5
11
run_ksoftirqd
vmlinux 0xffffffff854a9768
Build id: 3.10.0-1160.90.1.el7.x86_64
0x38
12
smpboot_thread_fn
vmlinux 0xffffffff854d43fc
Build id: 3.10.0-1160.90.1.el7.x86_64
0x14c
13
? lg_double_unlock
vmlinux 0xffffffff854d42b0
Build id: 3.10.0-1160.90.1.el7.x86_64
0x40
14
kthread
vmlinux 0xffffffff854cb621
Build id: 3.10.0-1160.90.1.el7.x86_64
0xd1
15
? insert_kthread_work
vmlinux 0xffffffff854cb550
Build id: 3.10.0-1160.90.1.el7.x86_64
0x40
16
ret_from_fork_nospec_begin
vmlinux 0xffffffff85bc51f7
Build id: 3.10.0-1160.90.1.el7.x86_64
0x21
17
? insert_kthread_work
vmlinux 0xffffffff854cb550
Build id: 3.10.0-1160.90.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [ksoftirqd/0:6]
Modules linked in: xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd nfit joydev libnvdimm sg pcspkr vmw_balloon i2c_piix4 parport_pc parport vmw_vmci sch_fq_codel ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic vmwgfx drm_kms_helper pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi
e1000 drm mptscsih crct10dif_pclmul crct10dif_common mptbase ata_piix crc32c_intel serio_raw libata drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.90.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff957e39315280 ti: ffff957e3934c000 task.ti: ffff957e3934c000
RIP: 0010:[<ffffffff857adcd8>]  [<ffffffff857adcd8>] memcpy+0x68/0x130
RSP: 0018:ffff957e3934fc08  EFLAGS: 00010246
RAX: ffff957f7200c440 RBX: ffffffff8550ea2f RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffff957d7e404820 RDI: ffff957f7200c480
RBP: ffff957e3934fc10 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000009702 R11: a8c0000000000000 R12: ffff957e3923d260
R13: ffff957f70205e00 R14: ffff957f70205e00 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff957f7fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007efe0e2c6000 CR3: 000000013d210000 CR4: 00000000000607f0
Call Trace:
[<ffffffff857c919b>] ? swiotlb_tbl_sync_single+0x7b/0xa0
[<ffffffff857c91ed>] swiotlb_sync_single+0x2d/0x40
[<ffffffff857c920c>] swiotlb_sync_single_for_cpu+0xc/0x20
[<ffffffffc02873ad>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc0288180>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffc065a0cd>] ? nf_ct_delete+0x18d/0x280 [nf_conntrack]
[<ffffffff85aef170>] ? tcp_init_xmit_timers+0x30/0x30
[<ffffffff85a80037>] net_rx_action+0x277/0x3a0
[<ffffffff854a9595>] __do_softirq+0xf5/0x290
[<ffffffff854a9768>] run_ksoftirqd+0x38/0x60
[<ffffffff854d43fc>] smpboot_thread_fn+0x14c/0x1b0
[<ffffffff854d42b0>] ? lg_double_unlock+0x40/0x40
[<ffffffff854cb621>] kthread+0xd1/0xe0
[<ffffffff854cb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff85bc51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff854cb550>] ? insert_kthread_work+0x40/0x40
Code: d4 83 c2 20 eb 44 48 01 d6 48 01 d7 48 83 ea 20 0f 1f 00 48 83 ea 20 4c 8b 46 f8 4c 8b 4e f0 4c 8b 56 e8 4c 8b 5e e0 48 8d 76 e0 <4c> 89 47 f8 4c 89 4f f0 4c 89 57 e8 4c 89 5f e0 48 8d 7f e0 73 
Complete report #1413326
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffffb5dadc99
Build id: 3.10.0-1160.90.1.el7.x86_64
0x29
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffffb5dc919b
Build id: 3.10.0-1160.90.1.el7.x86_64
0x7b
3
swiotlb_sync_single
vmlinux 0xffffffffb5dc91ed
Build id: 3.10.0-1160.90.1.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffffb5dc920c
Build id: 3.10.0-1160.90.1.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc00a23ad
Build id: 3.10.0-1160.90.1.el7.x86_64
0x19d
6
e1000_clean
e1000 0xffffffffc00a3180
Build id: 3.10.0-1160.90.1.el7.x86_64
0x280
7
? __wake_up_common_lock
vmlinux 0xffffffffb5ad8f31
Build id: 3.10.0-1160.90.1.el7.x86_64
0x91
8
net_rx_action
vmlinux 0xffffffffb6080037
Build id: 3.10.0-1160.90.1.el7.x86_64
0x277
9
__do_softirq
vmlinux 0xffffffffb5aa9595
Build id: 3.10.0-1160.90.1.el7.x86_64
0xf5
10
run_ksoftirqd
vmlinux 0xffffffffb5aa9768
Build id: 3.10.0-1160.90.1.el7.x86_64
0x38
11
smpboot_thread_fn
vmlinux 0xffffffffb5ad43fc
Build id: 3.10.0-1160.90.1.el7.x86_64
0x14c
12
? lg_double_unlock
vmlinux 0xffffffffb5ad42b0
Build id: 3.10.0-1160.90.1.el7.x86_64
0x40
13
kthread
vmlinux 0xffffffffb5acb621
Build id: 3.10.0-1160.90.1.el7.x86_64
0xd1
14
? insert_kthread_work
vmlinux 0xffffffffb5acb550
Build id: 3.10.0-1160.90.1.el7.x86_64
0x40
15
ret_from_fork_nospec_begin
vmlinux 0xffffffffb61c51dd
Build id: 3.10.0-1160.90.1.el7.x86_64
0x7
16
? insert_kthread_work
vmlinux 0xffffffffb5acb550
Build id: 3.10.0-1160.90.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [ksoftirqd/2:19]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 iosf_mbi snd_rawmidi crc32_pclmul snd_ac97_codec ghash_clmulni_intel aesni_intel lrw gf128mul ac97_bus glue_helper snd_seq ablk_helper snd_seq_device cryptd sg snd_pcm ppdev joydev snd_timer snd soundcore i2c_piix4 vmw_vmci vmw_balloon pcspkr parport_pc parport ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif
crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit libnvdimm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel e1000 serio_raw mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 2 PID: 19 Comm: ksoftirqd/2 Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.90.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8b4f758dd280 ti: ffff8b4f75904000 task.ti: ffff8b4f75904000
RIP: 0010:[<ffffffffb5dadc99>]  [<ffffffffb5dadc99>] memcpy+0x29/0x130
RSP: 0018:ffff8b4f75907c08  EFLAGS: 00010246
RAX: ffff8b4fbeb3bcc0 RBX: ffffffffb5b0ea2f RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffff8b4ebf0eb820 RDI: ffff8b4fbeb3bcc0
RBP: ffff8b4f75907c10 R08: 9b08f0ac72290c00 R09: 004500081d26014b
R10: 063e004040cd2800 R11: a8c0010aa8c098d9 R12: ffff8b522c47b260
R13: ffff8b4e10017d00 R14: ffffffffb60f9b00 R15: ffff8b4f75907b00
FS:  0000000000000000(0000) GS:ffff8b522d680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055f3e7242000 CR3: 00000003fa20e000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffffb5dc919b>] ? swiotlb_tbl_sync_single+0x7b/0xa0
[<ffffffffb5dc91ed>] swiotlb_sync_single+0x2d/0x40
[<ffffffffb5dc920c>] swiotlb_sync_single_for_cpu+0xc/0x20
[<ffffffffc00a23ad>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc00a3180>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffb5ad8f31>] ? __wake_up_common_lock+0x91/0xc0
[<ffffffffb6080037>] net_rx_action+0x277/0x3a0
[<ffffffffb5aa9595>] __do_softirq+0xf5/0x290
[<ffffffffb5aa9768>] run_ksoftirqd+0x38/0x60
[<ffffffffb5ad43fc>] smpboot_thread_fn+0x14c/0x1b0
[<ffffffffb5ad42b0>] ? lg_double_unlock+0x40/0x40
[<ffffffffb5acb621>] kthread+0xd1/0xe0
[<ffffffffb5acb550>] ? insert_kthread_work+0x40/0x40
[<ffffffffb61c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb5acb550>] ? insert_kthread_work+0x40/0x40
Code: 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 <4c> 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 
Complete report #948340
Frame # Function Binary Source or offset Line
1
swiotlb_tbl_sync_single
vmlinux 0xffffffffbd7a0539
Build id: 3.10.0-957.el7.x86_64
0x29
2
swiotlb_sync_single
vmlinux 0xffffffffbd7a05cd
Build id: 3.10.0-957.el7.x86_64
0x2d
3
swiotlb_sync_single_for_cpu
vmlinux 0xffffffffbd7a05ec
Build id: 3.10.0-957.el7.x86_64
0xc
4
e1000_clean_rx_irq
e1000 0xffffffffc035f31d
Build id: 3.10.0-957.el7.x86_64
0x19d
5
e1000_clean
e1000 0xffffffffc03600dd
Build id: 3.10.0-957.el7.x86_64
0x27d
6
? __queue_work
vmlinux 0xffffffffbd4b7c0c
Build id: 3.10.0-957.el7.x86_64
0x13c
7
net_rx_action
vmlinux 0xffffffffbda39e9f
Build id: 3.10.0-957.el7.x86_64
0x26f
8
__do_softirq
vmlinux 0xffffffffbd4a0f05
Build id: 3.10.0-957.el7.x86_64
0xf5
9
run_ksoftirqd
vmlinux 0xffffffffbd4a10c8
Build id: 3.10.0-957.el7.x86_64
0x38
10
smpboot_thread_fn
vmlinux 0xffffffffbd4ca5c4
Build id: 3.10.0-957.el7.x86_64
0x144
11
? lg_double_unlock
vmlinux 0xffffffffbd4ca480
Build id: 3.10.0-957.el7.x86_64
0x40
12
kthread
vmlinux 0xffffffffbd4c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
13
? insert_kthread_work
vmlinux 0xffffffffbd4c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
14
ret_from_fork_nospec_begin
vmlinux 0xffffffffbdb74c37
Build id: 3.10.0-957.el7.x86_64
0x21
15
? insert_kthread_work
vmlinux 0xffffffffbd4c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [ksoftirqd/2:19]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmhgfs(OE) vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event ppdev vmw_balloon crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore sg parport_pc parport i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod cdrom crct10dif_generic ata_generic pata_acpi crct10dif_pclmul
crct10dif_common crc32c_intel serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm e1000 drm ata_piix mptspi scsi_transport_spi mptscsih mptbase libata drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 19 Comm: ksoftirqd/2 Kdump: loaded Tainted: G           OEL ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff8f307a84d140 ti: ffff8f307a88c000 task.ti: ffff8f307a88c000
RIP: 0010:[<ffffffffbd7a0539>]  [<ffffffffbd7a0539>] swiotlb_tbl_sync_single+0x29/0x90
RSP: 0018:ffff8f307a88fc10  EFLAGS: 00000206
RAX: 00000000000025de RBX: 0000000000000000 RCX: 0000000000000002
RDX: 0000000000000040 RSI: 00000000bd1bf000 RDI: ffff8f323ff1f000
RBP: ffff8f307a88fc10 R08: 0000000000000000 R09: 00000002a6fd8fc0
R10: ffff8f323fbd5fc0 R11: fffff9b7cad1f500 R12: ffff8f323ffd9000
R13: 0000000200000003 R14: 0000014100000000 R15: ffff8f323ffd9008
FS:  0000000000000000(0000) GS:ffff8f3231680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7738d27000 CR3: 0000000034bd0000 CR4: 00000000003607e0
Call Trace:
[<ffffffffbd7a05cd>] swiotlb_sync_single+0x2d/0x40
[<ffffffffbd7a05ec>] swiotlb_sync_single_for_cpu+0xc/0x10
[<ffffffffc035f31d>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc03600dd>] e1000_clean+0x27d/0x8e0 [e1000]
[<ffffffffbd4b7c0c>] ? __queue_work+0x13c/0x3e0
[<ffffffffbda39e9f>] net_rx_action+0x26f/0x390
[<ffffffffbd4a0f05>] __do_softirq+0xf5/0x280
[<ffffffffbd4a10c8>] run_ksoftirqd+0x38/0x50
[<ffffffffbd4ca5c4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffbd4ca480>] ? lg_double_unlock+0x40/0x40
[<ffffffffbd4c1c31>] kthread+0xd1/0xe0
[<ffffffffbd4c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffbdb74c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffbd4c1b60>] ? insert_kthread_work+0x40/0x40
Code: 3b 00 48 89 f0 48 2b 05 ae d6 e8 00 48 8b 3d 77 d6 e8 00 55 49 89 f1 41 81 e1 ff 07 00 00 48 89 e5 48 c1 e8 0b 48 98 4c 03 0c c7 <45> 85 c0 74 2a 41 83 f8 01 75 1c 83 f9 01 77 3c 48 8b 05 30 4c 
Complete report #881382
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffff9cd978e9
Build id: 3.10.0-1160.76.1.el7.x86_64
0x29
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffff9cdb2273
Build id: 3.10.0-1160.76.1.el7.x86_64
0x73
3
swiotlb_sync_single
vmlinux 0xffffffff9cdb22bd
Build id: 3.10.0-1160.76.1.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffff9cdb22dc
Build id: 3.10.0-1160.76.1.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc00e733d
Build id: 3.10.0-1160.76.1.el7.x86_64
0x19d
6
e1000_clean
e1000 0xffffffffc00e8100
Build id: 3.10.0-1160.76.1.el7.x86_64
0x280
7
? sched_clock_cpu
vmlinux 0xffffffff9cade3a5
Build id: 3.10.0-1160.76.1.el7.x86_64
0x85
8
net_rx_action
vmlinux 0xffffffff9d05a1bf
Build id: 3.10.0-1160.76.1.el7.x86_64
0x26f
9
__do_softirq
vmlinux 0xffffffff9caa4d85
Build id: 3.10.0-1160.76.1.el7.x86_64
0xf5
10
run_ksoftirqd
vmlinux 0xffffffff9caa4f48
Build id: 3.10.0-1160.76.1.el7.x86_64
0x38
11
smpboot_thread_fn
vmlinux 0xffffffff9cace9e4
Build id: 3.10.0-1160.76.1.el7.x86_64
0x144
12
? lg_double_unlock
vmlinux 0xffffffff9cace8a0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
13
kthread
vmlinux 0xffffffff9cac5f91
Build id: 3.10.0-1160.76.1.el7.x86_64
0xd1
14
? insert_kthread_work
vmlinux 0xffffffff9cac5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
15
ret_from_fork_nospec_begin
vmlinux 0xffffffff9d199df7
Build id: 3.10.0-1160.76.1.el7.x86_64
0x21
16
? insert_kthread_work
vmlinux 0xffffffff9cac5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 25s! [ksoftirqd/2:19]
Modules linked in: veth nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) tcp_lp nls_utf8 isofs xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw snd_ens1371 gf128mul
glue_helper snd_rawmidi snd_ac97_codec ac97_bus snd_seq ablk_helper snd_seq_device cryptd ppdev snd_pcm snd_timer snd vmw_balloon soundcore sg joydev pcspkr parport_pc parport vmw_vmci i2c_piix4 sch_fq_codel ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod drm_kms_helper crc_t10dif crct10dif_generic syscopyarea ahci sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw libahci drm ata_piix libata vmw_pvscsi e1000 drm_panel_orientation_quirks floppy fuse
CPU: 2 PID: 19 Comm: ksoftirqd/2 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
task: ffff9dd7fab24200 ti: ffff9dd7fab3c000 task.ti: ffff9dd7fab3c000
RIP: 0010:[<ffffffff9cd978e9>]  [<ffffffff9cd978e9>] memcpy+0x29/0x110
RSP: 0018:ffff9dd7fab3fc08  EFLAGS: 00010246
RAX: ffff9dd876db1ec0 RBX: ffffffff9cb07fbf RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffff9dd73bf67820 RDI: ffff9dd876db1ec0
RBP: ffff9dd7fab3fc10 R08: 0c00ffffffffffff R09: 01000608e1b46f29
R10: 0c00010004060008 R11: d40ba8c0e1b46f29 R12: ffff9dd8b3b17260
R13: ffff9dd72cf46700 R14: ffff9dd8b3b17260 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff9dd8b6880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0ea66409d0 CR3: 00000002327d0000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9cdb2273>] ? swiotlb_tbl_sync_single+0x73/0x90
[<ffffffff9cdb22bd>] swiotlb_sync_single+0x2d/0x40
[<ffffffff9cdb22dc>] swiotlb_sync_single_for_cpu+0xc/0x10
[<ffffffffc00e733d>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc00e8100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff9cade3a5>] ? sched_clock_cpu+0x85/0xc0
[<ffffffff9d05a1bf>] net_rx_action+0x26f/0x390
[<ffffffff9caa4d85>] __do_softirq+0xf5/0x280
[<ffffffff9caa4f48>] run_ksoftirqd+0x38/0x50
[<ffffffff9cace9e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff9cace8a0>] ? lg_double_unlock+0x40/0x40
[<ffffffff9cac5f91>] kthread+0xd1/0xe0
[<ffffffff9cac5ec0>] ? insert_kthread_work+0x40/0x40
[<ffffffff9d199df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9cac5ec0>] ? insert_kthread_work+0x40/0x40
Code: 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 <4c> 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 
Complete report #800487
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffff813ad74d
Build id: 3.10.0-1160.81.1.el7.x86_64
0xd
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffff813c8c5b
Build id: 3.10.0-1160.81.1.el7.x86_64
0x7b
3
swiotlb_sync_single
vmlinux 0xffffffff813c8cad
Build id: 3.10.0-1160.81.1.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffff813c8ccc
Build id: 3.10.0-1160.81.1.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc023f3ad
Build id: 3.10.0-1160.81.1.el7.x86_64
0x19d
6
e1000_clean
e1000 0xffffffffc0240180
Build id: 3.10.0-1160.81.1.el7.x86_64
0x280
7
? sk_reset_timer
vmlinux 0xffffffff816632a8
Build id: 3.10.0-1160.81.1.el7.x86_64
0x18
8
? tcp_send_loss_probe
vmlinux 0xffffffff816ee00a
Build id: 3.10.0-1160.81.1.el7.x86_64
0x8a
9
net_rx_action
vmlinux 0xffffffff81680037
Build id: 3.10.0-1160.81.1.el7.x86_64
0x277
10
__do_softirq
vmlinux 0xffffffff810a9545
Build id: 3.10.0-1160.81.1.el7.x86_64
0xf5
11
run_ksoftirqd
vmlinux 0xffffffff810a9718
Build id: 3.10.0-1160.81.1.el7.x86_64
0x38
12
smpboot_thread_fn
vmlinux 0xffffffff810d42ac
Build id: 3.10.0-1160.81.1.el7.x86_64
0x14c
13
? lg_double_unlock
vmlinux 0xffffffff810d4160
Build id: 3.10.0-1160.81.1.el7.x86_64
0x40
14
kthread
vmlinux 0xffffffff810cb4d1
Build id: 3.10.0-1160.81.1.el7.x86_64
0xd1
15
? insert_kthread_work
vmlinux 0xffffffff810cb400
Build id: 3.10.0-1160.81.1.el7.x86_64
0x40
16
ret_from_fork_nospec_begin
vmlinux 0xffffffff817c51dd
Build id: 3.10.0-1160.81.1.el7.x86_64
0x7
17
? insert_kthread_work
vmlinux 0xffffffff810cb400
Build id: 3.10.0-1160.81.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [ksoftirqd/0:6]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon pcspkr edac_mce_amd joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_common vmwgfx drm_kms_helper ata_generic pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptsas ahci scsi_transport_sas libahci ata_piix mptscsih drm serio_raw libata e1000 mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff9e2979053180 ti: ffff9e2979060000 task.ti: ffff9e2979060000
RIP: 0010:[<ffffffff813ad74d>]  [<ffffffff813ad74d>] memcpy+0xd/0x130
RSP: 0000:ffff9e2979063c08  EFLAGS: 00000246
RAX: ffff9e2c87a29ec0 RBX: ffffffff8170bcf9 RCX: 0000000000000008
RDX: 0000000000000000 RSI: ffff9e28bf7ee000 RDI: ffff9e2c87a29ec0
RBP: ffff9e2979063c10 R08: 0000000000000000 R09: 0000000487a29ec0
R10: ffff9e2ae4a1f120 R11: fffff2240b72dd00 R12: ffff9e2979393260
R13: ffff9e2adcb74400 R14: ffffffff816f9b00 R15: ffff9e2979063b00
FS:  00007f6ef8789700(0000) GS:ffff9e2ae4a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f7224a8c1c8 CR3: 0000000614010000 CR4: 00000000000007f0
Call Trace:
[<ffffffff813c8c5b>] ? swiotlb_tbl_sync_single+0x7b/0xa0
[<ffffffff813c8cad>] swiotlb_sync_single+0x2d/0x40
[<ffffffff813c8ccc>] swiotlb_sync_single_for_cpu+0xc/0x20
[<ffffffffc023f3ad>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc0240180>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff816632a8>] ? sk_reset_timer+0x18/0x30
[<ffffffff816ee00a>] ? tcp_send_loss_probe+0x8a/0x250
[<ffffffff81680037>] net_rx_action+0x277/0x3a0
[<ffffffff810a9545>] __do_softirq+0xf5/0x290
[<ffffffff810a9718>] run_ksoftirqd+0x38/0x60
[<ffffffff810d42ac>] smpboot_thread_fn+0x14c/0x1b0
[<ffffffff810d4160>] ? lg_double_unlock+0x40/0x40
[<ffffffff810cb4d1>] kthread+0xd1/0xe0
[<ffffffff810cb400>] ? insert_kthread_work+0x40/0x40
[<ffffffff817c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810cb400>] ? insert_kthread_work+0x40/0x40
Code: c1 e2 20 89 c0 48 09 c2 48 31 d3 e9 73 ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 
Complete report #792938
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffff9f95ab26
Build id: 3.10.0-862.9.1.el7.x86_64
0x16
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffff9f974973
Build id: 3.10.0-862.9.1.el7.x86_64
0x73
3
swiotlb_sync_single
vmlinux 0xffffffff9f9749bd
Build id: 3.10.0-862.9.1.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffff9f9749dc
Build id: 3.10.0-862.9.1.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc013e32d
Build id: 3.10.0-862.9.1.el7.x86_64
0x19d
6
e1000_clean
e1000 0xffffffffc013f0eb
Build id: 3.10.0-862.9.1.el7.x86_64
0x27b
7
? tcp_init_xmit_timers
vmlinux 0xffffffff9fc54090
Build id: 3.10.0-862.9.1.el7.x86_64
0x30
8
net_rx_action
vmlinux 0xffffffff9fbed46f
Build id: 3.10.0-862.9.1.el7.x86_64
0x26f
9
__do_softirq
vmlinux 0xffffffff9f69b085
Build id: 3.10.0-862.9.1.el7.x86_64
0xf5
10
run_ksoftirqd
vmlinux 0xffffffff9f69b248
Build id: 3.10.0-862.9.1.el7.x86_64
0x38
11
smpboot_thread_fn
vmlinux 0xffffffff9f6c3dd4
Build id: 3.10.0-862.9.1.el7.x86_64
0x144
12
? lg_double_unlock
vmlinux 0xffffffff9f6c3c90
Build id: 3.10.0-862.9.1.el7.x86_64
0x40
13
kthread
vmlinux 0xffffffff9f6bb621
Build id: 3.10.0-862.9.1.el7.x86_64
0xd1
14
? insert_kthread_work
vmlinux 0xffffffff9f6bb550
Build id: 3.10.0-862.9.1.el7.x86_64
0x40
15
ret_from_fork_nospec_begin
vmlinux 0xffffffff9fd205f7
Build id: 3.10.0-862.9.1.el7.x86_64
0x21
16
? insert_kthread_work
vmlinux 0xffffffff9f6bb550
Build id: 3.10.0-862.9.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [ksoftirqd/7:44]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_15508(E) falcon_lsm_pinned_15402(E) falcon_lsm_pinned_15309(E) falcon_lsm_pinned_15110(E) falcon_lsm_pinned_15003(E) falcon_lsm_pinned_14812(E) falcon_lsm_pinned_14810(E) falcon_lsm_pinned_14713(E) falcon_lsm_pinned_14712(E) falcon_lsm_pinned_14604(E) falcon_lsm_pinned_14504(E) falcon_lsm_pinned_14306(E) falcon_lsm_pinned_14203(E) falcon_lsm_pinned_14006(E) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink falcon_lsm_pinned_13804(E) vmw_vsock_vmci_transport vsock
sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr joydev i2c_piix4 vmw_vmci sg nfit shpchp libnvdimm parport_pc parport nfsd nfs_acl lockd auth_rpcgss grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm crct10dif_pclmul crct10dif_common crc32c_intel libata mptspi serio_raw scsi_transport_spi mptscsih e1000 mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: falcon_kal]
CPU: 7 PID: 44 Comm: ksoftirqd/7 Tainted: P        W   EL ------------   3.10.0-862.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9eb03a389fa0 ti: ffff9eb03a398000 task.ti: ffff9eb03a398000
RIP: 0010:[<ffffffff9f95ab26>]  [<ffffffff9f95ab26>] memcpy+0x16/0x110
RSP: 0018:ffff9eb03a39bc08  EFLAGS: 00010206
RAX: ffff9eb19b1edac0 RBX: ffffffff9f6f944f RCX: 0000000000000002
RDX: 0000000000000006 RSI: ffff9eaf7f542800 RDI: ffff9eb19b1edac0
RBP: ffff9eb03a39bc10 R08: 0000000000000000 R09: 00000002db1edac0
R10: 000000001bf20000 R11: 0000000000000400 R12: ffff9eb2d0ae7260
R13: ffff9eb1f3df1700 R14: ffffffff9fc5e600 R15: ffff9eb03a39bb00
FS:  0000000000000000(0000) GS:ffff9eb2e1dc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5e4631a000 CR3: 00000000b9fa4000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9f974973>] ? swiotlb_tbl_sync_single+0x73/0x90
[<ffffffff9f9749bd>] swiotlb_sync_single+0x2d/0x40
[<ffffffff9f9749dc>] swiotlb_sync_single_for_cpu+0xc/0x10
[<ffffffffc013e32d>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc013f0eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffff9fc54090>] ? tcp_init_xmit_timers+0x30/0x30
[<ffffffff9fbed46f>] net_rx_action+0x26f/0x390
[<ffffffff9f69b085>] __do_softirq+0xf5/0x280
[<ffffffff9f69b248>] run_ksoftirqd+0x38/0x50
[<ffffffff9f6c3dd4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff9f6c3c90>] ? lg_double_unlock+0x40/0x40
[<ffffffff9f6bb621>] kthread+0xd1/0xe0
[<ffffffff9f6bb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff9fd205f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9f6bb550>] ? insert_kthread_work+0x40/0x40
Code: e2 20 89 c0 48 09 c2 48 31 d3 e9 7b ff ff ff 90 90 90 90 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 <4c> 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 4c 89 
Complete report #655484
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffffb1bad769
Build id: 3.10.0-1160.80.1.el7.x86_64
0x29
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffffb1bc8c5b
Build id: 3.10.0-1160.80.1.el7.x86_64
0x7b
3
swiotlb_sync_single
vmlinux 0xffffffffb1bc8cad
Build id: 3.10.0-1160.80.1.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffffb1bc8ccc
Build id: 3.10.0-1160.80.1.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc01703ad
Build id: 3.10.0-1160.80.1.el7.x86_64
0x19d
6
e1000_clean
e1000 0xffffffffc0171180
Build id: 3.10.0-1160.80.1.el7.x86_64
0x280
7
? __queue_work
vmlinux 0xffffffffb18c105c
Build id: 3.10.0-1160.80.1.el7.x86_64
0x13c
8
net_rx_action
vmlinux 0xffffffffb1e80037
Build id: 3.10.0-1160.80.1.el7.x86_64
0x277
9
__do_softirq
vmlinux 0xffffffffb18a9545
Build id: 3.10.0-1160.80.1.el7.x86_64
0xf5
10
run_ksoftirqd
vmlinux 0xffffffffb18a9718
Build id: 3.10.0-1160.80.1.el7.x86_64
0x38
11
smpboot_thread_fn
vmlinux 0xffffffffb18d42ac
Build id: 3.10.0-1160.80.1.el7.x86_64
0x14c
12
? lg_double_unlock
vmlinux 0xffffffffb18d4160
Build id: 3.10.0-1160.80.1.el7.x86_64
0x40
13
kthread
vmlinux 0xffffffffb18cb4d1
Build id: 3.10.0-1160.80.1.el7.x86_64
0xd1
14
? insert_kthread_work
vmlinux 0xffffffffb18cb400
Build id: 3.10.0-1160.80.1.el7.x86_64
0x40
15
ret_from_fork_nospec_begin
vmlinux 0xffffffffb1fc51f7
Build id: 3.10.0-1160.80.1.el7.x86_64
0x21
16
? insert_kthread_work
vmlinux 0xffffffffb18cb400
Build id: 3.10.0-1160.80.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 29s! [ksoftirqd/0:6]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache vmw_vsock_vmci_transport vsock coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr joydev sg vmw_balloon parport_pc parport vmw_vmci i2c_piix4 nfsd nfs_acl lockd sch_fq_codel grace auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi ata_piix scsi_transport_spi mptscsih crct10dif_pclmul crct10dif_common crc32c_intel mptbase libata e1000 serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.80.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff9ee9fb6f3180 ti: ffff9ee9fb704000 task.ti: ffff9ee9fb704000
RIP: 0010:[<ffffffffb1bad769>]  [<ffffffffb1bad769>] memcpy+0x29/0x130
RSP: 0000:ffff9ee9fb707c08  EFLAGS: 00010246
RAX: ffff9eeb2cc39ec0 RBX: ffffffffb190e8df RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffff9eea3f7c5020 RDI: ffff9eeb2cc39ec0
RBP: ffff9ee9fb707c10 R08: 78f0ffffffffffff R09: 010006086232f316
R10: 78f0010004060008 R11: 0149a8c06232f316 R12: ffff9ee9b5be9260
R13: ffff9ee9f20f4400 R14: ffff9ee9b5be9260 R15: ffff9ee9fb707b00
FS:  0000000000000000(0000) GS:ffff9ee9fd200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000021b1610 CR3: 000000003ca10000 CR4: 00000000000007f0
Call Trace:
[<ffffffffb1bc8c5b>] ? swiotlb_tbl_sync_single+0x7b/0xa0
[<ffffffffb1bc8cad>] swiotlb_sync_single+0x2d/0x40
[<ffffffffb1bc8ccc>] swiotlb_sync_single_for_cpu+0xc/0x20
[<ffffffffc01703ad>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc0171180>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffb18c105c>] ? __queue_work+0x13c/0x400
[<ffffffffb1e80037>] net_rx_action+0x277/0x3a0
[<ffffffffb18a9545>] __do_softirq+0xf5/0x290
[<ffffffffb18a9718>] run_ksoftirqd+0x38/0x60
[<ffffffffb18d42ac>] smpboot_thread_fn+0x14c/0x1b0
[<ffffffffb18d4160>] ? lg_double_unlock+0x40/0x40
[<ffffffffb18cb4d1>] kthread+0xd1/0xe0
[<ffffffffb18cb400>] ? insert_kthread_work+0x40/0x40
[<ffffffffb1fc51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb18cb400>] ? insert_kthread_work+0x40/0x40
Code: 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 <4c> 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 
Complete report #652988
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffffab7960a9
Build id: 3.10.0-1127.19.1.el7.x86_64
0x29
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffffab7b0943
Build id: 3.10.0-1127.19.1.el7.x86_64
0x73
3
swiotlb_sync_single
vmlinux 0xffffffffab7b098d
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffffab7b09ac
Build id: 3.10.0-1127.19.1.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc01b833d
Build id: 3.10.0-1127.19.1.el7.x86_64
0x19d
6
e1000_clean
e1000 0xffffffffc01b9100
Build id: 3.10.0-1127.19.1.el7.x86_64
0x280
7
? tcp_schedule_loss_probe
vmlinux 0xffffffffababc49d
Build id: 3.10.0-1127.19.1.el7.x86_64
0x1d
8
? tcp_write_xmit
vmlinux 0xffffffffababd523
Build id: 3.10.0-1127.19.1.el7.x86_64
0x353
9
net_rx_action
vmlinux 0xffffffffaba545bf
Build id: 3.10.0-1127.19.1.el7.x86_64
0x26f
10
__do_softirq
vmlinux 0xffffffffab4a5695
Build id: 3.10.0-1127.19.1.el7.x86_64
0xf5
11
run_ksoftirqd
vmlinux 0xffffffffab4a5858
Build id: 3.10.0-1127.19.1.el7.x86_64
0x38
12
smpboot_thread_fn
vmlinux 0xffffffffab4cf0e4
Build id: 3.10.0-1127.19.1.el7.x86_64
0x144
13
? lg_double_unlock
vmlinux 0xffffffffab4cefa0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
14
kthread
vmlinux 0xffffffffab4c6691
Build id: 3.10.0-1127.19.1.el7.x86_64
0xd1
15
? insert_kthread_work
vmlinux 0xffffffffab4c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
16
ret_from_fork_nospec_begin
vmlinux 0xffffffffabb92d37
Build id: 3.10.0-1127.19.1.el7.x86_64
0x21
17
? insert_kthread_work
vmlinux 0xffffffffab4c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [ksoftirqd/4:29]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 sb_edac snd_rawmidi snd_ac97_codec iosf_mbi ppdev crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq aesni_intel lrw gf128mul glue_helper vmw_balloon snd_seq_device ablk_helper snd_pcm cryptd snd_timer joydev snd pcspkr soundcore sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common
crc32c_intel nfit ata_piix libnvdimm mptspi serio_raw libata e1000 scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 29 Comm: ksoftirqd/4 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9b21398f1070 ti: ffff9b21398fc000 task.ti: ffff9b21398fc000
RIP: 0010:[<ffffffffab7960a9>]  [<ffffffffab7960a9>] memcpy+0x29/0x110
RSP: 0000:ffff9b21398ffc08  EFLAGS: 00010246
RAX: ffff9b238a359ec0 RBX: ffffffffab5082af RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffff9b207e849820 RDI: ffff9b238a359ec0
RBP: ffff9b21398ffc10 R08: 0c0066837b290c00 R09: 004500088a53ee29
R10: 06400040bad62800 R11: a8c0f501a8c0dfde R12: ffff9b23ec79f260
R13: ffff9b20cdbef400 R14: ffffffffabacaf00 R15: ffff9b21398ffb00
FS:  0000000000000000(0000) GS:ffff9b23ed700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f561a1e0bd0 CR3: 0000000421e2e000 CR4: 00000000003607e0
Call Trace:
[<ffffffffab7b0943>] ? swiotlb_tbl_sync_single+0x73/0x90
[<ffffffffab7b098d>] swiotlb_sync_single+0x2d/0x40
[<ffffffffab7b09ac>] swiotlb_sync_single_for_cpu+0xc/0x10
[<ffffffffc01b833d>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc01b9100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffababc49d>] ? tcp_schedule_loss_probe+0x1d/0x1e0
[<ffffffffababd523>] ? tcp_write_xmit+0x353/0xd40
[<ffffffffaba545bf>] net_rx_action+0x26f/0x390
[<ffffffffab4a5695>] __do_softirq+0xf5/0x280
[<ffffffffab4a5858>] run_ksoftirqd+0x38/0x50
[<ffffffffab4cf0e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffab4cefa0>] ? lg_double_unlock+0x40/0x40
[<ffffffffab4c6691>] kthread+0xd1/0xe0
[<ffffffffab4c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffabb92d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffab4c65c0>] ? insert_kthread_work+0x40/0x40
Code: 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 <4c> 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 
Complete report #624356
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffffa21ad74d
Build id: 3.10.0-1160.81.1.el7.x86_64
0xd
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffffa21c8c5b
Build id: 3.10.0-1160.81.1.el7.x86_64
0x7b
3
swiotlb_sync_single
vmlinux 0xffffffffa21c8cad
Build id: 3.10.0-1160.81.1.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffffa21c8ccc
Build id: 3.10.0-1160.81.1.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc01a63ad
Build id: 3.10.0-1160.81.1.el7.x86_64
0x19d
6
e1000_clean
e1000 0xffffffffc01a7180
Build id: 3.10.0-1160.81.1.el7.x86_64
0x280
7
net_rx_action
vmlinux 0xffffffffa2480037
Build id: 3.10.0-1160.81.1.el7.x86_64
0x277
8
__do_softirq
vmlinux 0xffffffffa1ea9545
Build id: 3.10.0-1160.81.1.el7.x86_64
0xf5
9
run_ksoftirqd
vmlinux 0xffffffffa1ea9718
Build id: 3.10.0-1160.81.1.el7.x86_64
0x38
10
smpboot_thread_fn
vmlinux 0xffffffffa1ed42ac
Build id: 3.10.0-1160.81.1.el7.x86_64
0x14c
11
? lg_double_unlock
vmlinux 0xffffffffa1ed4160
Build id: 3.10.0-1160.81.1.el7.x86_64
0x40
12
kthread
vmlinux 0xffffffffa1ecb4d1
Build id: 3.10.0-1160.81.1.el7.x86_64
0xd1
13
? insert_kthread_work
vmlinux 0xffffffffa1ecb400
Build id: 3.10.0-1160.81.1.el7.x86_64
0x40
14
ret_from_fork_nospec_begin
vmlinux 0xffffffffa25c51dd
Build id: 3.10.0-1160.81.1.el7.x86_64
0x7
15
? insert_kthread_work
vmlinux 0xffffffffa1ecb400
Build id: 3.10.0-1160.81.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ksoftirqd/0:6]
Modules linked in: tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon edac_mce_amd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic vmwgfx crct10dif_common drm_kms_helper syscopyarea ata_generic sysfillrect pata_acpi sysimgblt fb_sys_fops ttm ahci mptsas drm ata_piix libahci scsi_transport_sas mptscsih libata e1000 mptbase serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff8ad9b907b180 ti: ffff8ad9b908c000 task.ti: ffff8ad9b908c000
RIP: 0010:[<ffffffffa21ad74d>]  [<ffffffffa21ad74d>] memcpy+0xd/0x130
RSP: 0018:ffff8ad9b908fc08  EFLAGS: 00000206
RAX: ffff8ae32a820840 RBX: 00000000000005f2 RCX: 0000000000000009
RDX: 0000000000000006 RSI: ffff8ad8fc1a3800 RDI: ffff8ae32a820840
RBP: ffff8ad9b908fc10 R08: 0000000000000000 R09: 0000000aea820840
R10: ffff8adb24a1f120 R11: ffffe3b6210f6080 R12: 000000000001f120
R13: ffffffffa2466c7f R14: ffff8adb24a1f120 R15: ffffe3b6210f6080
FS:  00007fdceb0e4740(0000) GS:ffff8adb24a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f184ea64c80 CR3: 00000002d4830000 CR4: 00000000000007f0
Call Trace:
[<ffffffffa21c8c5b>] ? swiotlb_tbl_sync_single+0x7b/0xa0
[<ffffffffa21c8cad>] swiotlb_sync_single+0x2d/0x40
[<ffffffffa21c8ccc>] swiotlb_sync_single_for_cpu+0xc/0x20
[<ffffffffc01a63ad>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc01a7180>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffa2480037>] net_rx_action+0x277/0x3a0
[<ffffffffa1ea9545>] __do_softirq+0xf5/0x290
[<ffffffffa1ea9718>] run_ksoftirqd+0x38/0x60
[<ffffffffa1ed42ac>] smpboot_thread_fn+0x14c/0x1b0
[<ffffffffa1ed4160>] ? lg_double_unlock+0x40/0x40
[<ffffffffa1ecb4d1>] kthread+0xd1/0xe0
[<ffffffffa1ecb400>] ? insert_kthread_work+0x40/0x40
[<ffffffffa25c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffa1ecb400>] ? insert_kthread_work+0x40/0x40
Code: c1 e2 20 89 c0 48 09 c2 48 31 d3 e9 73 ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 
Complete report #373083
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffff99196029
Build id: 3.10.0-1127.8.2.el7.x86_64
0x29
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffff991b08c3
Build id: 3.10.0-1127.8.2.el7.x86_64
0x73
3
swiotlb_sync_single
vmlinux 0xffffffff991b090d
Build id: 3.10.0-1127.8.2.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffff991b092c
Build id: 3.10.0-1127.8.2.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc00af33d
Build id: 3.10.0-1127.8.2.el7.x86_64
0x19d
6
e1000_clean
e1000 0xffffffffc00b0100
Build id: 3.10.0-1127.8.2.el7.x86_64
0x280
7
? nf_ct_delete
nf_conntrack 0xffffffffc047602d
Build id: 3.10.0-1127.8.2.el7.x86_64
0x18d
8
? mod_timer
vmlinux 0xffffffff98eaf5ec
Build id: 3.10.0-1127.8.2.el7.x86_64
0x10c
9
net_rx_action
vmlinux 0xffffffff994545bf
Build id: 3.10.0-1127.8.2.el7.x86_64
0x26f
10
__do_softirq
vmlinux 0xffffffff98ea5695
Build id: 3.10.0-1127.8.2.el7.x86_64
0xf5
11
run_ksoftirqd
vmlinux 0xffffffff98ea5858
Build id: 3.10.0-1127.8.2.el7.x86_64
0x38
12
smpboot_thread_fn
vmlinux 0xffffffff98ecf0e4
Build id: 3.10.0-1127.8.2.el7.x86_64
0x144
13
? lg_double_unlock
vmlinux 0xffffffff98ecefa0
Build id: 3.10.0-1127.8.2.el7.x86_64
0x40
14
kthread
vmlinux 0xffffffff98ec6691
Build id: 3.10.0-1127.8.2.el7.x86_64
0xd1
15
? insert_kthread_work
vmlinux 0xffffffff98ec65c0
Build id: 3.10.0-1127.8.2.el7.x86_64
0x40
16
ret_from_fork_nospec_begin
vmlinux 0xffffffff99592d37
Build id: 3.10.0-1127.8.2.el7.x86_64
0x21
17
? insert_kthread_work
vmlinux 0xffffffff98ec65c0
Build id: 3.10.0-1127.8.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 21s! [ksoftirqd/4:29]
Modules linked in: talpa_vfshook(OE) talpa_pedconnector(OE) talpa_pedevice(OE) talpa_vcdevice(OE) talpa_core(OE) talpa_linux(OE) talpa_syscallhook(OE) ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_ipv4 nf_log_common ip6table_filter nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr vmw_balloon joydev parport_pc sg parport vmw_vmci i2c_piix4 tcp_htcp ip_tables xfs libcrc32c sr_mod
cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea sysfillrect sysimgblt crc_t10dif fb_sys_fops crct10dif_generic ttm ata_piix crct10dif_pclmul crct10dif_common libata drm crc32c_intel serio_raw e1000 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 29 Comm: ksoftirqd/4 Kdump: loaded Tainted: G        W  OEL ------------   3.10.0-1127.8.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff9acf39c88000 ti: ffff9acf39c84000 task.ti: ffff9acf39c84000
RIP: 0010:[<ffffffff99196029>]  [<ffffffff99196029>] memcpy+0x29/0x110
RSP: 0018:ffff9acf39c87c08  EFLAGS: 00010246
RAX: ffff9acf44590fc0 RBX: ffffffff98f0825f RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffff9ace7f513020 RDI: ffff9acf44590fc0
RBP: ffff9acf39c87c10 R08: ff00ffffffffffff R09: 01000608feffffff
R10: ff00010004060008 R11: fc7772a7feffffff R12: ffff9ad1ef2a3260
R13: ffff9ad1ee411400 R14: ffff9ad1ef2a3260 R15: ffff9acf39c87b00
FS:  0000000000000000(0000) GS:ffff9ad1ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9383b06390 CR3: 000000042ad6c000 CR4: 00000000000407e0
Call Trace:
[<ffffffff991b08c3>] ? swiotlb_tbl_sync_single+0x73/0x90
[<ffffffff991b090d>] swiotlb_sync_single+0x2d/0x40
[<ffffffff991b092c>] swiotlb_sync_single_for_cpu+0xc/0x10
[<ffffffffc00af33d>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc00b0100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffc047602d>] ? nf_ct_delete+0x18d/0x270 [nf_conntrack]
[<ffffffff98eaf5ec>] ? mod_timer+0x10c/0x230
[<ffffffff994545bf>] net_rx_action+0x26f/0x390
[<ffffffff98ea5695>] __do_softirq+0xf5/0x280
[<ffffffff98ea5858>] run_ksoftirqd+0x38/0x50
[<ffffffff98ecf0e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff98ecefa0>] ? lg_double_unlock+0x40/0x40
[<ffffffff98ec6691>] kthread+0xd1/0xe0
[<ffffffff98ec65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff99592d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff98ec65c0>] ? insert_kthread_work+0x40/0x40
Code: 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 <4c> 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 
Complete report #330870
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffff84596e76
Build id: 3.10.0-1160.el7.x86_64
0x16
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffff845b1813
Build id: 3.10.0-1160.el7.x86_64
0x73
3
swiotlb_sync_single
vmlinux 0xffffffff845b185d
Build id: 3.10.0-1160.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffff845b187c
Build id: 3.10.0-1160.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc03ec33d
Build id: 3.10.0-1160.el7.x86_64
0x19d
6
? sched_clock_cpu
vmlinux 0xffffffff842de185
Build id: 3.10.0-1160.el7.x86_64
0x85
7
e1000_clean
e1000 0xffffffffc03ed100
Build id: 3.10.0-1160.el7.x86_64
0x280
8
? update_curr
vmlinux 0xffffffff842e43cc
Build id: 3.10.0-1160.el7.x86_64
0x14c
9
net_rx_action
vmlinux 0xffffffff848555bf
Build id: 3.10.0-1160.el7.x86_64
0x26f
10
__do_softirq
vmlinux 0xffffffff842a4b95
Build id: 3.10.0-1160.el7.x86_64
0xf5
11
run_ksoftirqd
vmlinux 0xffffffff842a4d58
Build id: 3.10.0-1160.el7.x86_64
0x38
12
smpboot_thread_fn
vmlinux 0xffffffff842ce674
Build id: 3.10.0-1160.el7.x86_64
0x144
13
? lg_double_unlock
vmlinux 0xffffffff842ce530
Build id: 3.10.0-1160.el7.x86_64
0x40
14
kthread
vmlinux 0xffffffff842c5c21
Build id: 3.10.0-1160.el7.x86_64
0xd1
15
? insert_kthread_work
vmlinux 0xffffffff842c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
16
ret_from_fork_nospec_begin
vmlinux 0xffffffff84993df7
Build id: 3.10.0-1160.el7.x86_64
0x21
17
? insert_kthread_work
vmlinux 0xffffffff842c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [ksoftirqd/0:6]
Modules linked in: tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 sb_edac snd_rawmidi snd_ac97_codec coretemp ac97_bus iosf_mbi crc32_pclmul snd_seq ghash_clmulni_intel snd_seq_device aesni_intel lrw snd_pcm gf128mul glue_helper snd_timer ablk_helper cryptd snd
ppdev vmw_balloon soundcore pcspkr joydev sg i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm crct10dif_pclmul crct10dif_common crc32c_intel mptspi scsi_transport_spi libata mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff88f72375d280 ti: ffff88f72435c000 task.ti: ffff88f72435c000
RIP: 0010:[<ffffffff84596e76>]  [<ffffffff84596e76>] memcpy+0x16/0x110
RSP: 0018:ffff88f72435fc08  EFLAGS: 00010246
RAX: ffff89048a6969c0 RBX: ffffffff843078df RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffff88f67bf14000 RDI: ffff89048a6969c0
RBP: ffff88f72435fc10 R08: 0000000000000000 R09: 0000000eca6969c0
R10: 0000000000000003 R11: 0000000000000100 R12: ffff88f73d37f260
R13: ffff88fa0c507700 R14: ffff88fa0c507700 R15: ffff88f72435fb00
FS:  0000000000000000(0000) GS:ffff8905bd600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00001b28c45d0c00 CR3: 000000010e3ae000 CR4: 00000000001607f0
Call Trace:
[<ffffffff845b1813>] ? swiotlb_tbl_sync_single+0x73/0x90
[<ffffffff845b185d>] swiotlb_sync_single+0x2d/0x40
[<ffffffff845b187c>] swiotlb_sync_single_for_cpu+0xc/0x10
[<ffffffffc03ec33d>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffff842de185>] ? sched_clock_cpu+0x85/0xc0
[<ffffffffc03ed100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff842e43cc>] ? update_curr+0x14c/0x1e0
[<ffffffff848555bf>] net_rx_action+0x26f/0x390
[<ffffffff842a4b95>] __do_softirq+0xf5/0x280
[<ffffffff842a4d58>] run_ksoftirqd+0x38/0x50
[<ffffffff842ce674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff842ce530>] ? lg_double_unlock+0x40/0x40
[<ffffffff842c5c21>] kthread+0xd1/0xe0
[<ffffffff842c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff84993df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff842c5b50>] ? insert_kthread_work+0x40/0x40
Code: e2 20 89 c0 48 09 c2 48 31 d3 e9 7b ff ff ff 90 90 90 90 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 <4c> 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 4c 89 
Complete report #324458
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffffa8b5a089
Build id: 3.10.0-862.el7.x86_64
0x29
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffffa8b73ec3
Build id: 3.10.0-862.el7.x86_64
0x73
3
swiotlb_sync_single
vmlinux 0xffffffffa8b73f0d
Build id: 3.10.0-862.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffffa8b73f2c
Build id: 3.10.0-862.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc042232d
Build id: 3.10.0-862.el7.x86_64
0x19d
6
e1000_clean
e1000 0xffffffffc04230eb
Build id: 3.10.0-862.el7.x86_64
0x27b
7
? update_curr
vmlinux 0xffffffffa88d816c
Build id: 3.10.0-862.el7.x86_64
0x14c
8
? account_entity_dequeue
vmlinux 0xffffffffa88d4ade
Build id: 3.10.0-862.el7.x86_64
0xae
9
net_rx_action
vmlinux 0xffffffffa8dec3af
Build id: 3.10.0-862.el7.x86_64
0x26f
10
__do_softirq
vmlinux 0xffffffffa889a945
Build id: 3.10.0-862.el7.x86_64
0xf5
11
run_ksoftirqd
vmlinux 0xffffffffa889ab08
Build id: 3.10.0-862.el7.x86_64
0x38
12
smpboot_thread_fn
vmlinux 0xffffffffa88c35e4
Build id: 3.10.0-862.el7.x86_64
0x144
13
? lg_double_unlock
vmlinux 0xffffffffa88c34a0
Build id: 3.10.0-862.el7.x86_64
0x40
14
kthread
vmlinux 0xffffffffa88bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
15
? insert_kthread_work
vmlinux 0xffffffffa88bad60
Build id: 3.10.0-862.el7.x86_64
0x40
16
ret_from_fork_nospec_begin
vmlinux 0xffffffffa8f1f637
Build id: 3.10.0-862.el7.x86_64
0x21
17
? insert_kthread_work
vmlinux 0xffffffffa88bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [ksoftirqd/4:30]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ppdev vmw_balloon coretemp pcspkr joydev sg vmw_vmci parport_pc i2c_piix4 parport shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi vmwgfx
drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata mptspi scsi_transport_spi e1000 mptscsih serio_raw mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 30 Comm: ksoftirqd/4 Kdump: loaded Not tainted 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff88e02c01cf10 ti: ffff88e02c038000 task.ti: ffff88e02c038000
RIP: 0010:[<ffffffffa8b5a089>]  [<ffffffffa8b5a089>] memcpy+0x29/0x110
RSP: 0018:ffff88e02c03bc08  EFLAGS: 00010246
RAX: ffff88e5903f80c0 RBX: ffffffffa88f8c5f RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffff88ddfc087020 RDI: ffff88e5903f80c0
RBP: ffff88e02c03bc10 R08: 00c4ffffffffffff R09: 0100060853dca4ad
R10: 00c4010004060008 R11: 41b0dcdb53dca4ad R12: ffff88e40698d260
R13: ffff88e1ca76d400 R14: ffffffffa8e5d600 R15: ffff88e02c03bb00
FS:  0000000000000000(0000) GS:ffff88e1d7000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fa5e06d4003 CR3: 000000004880e000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa8b73ec3>] ? swiotlb_tbl_sync_single+0x73/0x90
[<ffffffffa8b73f0d>] swiotlb_sync_single+0x2d/0x40
[<ffffffffa8b73f2c>] swiotlb_sync_single_for_cpu+0xc/0x10
[<ffffffffc042232d>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc04230eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffffa88d816c>] ? update_curr+0x14c/0x1e0
[<ffffffffa88d4ade>] ? account_entity_dequeue+0xae/0xd0
[<ffffffffa8dec3af>] net_rx_action+0x26f/0x390
[<ffffffffa889a945>] __do_softirq+0xf5/0x280
[<ffffffffa889ab08>] run_ksoftirqd+0x38/0x50
[<ffffffffa88c35e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa88c34a0>] ? lg_double_unlock+0x40/0x40
[<ffffffffa88bae31>] kthread+0xd1/0xe0
[<ffffffffa88bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa8f1f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa88bad60>] ? insert_kthread_work+0x40/0x40
Code: 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 <4c> 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 
Complete report #152400
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffffba7926dd
Build id: 3.10.0-1062.12.1.el7.x86_64
0xd
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffffba7acea3
Build id: 3.10.0-1062.12.1.el7.x86_64
0x73
3
swiotlb_sync_single
vmlinux 0xffffffffba7aceed
Build id: 3.10.0-1062.12.1.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffffba7acf0c
Build id: 3.10.0-1062.12.1.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc042e33d
Build id: 3.10.0-1062.12.1.el7.x86_64
0x19d
6
e1000_clean
e1000 0xffffffffc042f100
Build id: 3.10.0-1062.12.1.el7.x86_64
0x280
7
net_rx_action
vmlinux 0xffffffffbaa5057f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x26f
8
__do_softirq
vmlinux 0xffffffffba4a5305
Build id: 3.10.0-1062.12.1.el7.x86_64
0xf5
9
run_ksoftirqd
vmlinux 0xffffffffba4a54c8
Build id: 3.10.0-1062.12.1.el7.x86_64
0x38
10
smpboot_thread_fn
vmlinux 0xffffffffba4cec24
Build id: 3.10.0-1062.12.1.el7.x86_64
0x144
11
? lg_double_unlock
vmlinux 0xffffffffba4ceae0
Build id: 3.10.0-1062.12.1.el7.x86_64
0x40
12
kthread
vmlinux 0xffffffffba4c61f1
Build id: 3.10.0-1062.12.1.el7.x86_64
0xd1
13
? insert_kthread_work
vmlinux 0xffffffffba4c6120
Build id: 3.10.0-1062.12.1.el7.x86_64
0x40
14
ret_from_fork_nospec_begin
vmlinux 0xffffffffbab8dd24
Build id: 3.10.0-1062.12.1.el7.x86_64
0xe
15
? insert_kthread_work
vmlinux 0xffffffffba4c6120
Build id: 3.10.0-1062.12.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 29s! [ksoftirqd/0:6]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus ppdev snd_seq edac_mce_amd crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel snd_seq_device snd_pcm lrw gf128mul glue_helper ablk_helper snd_timer cryptd snd sg pcspkr soundcore vmw_vmci joydev i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix mptspi libata scsi_transport_spi serio_raw e1000 mptscsih mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Not tainted 3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
task: ffff923dbcf0d230 ti: ffff923dbcfe8000 task.ti: ffff923dbcfe8000
RIP: 0010:[<ffffffffba7926dd>]  [<ffffffffba7926dd>] memcpy+0xd/0x110
RSP: 0018:ffff923dbcfebc08  EFLAGS: 00000246
RAX: ffff923dca276240 RBX: ffffffffbaad91d9 RCX: 000000000000000c
RDX: 0000000000000000 RSI: ffff923cff118800 RDI: ffff923dca276240
RBP: ffff923dbcfebc10 R08: 0000000000000000 R09: 000000018a276240
R10: ffff923e7661f120 R11: ffffefd3c6f7e880 R12: ffff923dbca51260
R13: ffff923dfdfa2000 R14: ffff923dbca51260 R15: ffffffffbb165100
FS:  00007f6e5f94f700(0000) GS:ffff923e76600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fe7786c8000 CR3: 0000000227f96000 CR4: 00000000000407f0
Call Trace:
[<ffffffffba7acea3>] ? swiotlb_tbl_sync_single+0x73/0x90
[<ffffffffba7aceed>] swiotlb_sync_single+0x2d/0x40
[<ffffffffba7acf0c>] swiotlb_sync_single_for_cpu+0xc/0x10
[<ffffffffc042e33d>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc042f100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffbaa5057f>] net_rx_action+0x26f/0x390
[<ffffffffba4a5305>] __do_softirq+0xf5/0x280
[<ffffffffba4a54c8>] run_ksoftirqd+0x38/0x50
[<ffffffffba4cec24>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffba4ceae0>] ? lg_double_unlock+0x40/0x40
[<ffffffffba4c61f1>] kthread+0xd1/0xe0
[<ffffffffba4c6120>] ? insert_kthread_work+0x40/0x40
[<ffffffffbab8dd24>] ret_from_fork_nospec_begin+0xe/0x21
[<ffffffffba4c6120>] ? insert_kthread_work+0x40/0x40
Code: d0 ff 0f ae e8 0f 31 48 c1 e2 20 89 c0 48 09 c2 48 31 d3 e9 7b ff ff ff 90 90 90 90 90 90 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 
Complete report #115897
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffff813269d8
Build id: 3.10.0-514.10.2.el7.x86_64
0x68
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffff8133ae8e
Build id: 3.10.0-514.10.2.el7.x86_64
0x7e
3
swiotlb_sync_single
vmlinux 0xffffffff8133aec0
Build id: 3.10.0-514.10.2.el7.x86_64
0x20
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffff8133aedc
Build id: 3.10.0-514.10.2.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffa008e1df
Build id: 3.10.0-514.10.2.el7.x86_64
0x18f
6
e1000_clean
e1000 0xffffffffa008ed31
Build id: 3.10.0-514.10.2.el7.x86_64
0x281
7
? __queue_work
vmlinux 0xffffffff810a5f20
Build id: 3.10.0-514.10.2.el7.x86_64
0x140
8
net_rx_action
vmlinux 0xffffffff815705e0
Build id: 3.10.0-514.10.2.el7.x86_64
0x170
9
__do_softirq
vmlinux 0xffffffff8108f2cf
Build id: 3.10.0-514.10.2.el7.x86_64
0xef
10
run_ksoftirqd
vmlinux 0xffffffff8108f498
Build id: 3.10.0-514.10.2.el7.x86_64
0x38
11
smpboot_thread_fn
vmlinux 0xffffffff810b927f
Build id: 3.10.0-514.10.2.el7.x86_64
0x12f
12
? lg_double_unlock
vmlinux 0xffffffff810b9150
Build id: 3.10.0-514.10.2.el7.x86_64
0x90
13
kthread
vmlinux 0xffffffff810b06ff
Build id: 3.10.0-514.10.2.el7.x86_64
0xcf
14
? kthread_create_on_node
vmlinux 0xffffffff810b0630
Build id: 3.10.0-514.10.2.el7.x86_64
0x140
15
ret_from_fork
vmlinux 0xffffffff81696a58
Build id: 3.10.0-514.10.2.el7.x86_64
0x58
16
? kthread_create_on_node
vmlinux 0xffffffff810b0630
Build id: 3.10.0-514.10.2.el7.x86_64
0x140
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 21s! [ksoftirqd/4:28]
Modules linked in: fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_hda_codec_generic intel_powerclamp coretemp snd_hda_intel crc32_pclmul snd_hda_codec ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper snd_hda_core ablk_helper cryptd snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer ppdev snd vmw_balloon pcspkr soundcore sg vmw_vmci i2c_piix4 parport_pc parport shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common
crc32c_intel vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt serio_raw fb_sys_fops ttm mptspi scsi_transport_spi drm ata_piix mptscsih libata e1000 mptbase i2c_core fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 28 Comm: ksoftirqd/4 Tainted: G             L ------------   3.10.0-514.10.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff880174f19f60 ti: ffff880174f28000 task.ti: ffff880174f28000
RIP: 0010:[<ffffffff813269d8>]  [<ffffffff813269d8>] memcpy+0x68/0x110
RSP: 0018:ffff880174f2bc08  EFLAGS: 00010206
RAX: ffff880138400840 RBX: ffffffff81aeae40 RCX: 0000000000000002
RDX: 0000000000000006 RSI: ffff8800be98d826 RDI: ffff880138400886
RBP: ffff880174f2bc10 R08: 0000000002040101 R09: 0803030178050402
R10: 00002682f0fa0280 R11: 00000000ff79e78c R12: ffff8806d3605c00
R13: 0000000000000246 R14: ffffffff815dc074 R15: ffffffff8156fab2
FS:  0000000000000000(0000) GS:ffff88079f700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f4910390965 CR3: 00000007933d0000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffff8133ae8e ffff880174f2bc20 ffffffff8133aec0 ffff880174f2bc30
ffffffff8133aedc ffff880174f2bcd8 ffffffffa008e1df ffff8807a00af000
ffff8801750fa8c0 ffff8807a00af098 0000003674f2bd88 ffff8801750fa000
Call Trace:
[<ffffffff8133ae8e>] ? swiotlb_tbl_sync_single+0x7e/0x90
[<ffffffff8133aec0>] swiotlb_sync_single+0x20/0x30
[<ffffffff8133aedc>] swiotlb_sync_single_for_cpu+0xc/0x10
[<ffffffffa008e1df>] e1000_clean_rx_irq+0x18f/0x4f0 [e1000]
[<ffffffffa008ed31>] e1000_clean+0x281/0x8f0 [e1000]
[<ffffffff810a5f20>] ? __queue_work+0x140/0x320
[<ffffffff815705e0>] net_rx_action+0x170/0x380
[<ffffffff8108f2cf>] __do_softirq+0xef/0x280
[<ffffffff8108f498>] run_ksoftirqd+0x38/0x50
[<ffffffff810b927f>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b9150>] ? lg_double_unlock+0x90/0x90
[<ffffffff810b06ff>] kthread+0xcf/0xe0
[<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81696a58>] ret_from_fork+0x58/0x90
[<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
Code: d4 83 c2 20 eb 44 48 01 d6 48 01 d7 48 83 ea 20 0f 1f 00 48 83 ea 20 4c 8b 46 f8 4c 8b 4e f0 4c 8b 56 e8 4c 8b 5e e0 48 8d 76 e0 <4c> 89 47 f8 4c 89 4f f0 4c 89 57 e8 4c 89 5f e0 48 8d 7f e0 73 
Complete report #24178
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffff813267a4
Build id: 3.10.0-514.el7.x86_64
0x54
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffff8133ac6e
Build id: 3.10.0-514.el7.x86_64
0x7e
3
swiotlb_sync_single
vmlinux 0xffffffff8133aca0
Build id: 3.10.0-514.el7.x86_64
0x20
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffff8133acbc
Build id: 3.10.0-514.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffa005e1df
Build id: 3.10.0-514.el7.x86_64
0x18f
6
e1000_clean
e1000 0xffffffffa005ed31
Build id: 3.10.0-514.el7.x86_64
0x281
7
net_rx_action
vmlinux 0xffffffff815700b0
Build id: 3.10.0-514.el7.x86_64
0x170
8
__do_softirq
vmlinux 0xffffffff8108f0ff
Build id: 3.10.0-514.el7.x86_64
0xef
9
run_ksoftirqd
vmlinux 0xffffffff8108f2c8
Build id: 3.10.0-514.el7.x86_64
0x38
10
smpboot_thread_fn
vmlinux 0xffffffff810b90af
Build id: 3.10.0-514.el7.x86_64
0x12f
11
? lg_double_unlock
vmlinux 0xffffffff810b8f80
Build id: 3.10.0-514.el7.x86_64
0x90
12
kthread
vmlinux 0xffffffff810b052f
Build id: 3.10.0-514.el7.x86_64
0xcf
13
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7.x86_64
0x140
14
ret_from_fork
vmlinux 0xffffffff81696518
Build id: 3.10.0-514.el7.x86_64
0x58
15
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7.x86_64
0x140
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [ksoftirqd/2:18]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_ftp ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod intel_powerclamp coretemp ppdev vmw_balloon pcspkr sg vmw_vmci i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic
sr_mod cdrom crct10dif_common ata_generic pata_acpi vmwgfx crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm ata_piix mptspi scsi_transport_spi mptscsih libata e1000 mptbase i2c_core fjes
CPU: 2 PID: 18 Comm: ksoftirqd/2 Tainted: G             L ------------   3.10.0-514.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff880184ab0000 ti: ffff880184ab8000 task.ti: ffff880184ab8000
RIP: 0010:[<ffffffff813267a4>]  [<ffffffff813267a4>] memcpy+0x54/0x110
RSP: 0018:ffff880184abbc08  EFLAGS: 00010246
RAX: ffff8801e46b6240 RBX: ffffffff81aeaea0 RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffff8800bf2dd040 RDI: ffff8801e46b6280
RBP: ffff880184abbc10 R08: 0000000000000000 R09: 00000001e46b6240
R10: ffff880234699b00 R11: ffffea0002e12100 R12: 0000000000000000
R13: 000000000000004e R14: ffff8800b8485a00 R15: ffffffff8156f582
FS:  0000000000000000(0000) GS:ffff880234680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f71d4d49640 CR3: 0000000034bc7000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffff8133ac6e ffff880184abbc20 ffffffff8133aca0 ffff880184abbc30
ffffffff8133acbc ffff880184abbcd8 ffffffffa005e1df ffff88023522d000
ffff88022f1988c0 ffff88023522d098 0000000d00000000 ffff88022f198000
Call Trace:
[<ffffffff8133ac6e>] ? swiotlb_tbl_sync_single+0x7e/0x90
[<ffffffff8133aca0>] swiotlb_sync_single+0x20/0x30
[<ffffffff8133acbc>] swiotlb_sync_single_for_cpu+0xc/0x10
[<ffffffffa005e1df>] e1000_clean_rx_irq+0x18f/0x4f0 [e1000]
[<ffffffffa005ed31>] e1000_clean+0x281/0x8f0 [e1000]
[<ffffffff815700b0>] net_rx_action+0x170/0x380
[<ffffffff8108f0ff>] __do_softirq+0xef/0x280
[<ffffffff8108f2c8>] run_ksoftirqd+0x38/0x50
[<ffffffff810b90af>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f80>] ? lg_double_unlock+0x90/0x90
[<ffffffff810b052f>] kthread+0xcf/0xe0
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81696518>] ret_from_fork+0x58/0x90
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
Code: 4c 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 83 c2 20 eb 44 48 01 d6 48 01 d7 48 83 ea 20 0f 1f 00 48 83 ea 20 <4c> 8b 46 f8 4c 8b 4e f0 4c 8b 56 e8 4c 8b 5e e0 48 8d 76 e0 4c 
Complete report #50454
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffff85f86348
Build id: 3.10.0-957.el7.x86_64
0x68
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffff85fa0583
Build id: 3.10.0-957.el7.x86_64
0x73
3
swiotlb_sync_single
vmlinux 0xffffffff85fa05cd
Build id: 3.10.0-957.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffff85fa05ec
Build id: 3.10.0-957.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc02e731d
Build id: 3.10.0-957.el7.x86_64
0x19d
6
e1000_clean
e1000 0xffffffffc02e80dd
Build id: 3.10.0-957.el7.x86_64
0x27d
7
? nf_ct_delete
nf_conntrack 0xffffffffc064c06d
Build id: 3.10.0-957.el7.x86_64
0x18d
8
? update_curr
vmlinux 0xffffffff85ce00bc
Build id: 3.10.0-957.el7.x86_64
0x14c
9
net_rx_action
vmlinux 0xffffffff86239e9f
Build id: 3.10.0-957.el7.x86_64
0x26f
10
__do_softirq
vmlinux 0xffffffff85ca0f05
Build id: 3.10.0-957.el7.x86_64
0xf5
11
run_ksoftirqd
vmlinux 0xffffffff85ca10c8
Build id: 3.10.0-957.el7.x86_64
0x38
12
smpboot_thread_fn
vmlinux 0xffffffff85cca5c4
Build id: 3.10.0-957.el7.x86_64
0x144
13
? lg_double_unlock
vmlinux 0xffffffff85cca480
Build id: 3.10.0-957.el7.x86_64
0x40
14
kthread
vmlinux 0xffffffff85cc1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
15
? insert_kthread_work
vmlinux 0xffffffff85cc1b60
Build id: 3.10.0-957.el7.x86_64
0x40
16
ret_from_fork_nospec_begin
vmlinux 0xffffffff86374c37
Build id: 3.10.0-957.el7.x86_64
0x21
17
? insert_kthread_work
vmlinux 0xffffffff85cc1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ksoftirqd/0:3]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 tun ebtable_filter ebtables ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype ip6table_filter ip6_tables devlink iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon pcspkr joydev parport_pc parport sg vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmwgfx drm_kms_helper floppy syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm e1000 drm ata_piix libata mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3 Comm: ksoftirqd/0 Kdump: loaded Tainted: G               ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 08/16/2013
task: ffff8bc234992080 ti: ffff8bc2349b4000 task.ti: ffff8bc2349b4000
RIP: 0010:[<ffffffff85f86348>]  [<ffffffff85f86348>] memcpy+0x68/0x110
RSP: 0018:ffff8bc2349b7c08  EFLAGS: 00010246
RAX: ffff8bc86e679740 RBX: ffffffff862c06c9 RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffff8bc17f981020 RDI: ffff8bc86e679780
RBP: ffff8bc2349b7c10 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000005201 R11: a8c0000000000000 R12: ffff8bc2348db260
R13: ffff8bc87fa35400 R14: ffff8bc87fa35400 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8bc8d8c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006ec0f8 CR3: 00000007f3bcc000 CR4: 00000000000407f0
Call Trace:
[<ffffffff85fa0583>] ? swiotlb_tbl_sync_single+0x73/0x90
[<ffffffff85fa05cd>] swiotlb_sync_single+0x2d/0x40
[<ffffffff85fa05ec>] swiotlb_sync_single_for_cpu+0xc/0x10
[<ffffffffc02e731d>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc02e80dd>] e1000_clean+0x27d/0x8e0 [e1000]
[<ffffffffc064c06d>] ? nf_ct_delete+0x18d/0x270 [nf_conntrack]
[<ffffffff85ce00bc>] ? update_curr+0x14c/0x1e0
[<ffffffff86239e9f>] net_rx_action+0x26f/0x390
[<ffffffff85ca0f05>] __do_softirq+0xf5/0x280
[<ffffffff85ca10c8>] run_ksoftirqd+0x38/0x50
[<ffffffff85cca5c4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff85cca480>] ? lg_double_unlock+0x40/0x40
[<ffffffff85cc1c31>] kthread+0xd1/0xe0
[<ffffffff85cc1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff86374c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff85cc1b60>] ? insert_kthread_work+0x40/0x40
Code: d4 83 c2 20 eb 44 48 01 d6 48 01 d7 48 83 ea 20 0f 1f 00 48 83 ea 20 4c 8b 46 f8 4c 8b 4e f0 4c 8b 56 e8 4c 8b 5e e0 48 8d 76 e0 <4c> 89 47 f8 4c 89 4f f0 4c 89 57 e8 4c 89 5f e0 48 8d 7f e0 73 
Complete report #17570
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffff9c596ec9
Build id: 3.10.0-1160.6.1.el7.x86_64
0x29
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffff9c5b1853
Build id: 3.10.0-1160.6.1.el7.x86_64
0x73
3
swiotlb_sync_single
vmlinux 0xffffffff9c5b189d
Build id: 3.10.0-1160.6.1.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffff9c5b18bc
Build id: 3.10.0-1160.6.1.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc039833d
Build id: 3.10.0-1160.6.1.el7.x86_64
0x19d
6
e1000_clean
e1000 0xffffffffc0399100
Build id: 3.10.0-1160.6.1.el7.x86_64
0x280
7
? update_curr
vmlinux 0xffffffff9c2e441c
Build id: 3.10.0-1160.6.1.el7.x86_64
0x14c
8
net_rx_action
vmlinux 0xffffffff9c85561f
Build id: 3.10.0-1160.6.1.el7.x86_64
0x26f
9
__do_softirq
vmlinux 0xffffffff9c2a4b95
Build id: 3.10.0-1160.6.1.el7.x86_64
0xf5
10
run_ksoftirqd
vmlinux 0xffffffff9c2a4d58
Build id: 3.10.0-1160.6.1.el7.x86_64
0x38
11
smpboot_thread_fn
vmlinux 0xffffffff9c2ce674
Build id: 3.10.0-1160.6.1.el7.x86_64
0x144
12
? lg_double_unlock
vmlinux 0xffffffff9c2ce530
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
13
kthread
vmlinux 0xffffffff9c2c5c21
Build id: 3.10.0-1160.6.1.el7.x86_64
0xd1
14
? insert_kthread_work
vmlinux 0xffffffff9c2c5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
15
ret_from_fork_nospec_begin
vmlinux 0xffffffff9c994df7
Build id: 3.10.0-1160.6.1.el7.x86_64
0x21
16
? insert_kthread_work
vmlinux 0xffffffff9c2c5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [ksoftirqd/0:6]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc coretemp crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper vmw_balloon ablk_helper cryptd joydev pcspkr sg vmw_vmci parport_pc i2c_piix4 parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix libata drm mptspi scsi_transport_spi e1000 mptscsih crct10dif_pclmul crct10dif_common crc32c_intel serio_raw mptbase
drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 10/13/2009
task: ffff88e2b8f7d280 ti: ffff88e2b9750000 task.ti: ffff88e2b9750000
RIP: 0010:[<ffffffff9c596ec9>]  [<ffffffff9c596ec9>] memcpy+0x29/0x110
RSP: 0018:ffff88e2b9753c08  EFLAGS: 00010246
RAX: ffff88e4bb592dc0 RBX: ffffffff9c30792f RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffff88e1fe3f6020 RDI: ffff88e4bb592dc0
RBP: ffff88e2b9753c10 R08: 39b4ffffffffffff R09: 01000608006af8d6
R10: 39b4010004060008 R11: 01cbb00a006af8d6 R12: ffff88e2b8745260
R13: ffff88e2afbd2900 R14: ffff88e2b8745260 R15: 000000000000fe00
FS:  0000000000000000(0000) GS:ffff88e57fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe16a080000 CR3: 0000000172398000 CR4: 00000000000006f0
Call Trace:
[<ffffffff9c5b1853>] ? swiotlb_tbl_sync_single+0x73/0x90
[<ffffffff9c5b189d>] swiotlb_sync_single+0x2d/0x40
[<ffffffff9c5b18bc>] swiotlb_sync_single_for_cpu+0xc/0x10
[<ffffffffc039833d>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc0399100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff9c2e441c>] ? update_curr+0x14c/0x1e0
[<ffffffff9c85561f>] net_rx_action+0x26f/0x390
[<ffffffff9c2a4b95>] __do_softirq+0xf5/0x280
[<ffffffff9c2a4d58>] run_ksoftirqd+0x38/0x50
[<ffffffff9c2ce674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff9c2ce530>] ? lg_double_unlock+0x40/0x40
[<ffffffff9c2c5c21>] kthread+0xd1/0xe0
[<ffffffff9c2c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff9c994df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9c2c5b50>] ? insert_kthread_work+0x40/0x40
Code: 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 <4c> 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 
Complete report #11798
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffff90586b89
Build id: 3.10.0-957.21.3.el7.x86_64
0x29
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffff905a0e03
Build id: 3.10.0-957.21.3.el7.x86_64
0x73
3
swiotlb_sync_single
vmlinux 0xffffffff905a0e4d
Build id: 3.10.0-957.21.3.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffff905a0e6c
Build id: 3.10.0-957.21.3.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc04ed31d
Build id: 3.10.0-957.21.3.el7.x86_64
0x19d
6
e1000_clean
e1000 0xffffffffc04ee0dd
Build id: 3.10.0-957.21.3.el7.x86_64
0x27d
7
? nf_ct_delete
nf_conntrack 0xffffffffc08b901d
Build id: 3.10.0-957.21.3.el7.x86_64
0x18d
8
? tcp_send_ack
vmlinux 0xffffffff908a416f
Build id: 3.10.0-957.21.3.el7.x86_64
0x11f
9
net_rx_action
vmlinux 0xffffffff9083af1f
Build id: 3.10.0-957.21.3.el7.x86_64
0x26f
10
__do_softirq
vmlinux 0xffffffff902a1075
Build id: 3.10.0-957.21.3.el7.x86_64
0xf5
11
run_ksoftirqd
vmlinux 0xffffffff902a1238
Build id: 3.10.0-957.21.3.el7.x86_64
0x38
12
smpboot_thread_fn
vmlinux 0xffffffff902ca734
Build id: 3.10.0-957.21.3.el7.x86_64
0x144
13
? lg_double_unlock
vmlinux 0xffffffff902ca5f0
Build id: 3.10.0-957.21.3.el7.x86_64
0x40
14
kthread
vmlinux 0xffffffff902c1da1
Build id: 3.10.0-957.21.3.el7.x86_64
0xd1
15
? insert_kthread_work
vmlinux 0xffffffff902c1cd0
Build id: 3.10.0-957.21.3.el7.x86_64
0x40
16
ret_from_fork_nospec_begin
vmlinux 0xffffffff90975c37
Build id: 3.10.0-957.21.3.el7.x86_64
0x21
17
? insert_kthread_work
vmlinux 0xffffffff902c1cd0
Build id: 3.10.0-957.21.3.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [ksoftirqd/1:14]
Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg vmw_balloon pcspkr joydev parport_pc i2c_piix4 parport vmw_vmci ip_tables
xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw mptspi scsi_transport_spi mptscsih drm ata_piix e1000 mptbase libata nfit libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 14 Comm: ksoftirqd/1 Kdump: loaded Tainted: G        W    L ------------   3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff97ab3d78a080 ti: ffff97ab3d798000 task.ti: ffff97ab3d798000
RIP: 0010:[<ffffffff90586b89>]  [<ffffffff90586b89>] memcpy+0x29/0x110
RSP: 0018:ffff97ab3d79bc08  EFLAGS: 00010202
RAX: ffff97aad50dadc0 RBX: ffffffff908c19a9 RCX: 0000000000000002
RDX: 0000000000000002 RSI: ffff97aa7c0eb020 RDI: ffff97aad50dadc0
RBP: ffff97ab3d79bc10 R08: 50005b2c5b3e16fa R09: 00450008b6be8756
R10: 0675004007013000 R11: a8c05567ee3dd19d R12: ffff97ab78be5260
R13: ffff97a9e986fe00 R14: ffffffff908b0000 R15: ffff97ab3d79bb00
FS:  0000000000000000(0000) GS:ffff97ab7fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f208001e000 CR3: 000000005b14a000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff905a0e03>] ? swiotlb_tbl_sync_single+0x73/0x90
[<ffffffff905a0e4d>] swiotlb_sync_single+0x2d/0x40
[<ffffffff905a0e6c>] swiotlb_sync_single_for_cpu+0xc/0x10
[<ffffffffc04ed31d>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc04ee0dd>] e1000_clean+0x27d/0x8e0 [e1000]
[<ffffffffc08b901d>] ? nf_ct_delete+0x18d/0x270 [nf_conntrack]
[<ffffffff908a416f>] ? tcp_send_ack+0x11f/0x170
[<ffffffff9083af1f>] net_rx_action+0x26f/0x390
[<ffffffff902a1075>] __do_softirq+0xf5/0x280
[<ffffffff902a1238>] run_ksoftirqd+0x38/0x50
[<ffffffff902ca734>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff902ca5f0>] ? lg_double_unlock+0x40/0x40
[<ffffffff902c1da1>] kthread+0xd1/0xe0
[<ffffffff902c1cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffff90975c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff902c1cd0>] ? insert_kthread_work+0x40/0x40
Code: 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 <4c> 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 
Complete report #4339
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffff9ab5ab39
Build id: 3.10.0-862.6.3.el7.x86_64
0x29
2
? swiotlb_tbl_sync_single
vmlinux 0xffffffff9ab74973
Build id: 3.10.0-862.6.3.el7.x86_64
0x73
3
swiotlb_sync_single
vmlinux 0xffffffff9ab749bd
Build id: 3.10.0-862.6.3.el7.x86_64
0x2d
4
swiotlb_sync_single_for_cpu
vmlinux 0xffffffff9ab749dc
Build id: 3.10.0-862.6.3.el7.x86_64
0xc
5
e1000_clean_rx_irq
e1000 0xffffffffc048a32d
Build id: 3.10.0-862.6.3.el7.x86_64
0x19d
6
e1000_clean
e1000 0xffffffffc048b0eb
Build id: 3.10.0-862.6.3.el7.x86_64
0x27b
7
? update_curr
vmlinux 0xffffffff9a8d895c
Build id: 3.10.0-862.6.3.el7.x86_64
0x14c
8
? account_entity_dequeue
vmlinux 0xffffffff9a8d52ce
Build id: 3.10.0-862.6.3.el7.x86_64
0xae
9
net_rx_action
vmlinux 0xffffffff9aded46f
Build id: 3.10.0-862.6.3.el7.x86_64
0x26f
10
__do_softirq
vmlinux 0xffffffff9a89b085
Build id: 3.10.0-862.6.3.el7.x86_64
0xf5
11
run_ksoftirqd
vmlinux 0xffffffff9a89b248
Build id: 3.10.0-862.6.3.el7.x86_64
0x38
12
smpboot_thread_fn
vmlinux 0xffffffff9a8c3dd4
Build id: 3.10.0-862.6.3.el7.x86_64
0x144
13
? lg_double_unlock
vmlinux 0xffffffff9a8c3c90
Build id: 3.10.0-862.6.3.el7.x86_64
0x40
14
kthread
vmlinux 0xffffffff9a8bb621
Build id: 3.10.0-862.6.3.el7.x86_64
0xd1
15
? insert_kthread_work
vmlinux 0xffffffff9a8bb550
Build id: 3.10.0-862.6.3.el7.x86_64
0x40
16
ret_from_fork_nospec_begin
vmlinux 0xffffffff9af205f7
Build id: 3.10.0-862.6.3.el7.x86_64
0x21
17
? insert_kthread_work
vmlinux 0xffffffff9a8bb550
Build id: 3.10.0-862.6.3.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [ksoftirqd/0:3]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr joydev vmw_balloon sg parport_pc parport shpchp vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul crct10dif_common mptspi crc32c_intel
scsi_transport_spi mptscsih e1000 serio_raw mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W    L ------------   3.10.0-862.6.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
task: ffff8fcf38370000 ti: ffff8fcf3835c000 task.ti: ffff8fcf3835c000
RIP: 0010:[<ffffffff9ab5ab39>]  [<ffffffff9ab5ab39>] memcpy+0x29/0x110
RSP: 0018:ffff8fcf3835fc08  EFLAGS: 00010202
RAX: ffff8fd1904278c0 RBX: ffffffff9a8f944f RCX: 0000000000000002
RDX: 000000000000000e RSI: ffff8fcfbfca2020 RDI: ffff8fd1904278c0
RBP: ffff8fcf3835fc10 R08: 5000b783a3565000 R09: 0045000835bca356
R10: 064000408dbf3c00 R11: 5d0a140c5d0a5a4e R12: ffff8fcf368a9260
R13: ffff8fcf3fee7500 R14: ffffffff9ae5e600 R15: ffff8fcf3835fb00
FS:  0000000000000000(0000) GS:ffff8fcf38c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb044896000 CR3: 0000000014830000 CR4: 00000000001607f0
Call Trace:
[<ffffffff9ab74973>] ? swiotlb_tbl_sync_single+0x73/0x90
[<ffffffff9ab749bd>] swiotlb_sync_single+0x2d/0x40
[<ffffffff9ab749dc>] swiotlb_sync_single_for_cpu+0xc/0x10
[<ffffffffc048a32d>] e1000_clean_rx_irq+0x19d/0x510 [e1000]
[<ffffffffc048b0eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffff9a8d895c>] ? update_curr+0x14c/0x1e0
[<ffffffff9a8d52ce>] ? account_entity_dequeue+0xae/0xd0
[<ffffffff9aded46f>] net_rx_action+0x26f/0x390
[<ffffffff9a89b085>] __do_softirq+0xf5/0x280
[<ffffffff9a89b248>] run_ksoftirqd+0x38/0x50
[<ffffffff9a8c3dd4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff9a8c3c90>] ? lg_double_unlock+0x40/0x40
[<ffffffff9a8bb621>] kthread+0xd1/0xe0
[<ffffffff9a8bb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff9af205f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9a8bb550>] ? insert_kthread_work+0x40/0x40
Code: 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 <4c> 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4