Info

Permalink
Function
smp_call_function_many
First occurrence
2023-02-27
Last occurrence
2024-09-23
State
NEW
Components
kernel
Quality
-21

Statistics

Operating system Count
CentOS 7 42
Architecture Count
x86_64 42
Related packages Count
kernel 42
 0:3.10.0-1160.119.1.el7 31
 0:3.10.0-1160.88.1.el7 4
 0:3.10.0-1062.el7 3
 0:3.10.0-1160.59.1.el7 1
 0:3.10.0-1160.83.1.el7 1
 0:3.10.0-1160.el7 1
 0:3.10.0-1160.92.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #1417170
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa591e25a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa5881fa8
Build id: 3.10.0-1160.119.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa588201b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa59fa0ce
Build id: 3.10.0-1160.119.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa59fbc4f
Build id: 3.10.0-1160.119.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa59fbd83
Build id: 3.10.0-1160.119.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffa59ff2ca
Build id: 3.10.0-1160.119.1.el7.x86_64
0xea
8
? futex_wake
vmlinux 0xffffffffa59196b0
Build id: 3.10.0-1160.119.1.el7.x86_64
0x90
9
SyS_madvise
vmlinux 0xffffffffa59f96bd
Build id: 3.10.0-1160.119.1.el7.x86_64
0x49d
10
? do_send_specific
vmlinux 0xffffffffa58b711d
Build id: 3.10.0-1160.119.1.el7.x86_64
0x7d
11
system_call_fastpath
vmlinux 0xffffffffa5fc539a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [claude:54614]
Modules linked in: tcp_lp rfcomm xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack vmw_vsock_vmci_transport vsock ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse bnep sunrpc snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 ppdev crc32_pclmul snd_ens1371 ghash_clmulni_intel vmw_balloon snd_rawmidi snd_ac97_codec btusb btrtl btbcm ac97_bus aesni_intel btintel snd_seq lrw gf128mul bluetooth glue_helper snd_seq_device
ablk_helper cryptd snd_pcm joydev pcspkr snd_timer rfkill snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix libata nfit drm libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel mptspi scsi_transport_spi mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 11 PID: 54614 Comm: claude Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff91ae77e51080 ti: ffff91a6c3e18000 task.ti: ffff91a6c3e18000
RIP: 0010:[<ffffffffa591e25a>]  [<ffffffffa591e25a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff91a6c3e1bc70  EFLAGS: 00000202
RAX: 0000000000000008 RBX: 00080000a5881b10 RCX: ffffcd4b3e000590
RDX: 0000000000000008 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff91a6c3e1bca8 R08: ffff91ae3b5e8400 R09: ffffffffa5b9c7f9
R10: ffff91ae7d8df160 R11: ffffe05d1a45b800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff91a6c3e1bc20 R15: 0000000000000001
FS:  00007ffa2ccac700(0000) GS:ffff91ae7d8c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f68a737ed70 CR3: 00000005dbb34000 CR4: 00000000007407e0
PKRU: 55555554
Call Trace:
[<ffffffffa5881fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa588201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa59fa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffa59fbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa59fbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffffa59ff2ca>] zap_page_range+0xea/0x140
[<ffffffffa59196b0>] ? futex_wake+0x90/0x190
[<ffffffffa59f96bd>] SyS_madvise+0x49d/0xac0
[<ffffffffa58b711d>] ? do_send_specific+0x7d/0xa0
[<ffffffffa5fc539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ae 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 c0 19 55 a6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 7d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1129844
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbb11e1ca
Build id: 3.10.0-1160.88.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbb081fa8
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbb08201b
Build id: 3.10.0-1160.88.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbb1fa00e
Build id: 3.10.0-1160.88.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbb1fbb8f
Build id: 3.10.0-1160.88.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbb1fbcc3
Build id: 3.10.0-1160.88.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffbb1f86f6
Build id: 3.10.0-1160.88.1.el7.x86_64
0x116
8
? wake_up_q
vmlinux 0xffffffffbb0e1118
Build id: 3.10.0-1160.88.1.el7.x86_64
0x58
9
? futex_wake
vmlinux 0xffffffffbb119620
Build id: 3.10.0-1160.88.1.el7.x86_64
0x90
10
SyS_madvise
vmlinux 0xffffffffbb1f9631
Build id: 3.10.0-1160.88.1.el7.x86_64
0x4d1
11
? do_send_specific
vmlinux 0xffffffffbb0b711d
Build id: 3.10.0-1160.88.1.el7.x86_64
0x7d
12
system_call_fastpath
vmlinux 0xffffffffbb7c539a
Build id: 3.10.0-1160.88.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [prometheus:3658]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink arc4 md4 nls_utf8 cifs ccm dns_resolver sunrpc ppdev sg virtio_balloon joydev parport_pc parport i2c_piix4 pcspkr ip_tables xfs libcrc32c sr_mod cdrom virtio_net virtio_console net_failover virtio_blk virtio_scsi failover ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci drm_panel_orientation_quirks virtio_ring serio_raw virtio floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 3658 Comm: prometheus Tainted: G             L ------------   3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
task: ffff9636c88be300 ti: ffff9639b3fd0000 task.ti: ffff9639b3fd0000
RIP: 0010:[<ffffffffbb11e1ca>]  [<ffffffffbb11e1ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9639b3fd3c70  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffffffffffff10 RCX: ffff9636d9e60450
RDX: 0000000000000001 RSI: 0000000000000060 RDI: 0000000000000000
RBP: ffff9639b3fd3ca8 R08: ffff962f2757b800 R09: ffffffffbb39c669
R10: ffff9636d9edf160 R11: ffffe886aebc9000 R12: 0000000000000060
R13: 00000000ffffffff R14: 0000000000000020 R15: 0000000000000060
FS:  00007fd6821fc700(0000) GS:ffff9636d9ec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000056047f129068 CR3: 0000000be530a000 CR4: 00000000000006e0
Call Trace:
[<ffffffffbb081fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbb08201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbb1fa00e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffbb1fbb8f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbb1fbcc3>] tlb_finish_mmu+0x23/0x40
[<ffffffffbb1f86f6>] madvise_free_single_vma+0x116/0x180
[<ffffffffbb0e1118>] ? wake_up_q+0x58/0x90
[<ffffffffbb119620>] ? futex_wake+0x90/0x190
[<ffffffffbb1f9631>] SyS_madvise+0x4d1/0xac0
[<ffffffffbb0b711d>] ? do_send_specific+0x7d/0xa0
[<ffffffffbb7c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 7e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 d5 bb f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #995806
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffacd150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffacc7d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffacc7d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffacdea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffacdebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffacdebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffacde8866
Build id: 3.10.0-1062.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffffacde97a1
Build id: 3.10.0-1062.el7.x86_64
0x4d1
9
? do_send_specific
vmlinux 0xffffffffaccb1455
Build id: 3.10.0-1062.el7.x86_64
0x75
10
system_call_fastpath
vmlinux 0xffffffffad38bede
Build id: 3.10.0-1062.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [containerd:3461]
Modules linked in: nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc ext4 snd_seq_midi snd_seq_midi_event mbcache jbd2 snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm iosf_mbi ppdev crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel snd_timer lrw gf128mul snd glue_helper ablk_helper cryptd joydev soundcore pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix nfit drm crct10dif_pclmul crct10dif_common libata mptspi crc32c_intel libnvdimm scsi_transport_spi e1000 mptscsih mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 3461 Comm: containerd Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff95bfb6b462a0 ti: ffff95bf33ac8000 task.ti: ffff95bf33ac8000
RIP: 0010:[<ffffffffacd150ca>]  [<ffffffffacd150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff95bf33acbc70  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000acc7d1d0 RCX: ffff95bfbd6606a0
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff95bf33acbca8 R08: ffff95b129664c00 R09: ffffffffacf7f629
R10: ffff95bfbd7df160 R11: fffff56ba44cba00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff95bf33acbc20 R15: 000000c0005ed000
FS:  00007f848677c700(0000) GS:ffff95bfbd7c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000012fceb4c6000 CR3: 0000000fe73f6000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffacc7d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffacc7d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffacdea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffacdebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffacdebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffffacde8866>] madvise_free_single_vma+0x116/0x180
[<ffffffffacde97a1>] SyS_madvise+0x4d1/0xac0
[<ffffffffaccb1455>] ? do_send_specific+0x75/0xa0
[<ffffffffad38bede>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 94 ad f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #941076
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbb11e1ca
Build id: 3.10.0-1160.88.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbb081fa8
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbb08201b
Build id: 3.10.0-1160.88.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbb1fa00e
Build id: 3.10.0-1160.88.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbb1fbb8f
Build id: 3.10.0-1160.88.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbb1fbcc3
Build id: 3.10.0-1160.88.1.el7.x86_64
0x23
7
madvise_free_single_vma
vmlinux 0xffffffffbb1f86f6
Build id: 3.10.0-1160.88.1.el7.x86_64
0x116
8
SyS_madvise
vmlinux 0xffffffffbb1f9631
Build id: 3.10.0-1160.88.1.el7.x86_64
0x4d1
9
? do_send_specific
vmlinux 0xffffffffbb0b711d
Build id: 3.10.0-1160.88.1.el7.x86_64
0x7d
10
system_call_fastpath
vmlinux 0xffffffffbb7c539a
Build id: 3.10.0-1160.88.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [node_exporter:3794]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink arc4 md4 nls_utf8 cifs ccm dns_resolver sunrpc ppdev sg virtio_balloon joydev parport_pc parport i2c_piix4 pcspkr ip_tables xfs libcrc32c sr_mod cdrom virtio_net virtio_console net_failover virtio_blk virtio_scsi failover ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci drm_panel_orientation_quirks virtio_ring serio_raw virtio floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 3794 Comm: node_exporter Tainted: G             L ------------   3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
task: ffff962e7a766300 ti: ffff9639ac3ac000 task.ti: ffff9639ac3ac000
RIP: 0010:[<ffffffffbb11e1ca>]  [<ffffffffbb11e1ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9639ac3afc70  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000bb081b10 RCX: ffff9636d9e60428
RDX: 0000000000000001 RSI: 0000000000000060 RDI: 0000000000000000
RBP: ffff9639ac3afca8 R08: ffff962f2757b400 R09: ffffffffbb39c669
R10: ffff9636d9e9f160 R11: ffffe886afcf2000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9639ac3afc20 R15: 000000c000855000
FS:  000000c000698090(0000) GS:ffff9636d9e80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2505e9a000 CR3: 0000000ba0248000 CR4: 00000000000006e0
Call Trace:
[<ffffffffbb081fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbb08201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbb1fa00e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffbb1fbb8f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbb1fbcc3>] tlb_finish_mmu+0x23/0x40
[<ffffffffbb1f86f6>] madvise_free_single_vma+0x116/0x180
[<ffffffffbb1f9631>] SyS_madvise+0x4d1/0xac0
[<ffffffffbb0b711d>] ? do_send_specific+0x7d/0xa0
[<ffffffffbb7c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 7e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 d5 bb f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #933413
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa131e1ca
Build id: 3.10.0-1160.92.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa1281fa8
Build id: 3.10.0-1160.92.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa128201b
Build id: 3.10.0-1160.92.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa13fa00e
Build id: 3.10.0-1160.92.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa13fbb8f
Build id: 3.10.0-1160.92.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa13fbcc3
Build id: 3.10.0-1160.92.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffa13ff20a
Build id: 3.10.0-1160.92.1.el7.x86_64
0xea
8
? wake_up_q
vmlinux 0xffffffffa12e1118
Build id: 3.10.0-1160.92.1.el7.x86_64
0x58
9
SyS_madvise
vmlinux 0xffffffffa13f95fd
Build id: 3.10.0-1160.92.1.el7.x86_64
0x49d
10
? do_send_specific
vmlinux 0xffffffffa12b711d
Build id: 3.10.0-1160.92.1.el7.x86_64
0x7d
11
system_call_fastpath
vmlinux 0xffffffffa19c539a
Build id: 3.10.0-1160.92.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [containerd-shim:2304]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod ppdev vmw_balloon crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc
parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic crct10dif_pclmul vmwgfx crct10dif_common crc32c_intel pata_acpi e1000 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit mptspi serio_raw scsi_transport_spi libnvdimm ata_piix drm mptscsih libata mptbase drm_panel_orientation_quirks fuse
CPU: 5 PID: 2304 Comm: containerd-shim Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.92.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9287b5f02100 ti: ffff928f72c20000 task.ti: ffff928f72c20000
RIP: 0010:[<ffffffffa131e1ca>]  [<ffffffffa131e1ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff928f72c23c70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a1281b10 RCX: ffff928f9d6206a0
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff928f72c23ca8 R08: ffff9288ec55c800 R09: ffffffffa159c669
R10: ffff928f9d75f160 R11: ffffe196e06fba00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff928f72c23c20 R15: ffffffffa240db80
FS:  000000c000190890(0000) GS:ffff928f9d740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000141010 CR3: 00000007f2d6c000 CR4: 0000000000360fe0
Call Trace:
[<ffffffffa1281fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa128201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa13fa00e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffa13fbb8f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa13fbcc3>] tlb_finish_mmu+0x23/0x40
[<ffffffffa13ff20a>] zap_page_range+0xea/0x140
[<ffffffffa12e1118>] ? wake_up_q+0x58/0x90
[<ffffffffa13f95fd>] SyS_madvise+0x49d/0xac0
[<ffffffffa12b711d>] ? do_send_specific+0x7d/0xa0
[<ffffffffa19c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 7e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 f5 a1 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #632400
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9c716c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9c67e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9c67e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9c7edd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9c7ef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9c7ef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff9c7f2dba
Build id: 3.10.0-1160.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff9c7ed2ad
Build id: 3.10.0-1160.el7.x86_64
0x49d
9
? do_send_specific
vmlinux 0xffffffff9c6b1fa5
Build id: 3.10.0-1160.el7.x86_64
0x75
10
system_call_fastpath
vmlinux 0xffffffff9cd93f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [worker:6870]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq pcspkr snd_seq_device snd_pcm iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper joydev snd_timer sg snd ablk_helper cryptd soundcore vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
ttm nfit ata_piix drm mptspi scsi_transport_spi libnvdimm mptscsih mptbase crct10dif_pclmul crct10dif_common e1000 libata crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 6870 Comm: worker Kdump: loaded Not tainted 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9c58ad044200 ti: ffff9c5abfe4c000 task.ti: ffff9c5abfe4c000
RIP: 0010:[<ffffffff9c716c0a>]  [<ffffffff9c716c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9c5abfe4fc70  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 000800009c67e6a0 RCX: ffff9c5ac0660628
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9c5abfe4fca8 R08: ffff9c59b13cb400 R09: ffffffff9c985f99
R10: ffff9c5ac069f160 R11: ffffe4e542cc4800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9c5abfe4fc20 R15: 0000000000000001
FS:  00002b689aa30700(0000) GS:ffff9c5ac0680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001626d68 CR3: 00000002305bc000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9c67e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9c67e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9c7edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9c7ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9c7ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff9c7f2dba>] zap_page_range+0xea/0x130
[<ffffffff9c7ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffff9c6b1fa5>] ? do_send_specific+0x75/0xa0
[<ffffffff9cd93f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 35 9d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89