Info

Permalink
Function
smp_call_function_many
First occurrence
2021-11-02
Last occurrence
2023-12-22
State
NEW
Components
kernel
Quality
-24

Statistics

Operating system Count
CentOS 7 21
Architecture Count
x86_64 21
Related packages Count
kernel 21
 0:3.10.0-1160.119.1.el7 17
 0:3.10.0-1160.45.1.el7 2
 0:3.10.0-1160.71.1.el7 1
 0:3.10.0-1160.92.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1417170
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa591e25a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa5881fa8
Build id: 3.10.0-1160.119.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa588201b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa59fa0ce
Build id: 3.10.0-1160.119.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa59fbc4f
Build id: 3.10.0-1160.119.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa59fbd83
Build id: 3.10.0-1160.119.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffa59ff2ca
Build id: 3.10.0-1160.119.1.el7.x86_64
0xea
8
? futex_wake
vmlinux 0xffffffffa59196b0
Build id: 3.10.0-1160.119.1.el7.x86_64
0x90
9
SyS_madvise
vmlinux 0xffffffffa59f96bd
Build id: 3.10.0-1160.119.1.el7.x86_64
0x49d
10
? do_send_specific
vmlinux 0xffffffffa58b711d
Build id: 3.10.0-1160.119.1.el7.x86_64
0x7d
11
system_call_fastpath
vmlinux 0xffffffffa5fc539a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [claude:54614]
Modules linked in: tcp_lp rfcomm xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack vmw_vsock_vmci_transport vsock ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse bnep sunrpc snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 ppdev crc32_pclmul snd_ens1371 ghash_clmulni_intel vmw_balloon snd_rawmidi snd_ac97_codec btusb btrtl btbcm ac97_bus aesni_intel btintel snd_seq lrw gf128mul bluetooth glue_helper snd_seq_device
ablk_helper cryptd snd_pcm joydev pcspkr snd_timer rfkill snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix libata nfit drm libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel mptspi scsi_transport_spi mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 11 PID: 54614 Comm: claude Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff91ae77e51080 ti: ffff91a6c3e18000 task.ti: ffff91a6c3e18000
RIP: 0010:[<ffffffffa591e25a>]  [<ffffffffa591e25a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff91a6c3e1bc70  EFLAGS: 00000202
RAX: 0000000000000008 RBX: 00080000a5881b10 RCX: ffffcd4b3e000590
RDX: 0000000000000008 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff91a6c3e1bca8 R08: ffff91ae3b5e8400 R09: ffffffffa5b9c7f9
R10: ffff91ae7d8df160 R11: ffffe05d1a45b800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff91a6c3e1bc20 R15: 0000000000000001
FS:  00007ffa2ccac700(0000) GS:ffff91ae7d8c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f68a737ed70 CR3: 00000005dbb34000 CR4: 00000000007407e0
PKRU: 55555554
Call Trace:
[<ffffffffa5881fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa588201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa59fa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffa59fbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa59fbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffffa59ff2ca>] zap_page_range+0xea/0x140
[<ffffffffa59196b0>] ? futex_wake+0x90/0x190
[<ffffffffa59f96bd>] SyS_madvise+0x49d/0xac0
[<ffffffffa58b711d>] ? do_send_specific+0x7d/0xa0
[<ffffffffa5fc539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ae 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 c0 19 55 a6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 7d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #279498
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff81d170da
Build id: 3.10.0-1160.45.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff81c7e258
Build id: 3.10.0-1160.45.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff81c7e2cb
Build id: 3.10.0-1160.45.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff81dee676
Build id: 3.10.0-1160.45.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff81df013f
Build id: 3.10.0-1160.45.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff81df0263
Build id: 3.10.0-1160.45.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff81df372a
Build id: 3.10.0-1160.45.1.el7.x86_64
0xea
8
? wake_up_q
vmlinux 0xffffffff81cdada8
Build id: 3.10.0-1160.45.1.el7.x86_64
0x58
9
SyS_madvise
vmlinux 0xffffffff81dedc1d
Build id: 3.10.0-1160.45.1.el7.x86_64
0x49d
10
? poll_select_copy_remaining
vmlinux 0xffffffff81e64924
Build id: 3.10.0-1160.45.1.el7.x86_64
0x144
11
? SyS_pselect6
vmlinux 0xffffffff81e65a27
Build id: 3.10.0-1160.45.1.el7.x86_64
0x257
12
system_call_fastpath
vmlinux 0xffffffff82395f92
Build id: 3.10.0-1160.45.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [dockerd-current:1501]
Modules linked in: tcp_lp xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event sb_edac coretemp snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device iosf_mbi snd_pcm crc32_pclmul ghash_clmulni_intel aesni_intel
lrw snd_timer gf128mul glue_helper snd ablk_helper soundcore ppdev cryptd pcspkr joydev sg vmw_balloon parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common ata_piix nfit crc32c_intel libata libnvdimm mptspi scsi_transport_spi serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks fuse
CPU: 8 PID: 1501 Comm: dockerd-current Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8ecc37022100 ti: ffff8ece2093c000 task.ti: ffff8ece2093c000
RIP: 0010:[<ffffffff81d170da>]  [<ffffffff81d170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8ece2093fc70  EFLAGS: 00000202
RAX: 000000000000000f RBX: 0008000081c7ddd0 RCX: ffff8ece339e0718
RDX: 000000000000000f RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8ece2093fca8 R08: ffff8eccf8ccf000 R09: ffffffff81f86bf9
R10: ffff8ece3381f160 R11: ffffceee0a473000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8ece2093fc20 R15: 0000000000000001
FS:  00007ff5c52ef700(0000) GS:ffff8ece33800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f91210d0000 CR3: 00000002a0cb2000 CR4: 00000000001607e0
Call Trace:
[<ffffffff81c7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81c7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff81dee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff81df013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff81df0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff81df372a>] zap_page_range+0xea/0x130
[<ffffffff81cdada8>] ? wake_up_q+0x58/0x80
[<ffffffff81dedc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff81e64924>] ? poll_select_copy_remaining+0x144/0x180
[<ffffffff81e65a27>] ? SyS_pselect6+0x257/0x260
[<ffffffff82395f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 95 82 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #279316
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8ed170da
Build id: 3.10.0-1160.45.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8ec7e258
Build id: 3.10.0-1160.45.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8ec7e2cb
Build id: 3.10.0-1160.45.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8edee676
Build id: 3.10.0-1160.45.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8edf013f
Build id: 3.10.0-1160.45.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8edf0263
Build id: 3.10.0-1160.45.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff8edf372a
Build id: 3.10.0-1160.45.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff8ededc1d
Build id: 3.10.0-1160.45.1.el7.x86_64
0x49d
9
? poll_select_copy_remaining
vmlinux 0xffffffff8ee64924
Build id: 3.10.0-1160.45.1.el7.x86_64
0x144
10
? SyS_pselect6
vmlinux 0xffffffff8ee65a27
Build id: 3.10.0-1160.45.1.el7.x86_64
0x257
11
system_call_fastpath
vmlinux 0xffffffff8f395f92
Build id: 3.10.0-1160.45.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [dockerd-current:1405]
Modules linked in: tcp_lp xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock overlay(T) sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq sb_edac coretemp snd_seq_device snd_pcm snd_timer iosf_mbi snd crc32_pclmul soundcore
ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev i2c_piix4 vmw_balloon sg joydev pcspkr vmw_vmci parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect crct10dif_pclmul crct10dif_common sysimgblt fb_sys_fops crc32c_intel ttm drm ata_piix nfit mptspi libata libnvdimm scsi_transport_spi serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks fuse
CPU: 5 PID: 1405 Comm: dockerd-current Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9b1cdfe94200 ti: ffff9b1cdfec4000 task.ti: ffff9b1cdfec4000
RIP: 0010:[<ffffffff8ed170da>]  [<ffffffff8ed170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9b1cdfec7c70  EFLAGS: 00000202
RAX: 000000000000000d RBX: 000800008ec7ddd0 RCX: ffff9b1cf39606a0
RDX: 000000000000000d RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9b1cdfec7ca8 R08: ffff9b1bb8ccd800 R09: ffffffff8ef86bf9
R10: ffff9b1cf375f160 R11: ffffd7a002eeea00 R12: 0000000000000282
R13: 0000000000000018 R14: ffff9b1cdfec7be0 R15: 0000000000000282
FS:  00007fa66dee5700(0000) GS:ffff9b1cf3740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f00e5881d70 CR3: 00000000b7b40000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8ec7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8ec7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8edee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8edf013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8edf0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff8edf372a>] zap_page_range+0xea/0x130
[<ffffffff8ededc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff8ee64924>] ? poll_select_copy_remaining+0x144/0x180
[<ffffffff8ee65a27>] ? SyS_pselect6+0x257/0x260
[<ffffffff8f395f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 95 8f f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89