Info

Permalink
Function
_raw_spin_lock
First occurrence
2023-01-09
Last occurrence
2023-01-09
State
NEW
Components
kernel
Quality
-75

Statistics

Operating system Count
CentOS 7 17
Architecture Count
x86_64 17
Related packages Count
kernel 17
 0:3.10.0-514.26.2.el7 15
 0:3.10.0-693.5.2.el7 1
 0:3.10.0-693.21.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #1419338
Frame # Function Binary Source or offset Line
1
ext4_inode_touch_time_cmp
ext4 0xffffffffc02715b0
Build id: 3.10.0-693.21.1.el7.x86_64
0x40
2
merge
vmlinux 0xffffffff8133c2e2
Build id: 3.10.0-693.21.1.el7.x86_64
0x62
3
? ext4_ind_remove_space
ext4 0xffffffffc0271570
Build id: 3.10.0-693.21.1.el7.x86_64
0x8c0
4
list_sort
vmlinux 0xffffffff8133c3db
Build id: 3.10.0-693.21.1.el7.x86_64
0x9b
5
__ext4_es_shrink
ext4 0xffffffffc0271a6e
Build id: 3.10.0-693.21.1.el7.x86_64
0x1ce
6
ext4_es_shrink
ext4 0xffffffffc0271bf4
Build id: 3.10.0-693.21.1.el7.x86_64
0xb4
7
shrink_slab
vmlinux 0xffffffff811999a5
Build id: 3.10.0-693.21.1.el7.x86_64
0x175
8
? vmpressure
vmlinux 0xffffffff811fbf31
Build id: 3.10.0-693.21.1.el7.x86_64
0x21
9
do_try_to_free_pages
vmlinux 0xffffffff8119cb22
Build id: 3.10.0-693.21.1.el7.x86_64
0x3c2
10
try_to_free_pages
vmlinux 0xffffffff8119cd3c
Build id: 3.10.0-693.21.1.el7.x86_64
0xfc
11
__alloc_pages_slowpath
vmlinux 0xffffffff816aa515
Build id: 3.10.0-693.21.1.el7.x86_64
0x457
12
__alloc_pages_nodemask
vmlinux 0xffffffff811912a5
Build id: 3.10.0-693.21.1.el7.x86_64
0x405
13
alloc_pages_current
vmlinux 0xffffffff811d5a38
Build id: 3.10.0-693.21.1.el7.x86_64
0x98
14
__get_free_pages
vmlinux 0xffffffff8118bb0e
Build id: 3.10.0-693.21.1.el7.x86_64
0xe
15
kmalloc_order_trace
vmlinux 0xffffffff811e146e
Build id: 3.10.0-693.21.1.el7.x86_64
0x2e
16
__kmalloc
vmlinux 0xffffffff811e5011
Build id: 3.10.0-693.21.1.el7.x86_64
0x211
17
? get_iowait_time
vmlinux 0xffffffff8127f4b0
Build id: 3.10.0-693.21.1.el7.x86_64
0x60
18
seq_buf_alloc
vmlinux 0xffffffff8122ac87
Build id: 3.10.0-693.21.1.el7.x86_64
0x17
19
single_open_size
vmlinux 0xffffffff8122b02a
Build id: 3.10.0-693.21.1.el7.x86_64
0x2a
20
? single_open
vmlinux 0xffffffff8122abe0
Build id: 3.10.0-693.21.1.el7.x86_64
0xb0
21
stat_open
vmlinux 0xffffffff8127fbb5
Build id: 3.10.0-693.21.1.el7.x86_64
0x45
22
proc_reg_open
vmlinux 0xffffffff81275b42
Build id: 3.10.0-693.21.1.el7.x86_64
0x72
23
do_dentry_open
vmlinux 0xffffffff81202f8a
Build id: 3.10.0-693.21.1.el7.x86_64
0x1aa
24
? security_inode_permission
vmlinux 0xffffffff812b7802
Build id: 3.10.0-693.21.1.el7.x86_64
0x22
25
? proc_reg_release
vmlinux 0xffffffff81275ad0
Build id: 3.10.0-693.21.1.el7.x86_64
0x80
26
vfs_open
vmlinux 0xffffffff8120315a
Build id: 3.10.0-693.21.1.el7.x86_64
0x5a
27
? may_open
vmlinux 0xffffffff81210fc8
Build id: 3.10.0-693.21.1.el7.x86_64
0x68
28
do_last
vmlinux 0xffffffff812144bd
Build id: 3.10.0-693.21.1.el7.x86_64
0x1ed
29
path_openat
vmlinux 0xffffffff81215652
Build id: 3.10.0-693.21.1.el7.x86_64
0xc2
30
? do_numa_page
vmlinux 0xffffffff811b53b9
Build id: 3.10.0-693.21.1.el7.x86_64
0x159
31
do_filp_open
vmlinux 0xffffffff81217bfb
Build id: 3.10.0-693.21.1.el7.x86_64
0x4b
32
? __alloc_fd
vmlinux 0xffffffff81224f1a
Build id: 3.10.0-693.21.1.el7.x86_64
0x8a
33
do_sys_open
vmlinux 0xffffffff81204523
Build id: 3.10.0-693.21.1.el7.x86_64
0xf3
34
SyS_open
vmlinux 0xffffffff8120463e
Build id: 3.10.0-693.21.1.el7.x86_64
0x1e
35
system_call_fastpath
vmlinux 0xffffffff816c0715
Build id: 3.10.0-693.21.1.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [sfavsrv:125507]
Modules linked in: udp_diag tcp_diag inet_diag binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bonding dm_mirror dm_region_hash dm_log dm_mod skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support vfat fat pcspkr cdc_ether usbnet mii uas sg usb_storage i2c_i801 shpchp lpc_ich mei_me mei wmi ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter nfsd auth_rpcgss nfs_acl
lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops i40e ttm igb drm ahci libahci dca ptp i2c_algo_bit pps_core libata megaraid_sas(OE) i2c_core
CPU: 26 PID: 125507 Comm: sfavsrv Tainted: G           OEL ------------   3.10.0-693.21.1.el7.x86_64 #1
Hardware name: Lenovo ThinkSystem SR850 -[7X19CTO1WW]-/-[7X19CTO1WW]-, BIOS -[TEE164L-2.80]- 10/23/2020
task: ffff883041b18fd0 ti: ffff8830f7c74000 task.ti: ffff8830f7c74000
RIP: 0010:[<ffffffffc02715b0>]  [<ffffffffc02715b0>] ext4_inode_touch_time_cmp+0x40/0x90 [ext4]
RSP: 0018:ffff8830f7c77588  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffffffff811e1682 RCX: 000000053863ba77
RDX: ffff8832e0ed37d0 RSI: ffff883a59607850 RDI: 0000000000000000
RBP: ffff8830f7c77588 R08: ffff8830f7c77708 R09: ffff8817deb19600
R10: 00000000fe163101 R11: ffffea0053f858c0 R12: 0000000000000000
R13: ffff882fde26cd00 R14: ffff884ff3e88400 R15: ffffffff8119a9e1
FS:  00007fa65f7fe700(0000) GS:ffff882fde500000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0b56dd84d8 CR3: 00000030f8ce0000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8133c2e2>] merge+0x62/0xc0
[<ffffffffc0271570>] ? ext4_ind_remove_space+0x8c0/0x8c0 [ext4]
[<ffffffff8133c3db>] list_sort+0x9b/0x250
[<ffffffffc0271a6e>] __ext4_es_shrink+0x1ce/0x2a0 [ext4]
[<ffffffffc0271bf4>] ext4_es_shrink+0xb4/0x130 [ext4]
[<ffffffff811999a5>] shrink_slab+0x175/0x340
[<ffffffff811fbf31>] ? vmpressure+0x21/0x90
[<ffffffff8119cb22>] do_try_to_free_pages+0x3c2/0x4e0
[<ffffffff8119cd3c>] try_to_free_pages+0xfc/0x180
[<ffffffff816aa515>] __alloc_pages_slowpath+0x457/0x724
[<ffffffff811912a5>] __alloc_pages_nodemask+0x405/0x420
[<ffffffff811d5a38>] alloc_pages_current+0x98/0x110
[<ffffffff8118bb0e>] __get_free_pages+0xe/0x40
[<ffffffff811e146e>] kmalloc_order_trace+0x2e/0xa0
[<ffffffff811e5011>] __kmalloc+0x211/0x230
[<ffffffff8127f4b0>] ? get_iowait_time+0x60/0x60
[<ffffffff8122ac87>] seq_buf_alloc+0x17/0x40
[<ffffffff8122b02a>] single_open_size+0x2a/0xa0
[<ffffffff8122abe0>] ? single_open+0xb0/0xb0
[<ffffffff8127fbb5>] stat_open+0x45/0x50
[<ffffffff81275b42>] proc_reg_open+0x72/0x100
[<ffffffff81202f8a>] do_dentry_open+0x1aa/0x2e0
[<ffffffff812b7802>] ? security_inode_permission+0x22/0x30
[<ffffffff81275ad0>] ? proc_reg_release+0x80/0x80
[<ffffffff8120315a>] vfs_open+0x5a/0xb0
[<ffffffff81210fc8>] ? may_open+0x68/0x110
[<ffffffff812144bd>] do_last+0x1ed/0x12c0
[<ffffffff81215652>] path_openat+0xc2/0x490
[<ffffffff811b53b9>] ? do_numa_page+0x159/0x1e0
[<ffffffff81217bfb>] do_filp_open+0x4b/0xb0
[<ffffffff81224f1a>] ? __alloc_fd+0x8a/0x130
[<ffffffff81204523>] do_sys_open+0xf3/0x1f0
[<ffffffff8120463e>] SyS_open+0x1e/0x20
[<ffffffff816c0715>] system_call_fastpath+0x1c/0x21
Code: 01 74 15 48 8b 8a e8 fc ff ff b8 01 00 00 00 48 c1 e9 2c 83 e1 01 74 29 48 8b 86 e8 fc ff ff 48 c1 e8 2c a8 01 74 24 48 8b 4e 18 <48> 8b 42 18 48 39 c1 74 37 48 29 c8 48 c1 f8 3f 83 e0 02 83 e8 
Complete report #1271994
Frame # Function Binary Source or offset Line
1
dump_stack
vmlinux 0xffffffff816a3e51
Build id: 3.10.0-693.5.2.el7.x86_64
0x19
2
__warn
vmlinux 0xffffffff810879d8
Build id: 3.10.0-693.5.2.el7.x86_64
0xd8
3
warn_slowpath_null
vmlinux 0xffffffff81087b1d
Build id: 3.10.0-693.5.2.el7.x86_64
0x1d
4
nfs_clear_inode
nfs 0xffffffffc0883a9c
Build id: 3.10.0-693.5.2.el7.x86_64
0x5c
5
nfs_evict_inode
nfs 0xffffffffc0885fd9
Build id: 3.10.0-693.5.2.el7.x86_64
0x29
6
evict
vmlinux 0xffffffff8121d729
Build id: 3.10.0-693.5.2.el7.x86_64
0xa9
7
dispose_list
vmlinux 0xffffffff8121d83e
Build id: 3.10.0-693.5.2.el7.x86_64
0x3e
8
prune_icache_sb
vmlinux 0xffffffff8121e834
Build id: 3.10.0-693.5.2.el7.x86_64
0x174
9
prune_super
vmlinux 0xffffffff81203863
Build id: 3.10.0-693.5.2.el7.x86_64
0x143
10
shrink_slab
vmlinux 0xffffffff81195443
Build id: 3.10.0-693.5.2.el7.x86_64
0x163
11
? vmpressure
vmlinux 0xffffffff811f7451
Build id: 3.10.0-693.5.2.el7.x86_64
0x21
12
do_try_to_free_pages
vmlinux 0xffffffff811985a2
Build id: 3.10.0-693.5.2.el7.x86_64
0x3c2
13
try_to_free_pages
vmlinux 0xffffffff811987bc
Build id: 3.10.0-693.5.2.el7.x86_64
0xfc
14
__alloc_pages_slowpath
vmlinux 0xffffffff8169fbcb
Build id: 3.10.0-693.5.2.el7.x86_64
0x457
15
__alloc_pages_nodemask
vmlinux 0xffffffff8118cdb5
Build id: 3.10.0-693.5.2.el7.x86_64
0x405
16
alloc_pages_current
vmlinux 0xffffffff811d1078
Build id: 3.10.0-693.5.2.el7.x86_64
0x98
17
__get_free_pages
vmlinux 0xffffffff8118761e
Build id: 3.10.0-693.5.2.el7.x86_64
0xe
18
kmalloc_order_trace
vmlinux 0xffffffff811dca2e
Build id: 3.10.0-693.5.2.el7.x86_64
0x2e
19
__kmalloc
vmlinux 0xffffffff811e05c1
Build id: 3.10.0-693.5.2.el7.x86_64
0x211
20
? get_iowait_time
vmlinux 0xffffffff8127a030
Build id: 3.10.0-693.5.2.el7.x86_64
0x60
21
seq_buf_alloc
vmlinux 0xffffffff81225f07
Build id: 3.10.0-693.5.2.el7.x86_64
0x17
22
single_open_size
vmlinux 0xffffffff8122628a
Build id: 3.10.0-693.5.2.el7.x86_64
0x2a
23
? single_open
vmlinux 0xffffffff81225e60
Build id: 3.10.0-693.5.2.el7.x86_64
0xb0
24
stat_open
vmlinux 0xffffffff8127a735
Build id: 3.10.0-693.5.2.el7.x86_64
0x45
25
proc_reg_open
vmlinux 0xffffffff8127073f
Build id: 3.10.0-693.5.2.el7.x86_64
0x6f
26
do_dentry_open
vmlinux 0xffffffff811fe5e7
Build id: 3.10.0-693.5.2.el7.x86_64
0x1a7
27
? security_inode_permission
vmlinux 0xffffffff812b1fbc
Build id: 3.10.0-693.5.2.el7.x86_64
0x1c
28
? proc_reg_release
vmlinux 0xffffffff812706d0
Build id: 3.10.0-693.5.2.el7.x86_64
0x80
29
vfs_open
vmlinux 0xffffffff811fe7ba
Build id: 3.10.0-693.5.2.el7.x86_64
0x5a
30
? may_open
vmlinux 0xffffffff8120c398
Build id: 3.10.0-693.5.2.el7.x86_64
0x68
31
do_last
vmlinux 0xffffffff8120f80d
Build id: 3.10.0-693.5.2.el7.x86_64
0x1ed
32
path_openat
vmlinux 0xffffffff812109a2
Build id: 3.10.0-693.5.2.el7.x86_64
0xc2
33
? user_path_at_empty
vmlinux 0xffffffff81212e42
Build id: 3.10.0-693.5.2.el7.x86_64
0x72
34
do_filp_open
vmlinux 0xffffffff81212f3b
Build id: 3.10.0-693.5.2.el7.x86_64
0x4b
35
? __alloc_fd
vmlinux 0xffffffff8122019a
Build id: 3.10.0-693.5.2.el7.x86_64
0x8a
36
do_sys_open
vmlinux 0xffffffff811ffb83
Build id: 3.10.0-693.5.2.el7.x86_64
0xf3
37
SyS_open
vmlinux 0xffffffff811ffc9e
Build id: 3.10.0-693.5.2.el7.x86_64
0x1e
38
system_call_fastpath
vmlinux 0xffffffff816b5089
Build id: 3.10.0-693.5.2.el7.x86_64
0x16
WARNING: CPU: 17 PID: 165508 at fs/nfs/inode.c:127 nfs_clear_inode+0x5c/0x90 [nfs]
Modules linked in: fuse ses enclosure scsi_transport_sas uas usb_storage tcp_diag udp_diag inet_diag nf_conntrack_ftp nf_conntrack libcrc32c 8021q garp mrp stp llc nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6_tables iptable_filter rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel dell_smbios aesni_intel lrw gf128mul glue_helper ablk_helper dcdbas sparse_keymap iTCO_wdt cryptd iTCO_vendor_support joydev sg pcspkr ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad mei_me mei lpc_ich acpi_power_meter
i2c_i801 wmi shpchp nfsd binfmt_misc auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea crct10dif_pclmul sysfillrect crct10dif_common sysimgblt crc32c_intel fb_sys_fops ttm drm ahci libahci tg3 bnxt_en libata megaraid_sas ptp i2c_core pps_core
CPU: 17 PID: 165508 Comm: zabbix_agentd Not tainted 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0XP8V5, BIOS 2.2.11 06/14/2019
0000000000000000 0000000008621460 ffff8818e69af650 ffffffff816a3e51
ffff8818e69af690 ffffffff810879d8 0000007f00000000 ffff88179e772288
ffff88179e772390 ffffffffc089cc60 ffff8817da218908 ffff880142c2f148
Call Trace:
[<ffffffff816a3e51>] dump_stack+0x19/0x1b
[<ffffffff810879d8>] __warn+0xd8/0x100
[<ffffffff81087b1d>] warn_slowpath_null+0x1d/0x20
[<ffffffffc0883a9c>] nfs_clear_inode+0x5c/0x90 [nfs]
[<ffffffffc0885fd9>] nfs_evict_inode+0x29/0x30 [nfs]
[<ffffffff8121d729>] evict+0xa9/0x180
[<ffffffff8121d83e>] dispose_list+0x3e/0x50
[<ffffffff8121e834>] prune_icache_sb+0x174/0x340
[<ffffffff81203863>] prune_super+0x143/0x170
[<ffffffff81195443>] shrink_slab+0x163/0x330
[<ffffffff811f7451>] ? vmpressure+0x21/0x90
[<ffffffff811985a2>] do_try_to_free_pages+0x3c2/0x4e0
[<ffffffff811987bc>] try_to_free_pages+0xfc/0x180
[<ffffffff8169fbcb>] __alloc_pages_slowpath+0x457/0x724
[<ffffffff8118cdb5>] __alloc_pages_nodemask+0x405/0x420
[<ffffffff811d1078>] alloc_pages_current+0x98/0x110
[<ffffffff8118761e>] __get_free_pages+0xe/0x40
[<ffffffff811dca2e>] kmalloc_order_trace+0x2e/0xa0
[<ffffffff811e05c1>] __kmalloc+0x211/0x230
[<ffffffff8127a030>] ? get_iowait_time+0x60/0x60
[<ffffffff81225f07>] seq_buf_alloc+0x17/0x40
[<ffffffff8122628a>] single_open_size+0x2a/0xa0
[<ffffffff81225e60>] ? single_open+0xb0/0xb0
[<ffffffff8127a735>] stat_open+0x45/0x50
[<ffffffff8127073f>] proc_reg_open+0x6f/0x100
[<ffffffff811fe5e7>] do_dentry_open+0x1a7/0x2e0
[<ffffffff812b1fbc>] ? security_inode_permission+0x1c/0x30
[<ffffffff812706d0>] ? proc_reg_release+0x80/0x80
[<ffffffff811fe7ba>] vfs_open+0x5a/0xb0
[<ffffffff8120c398>] ? may_open+0x68/0x110
[<ffffffff8120f80d>] do_last+0x1ed/0x12c0
[<ffffffff812109a2>] path_openat+0xc2/0x490
[<ffffffff81212e42>] ? user_path_at_empty+0x72/0xc0
[<ffffffff81212f3b>] do_filp_open+0x4b/0xb0
[<ffffffff8122019a>] ? __alloc_fd+0x8a/0x130
[<ffffffff811ffb83>] do_sys_open+0xf3/0x1f0
[<ffffffff811ffc9e>] SyS_open+0x1e/0x20
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Complete report #594519
Frame # Function Binary Source or offset Line
1
_raw_spin_lock
vmlinux 0xffffffff8168ea6e
Build id: 3.10.0-514.26.2.el7.x86_64
0xe
2
? shrink_dentry_list
vmlinux 0xffffffff8121757f
Build id: 3.10.0-514.26.2.el7.x86_64
0x4f
3
prune_dcache_sb
vmlinux 0xffffffff81217e4e
Build id: 3.10.0-514.26.2.el7.x86_64
0x13e
4
prune_super
vmlinux 0xffffffff81201128
Build id: 3.10.0-514.26.2.el7.x86_64
0x138
5
shrink_slab
vmlinux 0xffffffff811947b3
Build id: 3.10.0-514.26.2.el7.x86_64
0x163
6
? vmpressure
vmlinux 0xffffffff811f5ad1
Build id: 3.10.0-514.26.2.el7.x86_64
0x21
7
do_try_to_free_pages
vmlinux 0xffffffff81197ab2
Build id: 3.10.0-514.26.2.el7.x86_64
0x3c2
8
try_to_free_pages
vmlinux 0xffffffff81197ccc
Build id: 3.10.0-514.26.2.el7.x86_64
0xfc
9
__alloc_pages_slowpath
vmlinux 0xffffffff81682a68
Build id: 3.10.0-514.26.2.el7.x86_64
0x458
10
__alloc_pages_nodemask
vmlinux 0xffffffff8118b655
Build id: 3.10.0-514.26.2.el7.x86_64
0x405
11
alloc_pages_current
vmlinux 0xffffffff811cf9ca
Build id: 3.10.0-514.26.2.el7.x86_64
0xaa
12
__get_free_pages
vmlinux 0xffffffff81185f7e
Build id: 3.10.0-514.26.2.el7.x86_64
0xe
13
kmalloc_order_trace
vmlinux 0xffffffff811db26e
Build id: 3.10.0-514.26.2.el7.x86_64
0x2e
14
__kmalloc
vmlinux 0xffffffff811dda41
Build id: 3.10.0-514.26.2.el7.x86_64
0x221
15
? get_iowait_time
vmlinux 0xffffffff81275c20
Build id: 3.10.0-514.26.2.el7.x86_64
0x60
16
seq_buf_alloc
vmlinux 0xffffffff812227c7
Build id: 3.10.0-514.26.2.el7.x86_64
0x17
17
single_open_size
vmlinux 0xffffffff81222b4a
Build id: 3.10.0-514.26.2.el7.x86_64
0x2a
18
? single_open
vmlinux 0xffffffff81222720
Build id: 3.10.0-514.26.2.el7.x86_64
0xb0
19
stat_open
vmlinux 0xffffffff81276325
Build id: 3.10.0-514.26.2.el7.x86_64
0x45
20
proc_reg_open
vmlinux 0xffffffff8126c2ef
Build id: 3.10.0-514.26.2.el7.x86_64
0x6f
21
do_dentry_open
vmlinux 0xffffffff811fc687
Build id: 3.10.0-514.26.2.el7.x86_64
0x1a7
22
? security_inode_permission
vmlinux 0xffffffff812a91ec
Build id: 3.10.0-514.26.2.el7.x86_64
0x1c
23
? proc_reg_release
vmlinux 0xffffffff8126c280
Build id: 3.10.0-514.26.2.el7.x86_64
0x70
24
vfs_open
vmlinux 0xffffffff811fc85f
Build id: 3.10.0-514.26.2.el7.x86_64
0x5f
25
? may_open
vmlinux 0xffffffff8120a228
Build id: 3.10.0-514.26.2.el7.x86_64
0x68
26
do_last
vmlinux 0xffffffff8120d3cd
Build id: 3.10.0-514.26.2.el7.x86_64
0x1ed
27
path_openat
vmlinux 0xffffffff8120e542
Build id: 3.10.0-514.26.2.el7.x86_64
0xc2
28
? user_path_at_empty
vmlinux 0xffffffff812105c2
Build id: 3.10.0-514.26.2.el7.x86_64
0x72
29
do_filp_open
vmlinux 0xffffffff812106bb
Build id: 3.10.0-514.26.2.el7.x86_64
0x4b
30
? __alloc_fd
vmlinux 0xffffffff8121d337
Build id: 3.10.0-514.26.2.el7.x86_64
0xa7
31
do_sys_open
vmlinux 0xffffffff811fdbc3
Build id: 3.10.0-514.26.2.el7.x86_64
0xf3
32
SyS_open
vmlinux 0xffffffff811fdcde
Build id: 3.10.0-514.26.2.el7.x86_64
0x1e
33
system_call_fastpath
vmlinux 0xffffffff81697809
Build id: 3.10.0-514.26.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#39 stuck for 67s! [zabbix_agentd:2176]
Modules linked in: rpcsec_gss_krb5 mpt3sas mpt2sas raid_class xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter mptctl mptbase vfat fat iTCO_wdt iTCO_vendor_support mxm_wmi dcdbas intel_powerclamp ipmi_devintf coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel ixgbe aesni_intel lrw gf128mul ses mei_me mdio enclosure dca glue_helper ablk_helper mei scsi_transport_sas cryptd sg pcspkr ipmi_ssif lpc_ich sb_edac edac_core shpchp ipmi_si ipmi_msghandler wmi acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod
crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_pclmul crct10dif_common crc32c_intel mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci tg3 i2c_core libata ptp megaraid_sas pps_core fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 39 PID: 2176 Comm: zabbix_agentd Tainted: G             L ------------   3.10.0-514.26.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R730/0WCJNT, BIOS 2.4.3 01/17/2017
task: ffff885efa788fb0 ti: ffff885efd8e0000 task.ti: ffff885efd8e0000
RIP: 0010:[<ffffffff8168ea6e>]  [<ffffffff8168ea6e>] _raw_spin_lock+0xe/0x50
RSP: 0018:ffff885efd8e36c8  EFLAGS: 00000206
RAX: 00000000000a000a RBX: 0000000000000000 RCX: 000aeefe74ea74ea
RDX: 000aeefd74ea74ea RSI: 00000000000aeefe RDI: ffff88366d44bb58
RBP: ffff885efd8e3718 R08: 0000000000000000 R09: 0000000000000000
R10: ffff88bd02b80fc0 R11: 0000000000000001 R12: ffff88395ca25740
R13: 0000000000000001 R14: ffffffff81215120 R15: ffff885efd8e36a0
FS:  00007f0ac4b2fd80(0000) GS:ffff88befe2c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1c14af1280 CR3: 0000005ed9727000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff8121757f ffff88bd02b80fc0 ffff88bd02b80fc0 ffff88366d44bb80
00000000f1d20558 ffff88befd0c9000 ffff88276acadf18 0000000092c2c9f8
ffff88befd0c90d0 ffff88276acadf40 ffff885efd8e3778 ffffffff81217e4e
Call Trace:
[<ffffffff8121757f>] ? shrink_dentry_list+0x4f/0x480
[<ffffffff81217e4e>] prune_dcache_sb+0x13e/0x180
[<ffffffff81201128>] prune_super+0x138/0x170
[<ffffffff811947b3>] shrink_slab+0x163/0x330
[<ffffffff811f5ad1>] ? vmpressure+0x21/0x90
[<ffffffff81197ab2>] do_try_to_free_pages+0x3c2/0x4e0
[<ffffffff81197ccc>] try_to_free_pages+0xfc/0x180
[<ffffffff81682a68>] __alloc_pages_slowpath+0x458/0x725
[<ffffffff8118b655>] __alloc_pages_nodemask+0x405/0x420
[<ffffffff811cf9ca>] alloc_pages_current+0xaa/0x170
[<ffffffff81185f7e>] __get_free_pages+0xe/0x50
[<ffffffff811db26e>] kmalloc_order_trace+0x2e/0xa0
[<ffffffff811dda41>] __kmalloc+0x221/0x240
[<ffffffff81275c20>] ? get_iowait_time+0x60/0x60
[<ffffffff812227c7>] seq_buf_alloc+0x17/0x40
[<ffffffff81222b4a>] single_open_size+0x2a/0xa0
[<ffffffff81222720>] ? single_open+0xb0/0xb0
[<ffffffff81276325>] stat_open+0x45/0x50
[<ffffffff8126c2ef>] proc_reg_open+0x6f/0x100
[<ffffffff811fc687>] do_dentry_open+0x1a7/0x2e0
[<ffffffff812a91ec>] ? security_inode_permission+0x1c/0x30
[<ffffffff8126c280>] ? proc_reg_release+0x70/0x70
[<ffffffff811fc85f>] vfs_open+0x5f/0xe0
[<ffffffff8120a228>] ? may_open+0x68/0x110
[<ffffffff8120d3cd>] do_last+0x1ed/0x12a0
[<ffffffff8120e542>] path_openat+0xc2/0x490
[<ffffffff812105c2>] ? user_path_at_empty+0x72/0xc0
[<ffffffff812106bb>] do_filp_open+0x4b/0xb0
[<ffffffff8121d337>] ? __alloc_fd+0xa7/0x130
[<ffffffff811fdbc3>] do_sys_open+0xf3/0x1f0
[<ffffffff811fdcde>] SyS_open+0x1e/0x20
[<ffffffff81697809>] system_call_fastpath+0x16/0x1b
Code: 66 83 07 02 f6 47 02 01 74 e5 0f 1f 00 e8 38 28 ff ff eb db 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 00 00 02 00 f0 0f c1 07 <89> c2 c1 ea 10 66 39 c2 75 01 c3 55 83 e2 fe 0f b7 f2 48 89 e5 
Complete report #591910
Frame # Function Binary Source or offset Line
1
_raw_spin_lock
vmlinux 0xffffffff8168ea9a
Build id: 3.10.0-514.26.2.el7.x86_64
0x3a
2
dentry_lru_del
vmlinux 0xffffffff812150db
Build id: 3.10.0-514.26.2.el7.x86_64
0x2b
3
shrink_dentry_list
vmlinux 0xffffffff8121765b
Build id: 3.10.0-514.26.2.el7.x86_64
0x12b
4
prune_dcache_sb
vmlinux 0xffffffff81217e4e
Build id: 3.10.0-514.26.2.el7.x86_64
0x13e
5
prune_super
vmlinux 0xffffffff81201128
Build id: 3.10.0-514.26.2.el7.x86_64
0x138
6
shrink_slab
vmlinux 0xffffffff811947b3
Build id: 3.10.0-514.26.2.el7.x86_64
0x163
7
? vmpressure
vmlinux 0xffffffff811f5ad1
Build id: 3.10.0-514.26.2.el7.x86_64
0x21
8
do_try_to_free_pages
vmlinux 0xffffffff81197ab2
Build id: 3.10.0-514.26.2.el7.x86_64
0x3c2
9
try_to_free_pages
vmlinux 0xffffffff81197ccc
Build id: 3.10.0-514.26.2.el7.x86_64
0xfc
10
__alloc_pages_slowpath
vmlinux 0xffffffff81682a68
Build id: 3.10.0-514.26.2.el7.x86_64
0x458
11
__alloc_pages_nodemask
vmlinux 0xffffffff8118b655
Build id: 3.10.0-514.26.2.el7.x86_64
0x405
12
alloc_pages_current
vmlinux 0xffffffff811cf9ca
Build id: 3.10.0-514.26.2.el7.x86_64
0xaa
13
__get_free_pages
vmlinux 0xffffffff81185f7e
Build id: 3.10.0-514.26.2.el7.x86_64
0xe
14
kmalloc_order_trace
vmlinux 0xffffffff811db26e
Build id: 3.10.0-514.26.2.el7.x86_64
0x2e
15
__kmalloc
vmlinux 0xffffffff811dda41
Build id: 3.10.0-514.26.2.el7.x86_64
0x221
16
? get_iowait_time
vmlinux 0xffffffff81275c20
Build id: 3.10.0-514.26.2.el7.x86_64
0x60
17
seq_buf_alloc
vmlinux 0xffffffff812227c7
Build id: 3.10.0-514.26.2.el7.x86_64
0x17
18
single_open_size
vmlinux 0xffffffff81222b4a
Build id: 3.10.0-514.26.2.el7.x86_64
0x2a
19
? single_open
vmlinux 0xffffffff81222720
Build id: 3.10.0-514.26.2.el7.x86_64
0xb0
20
stat_open
vmlinux 0xffffffff81276325
Build id: 3.10.0-514.26.2.el7.x86_64
0x45
21
proc_reg_open
vmlinux 0xffffffff8126c2ef
Build id: 3.10.0-514.26.2.el7.x86_64
0x6f
22
do_dentry_open
vmlinux 0xffffffff811fc687
Build id: 3.10.0-514.26.2.el7.x86_64
0x1a7
23
? security_inode_permission
vmlinux 0xffffffff812a91ec
Build id: 3.10.0-514.26.2.el7.x86_64
0x1c
24
? proc_reg_release
vmlinux 0xffffffff8126c280
Build id: 3.10.0-514.26.2.el7.x86_64
0x70
25
vfs_open
vmlinux 0xffffffff811fc85f
Build id: 3.10.0-514.26.2.el7.x86_64
0x5f
26
? may_open
vmlinux 0xffffffff8120a228
Build id: 3.10.0-514.26.2.el7.x86_64
0x68
27
do_last
vmlinux 0xffffffff8120d3cd
Build id: 3.10.0-514.26.2.el7.x86_64
0x1ed
28
path_openat
vmlinux 0xffffffff8120e542
Build id: 3.10.0-514.26.2.el7.x86_64
0xc2
29
? user_path_at_empty
vmlinux 0xffffffff812105c2
Build id: 3.10.0-514.26.2.el7.x86_64
0x72
30
do_filp_open
vmlinux 0xffffffff812106bb
Build id: 3.10.0-514.26.2.el7.x86_64
0x4b
31
? __alloc_fd
vmlinux 0xffffffff8121d337
Build id: 3.10.0-514.26.2.el7.x86_64
0xa7
32
do_sys_open
vmlinux 0xffffffff811fdbc3
Build id: 3.10.0-514.26.2.el7.x86_64
0xf3
33
SyS_open
vmlinux 0xffffffff811fdcde
Build id: 3.10.0-514.26.2.el7.x86_64
0x1e
34
system_call_fastpath
vmlinux 0xffffffff81697809
Build id: 3.10.0-514.26.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [zabbix_agentd:2209]
Modules linked in: binfmt_misc mpt3sas mpt2sas raid_class xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter mptctl mptbase vfat fat mxm_wmi iTCO_wdt iTCO_vendor_support ipmi_devintf dcdbas intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses ixgbe sb_edac enclosure scsi_transport_sas ipmi_ssif pcspkr mei_me mdio dca sg lpc_ich mei edac_core ipmi_si ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod
sd_mod cdrom crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci libata i2c_core tg3 megaraid_sas ptp pps_core fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 2209 Comm: zabbix_agentd Tainted: G        W    L ------------   3.10.0-514.26.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R730/0WCJNT, BIOS 2.4.3 01/17/2017
task: ffff885efa10af10 ti: ffff885eefbb4000 task.ti: ffff885eefbb4000
RIP: 0010:[<ffffffff8168ea9a>]  [<ffffffff8168ea9a>] _raw_spin_lock+0x3a/0x50
RSP: 0000:ffff885eefbb76a0  EFLAGS: 00000202
RAX: 0000000000007f58 RBX: ffffffff81215120 RCX: 0000000000008984
RDX: 000000000000898c RSI: 000000000000898c RDI: ffffffff819b7400
RBP: ffff885eefbb76a0 R08: 0000000000000000 R09: 0000000000000000
R10: ffff887b580f2000 R11: 0000000000000001 R12: ffffffff81ae6300
R13: 00000000004f8198 R14: 0000000000000000 R15: 0000000000000000
FS:  00007f6104164d80(0000) GS:ffff88befde40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb7d7d47b80 CR3: 0000005ef7465000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff885eefbb76c0 ffffffff812150db ffff8822e39a2000 ffff887b580f2080
ffff885eefbb7718 ffffffff8121765b ffff887b580f2000 ffff887b580f2000
ffff887b580f2080 0000000091fa88ac ffff88befd5a0800 ffff887e37b2f0d8
Call Trace:
[<ffffffff812150db>] dentry_lru_del+0x2b/0x70
[<ffffffff8121765b>] shrink_dentry_list+0x12b/0x480
[<ffffffff81217e4e>] prune_dcache_sb+0x13e/0x180
[<ffffffff81201128>] prune_super+0x138/0x170
[<ffffffff811947b3>] shrink_slab+0x163/0x330
[<ffffffff811f5ad1>] ? vmpressure+0x21/0x90
[<ffffffff81197ab2>] do_try_to_free_pages+0x3c2/0x4e0
[<ffffffff81197ccc>] try_to_free_pages+0xfc/0x180
[<ffffffff81682a68>] __alloc_pages_slowpath+0x458/0x725
[<ffffffff8118b655>] __alloc_pages_nodemask+0x405/0x420
[<ffffffff811cf9ca>] alloc_pages_current+0xaa/0x170
[<ffffffff81185f7e>] __get_free_pages+0xe/0x50
[<ffffffff811db26e>] kmalloc_order_trace+0x2e/0xa0
[<ffffffff811dda41>] __kmalloc+0x221/0x240
[<ffffffff81275c20>] ? get_iowait_time+0x60/0x60
[<ffffffff812227c7>] seq_buf_alloc+0x17/0x40
[<ffffffff81222b4a>] single_open_size+0x2a/0xa0
[<ffffffff81222720>] ? single_open+0xb0/0xb0
[<ffffffff81276325>] stat_open+0x45/0x50
[<ffffffff8126c2ef>] proc_reg_open+0x6f/0x100
[<ffffffff811fc687>] do_dentry_open+0x1a7/0x2e0
[<ffffffff812a91ec>] ? security_inode_permission+0x1c/0x30
[<ffffffff8126c280>] ? proc_reg_release+0x70/0x70
[<ffffffff811fc85f>] vfs_open+0x5f/0xe0
[<ffffffff8120a228>] ? may_open+0x68/0x110
[<ffffffff8120d3cd>] do_last+0x1ed/0x12a0
[<ffffffff8120e542>] path_openat+0xc2/0x490
[<ffffffff812105c2>] ? user_path_at_empty+0x72/0xc0
[<ffffffff812106bb>] do_filp_open+0x4b/0xb0
[<ffffffff8121d337>] ? __alloc_fd+0xa7/0x130
[<ffffffff811fdbc3>] do_sys_open+0xf3/0x1f0
[<ffffffff811fdcde>] SyS_open+0x1e/0x20
[<ffffffff81697809>] system_call_fastpath+0x16/0x1b
Code: c2 c1 ea 10 66 39 c2 75 01 c3 55 83 e2 fe 0f b7 f2 48 89 e5 b8 00 80 00 00 eb 0d 66 0f 1f 44 00 00 f3 90 83 e8 01 74 0a 0f b7 0f <66> 39 ca 75 f1 5d c3 0f 1f 80 00 00 00 00 eb d9 66 0f 1f 44 00