We would like to report an urgent issue with our linked environment.
We ve a cent OS physical content node with the following versions:
cn1.vg1ath1c1.cdn ~# hostnamectl
Static hostname: cn1.vg1ath1c1.cdn
Icon name: computer-server
Chassis: server
Machine ID: 0854094725eb46dcabbe12de0106e9a2
Boot ID: c1bb5d42fd5f40f9839ec5d1654f50a8
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-693.el7.x86_64
Architecture: x86-64
And inbuilt on it we ve a virtual machine with following versions:
oa1.vg1ath1c1.cdn ~# hostnamectl
Static hostname: oa1.vg1ath1c1.cdn
Icon name: computer-vm
Chassis: vm
Machine ID: f390795d9392450ab54f86ce8633c607
Boot ID: ce43d9694584465292bdb51cf4b38efe
Virtualization: kvm
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-693.el7.x86_64
Architecture: x86-64
We re finding issues on the connectivity between the physical content node and the inbuilt origin, described below:
All NFS threads are blocked as per the attached CN1 DMESG output:
Code: Select all
[Wed Sep 25 17:50:44 2019] INFO: task nfsd:4900 blocked for more than 120 seconds.
[Wed Sep 25 17:50:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:50:44 2019] nfsd D ffff8841e068fa90 0 4900 2 0x00000080
[Wed Sep 25 17:50:44 2019] ffff8841e068f930 0000000000000046 ffff881feb70af70 ffff8841e068ffd8
[Wed Sep 25 17:50:44 2019] ffff8841e068ffd8 ffff8841e068ffd8 ffff881feb70af70 ffff881fffa96cc0
[Wed Sep 25 17:50:44 2019] 0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff8841e068fa90
[Wed Sep 25 17:50:44 2019] Call Trace:
[Wed Sep 25 17:50:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:50:44 2019] [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:50:44 2019] [<ffffffff815d0293>] ? ip_output+0x73/0xe0
[Wed Sep 25 17:50:44 2019] [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:50:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:50:44 2019] [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:50:44 2019] [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:50:44 2019] [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:50:44 2019] [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:50:44 2019] [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:50:44 2019] [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:50:44 2019] [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:50:44 2019] [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:50:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] INFO: task nfsd:4903 blocked for more than 120 seconds.
[Wed Sep 25 17:50:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:50:44 2019] nfsd D ffff8841d8ba3a90 0 4903 2 0x00000080
[Wed Sep 25 17:50:44 2019] ffff8841d8ba3930 0000000000000046 ffff881feb708000 ffff8841d8ba3fd8
[Wed Sep 25 17:50:44 2019] ffff8841d8ba3fd8 ffff8841d8ba3fd8 ffff881feb708000 ffff881fffd56cc0
[Wed Sep 25 17:50:44 2019] 0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff8841d8ba3a90
[Wed Sep 25 17:50:44 2019] Call Trace:
[Wed Sep 25 17:50:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:50:44 2019] [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:50:44 2019] [<ffffffff811de581>] ? __slab_free+0x81/0x2f0
[Wed Sep 25 17:50:44 2019] [<ffffffff815d0293>] ? ip_output+0x73/0xe0
[Wed Sep 25 17:50:44 2019] [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:50:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:50:44 2019] [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:50:44 2019] [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:50:44 2019] [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:50:44 2019] [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:50:44 2019] [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:50:44 2019] [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:50:44 2019] [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:50:44 2019] [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:50:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] INFO: task nfsd:4904 blocked for more than 120 seconds.
[Wed Sep 25 17:50:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:50:44 2019] nfsd D ffffffff816a75f0 0 4904 2 0x00000080
[Wed Sep 25 17:50:44 2019] ffff8841d9bab930 0000000000000046 ffff881feb70bf40 ffff8841d9babfd8
[Wed Sep 25 17:50:44 2019] ffff8841d9babfd8 ffff8841d9babfd8 ffff881feb70bf40 ffff881fffa56cc0
[Wed Sep 25 17:50:44 2019] 0000000000000000 7fffffffffffffff ffff88207ff8b0e8 ffffffff816a75f0
[Wed Sep 25 17:50:44 2019] Call Trace:
[Wed Sep 25 17:50:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:50:44 2019] [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:50:44 2019] [<ffffffffc0010847>] ? dm_merge_bvec+0x87/0xd0 [dm_mod]
[Wed Sep 25 17:50:44 2019] [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:50:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:50:44 2019] [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:50:44 2019] [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a7125>] __wait_on_bit+0x65/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffff81181cb1>] wait_on_page_bit+0x81/0xa0
[Wed Sep 25 17:50:44 2019] [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff8118dd59>] write_cache_pages+0x319/0x4d0
[Wed Sep 25 17:50:44 2019] [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:50:44 2019] [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:50:44 2019] [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:50:44 2019] [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:50:44 2019] [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:50:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] INFO: task nfsd:4905 blocked for more than 120 seconds.
[Wed Sep 25 17:50:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:50:44 2019] nfsd D ffffffff816a75f0 0 4905 2 0x00000080
[Wed Sep 25 17:50:44 2019] ffff8841eb743930 0000000000000046 ffff881feb70eeb0 ffff8841eb743fd8
[Wed Sep 25 17:50:44 2019] ffff8841eb743fd8 ffff8841eb743fd8 ffff881feb70eeb0 ffff881fffc96cc0
[Wed Sep 25 17:50:44 2019] 0000000000000000 7fffffffffffffff ffff88207ff83be8 ffffffff816a75f0
[Wed Sep 25 17:50:44 2019] Call Trace:
[Wed Sep 25 17:50:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:50:44 2019] [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:50:44 2019] [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:50:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:50:44 2019] [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:50:44 2019] [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a7125>] __wait_on_bit+0x65/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffff81181cb1>] wait_on_page_bit+0x81/0xa0
[Wed Sep 25 17:50:44 2019] [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff8118dd59>] write_cache_pages+0x319/0x4d0
[Wed Sep 25 17:50:44 2019] [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:50:44 2019] [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:50:44 2019] [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:50:44 2019] [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:50:44 2019] [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:50:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] INFO: task nfsd:4907 blocked for more than 120 seconds.
[Wed Sep 25 17:50:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:50:44 2019] nfsd D ffff8841f326ba90 0 4907 2 0x00000080
[Wed Sep 25 17:50:44 2019] ffff8841f326b930 0000000000000046 ffff881feb70cf10 ffff8841f326bfd8
[Wed Sep 25 17:50:44 2019] ffff8841f326bfd8 ffff8841f326bfd8 ffff881feb70cf10 ffff881fffb96cc0
[Wed Sep 25 17:50:44 2019] 0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff8841f326ba90
[Wed Sep 25 17:50:44 2019] Call Trace:
[Wed Sep 25 17:50:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:50:44 2019] [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:50:44 2019] [<ffffffff815d0293>] ? ip_output+0x73/0xe0
[Wed Sep 25 17:50:44 2019] [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:50:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:50:44 2019] [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:50:44 2019] [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:50:44 2019] [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:50:44 2019] [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:50:44 2019] [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:50:44 2019] [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:50:44 2019] [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:50:44 2019] [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:50:44 2019] [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:50:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] INFO: task kworker/u57:2:602765 blocked for more than 120 seconds.
[Wed Sep 25 17:50:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:50:44 2019] kworker/u57:2 D ffff881cdefaf9d8 0 602765 2 0x00000080
[Wed Sep 25 17:50:44 2019] Workqueue: writeback bdi_writeback_workfn (flush-253:1)
[Wed Sep 25 17:50:44 2019] ffff881cdefaf870 0000000000000046 ffff881ffd1f3f40 ffff881cdefaffd8
[Wed Sep 25 17:50:44 2019] ffff881cdefaffd8 ffff881cdefaffd8 ffff881ffd1f3f40 ffff881fffa16cc0
[Wed Sep 25 17:50:44 2019] 0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff881cdefaf9d8
[Wed Sep 25 17:50:44 2019] Call Trace:
[Wed Sep 25 17:50:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:50:44 2019] [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:50:44 2019] [<ffffffffc0010847>] ? dm_merge_bvec+0x87/0xd0 [dm_mod]
[Wed Sep 25 17:50:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:50:44 2019] [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:50:44 2019] [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:50:44 2019] [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:50:44 2019] [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:50:44 2019] [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:50:44 2019] [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff8122d7d0>] __writeback_single_inode+0x40/0x220
[Wed Sep 25 17:50:44 2019] [<ffffffff8122e414>] writeback_sb_inodes+0x1c4/0x490
[Wed Sep 25 17:50:44 2019] [<ffffffff8122e77f>] __writeback_inodes_wb+0x9f/0xd0
[Wed Sep 25 17:50:44 2019] [<ffffffff8122efb3>] wb_writeback+0x263/0x2f0
[Wed Sep 25 17:50:44 2019] [<ffffffff8122f45b>] bdi_writeback_workfn+0x2cb/0x460
[Wed Sep 25 17:50:44 2019] [<ffffffff810a881a>] process_one_work+0x17a/0x440
[Wed Sep 25 17:50:44 2019] [<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[Wed Sep 25 17:50:44 2019] [<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[Wed Sep 25 17:50:44 2019] [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:50:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:50:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019] INFO: task nfsd:4898 blocked for more than 120 seconds.
[Wed Sep 25 17:52:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:52:44 2019] nfsd D ffff8841e6fa3a90 0 4898 2 0x00000080
[Wed Sep 25 17:52:44 2019] ffff8841e6fa3930 0000000000000046 ffff881fc6f5af70 ffff8841e6fa3fd8
[Wed Sep 25 17:52:44 2019] ffff8841e6fa3fd8 ffff8841e6fa3fd8 ffff881fc6f5af70 ffff881fffa16cc0
[Wed Sep 25 17:52:44 2019] 0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff8841e6fa3a90
[Wed Sep 25 17:52:44 2019] Call Trace:
[Wed Sep 25 17:52:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019] [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:52:44 2019] [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:52:44 2019] [<ffffffff810cd794>] ? update_curr+0x104/0x190
[Wed Sep 25 17:52:44 2019] [<ffffffff810ca29e>] ? account_entity_dequeue+0xae/0xd0
[Wed Sep 25 17:52:44 2019] [<ffffffff810cdc7c>] ? dequeue_entity+0x11c/0x5d0
[Wed Sep 25 17:52:44 2019] [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:52:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019] [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:52:44 2019] [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:52:44 2019] [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:52:44 2019] [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:52:44 2019] [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:52:44 2019] [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:52:44 2019] [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:52:44 2019] [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:52:44 2019] [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:52:44 2019] [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:52:44 2019] [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:52:44 2019] [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:52:44 2019] [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:52:44 2019] [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:52:44 2019] [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:52:44 2019] [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:52:44 2019] [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:52:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019] [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:52:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019] INFO: task nfsd:4899 blocked for more than 120 seconds.
[Wed Sep 25 17:52:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:52:44 2019] nfsd D ffff8841def7ba90 0 4899 2 0x00000080
[Wed Sep 25 17:52:44 2019] ffff8841def7b930 0000000000000046 ffff881fc6f5cf10 ffff8841def7bfd8
[Wed Sep 25 17:52:44 2019] ffff8841def7bfd8 ffff8841def7bfd8 ffff881fc6f5cf10 ffff881fffa16cc0
[Wed Sep 25 17:52:44 2019] 0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff8841def7ba90
[Wed Sep 25 17:52:44 2019] Call Trace:
[Wed Sep 25 17:52:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019] [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:52:44 2019] [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:52:44 2019] [<ffffffff810ce54e>] ? dequeue_task_fair+0x41e/0x660
[Wed Sep 25 17:52:44 2019] [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:52:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019] [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:52:44 2019] [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:52:44 2019] [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:52:44 2019] [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:52:44 2019] [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:52:44 2019] [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:52:44 2019] [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:52:44 2019] [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:52:44 2019] [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:52:44 2019] [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:52:44 2019] [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:52:44 2019] [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:52:44 2019] [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:52:44 2019] [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:52:44 2019] [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:52:44 2019] [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:52:44 2019] [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:52:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019] [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:52:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019] INFO: task nfsd:4900 blocked for more than 120 seconds.
[Wed Sep 25 17:52:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:52:44 2019] nfsd D ffff8841e068fa90 0 4900 2 0x00000080
[Wed Sep 25 17:52:44 2019] ffff8841e068f930 0000000000000046 ffff881feb70af70 ffff8841e068ffd8
[Wed Sep 25 17:52:44 2019] ffff8841e068ffd8 ffff8841e068ffd8 ffff881feb70af70 ffff881fffa96cc0
[Wed Sep 25 17:52:44 2019] 0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff8841e068fa90
[Wed Sep 25 17:52:44 2019] Call Trace:
[Wed Sep 25 17:52:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019] [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:52:44 2019] [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:52:44 2019] [<ffffffff815d0293>] ? ip_output+0x73/0xe0
[Wed Sep 25 17:52:44 2019] [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:52:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019] [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:52:44 2019] [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:52:44 2019] [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:52:44 2019] [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:52:44 2019] [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:52:44 2019] [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:52:44 2019] [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:52:44 2019] [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:52:44 2019] [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:52:44 2019] [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:52:44 2019] [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:52:44 2019] [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:52:44 2019] [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:52:44 2019] [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:52:44 2019] [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:52:44 2019] [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:52:44 2019] [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:52:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019] [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:52:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019] INFO: task nfsd:4901 blocked for more than 120 seconds.
[Wed Sep 25 17:52:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:52:44 2019] nfsd D ffff8841e06aba90 0 4901 2 0x00000080
[Wed Sep 25 17:52:44 2019] ffff8841e06ab930 0000000000000046 ffff881feb709fa0 ffff8841e06abfd8
[Wed Sep 25 17:52:44 2019] ffff8841e06abfd8 ffff8841e06abfd8 ffff881feb709fa0 ffff881fffa16cc0
[Wed Sep 25 17:52:44 2019] 0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff8841e06aba90
[Wed Sep 25 17:52:44 2019] Call Trace:
[Wed Sep 25 17:52:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019] [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:52:44 2019] [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:52:44 2019] [<ffffffff81231b32>] ? __generic_file_splice_read+0x542/0x5e0
[Wed Sep 25 17:52:44 2019] [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:52:44 2019] [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019] [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:52:44 2019] [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:52:44 2019] [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:52:44 2019] [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:52:44 2019] [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:52:44 2019] [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:52:44 2019] [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:52:44 2019] [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:52:44 2019] [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:52:44 2019] [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:52:44 2019] [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:52:44 2019] [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:52:44 2019] [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:52:44 2019] [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:52:44 2019] [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:52:44 2019] [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:52:44 2019] [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:52:44 2019] [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:52:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019] [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:52:44 2019] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: Select all
cn1.vg1ath1c1.cdn ~# for p in $(ps -efl | grep \\[nfsd\\] | awk '{ print $4 }'); do echo "stack for pid $p"; cat /proc/$p/stack; done
stack for pid 4898
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
stack for pid 4899
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
stack for pid 4900
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
stack for pid 4901
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
stack for pid 4903
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
stack for pid 4904
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
stack for pid 4905
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
stack for pid 4907
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
This connectivity is important as content is being ingested from the content node into the origin.
We ve collected some additional information (while the issue was occurring) below:
Code: Select all
CN1:
cn1.vg1ath1c1.cdn ~# systemctl status nfs.service
● nfs-server.service - NFS server and services
Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
Drop-In: /run/systemd/generator/nfs-server.service.d
└─order-with-mounts.conf
Active: active (exited) since Thu 2019-08-29 19:42:51 UTC; 14h ago
Process: 2752302 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS)
Process: 2752298 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS)
Process: 2752296 ExecStop=/usr/sbin/rpc.nfsd 0 (code=exited, status=0/SUCCESS)
Process: 2758399 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
Process: 2758394 ExecStartPre=/bin/sh -c /bin/kill -HUP `cat /run/gssproxy.pid` (code=exited, status=0/SUCCESS)
Process: 2758392 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
Main PID: 2758399 (code=exited, status=0/SUCCESS)
Tasks: 0
CGroup: /system.slice/nfs-server.service
Aug 29 19:42:50 cn1.vg1ath1c1.cdn systemd[1]: Starting NFS server and services...
Aug 29 19:42:51 cn1.vg1ath1c1.cdn systemd[1]: Started NFS server and services.
cn1.vg1ath1c1.cdn ~# ps -ef | grep nfs
root 5023 2 0 Jul23 ? 00:00:00 [nfsd4_callbacks]
root 5033 2 0 Jul23 ? 00:03:19 [nfsd]
root 5034 2 0 Jul23 ? 00:03:53 [nfsd]
root 5035 2 0 Jul23 ? 00:04:33 [nfsd]
root 5036 2 0 Jul23 ? 00:05:27 [nfsd]
root 5037 2 0 Jul23 ? 00:07:09 [nfsd]
root 5038 2 0 Jul23 ? 00:09:58 [nfsd]
root 5039 2 0 Jul23 ? 00:23:14 [nfsd]
root 5040 2 0 Jul23 ? 01:08:31 [nfsd]
root 3142534 3141879 0 10:33 pts/1 00:00:00 grep --color=auto nfs
OA1:
oa1.vg1ath1c1.cdn ~# systemctl status nfs-client.target
● nfs-client.target - NFS client services
Loaded: loaded (/usr/lib/systemd/system/nfs-client.target; enabled; vendor preset: disabled)
Active: active since Thu 2019-08-29 18:40:47 UTC; 15h ago
Aug 29 18:40:47 oa1.vg1ath1c1.cdn systemd[1]: Reached target NFS client services.
Aug 29 18:40:47 oa1.vg1ath1c1.cdn systemd[1]: Starting NFS client services.
oa1.vg1ath1c1.cdn ~# ps -ef | grep nfs
root 3654 2 0 Jul23 ? 00:00:00 [nfsiod]
root 19072 18748 0 10:34 pts/7 00:00:00 grep --color=auto nfs
root 30578 1 0 Aug29 ? 00:00:00 ls --color=auto /mnt/nfscn
Code: Select all
CN1:
cn1.vg1ath1c1.cdn ~# cat /etc/exports
/export/pantry 10.139.4.8(rw,wdelay,no_root_squash,no_subtree_check,sec=sys,secure,no_all_squash,mountpoint) 10.139.4.9(rw,wdelay,no_root_squash,no_subtree_check,sec=sys,secure,no_all_squash,mountpoint) 10.139.4.10(rw,wdelay,no_root_squash,no_subtree_check,sec=sys,secure,no_all_squash,mountpoint) 10.139.4.11(rw,wdelay,no_root_squash,no_subtree_check,sec=sys,secure,no_all_squash,mountpoint) 10.139.4.7(rw,wdelay,no_root_squash,no_subtree_check,sec=sys,secure,no_all_squash,mountpoint)
cn1.vg1ath1c1.cdn ~# cat /proc/net/rpc/nfsd
rc 0 4648421 19728575
fh 7 0 0 0 0
io 1639158320 2077816748
th 8 0 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
ra 32 15076466 0 0 0 0 0 0 0 0 0 33990
net 24378984 0 24379062 23
rpc 24378267 0 0 0 0
proc3 22 14 1887176 6302 14000 2588495 0 15110219 4627770 6153 1027 0 0 6157 1027 0 0 0 1090 96230 14 7 31485
proc4 2 0 0
proc4ops 72 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
OA1:
oa1.vg1ath1c1.cdn ~# showmount -e
clnt_create: RPC: Program not registered
oa1.vg1ath1c1.cdn ~# systemctl status mnt-nfscn.mount
● mnt-nfscn.mount - mnt-nfscn
Loaded: loaded (/etc/systemd/system/mnt-nfscn.mount; enabled; vendor preset: disabled)
Active: active (mounted) since Tue 2019-07-23 23:06:29 UTC; 1 months 7 days ago
Where: /mnt/nfscn
What: 10.139.4.2:/export/pantry
Memory: 12.0K
Jul 23 23:06:21 oa1.vg1ath1c1.cdn systemd[1]: Mounting mnt-nfscn...
Jul 23 23:06:29 oa1.vg1ath1c1.cdn systemd[1]: Mounted mnt-nfscn.
Code: Select all
I’ve tried to stop mnt-nfscn.mount service in VxOA but it hangs.
oa1.vg1ath1c1.cdn ~# systemctl stop mnt-nfscn.mount
Opening a new session and checking the service status:
oa1.vg1ath1c1.cdn ~# systemctl status mnt-nfscn.mount
● mnt-nfscn.mount - mnt-nfscn
Loaded: loaded (/etc/systemd/system/mnt-nfscn.mount; enabled; vendor preset: disabled)
Active: deactivating (unmounting) (Result: timeout) since Fri 2019-08-30 10:41:09 UTC; 2min 45s ago
Where: /mnt/nfscn
What: 10.139.4.2:/export/pantry
Control: 28506 (umount)
Memory: 228.0K
CGroup: /system.slice/mnt-nfscn.mount
└─28506 /bin/umount /mnt/nfscn
Jul 23 23:06:21 oa1.vg1ath1c1.cdn systemd[1]: Mounting mnt-nfscn...
Jul 23 23:06:29 oa1.vg1ath1c1.cdn systemd[1]: Mounted mnt-nfscn.
Aug 30 10:41:09 oa1.vg1ath1c1.cdn systemd[1]: Unmounting mnt-nfscn...
Aug 30 10:42:40 oa1.vg1ath1c1.cdn systemd[1]: mnt-nfscn.mount unmounting timed out. Stopping.
Code: Select all
I was able to umount the /mnt/nfscn partition by using -l (lazy) option. So now I can list the partitions in OA1:
oa1.vg1ath1c1.cdn ~# df -hk
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_root-root 39040384 3491376 35549008 9% /
devtmpfs 29796120 0 29796120 0% /dev
tmpfs 29807940 1416 29806524 1% /dev/shm
tmpfs 29807940 1499572 28308368 6% /run
tmpfs 29807940 0 29807940 0% /sys/fs/cgroup
vxoa_tmpfs 145408 0 145408 0% /mnt/ramdisk
/dev/vda2 943128 159808 734580 18% /boot
/dev/mapper/vg_root-eve_internal_storage 8775680 32928 8742752 1% /cache1/storage
/dev/mapper/vg_root-var_log 79943540 919504 79024036 2% /var/log
/dev/mapper/vg_root-eve_logs 144050332 7017092 129692800 6% /var/log/httpd
/dev/mapper/vg_root-home 17565696 32968 17532728 1% /home
However, I’m not able to re-mount the nfs filesystem because the connectivity to NFS server appears to be still broken. I used the following command and it hung:
mount -t nfs -o rw 10.139.4.2:/export/pantry /mnt/nfscn/
So my perspective everything points to a problem in NFS server (CN1). I took a look into the threads status (previously attached) and they are all busy, which based on a RedHat article, they should not be blocked in uninterruptible sleep for long periods of time, making no forward progress.
Could you help us understand why this issue is occurring and why the threads become scrambled thus so affecting the service and creating interruption?
Thank you,
Regs,
Joao