NFS client (OA) cannot communicate to NFS server (CN

Issues related to applications and software problems
Post Reply
jlrpires
Posts: 2
Joined: 2019/10/04 14:09:27

NFS client (OA) cannot communicate to NFS server (CN

Post by jlrpires » 2019/10/04 14:52:57

Dear CentOS support,

We would like to report an urgent issue with our linked environment.

We ve a cent OS physical content node with the following versions:

cn1.vg1ath1c1.cdn ~# hostnamectl
Static hostname: cn1.vg1ath1c1.cdn
Icon name: computer-server
Chassis: server
Machine ID: 0854094725eb46dcabbe12de0106e9a2
Boot ID: c1bb5d42fd5f40f9839ec5d1654f50a8
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-693.el7.x86_64
Architecture: x86-64

And inbuilt on it we ve a virtual machine with following versions:

oa1.vg1ath1c1.cdn ~# hostnamectl
Static hostname: oa1.vg1ath1c1.cdn
Icon name: computer-vm
Chassis: vm
Machine ID: f390795d9392450ab54f86ce8633c607
Boot ID: ce43d9694584465292bdb51cf4b38efe
Virtualization: kvm
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-693.el7.x86_64
Architecture: x86-64


We re finding issues on the connectivity between the physical content node and the inbuilt origin, described below:

All NFS threads are blocked as per the attached CN1 DMESG output:

Code: Select all

[Wed Sep 25 17:50:44 2019] INFO: task nfsd:4900 blocked for more than 120 seconds.
[Wed Sep 25 17:50:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:50:44 2019] nfsd            D ffff8841e068fa90     0  4900      2 0x00000080
[Wed Sep 25 17:50:44 2019]  ffff8841e068f930 0000000000000046 ffff881feb70af70 ffff8841e068ffd8
[Wed Sep 25 17:50:44 2019]  ffff8841e068ffd8 ffff8841e068ffd8 ffff881feb70af70 ffff881fffa96cc0
[Wed Sep 25 17:50:44 2019]  0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff8841e068fa90
[Wed Sep 25 17:50:44 2019] Call Trace:
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:50:44 2019]  [<ffffffff815d0293>] ? ip_output+0x73/0xe0
[Wed Sep 25 17:50:44 2019]  [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:50:44 2019]  [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:50:44 2019]  [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:50:44 2019]  [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:50:44 2019]  [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] INFO: task nfsd:4903 blocked for more than 120 seconds.
[Wed Sep 25 17:50:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:50:44 2019] nfsd            D ffff8841d8ba3a90     0  4903      2 0x00000080
[Wed Sep 25 17:50:44 2019]  ffff8841d8ba3930 0000000000000046 ffff881feb708000 ffff8841d8ba3fd8
[Wed Sep 25 17:50:44 2019]  ffff8841d8ba3fd8 ffff8841d8ba3fd8 ffff881feb708000 ffff881fffd56cc0
[Wed Sep 25 17:50:44 2019]  0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff8841d8ba3a90
[Wed Sep 25 17:50:44 2019] Call Trace:
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:50:44 2019]  [<ffffffff811de581>] ? __slab_free+0x81/0x2f0
[Wed Sep 25 17:50:44 2019]  [<ffffffff815d0293>] ? ip_output+0x73/0xe0
[Wed Sep 25 17:50:44 2019]  [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:50:44 2019]  [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:50:44 2019]  [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:50:44 2019]  [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:50:44 2019]  [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] INFO: task nfsd:4904 blocked for more than 120 seconds.
[Wed Sep 25 17:50:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:50:44 2019] nfsd            D ffffffff816a75f0     0  4904      2 0x00000080
[Wed Sep 25 17:50:44 2019]  ffff8841d9bab930 0000000000000046 ffff881feb70bf40 ffff8841d9babfd8
[Wed Sep 25 17:50:44 2019]  ffff8841d9babfd8 ffff8841d9babfd8 ffff881feb70bf40 ffff881fffa56cc0
[Wed Sep 25 17:50:44 2019]  0000000000000000 7fffffffffffffff ffff88207ff8b0e8 ffffffff816a75f0
[Wed Sep 25 17:50:44 2019] Call Trace:
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0010847>] ? dm_merge_bvec+0x87/0xd0 [dm_mod]
[Wed Sep 25 17:50:44 2019]  [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a7125>] __wait_on_bit+0x65/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffff81181cb1>] wait_on_page_bit+0x81/0xa0
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff8118dd59>] write_cache_pages+0x319/0x4d0
[Wed Sep 25 17:50:44 2019]  [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:50:44 2019]  [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:50:44 2019]  [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] INFO: task nfsd:4905 blocked for more than 120 seconds.
[Wed Sep 25 17:50:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:50:44 2019] nfsd            D ffffffff816a75f0     0  4905      2 0x00000080
[Wed Sep 25 17:50:44 2019]  ffff8841eb743930 0000000000000046 ffff881feb70eeb0 ffff8841eb743fd8
[Wed Sep 25 17:50:44 2019]  ffff8841eb743fd8 ffff8841eb743fd8 ffff881feb70eeb0 ffff881fffc96cc0
[Wed Sep 25 17:50:44 2019]  0000000000000000 7fffffffffffffff ffff88207ff83be8 ffffffff816a75f0
[Wed Sep 25 17:50:44 2019] Call Trace:
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:50:44 2019]  [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a7125>] __wait_on_bit+0x65/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffff81181cb1>] wait_on_page_bit+0x81/0xa0
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff8118dd59>] write_cache_pages+0x319/0x4d0
[Wed Sep 25 17:50:44 2019]  [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:50:44 2019]  [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:50:44 2019]  [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] INFO: task nfsd:4907 blocked for more than 120 seconds.
[Wed Sep 25 17:50:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:50:44 2019] nfsd            D ffff8841f326ba90     0  4907      2 0x00000080
[Wed Sep 25 17:50:44 2019]  ffff8841f326b930 0000000000000046 ffff881feb70cf10 ffff8841f326bfd8
[Wed Sep 25 17:50:44 2019]  ffff8841f326bfd8 ffff8841f326bfd8 ffff881feb70cf10 ffff881fffb96cc0
[Wed Sep 25 17:50:44 2019]  0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff8841f326ba90
[Wed Sep 25 17:50:44 2019] Call Trace:
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:50:44 2019]  [<ffffffff815d0293>] ? ip_output+0x73/0xe0
[Wed Sep 25 17:50:44 2019]  [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:50:44 2019]  [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:50:44 2019]  [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:50:44 2019]  [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:50:44 2019]  [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019] INFO: task kworker/u57:2:602765 blocked for more than 120 seconds.
[Wed Sep 25 17:50:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:50:44 2019] kworker/u57:2   D ffff881cdefaf9d8     0 602765      2 0x00000080
[Wed Sep 25 17:50:44 2019] Workqueue: writeback bdi_writeback_workfn (flush-253:1)
[Wed Sep 25 17:50:44 2019]  ffff881cdefaf870 0000000000000046 ffff881ffd1f3f40 ffff881cdefaffd8
[Wed Sep 25 17:50:44 2019]  ffff881cdefaffd8 ffff881cdefaffd8 ffff881ffd1f3f40 ffff881fffa16cc0
[Wed Sep 25 17:50:44 2019]  0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff881cdefaf9d8
[Wed Sep 25 17:50:44 2019] Call Trace:
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:50:44 2019]  [<ffffffffc0010847>] ? dm_merge_bvec+0x87/0xd0 [dm_mod]
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:50:44 2019]  [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:50:44 2019]  [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:50:44 2019]  [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:50:44 2019]  [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff8122d7d0>] __writeback_single_inode+0x40/0x220
[Wed Sep 25 17:50:44 2019]  [<ffffffff8122e414>] writeback_sb_inodes+0x1c4/0x490
[Wed Sep 25 17:50:44 2019]  [<ffffffff8122e77f>] __writeback_inodes_wb+0x9f/0xd0
[Wed Sep 25 17:50:44 2019]  [<ffffffff8122efb3>] wb_writeback+0x263/0x2f0
[Wed Sep 25 17:50:44 2019]  [<ffffffff8122f45b>] bdi_writeback_workfn+0x2cb/0x460
[Wed Sep 25 17:50:44 2019]  [<ffffffff810a881a>] process_one_work+0x17a/0x440
[Wed Sep 25 17:50:44 2019]  [<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[Wed Sep 25 17:50:44 2019]  [<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:50:44 2019]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:50:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019] INFO: task nfsd:4898 blocked for more than 120 seconds.
[Wed Sep 25 17:52:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:52:44 2019] nfsd            D ffff8841e6fa3a90     0  4898      2 0x00000080
[Wed Sep 25 17:52:44 2019]  ffff8841e6fa3930 0000000000000046 ffff881fc6f5af70 ffff8841e6fa3fd8
[Wed Sep 25 17:52:44 2019]  ffff8841e6fa3fd8 ffff8841e6fa3fd8 ffff881fc6f5af70 ffff881fffa16cc0
[Wed Sep 25 17:52:44 2019]  0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff8841e6fa3a90
[Wed Sep 25 17:52:44 2019] Call Trace:
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:52:44 2019]  [<ffffffff810cd794>] ? update_curr+0x104/0x190
[Wed Sep 25 17:52:44 2019]  [<ffffffff810ca29e>] ? account_entity_dequeue+0xae/0xd0
[Wed Sep 25 17:52:44 2019]  [<ffffffff810cdc7c>] ? dequeue_entity+0x11c/0x5d0
[Wed Sep 25 17:52:44 2019]  [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:52:44 2019]  [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:52:44 2019]  [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:52:44 2019]  [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:52:44 2019]  [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:52:44 2019]  [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:52:44 2019]  [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:52:44 2019]  [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:52:44 2019]  [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:52:44 2019]  [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019] INFO: task nfsd:4899 blocked for more than 120 seconds.
[Wed Sep 25 17:52:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:52:44 2019] nfsd            D ffff8841def7ba90     0  4899      2 0x00000080
[Wed Sep 25 17:52:44 2019]  ffff8841def7b930 0000000000000046 ffff881fc6f5cf10 ffff8841def7bfd8
[Wed Sep 25 17:52:44 2019]  ffff8841def7bfd8 ffff8841def7bfd8 ffff881fc6f5cf10 ffff881fffa16cc0
[Wed Sep 25 17:52:44 2019]  0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff8841def7ba90
[Wed Sep 25 17:52:44 2019] Call Trace:
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:52:44 2019]  [<ffffffff810ce54e>] ? dequeue_task_fair+0x41e/0x660
[Wed Sep 25 17:52:44 2019]  [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:52:44 2019]  [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:52:44 2019]  [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:52:44 2019]  [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:52:44 2019]  [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:52:44 2019]  [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:52:44 2019]  [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:52:44 2019]  [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:52:44 2019]  [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:52:44 2019]  [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019] INFO: task nfsd:4900 blocked for more than 120 seconds.
[Wed Sep 25 17:52:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:52:44 2019] nfsd            D ffff8841e068fa90     0  4900      2 0x00000080
[Wed Sep 25 17:52:44 2019]  ffff8841e068f930 0000000000000046 ffff881feb70af70 ffff8841e068ffd8
[Wed Sep 25 17:52:44 2019]  ffff8841e068ffd8 ffff8841e068ffd8 ffff881feb70af70 ffff881fffa96cc0
[Wed Sep 25 17:52:44 2019]  0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff8841e068fa90
[Wed Sep 25 17:52:44 2019] Call Trace:
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:52:44 2019]  [<ffffffff815d0293>] ? ip_output+0x73/0xe0
[Wed Sep 25 17:52:44 2019]  [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:52:44 2019]  [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:52:44 2019]  [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:52:44 2019]  [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:52:44 2019]  [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:52:44 2019]  [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:52:44 2019]  [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:52:44 2019]  [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:52:44 2019]  [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:52:44 2019]  [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019] INFO: task nfsd:4901 blocked for more than 120 seconds.
[Wed Sep 25 17:52:44 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Sep 25 17:52:44 2019] nfsd            D ffff8841e06aba90     0  4901      2 0x00000080
[Wed Sep 25 17:52:44 2019]  ffff8841e06ab930 0000000000000046 ffff881feb709fa0 ffff8841e06abfd8
[Wed Sep 25 17:52:44 2019]  ffff8841e06abfd8 ffff8841e06abfd8 ffff881feb709fa0 ffff881fffa16cc0
[Wed Sep 25 17:52:44 2019]  0000000000000000 7fffffffffffffff ffffffff816a75f0 ffff8841e06aba90
[Wed Sep 25 17:52:44 2019] Call Trace:
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a94c9>] schedule+0x29/0x70
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[Wed Sep 25 17:52:44 2019]  [<ffffffff81231b32>] ? __generic_file_splice_read+0x542/0x5e0
[Wed Sep 25 17:52:44 2019]  [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a75f0>] ? bit_wait+0x50/0x50
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a8be8>] io_schedule+0x18/0x20
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a7601>] bit_wait_io+0x11/0x50
[Wed Sep 25 17:52:44 2019]  [<ffffffff816a731f>] __wait_on_bit_lock+0x5f/0xc0
[Wed Sep 25 17:52:44 2019]  [<ffffffff81182094>] __lock_page+0x74/0x90
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b19d0>] ? wake_bit_function+0x40/0x40
[Wed Sep 25 17:52:44 2019]  [<ffffffff8118de60>] write_cache_pages+0x420/0x4d0
[Wed Sep 25 17:52:44 2019]  [<ffffffffc028f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc028f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[Wed Sep 25 17:52:44 2019]  [<ffffffff8118effe>] do_writepages+0x1e/0x40
[Wed Sep 25 17:52:44 2019]  [<ffffffff81183c55>] __filemap_fdatawrite_range+0x65/0x80
[Wed Sep 25 17:52:44 2019]  [<ffffffff81183da1>] filemap_write_and_wait_range+0x41/0x90
[Wed Sep 25 17:52:44 2019]  [<ffffffffc029ab16>] xfs_file_fsync+0x66/0x1e0 [xfs]
[Wed Sep 25 17:52:44 2019]  [<ffffffff8123327d>] vfs_fsync_range+0x1d/0x30
[Wed Sep 25 17:52:44 2019]  [<ffffffffc042a1f9>] nfsd_commit+0xb9/0xe0 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc04338f6>] nfsd3_proc_commit+0x96/0xf0 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc0424593>] nfsd_dispatch+0xd3/0x280 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc034f453>] svc_process_common+0x453/0x6f0 [sunrpc]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc034f7f3>] svc_process+0x103/0x190 [sunrpc]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc0423eff>] nfsd+0xdf/0x150 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffffc0423e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[Wed Sep 25 17:52:44 2019]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[Wed Sep 25 17:52:44 2019]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
For restoring the connectivity we ve had to shutdown the VM initially and then reboot the physical content node via iLO. After that the NFS threads became normal:

Code: Select all

cn1.vg1ath1c1.cdn ~# for p in $(ps -efl | grep \\[nfsd\\] | awk '{ print $4 }'); do echo "stack for pid $p"; cat /proc/$p/stack; done
stack for pid 4898
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
stack for pid 4899
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
stack for pid 4900
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
stack for pid 4901
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
stack for pid 4903
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
stack for pid 4904
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
stack for pid 4905
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
stack for pid 4907
[<ffffffffc035f8c1>] svc_recv+0x971/0xb60 [sunrpc]
[<ffffffffc0423eed>] nfsd+0xcd/0x150 [nfsd]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff
This has already happened several times up to now. After awhile (usually 3.4 weeks) the threads scramble again not allowing us to have connectivity between the two machines.

This connectivity is important as content is being ingested from the content node into the origin.

We ve collected some additional information (while the issue was occurring) below:

Code: Select all

CN1:

cn1.vg1ath1c1.cdn ~# systemctl status nfs.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
  Drop-In: /run/systemd/generator/nfs-server.service.d
           └─order-with-mounts.conf
   Active: active (exited) since Thu 2019-08-29 19:42:51 UTC; 14h ago
  Process: 2752302 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS)
  Process: 2752298 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS)
  Process: 2752296 ExecStop=/usr/sbin/rpc.nfsd 0 (code=exited, status=0/SUCCESS)
  Process: 2758399 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
  Process: 2758394 ExecStartPre=/bin/sh -c /bin/kill -HUP `cat /run/gssproxy.pid` (code=exited, status=0/SUCCESS)
  Process: 2758392 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
 Main PID: 2758399 (code=exited, status=0/SUCCESS)
    Tasks: 0
   CGroup: /system.slice/nfs-server.service

Aug 29 19:42:50 cn1.vg1ath1c1.cdn systemd[1]: Starting NFS server and services...
Aug 29 19:42:51 cn1.vg1ath1c1.cdn systemd[1]: Started NFS server and services.
cn1.vg1ath1c1.cdn ~# ps -ef | grep nfs
root        5023       2  0 Jul23 ?        00:00:00 [nfsd4_callbacks]
root        5033       2  0 Jul23 ?        00:03:19 [nfsd]
root        5034       2  0 Jul23 ?        00:03:53 [nfsd]
root        5035       2  0 Jul23 ?        00:04:33 [nfsd]
root        5036       2  0 Jul23 ?        00:05:27 [nfsd]
root        5037       2  0 Jul23 ?        00:07:09 [nfsd]
root        5038       2  0 Jul23 ?        00:09:58 [nfsd]
root        5039       2  0 Jul23 ?        00:23:14 [nfsd]
root        5040       2  0 Jul23 ?        01:08:31 [nfsd]
root     3142534 3141879  0 10:33 pts/1    00:00:00 grep --color=auto nfs
OA1:

oa1.vg1ath1c1.cdn ~# systemctl status nfs-client.target
● nfs-client.target - NFS client services
   Loaded: loaded (/usr/lib/systemd/system/nfs-client.target; enabled; vendor preset: disabled)
   Active: active since Thu 2019-08-29 18:40:47 UTC; 15h ago

Aug 29 18:40:47 oa1.vg1ath1c1.cdn systemd[1]: Reached target NFS client services.
Aug 29 18:40:47 oa1.vg1ath1c1.cdn systemd[1]: Starting NFS client services.
oa1.vg1ath1c1.cdn ~# ps -ef | grep nfs
root      3654     2  0 Jul23 ?        00:00:00 [nfsiod]
root     19072 18748  0 10:34 pts/7    00:00:00 grep --color=auto nfs
root     30578     1  0 Aug29 ?        00:00:00 ls --color=auto /mnt/nfscn

Code: Select all

CN1:

cn1.vg1ath1c1.cdn ~# cat /etc/exports
/export/pantry 10.139.4.8(rw,wdelay,no_root_squash,no_subtree_check,sec=sys,secure,no_all_squash,mountpoint) 10.139.4.9(rw,wdelay,no_root_squash,no_subtree_check,sec=sys,secure,no_all_squash,mountpoint) 10.139.4.10(rw,wdelay,no_root_squash,no_subtree_check,sec=sys,secure,no_all_squash,mountpoint) 10.139.4.11(rw,wdelay,no_root_squash,no_subtree_check,sec=sys,secure,no_all_squash,mountpoint) 10.139.4.7(rw,wdelay,no_root_squash,no_subtree_check,sec=sys,secure,no_all_squash,mountpoint)
cn1.vg1ath1c1.cdn ~# cat /proc/net/rpc/nfsd
rc 0 4648421 19728575
fh 7 0 0 0 0
io 1639158320 2077816748
th 8 0 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
ra 32 15076466 0 0 0 0 0 0 0 0 0 33990
net 24378984 0 24379062 23
rpc 24378267 0 0 0 0
proc3 22 14 1887176 6302 14000 2588495 0 15110219 4627770 6153 1027 0 0 6157 1027 0 0 0 1090 96230 14 7 31485
proc4 2 0 0
proc4ops 72 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
OA1:

oa1.vg1ath1c1.cdn ~# showmount -e
clnt_create: RPC: Program not registered
oa1.vg1ath1c1.cdn ~# systemctl status mnt-nfscn.mount
● mnt-nfscn.mount - mnt-nfscn
   Loaded: loaded (/etc/systemd/system/mnt-nfscn.mount; enabled; vendor preset: disabled)
   Active: active (mounted) since Tue 2019-07-23 23:06:29 UTC; 1 months 7 days ago
    Where: /mnt/nfscn
     What: 10.139.4.2:/export/pantry
   Memory: 12.0K

Jul 23 23:06:21 oa1.vg1ath1c1.cdn systemd[1]: Mounting mnt-nfscn...
Jul 23 23:06:29 oa1.vg1ath1c1.cdn systemd[1]: Mounted mnt-nfscn.

Code: Select all

I’ve tried to stop mnt-nfscn.mount service in VxOA but it hangs.

oa1.vg1ath1c1.cdn ~# systemctl stop mnt-nfscn.mount

Opening a new session and checking the service status:

oa1.vg1ath1c1.cdn ~# systemctl status mnt-nfscn.mount
● mnt-nfscn.mount - mnt-nfscn
   Loaded: loaded (/etc/systemd/system/mnt-nfscn.mount; enabled; vendor preset: disabled)
   Active: deactivating (unmounting) (Result: timeout) since Fri 2019-08-30 10:41:09 UTC; 2min 45s ago
    Where: /mnt/nfscn
     What: 10.139.4.2:/export/pantry
  Control: 28506 (umount)
   Memory: 228.0K
   CGroup: /system.slice/mnt-nfscn.mount
           └─28506 /bin/umount /mnt/nfscn

Jul 23 23:06:21 oa1.vg1ath1c1.cdn systemd[1]: Mounting mnt-nfscn...
Jul 23 23:06:29 oa1.vg1ath1c1.cdn systemd[1]: Mounted mnt-nfscn.
Aug 30 10:41:09 oa1.vg1ath1c1.cdn systemd[1]: Unmounting mnt-nfscn...
Aug 30 10:42:40 oa1.vg1ath1c1.cdn systemd[1]: mnt-nfscn.mount unmounting timed out. Stopping.

Code: Select all

I was able to umount the /mnt/nfscn partition by using -l (lazy) option. So now I can list the partitions in OA1:

oa1.vg1ath1c1.cdn ~# df -hk
Filesystem                               1K-blocks    Used Available Use% Mounted on
/dev/mapper/vg_root-root                  39040384 3491376  35549008   9% /
devtmpfs                                  29796120       0  29796120   0% /dev
tmpfs                                     29807940    1416  29806524   1% /dev/shm
tmpfs                                     29807940 1499572  28308368   6% /run
tmpfs                                     29807940       0  29807940   0% /sys/fs/cgroup
vxoa_tmpfs                                  145408       0    145408   0% /mnt/ramdisk
/dev/vda2                                   943128  159808    734580  18% /boot
/dev/mapper/vg_root-eve_internal_storage   8775680   32928   8742752   1% /cache1/storage
/dev/mapper/vg_root-var_log               79943540  919504  79024036   2% /var/log
/dev/mapper/vg_root-eve_logs             144050332 7017092 129692800   6% /var/log/httpd
/dev/mapper/vg_root-home                  17565696   32968  17532728   1% /home
However, I’m not able to re-mount the nfs filesystem because the connectivity to NFS server appears to be still broken. I used the following command and it hung:

mount -t nfs -o rw 10.139.4.2:/export/pantry /mnt/nfscn/
So my perspective everything points to a problem in NFS server (CN1). I took a look into the threads status (previously attached) and they are all busy, which based on a RedHat article, they should not be blocked in uninterruptible sleep for long periods of time, making no forward progress.
I tried manually killing the nfs threads but no success, either by stopping the nfs server service or using kill -9 <pid> command. I did some research online but didn’t find any solution to kill NFS threads without rebooting the machine.

Could you help us understand why this issue is occurring and why the threads become scrambled thus so affecting the service and creating interruption?

Thank you,
Regs,
Joao

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: NFS client (OA) cannot communicate to NFS server (CN

Post by TrevorH » 2019/10/04 18:32:09

Start by updating to 7.7 which is the current and only supported CentOS 7 version.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

jlrpires
Posts: 2
Joined: 2019/10/04 14:09:27

Re: NFS client (OA) cannot communicate to NFS server (CN

Post by jlrpires » 2019/10/04 19:27:51

Hi TrevorH,

Will that effectively solve this issue?

Please clarify.

Thank you,
Regs,
Joao

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: NFS client (OA) cannot communicate to NFS server (CN

Post by TrevorH » 2019/10/04 22:41:19

No idea but old versions are not supported so you have a choice, stick with what you have now or update and hope that it's been fixed and if not then you get to raise a ticket on bugzilla.redhat.com
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply