CentOS 7 : xfs problem

[itsbig]

Please help me , what is root cause and how to fix this problem.
Kernel Version : Linux 3.10.0-1160.88.1.el7.x86_64

Sep 26 20:30:40 kernel: [<ffffffffc060d320>] ? __xfs_buf_submit+0x130/0x250 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060b9f7>] xfs_buf_iowait+0x27/0xc0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d769>] ? xfs_bwrite+0x29/0x60 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d320>] __xfs_buf_submit+0x130/0x250 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d769>] xfs_bwrite+0x29/0x60 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0616011>] xfs_reclaim_inode+0x341/0x370 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc06162a7>] xfs_reclaim_inodes_ag+0x267/0x3a0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0617393>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0627d55>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc062dc67>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] ? _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060c83c>] xfs_buf_lock+0x3c/0xe0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060ccb5>] xfs_buf_get_map+0x35/0x260 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d4a0>] xfs_buf_read_map+0x30/0x160 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc063f2a9>] xfs_trans_read_buf_map+0xf9/0x2d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05cbc53>] xfs_read_agf+0x93/0x120 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05cbd2b>] xfs_alloc_read_agf+0x4b/0x1b0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ce68b>] xfs_alloc_fix_freelist+0x4ab/0x500 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc06024aa>] ? xfs_perag_get+0x2a/0xc0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ce9fc>] xfs_alloc_vextent+0x10c/0x520 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05dcc33>] xfs_bmap_btalloc+0x203/0x7e0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05dd21e>] xfs_bmap_alloc+0xe/0x20 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05debf7>] xfs_bmapi_write+0x4c7/0xb70 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc062dc67>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061c6f2>] xfs_iomap_write_allocate+0x182/0x390 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc06057b6>] xfs_map_blocks+0x1a6/0x220 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0606834>] xfs_do_writepage+0x174/0x550 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0606c4b>] xfs_vm_writepage+0x3b/0x70 [xfs]
Sep 26 20:30:40 kernel: INFO: task xfs-data/dm-6:1909 blocked for more than 120 seconds.
Sep 26 20:30:40 kernel: xfs-data/dm-6 D ffff9f94a2623180 0 1909 2 0x00000000
Sep 26 20:30:40 kernel: Workqueue: xfs-data/dm-6 xfs_end_io [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060501d>] ? __xfs_setfilesize+0x2d/0x100 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061fb11>] xfs_ilock+0xc1/0x120 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060501d>] __xfs_setfilesize+0x2d/0x100 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0605136>] xfs_setfilesize_ioend+0x46/0x70 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0606f2b>] xfs_end_io+0x4b/0x90 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] ? _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060c83c>] xfs_buf_lock+0x3c/0xe0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060ccb5>] xfs_buf_get_map+0x35/0x260 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d4a0>] xfs_buf_read_map+0x30/0x160 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc063f2a9>] xfs_trans_read_buf_map+0xf9/0x2d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05e9fab>] xfs_da_read_buf+0xdb/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ea023>] xfs_da3_node_read+0x23/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc062dc67>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05eb322>] xfs_da3_node_lookup_int+0x72/0x350 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05f759d>] xfs_dir2_node_lookup+0x4d/0x170 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ee64d>] xfs_dir_lookup+0x1bd/0x1e0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0620749>] xfs_lookup+0x69/0x140 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061d458>] xfs_vn_lookup+0x78/0xc0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0611685>] ? xfs_file_buffered_aio_read+0xa5/0x110 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] ? _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060c83c>] xfs_buf_lock+0x3c/0xe0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060ccb5>] xfs_buf_get_map+0x35/0x260 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d4a0>] xfs_buf_read_map+0x30/0x160 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc063f2a9>] xfs_trans_read_buf_map+0xf9/0x2d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05e9fab>] xfs_da_read_buf+0xdb/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ea023>] xfs_da3_node_read+0x23/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc062dc67>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05eb322>] xfs_da3_node_lookup_int+0x72/0x350 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05f759d>] xfs_dir2_node_lookup+0x4d/0x170 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ee64d>] xfs_dir_lookup+0x1bd/0x1e0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0620749>] xfs_lookup+0x69/0x140 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061d458>] xfs_vn_lookup+0x78/0xc0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0611685>] ? xfs_file_buffered_aio_read+0xa5/0x110 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] ? _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060c83c>] xfs_buf_lock+0x3c/0xe0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060ccb5>] xfs_buf_get_map+0x35/0x260 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d4a0>] xfs_buf_read_map+0x30/0x160 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc063f2a9>] xfs_trans_read_buf_map+0xf9/0x2d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05e9fab>] xfs_da_read_buf+0xdb/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ea023>] xfs_da3_node_read+0x23/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc062dc67>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05eb322>] xfs_da3_node_lookup_int+0x72/0x350 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05f759d>] xfs_dir2_node_lookup+0x4d/0x170 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ee64d>] xfs_dir_lookup+0x1bd/0x1e0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0620749>] xfs_lookup+0x69/0x140 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061d458>] xfs_vn_lookup+0x78/0xc0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0611685>] ? xfs_file_buffered_aio_read+0xa5/0x110 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] ? _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060c83c>] xfs_buf_lock+0x3c/0xe0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060ccb5>] xfs_buf_get_map+0x35/0x260 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d4a0>] xfs_buf_read_map+0x30/0x160 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc063f2a9>] xfs_trans_read_buf_map+0xf9/0x2d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05fa9dd>] xfs_read_agi+0x9d/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05faaa4>] xfs_ialloc_read_agi+0x34/0xd0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05fca08>] xfs_dialloc+0xf8/0x290 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e881>] xfs_ialloc+0x81/0x590 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0620887>] xfs_dir_ialloc+0x67/0x1d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0620e52>] xfs_create+0x462/0x6b0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e433>] xfs_generic_create+0xd3/0x2a0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e634>] xfs_vn_mknod+0x14/0x20 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e673>] xfs_vn_create+0x13/0x20 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0611685>] ? xfs_file_buffered_aio_read+0xa5/0x110 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc062dc67>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] ? _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060c83c>] xfs_buf_lock+0x3c/0xe0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060ccb5>] xfs_buf_get_map+0x35/0x260 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d4a0>] xfs_buf_read_map+0x30/0x160 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc063f2a9>] xfs_trans_read_buf_map+0xf9/0x2d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05e21ac>] xfs_btree_read_buf_block.constprop.33+0xac/0xe0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05e4d9e>] xfs_btree_increment+0x21e/0x260 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05fbfd5>] xfs_dialloc_ag_inobt+0x4d5/0x680 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05fc6ee>] xfs_dialloc_ag+0x6e/0x290 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc06024aa>] ? xfs_perag_get+0x2a/0xc0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05fc96d>] xfs_dialloc+0x5d/0x290 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e881>] xfs_ialloc+0x81/0x590 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0620887>] xfs_dir_ialloc+0x67/0x1d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0620e52>] xfs_create+0x462/0x6b0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e433>] xfs_generic_create+0xd3/0x2a0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e634>] xfs_vn_mknod+0x14/0x20 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e673>] xfs_vn_create+0x13/0x20 [xfs]

[TrevorH]

That does not look like it is a complete list of everything that was logged. Is there more?

[tunk]

Probably won't make any difference, but your kernel is 6 months old.

[itsbig]

That does not look like it is a complete list of everything that was logged. Is there more?

Hi TrevorH

Here is full log

Sep 26 20:30:40 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 26 20:30:40 kernel: kthreadd D ffff9f7573949660 0 2 0 0x00000000
Sep 26 20:30:40 kernel: Call Trace:
Sep 26 20:30:40 kernel: [<ffffffff929b7ca9>] schedule+0x29/0x70
Sep 26 20:30:40 kernel: [<ffffffff929b5831>] schedule_timeout+0x221/0x2d0
Sep 26 20:30:40 kernel: [<ffffffff929b805d>] wait_for_completion+0xfd/0x140
Sep 26 20:30:40 kernel: [<ffffffff922e1170>] ? wake_up_state+0x20/0x20
Sep 26 20:30:40 kernel: [<ffffffffc060d320>] ? __xfs_buf_submit+0x130/0x250 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060b9f7>] xfs_buf_iowait+0x27/0xc0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d769>] ? xfs_bwrite+0x29/0x60 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d320>] __xfs_buf_submit+0x130/0x250 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d769>] xfs_bwrite+0x29/0x60 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0616011>] xfs_reclaim_inode+0x341/0x370 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc06162a7>] xfs_reclaim_inodes_ag+0x267/0x3a0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff922e10b5>] ? wake_up_process+0x15/0x20
Sep 26 20:30:40 kernel: [<ffffffffc0617393>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0627d55>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff9245e879>] prune_super+0xf9/0x1a0
Sep 26 20:30:40 kernel: [<ffffffff923dd6d5>] shrink_slab+0x175/0x360
Sep 26 20:30:40 kernel: [<ffffffff924505f1>] ? vmpressure+0x21/0x90
Sep 26 20:30:40 kernel: [<ffffffff923e09ea>] do_try_to_free_pages+0x3ca/0x520
Sep 26 20:30:40 kernel: [<ffffffff923e0c3c>] try_to_free_pages+0xfc/0x180
Sep 26 20:30:40 kernel: [<ffffffff923d45a1>] __alloc_pages_nodemask+0x841/0xbf0
Sep 26 20:30:40 kernel: [<ffffffff92434266>] new_slab+0xa6/0x4e0
Sep 26 20:30:40 kernel: [<ffffffff92434a6c>] ___slab_alloc+0x3cc/0x520
Sep 26 20:30:40 kernel: [<ffffffff9229ccf6>] ? copy_process+0x196/0x1a80
Sep 26 20:30:40 kernel: [<ffffffff922e63e6>] ? sched_slice.isra.56+0x66/0xd0
Sep 26 20:30:40 kernel: [<ffffffff9229ccf6>] ? copy_process+0x196/0x1a80
Sep 26 20:30:40 kernel: [<ffffffff929ae458>] __slab_alloc+0x40/0x60
Sep 26 20:30:40 kernel: [<ffffffff92434c4b>] kmem_cache_alloc_node+0x8b/0x210
Sep 26 20:30:40 kernel: [<ffffffff922cb550>] ? insert_kthread_work+0x40/0x40
Sep 26 20:30:40 kernel: [<ffffffff922cb550>] ? insert_kthread_work+0x40/0x40
Sep 26 20:30:40 kernel: [<ffffffff9229ccf6>] copy_process+0x196/0x1a80
Sep 26 20:30:40 kernel: [<ffffffff9229e791>] do_fork+0x91/0x340
Sep 26 20:30:40 kernel: [<ffffffff929b7920>] ? __schedule+0x320/0x680
Sep 26 20:30:40 kernel: [<ffffffff9229ea66>] kernel_thread+0x26/0x30
Sep 26 20:30:40 kernel: [<ffffffff922cc051>] kthreadd+0x2e1/0x320
Sep 26 20:30:40 kernel: [<ffffffff922cbd70>] ? kthread_create_on_cpu+0x70/0x70
Sep 26 20:30:40 kernel: [<ffffffff929c51dd>] ret_from_fork_nospec_begin+0x7/0x21
Sep 26 20:30:40 kernel: [<ffffffff922cbd70>] ? kthread_create_on_cpu+0x70/0x70
Sep 26 20:30:40 kernel: INFO: task kswapd0:510 blocked for more than 120 seconds.
Sep 26 20:30:40 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 26 20:30:40 kernel: kswapd0 D ffff9f94bd29d860 0 510 2 0x00000000
Sep 26 20:30:40 kernel: Call Trace:
Sep 26 20:30:40 kernel: [<ffffffff929b7ca9>] schedule+0x29/0x70
Sep 26 20:30:40 kernel: [<ffffffff929b5831>] schedule_timeout+0x221/0x2d0
Sep 26 20:30:40 kernel: [<ffffffffc062dc67>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff923d0e0a>] ? __rmqueue+0x8a/0x460
Sep 26 20:30:40 kernel: [<ffffffff925a39d4>] ? __radix_tree_lookup+0x84/0x100
Sep 26 20:30:40 kernel: [<ffffffff929b722b>] __down_common+0xaa/0x108
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] ? _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff929b72a6>] __down+0x1d/0x23
Sep 26 20:30:40 kernel: [<ffffffff922d17e5>] down+0x45/0x50
Sep 26 20:30:40 kernel: [<ffffffffc060c83c>] xfs_buf_lock+0x3c/0xe0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060ccb5>] xfs_buf_get_map+0x35/0x260 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d4a0>] xfs_buf_read_map+0x30/0x160 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc063f2a9>] xfs_trans_read_buf_map+0xf9/0x2d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05cbc53>] xfs_read_agf+0x93/0x120 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05cbd2b>] xfs_alloc_read_agf+0x4b/0x1b0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ce68b>] xfs_alloc_fix_freelist+0x4ab/0x500 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff925a39d4>] ? __radix_tree_lookup+0x84/0x100
Sep 26 20:30:40 kernel: [<ffffffff925a39d4>] ? __radix_tree_lookup+0x84/0x100
Sep 26 20:30:40 kernel: [<ffffffff925a3aad>] ? radix_tree_lookup+0xd/0x20
Sep 26 20:30:40 kernel: [<ffffffffc06024aa>] ? xfs_perag_get+0x2a/0xc0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ce9fc>] xfs_alloc_vextent+0x10c/0x520 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05dcc33>] xfs_bmap_btalloc+0x203/0x7e0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05dd21e>] xfs_bmap_alloc+0xe/0x20 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05debf7>] xfs_bmapi_write+0x4c7/0xb70 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc062dc67>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061c6f2>] xfs_iomap_write_allocate+0x182/0x390 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc06057b6>] xfs_map_blocks+0x1a6/0x220 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0606834>] xfs_do_writepage+0x174/0x550 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0606c4b>] xfs_vm_writepage+0x3b/0x70 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff923de942>] shrink_page_list+0x892/0xc40
Sep 26 20:30:40 kernel: [<ffffffff923df306>] shrink_inactive_list+0x1b6/0x5c0
Sep 26 20:30:40 kernel: [<ffffffff923dfde5>] shrink_lruvec+0x375/0x730
Sep 26 20:30:40 kernel: [<ffffffff923e0216>] shrink_zone+0x76/0x1a0
Sep 26 20:30:40 kernel: [<ffffffff923e1443>] balance_pgdat+0x383/0x5e0
Sep 26 20:30:40 kernel: [<ffffffff923e1813>] kswapd+0x173/0x430
Sep 26 20:30:40 kernel: [<ffffffff922cc790>] ? wake_up_atomic_t+0x40/0x40
Sep 26 20:30:40 kernel: [<ffffffff923e16a0>] ? balance_pgdat+0x5e0/0x5e0
Sep 26 20:30:40 kernel: [<ffffffff922cb621>] kthread+0xd1/0xe0
Sep 26 20:30:40 kernel: [<ffffffff922cb550>] ? insert_kthread_work+0x40/0x40
Sep 26 20:30:40 kernel: [<ffffffff929c51dd>] ret_from_fork_nospec_begin+0x7/0x21
Sep 26 20:30:40 kernel: [<ffffffff922cb550>] ? insert_kthread_work+0x40/0x40
Sep 26 20:30:40 kernel: INFO: task xfs-data/dm-6:1909 blocked for more than 120 seconds.
Sep 26 20:30:40 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 26 20:30:40 kernel: xfs-data/dm-6 D ffff9f94a2623180 0 1909 2 0x00000000
Sep 26 20:30:40 kernel: Workqueue: xfs-data/dm-6 xfs_end_io [xfs]
Sep 26 20:30:40 kernel: Call Trace:
Sep 26 20:30:40 kernel: [<ffffffff929b7ca9>] schedule+0x29/0x70
Sep 26 20:30:40 kernel: [<ffffffff929b9455>] rwsem_down_write_failed+0x215/0x3c0
Sep 26 20:30:40 kernel: [<ffffffffc060501d>] ? __xfs_setfilesize+0x2d/0x100 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff925ae417>] call_rwsem_down_write_failed+0x17/0x30
Sep 26 20:30:40 kernel: [<ffffffff929b716d>] down_write+0x2d/0x41
Sep 26 20:30:40 kernel: [<ffffffffc061fb11>] xfs_ilock+0xc1/0x120 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060501d>] __xfs_setfilesize+0x2d/0x100 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0605136>] xfs_setfilesize_ioend+0x46/0x70 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0606f2b>] xfs_end_io+0x4b/0x90 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff922c32ef>] process_one_work+0x17f/0x440
Sep 26 20:30:40 kernel: [<ffffffff922c3844>] rescuer_thread+0x294/0x3c0
Sep 26 20:30:40 kernel: [<ffffffff922c35b0>] ? process_one_work+0x440/0x440
Sep 26 20:30:40 kernel: [<ffffffff922cb621>] kthread+0xd1/0xe0
Sep 26 20:30:40 kernel: [<ffffffff922cb550>] ? insert_kthread_work+0x40/0x40
Sep 26 20:30:40 kernel: [<ffffffff929c51dd>] ret_from_fork_nospec_begin+0x7/0x21
Sep 26 20:30:40 kernel: [<ffffffff922cb550>] ? insert_kthread_work+0x40/0x40
Sep 26 20:30:40 kernel: INFO: task nginx:3437 blocked for more than 120 seconds.
Sep 26 20:30:40 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 26 20:30:40 kernel: nginx D ffff9f94b67326e0 0 3437 3435 0x00000080
Sep 26 20:30:40 kernel: Call Trace:
Sep 26 20:30:40 kernel: [<ffffffff929b8bf9>] schedule_preempt_disabled+0x29/0x70
Sep 26 20:30:40 kernel: [<ffffffff929b6ca7>] __mutex_lock_slowpath+0xc7/0x1e0
Sep 26 20:30:40 kernel: [<ffffffff929b602f>] mutex_lock+0x1f/0x33
Sep 26 20:30:40 kernel: [<ffffffff9246c8f8>] do_last+0x298/0x1380
Sep 26 20:30:40 kernel: [<ffffffff92435e96>] ? kmem_cache_alloc_trace+0x1d6/0x200
Sep 26 20:30:40 kernel: [<ffffffff9246daad>] path_openat+0xcd/0x5a0
Sep 26 20:30:40 kernel: [<ffffffffc048ed4e>] ? 0xffffffffc048ed4d
Sep 26 20:30:40 kernel: [<ffffffffc0e99242>] ? cshook_security_inode_free_security+0x7592/0x9ac0 [falcon_lsm_serviceable]
Sep 26 20:30:40 kernel: [<ffffffffc0efdf67>] ? cshook_systemcalltable_pre_compat_sys_ioctl+0x2a6e7/0x34460 [falcon_lsm_serviceable]
Sep 26 20:30:40 kernel: [<ffffffffc0efdf89>] ? cshook_systemcalltable_pre_compat_sys_ioctl+0x2a709/0x34460 [falcon_lsm_serviceable]
Sep 26 20:30:40 kernel: [<ffffffff9246f46d>] do_filp_open+0x4d/0xc0
Sep 26 20:30:40 kernel: [<ffffffff9247dcd7>] ? __alloc_fd+0x47/0x180
Sep 26 20:30:40 kernel: [<ffffffff9245ab34>] do_sys_open+0x124/0x220
Sep 26 20:30:40 kernel: [<ffffffff9245ac4e>] SyS_open+0x1e/0x30
Sep 26 20:30:40 kernel: [<ffffffffc0719cac>] unload_network_ops_symbols+0x93bc/0x99d0 [falcon_lsm_pinned_15508]
Sep 26 20:30:40 kernel: [<ffffffffc0783cac>] unload_network_ops_symbols+0x93bc/0x99d0 [falcon_lsm_pinned_15604]
Sep 26 20:30:40 kernel: [<ffffffff929c539a>] system_call_fastpath+0x25/0x2a
Sep 26 20:30:40 kernel: INFO: task nginx:3438 blocked for more than 120 seconds.
Sep 26 20:30:40 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 26 20:30:40 kernel: nginx D ffff9f94b6733760 0 3438 3435 0x00000080
Sep 26 20:30:40 kernel: Call Trace:
Sep 26 20:30:40 kernel: [<ffffffff929b7ca9>] schedule+0x29/0x70
Sep 26 20:30:40 kernel: [<ffffffff929b5831>] schedule_timeout+0x221/0x2d0
Sep 26 20:30:40 kernel: [<ffffffff925684ff>] ? generic_make_request_checks+0x28f/0x3a0
Sep 26 20:30:40 kernel: [<ffffffff925bd2fd>] ? list_del+0xd/0x30
Sep 26 20:30:40 kernel: [<ffffffff929b722b>] __down_common+0xaa/0x108
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] ? _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff929b72a6>] __down+0x1d/0x23
Sep 26 20:30:40 kernel: [<ffffffff922d17e5>] down+0x45/0x50
Sep 26 20:30:40 kernel: [<ffffffffc060c83c>] xfs_buf_lock+0x3c/0xe0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060ccb5>] xfs_buf_get_map+0x35/0x260 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d4a0>] xfs_buf_read_map+0x30/0x160 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc063f2a9>] xfs_trans_read_buf_map+0xf9/0x2d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05e9fab>] xfs_da_read_buf+0xdb/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ea023>] xfs_da3_node_read+0x23/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc062dc67>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05eb322>] xfs_da3_node_lookup_int+0x72/0x350 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05f759d>] xfs_dir2_node_lookup+0x4d/0x170 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ee64d>] xfs_dir_lookup+0x1bd/0x1e0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0620749>] xfs_lookup+0x69/0x140 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061d458>] xfs_vn_lookup+0x78/0xc0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff92466d23>] lookup_real+0x23/0x60
Sep 26 20:30:40 kernel: [<ffffffff9246d21a>] do_last+0xbba/0x1380
Sep 26 20:30:40 kernel: [<ffffffff92435e96>] ? kmem_cache_alloc_trace+0x1d6/0x200
Sep 26 20:30:40 kernel: [<ffffffff9246daad>] path_openat+0xcd/0x5a0
Sep 26 20:30:40 kernel: [<ffffffffc0611685>] ? xfs_file_buffered_aio_read+0xa5/0x110 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff922ec27b>] ? should_numa_migrate_memory+0x5b/0x170
Sep 26 20:30:40 kernel: [<ffffffff9246f46d>] do_filp_open+0x4d/0xc0
Sep 26 20:30:40 kernel: [<ffffffff9247dcd7>] ? __alloc_fd+0x47/0x180
Sep 26 20:30:40 kernel: [<ffffffff9245ab34>] do_sys_open+0x124/0x220
Sep 26 20:30:40 kernel: [<ffffffff9245ac4e>] SyS_open+0x1e/0x30
Sep 26 20:30:40 kernel: [<ffffffffc0719cac>] unload_network_ops_symbols+0x93bc/0x99d0 [falcon_lsm_pinned_15508]
Sep 26 20:30:40 kernel: [<ffffffffc0783cac>] unload_network_ops_symbols+0x93bc/0x99d0 [falcon_lsm_pinned_15604]
Sep 26 20:30:40 kernel: [<ffffffff929c539a>] system_call_fastpath+0x25/0x2a
Sep 26 20:30:40 kernel: INFO: task nginx:3443 blocked for more than 120 seconds.
Sep 26 20:30:40 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 26 20:30:40 kernel: nginx D ffff9f94b67368e0 0 3443 3435 0x00000080
Sep 26 20:30:40 kernel: Call Trace:
Sep 26 20:30:40 kernel: [<ffffffff929b7ca9>] schedule+0x29/0x70
Sep 26 20:30:40 kernel: [<ffffffff929b5831>] schedule_timeout+0x221/0x2d0
Sep 26 20:30:40 kernel: [<ffffffff9287edc6>] ? dev_hard_start_xmit+0x246/0x3b0
Sep 26 20:30:40 kernel: [<ffffffff929b722b>] __down_common+0xaa/0x108
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] ? _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff929b72a6>] __down+0x1d/0x23
Sep 26 20:30:40 kernel: [<ffffffff922d17e5>] down+0x45/0x50
Sep 26 20:30:40 kernel: [<ffffffffc060c83c>] xfs_buf_lock+0x3c/0xe0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060ccb5>] xfs_buf_get_map+0x35/0x260 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d4a0>] xfs_buf_read_map+0x30/0x160 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc063f2a9>] xfs_trans_read_buf_map+0xf9/0x2d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05e9fab>] xfs_da_read_buf+0xdb/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ea023>] xfs_da3_node_read+0x23/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc062dc67>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05eb322>] xfs_da3_node_lookup_int+0x72/0x350 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05f759d>] xfs_dir2_node_lookup+0x4d/0x170 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ee64d>] xfs_dir_lookup+0x1bd/0x1e0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0620749>] xfs_lookup+0x69/0x140 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061d458>] xfs_vn_lookup+0x78/0xc0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff92466d23>] lookup_real+0x23/0x60
Sep 26 20:30:40 kernel: [<ffffffff9246d21a>] do_last+0xbba/0x1380
Sep 26 20:30:40 kernel: [<ffffffff92435e96>] ? kmem_cache_alloc_trace+0x1d6/0x200
Sep 26 20:30:40 kernel: [<ffffffff9246daad>] path_openat+0xcd/0x5a0
Sep 26 20:30:40 kernel: [<ffffffffc0611685>] ? xfs_file_buffered_aio_read+0xa5/0x110 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff929b7152>] ? down_write+0x12/0x41
Sep 26 20:30:40 kernel: [<ffffffff9246f46d>] do_filp_open+0x4d/0xc0
Sep 26 20:30:40 kernel: [<ffffffff9247dcd7>] ? __alloc_fd+0x47/0x180
Sep 26 20:30:40 kernel: [<ffffffff9245ab34>] do_sys_open+0x124/0x220
Sep 26 20:30:40 kernel: [<ffffffff9245ac4e>] SyS_open+0x1e/0x30
Sep 26 20:30:40 kernel: [<ffffffffc0719cac>] unload_network_ops_symbols+0x93bc/0x99d0 [falcon_lsm_pinned_15508]
Sep 26 20:30:40 kernel: [<ffffffffc0783cac>] unload_network_ops_symbols+0x93bc/0x99d0 [falcon_lsm_pinned_15604]
Sep 26 20:30:40 kernel: [<ffffffff929c539a>] system_call_fastpath+0x25/0x2a
Sep 26 20:30:40 kernel: INFO: task nginx:3444 blocked for more than 120 seconds.
Sep 26 20:30:40 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 26 20:30:40 kernel: nginx D ffff9f94b6731660 0 3444 3435 0x00000080
Sep 26 20:30:40 kernel: Call Trace:
Sep 26 20:30:40 kernel: [<ffffffff929b7ca9>] schedule+0x29/0x70
Sep 26 20:30:40 kernel: [<ffffffff929b5831>] schedule_timeout+0x221/0x2d0
Sep 26 20:30:40 kernel: [<ffffffff9287edc6>] ? dev_hard_start_xmit+0x246/0x3b0
Sep 26 20:30:40 kernel: [<ffffffff929b722b>] __down_common+0xaa/0x108
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] ? _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff929b72a6>] __down+0x1d/0x23
Sep 26 20:30:40 kernel: [<ffffffff922d17e5>] down+0x45/0x50
Sep 26 20:30:40 kernel: [<ffffffffc060c83c>] xfs_buf_lock+0x3c/0xe0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060ccb5>] xfs_buf_get_map+0x35/0x260 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d4a0>] xfs_buf_read_map+0x30/0x160 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc063f2a9>] xfs_trans_read_buf_map+0xf9/0x2d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05e9fab>] xfs_da_read_buf+0xdb/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ea023>] xfs_da3_node_read+0x23/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc062dc67>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05eb322>] xfs_da3_node_lookup_int+0x72/0x350 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05f759d>] xfs_dir2_node_lookup+0x4d/0x170 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05ee64d>] xfs_dir_lookup+0x1bd/0x1e0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0620749>] xfs_lookup+0x69/0x140 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061d458>] xfs_vn_lookup+0x78/0xc0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff92466d23>] lookup_real+0x23/0x60
Sep 26 20:30:40 kernel: [<ffffffff9246d21a>] do_last+0xbba/0x1380
Sep 26 20:30:40 kernel: [<ffffffff92435e96>] ? kmem_cache_alloc_trace+0x1d6/0x200
Sep 26 20:30:40 kernel: [<ffffffff9246daad>] path_openat+0xcd/0x5a0
Sep 26 20:30:40 kernel: [<ffffffffc0611685>] ? xfs_file_buffered_aio_read+0xa5/0x110 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff929b7152>] ? down_write+0x12/0x41
Sep 26 20:30:40 kernel: [<ffffffff9246f46d>] do_filp_open+0x4d/0xc0
Sep 26 20:30:40 kernel: [<ffffffff9247dcd7>] ? __alloc_fd+0x47/0x180
Sep 26 20:30:40 kernel: [<ffffffff9245ab34>] do_sys_open+0x124/0x220
Sep 26 20:30:40 kernel: [<ffffffff9245ac4e>] SyS_open+0x1e/0x30
Sep 26 20:30:40 kernel: [<ffffffffc0719cac>] unload_network_ops_symbols+0x93bc/0x99d0 [falcon_lsm_pinned_15508]
Sep 26 20:30:40 kernel: [<ffffffffc0783cac>] unload_network_ops_symbols+0x93bc/0x99d0 [falcon_lsm_pinned_15604]
Sep 26 20:30:40 kernel: [<ffffffff929c539a>] system_call_fastpath+0x25/0x2a
Sep 26 20:30:40 kernel: INFO: task nginx:3446 blocked for more than 120 seconds.
Sep 26 20:30:40 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 26 20:30:40 kernel: nginx D ffff9f947d2a05e0 0 3446 3435 0x00000080
Sep 26 20:30:40 kernel: Call Trace:
Sep 26 20:30:40 kernel: [<ffffffff929b7ca9>] schedule+0x29/0x70
Sep 26 20:30:40 kernel: [<ffffffff929b5831>] schedule_timeout+0x221/0x2d0
Sep 26 20:30:40 kernel: [<ffffffff922a89b7>] ? local_bh_enable+0x17/0x20
Sep 26 20:30:40 kernel: [<ffffffff929b722b>] __down_common+0xaa/0x108
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] ? _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff929b72a6>] __down+0x1d/0x23
Sep 26 20:30:40 kernel: [<ffffffff922d17e5>] down+0x45/0x50
Sep 26 20:30:40 kernel: [<ffffffffc060c83c>] xfs_buf_lock+0x3c/0xe0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060ccb5>] xfs_buf_get_map+0x35/0x260 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d4a0>] xfs_buf_read_map+0x30/0x160 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc063f2a9>] xfs_trans_read_buf_map+0xf9/0x2d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05fa9dd>] xfs_read_agi+0x9d/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05faaa4>] xfs_ialloc_read_agi+0x34/0xd0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05fca08>] xfs_dialloc+0xf8/0x290 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e881>] xfs_ialloc+0x81/0x590 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0620887>] xfs_dir_ialloc+0x67/0x1d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff929b7152>] ? down_write+0x12/0x41
Sep 26 20:30:40 kernel: [<ffffffffc0620e52>] xfs_create+0x462/0x6b0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e433>] xfs_generic_create+0xd3/0x2a0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e634>] xfs_vn_mknod+0x14/0x20 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e673>] xfs_vn_create+0x13/0x20 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff924695ca>] vfs_create+0x15a/0x1e0
Sep 26 20:30:40 kernel: [<ffffffff9246d835>] do_last+0x11d5/0x1380
Sep 26 20:30:40 kernel: [<ffffffff92435e96>] ? kmem_cache_alloc_trace+0x1d6/0x200
Sep 26 20:30:40 kernel: [<ffffffff9246daad>] path_openat+0xcd/0x5a0
Sep 26 20:30:40 kernel: [<ffffffffc048ed4e>] ? 0xffffffffc048ed4d
Sep 26 20:30:40 kernel: [<ffffffffc0e99242>] ? cshook_security_inode_free_security+0x7592/0x9ac0 [falcon_lsm_serviceable]
Sep 26 20:30:40 kernel: [<ffffffffc0efdf67>] ? cshook_systemcalltable_pre_compat_sys_ioctl+0x2a6e7/0x34460 [falcon_lsm_serviceable]
Sep 26 20:30:40 kernel: [<ffffffffc0efdf89>] ? cshook_systemcalltable_pre_compat_sys_ioctl+0x2a709/0x34460 [falcon_lsm_serviceable]
Sep 26 20:30:40 kernel: [<ffffffff9246f46d>] do_filp_open+0x4d/0xc0
Sep 26 20:30:40 kernel: [<ffffffff9247dcd7>] ? __alloc_fd+0x47/0x180
Sep 26 20:30:40 kernel: [<ffffffff9245ab34>] do_sys_open+0x124/0x220
Sep 26 20:30:40 kernel: [<ffffffff9245ac4e>] SyS_open+0x1e/0x30
Sep 26 20:30:40 kernel: [<ffffffffc0719cac>] unload_network_ops_symbols+0x93bc/0x99d0 [falcon_lsm_pinned_15508]
Sep 26 20:30:40 kernel: [<ffffffffc0783cac>] unload_network_ops_symbols+0x93bc/0x99d0 [falcon_lsm_pinned_15604]
Sep 26 20:30:40 kernel: [<ffffffff929c539a>] system_call_fastpath+0x25/0x2a
Sep 26 20:30:40 kernel: INFO: task nginx:3448 blocked for more than 120 seconds.
Sep 26 20:30:40 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 26 20:30:40 kernel: nginx D ffff9f947d2a1660 0 3448 3435 0x00000080
Sep 26 20:30:40 kernel: Call Trace:
Sep 26 20:30:40 kernel: [<ffffffff929b8bf9>] schedule_preempt_disabled+0x29/0x70
Sep 26 20:30:40 kernel: [<ffffffff929b6ca7>] __mutex_lock_slowpath+0xc7/0x1e0
Sep 26 20:30:40 kernel: [<ffffffff929b602f>] mutex_lock+0x1f/0x33
Sep 26 20:30:40 kernel: [<ffffffff9246c8f8>] do_last+0x298/0x1380
Sep 26 20:30:40 kernel: [<ffffffff92435e96>] ? kmem_cache_alloc_trace+0x1d6/0x200
Sep 26 20:30:40 kernel: [<ffffffff9246daad>] path_openat+0xcd/0x5a0
Sep 26 20:30:40 kernel: [<ffffffffc0611685>] ? xfs_file_buffered_aio_read+0xa5/0x110 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff922ec27b>] ? should_numa_migrate_memory+0x5b/0x170
Sep 26 20:30:40 kernel: [<ffffffff9246f46d>] do_filp_open+0x4d/0xc0
Sep 26 20:30:40 kernel: [<ffffffff9247dcd7>] ? __alloc_fd+0x47/0x180
Sep 26 20:30:40 kernel: [<ffffffff9245ab34>] do_sys_open+0x124/0x220
Sep 26 20:30:40 kernel: [<ffffffff9245ac4e>] SyS_open+0x1e/0x30
Sep 26 20:30:40 kernel: [<ffffffffc0719cac>] unload_network_ops_symbols+0x93bc/0x99d0 [falcon_lsm_pinned_15508]
Sep 26 20:30:40 kernel: [<ffffffffc0783cac>] unload_network_ops_symbols+0x93bc/0x99d0 [falcon_lsm_pinned_15604]
Sep 26 20:30:40 kernel: [<ffffffff929c539a>] system_call_fastpath+0x25/0x2a
Sep 26 20:30:40 kernel: INFO: task nginx:3450 blocked for more than 120 seconds.
Sep 26 20:30:40 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 26 20:30:40 kernel: nginx D ffff9f947d2a3760 0 3450 3435 0x00000080
Sep 26 20:30:40 kernel: Call Trace:
Sep 26 20:30:40 kernel: [<ffffffff929b7ca9>] schedule+0x29/0x70
Sep 26 20:30:40 kernel: [<ffffffff929b5831>] schedule_timeout+0x221/0x2d0
Sep 26 20:30:40 kernel: [<ffffffffc062dc67>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0751de2>] ? bond_dev_queue_xmit+0x32/0x80 [bonding]
Sep 26 20:30:40 kernel: [<ffffffff925a39d4>] ? __radix_tree_lookup+0x84/0x100
Sep 26 20:30:40 kernel: [<ffffffff929b722b>] __down_common+0xaa/0x108
Sep 26 20:30:40 kernel: [<ffffffff925a39d4>] ? __radix_tree_lookup+0x84/0x100
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] ? _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff929b72a6>] __down+0x1d/0x23
Sep 26 20:30:40 kernel: [<ffffffff922d17e5>] down+0x45/0x50
Sep 26 20:30:40 kernel: [<ffffffffc060c83c>] xfs_buf_lock+0x3c/0xe0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060caf6>] _xfs_buf_find.isra.9+0x216/0x340 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060ccb5>] xfs_buf_get_map+0x35/0x260 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc060d4a0>] xfs_buf_read_map+0x30/0x160 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc063f2a9>] xfs_trans_read_buf_map+0xf9/0x2d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05e21ac>] xfs_btree_read_buf_block.constprop.33+0xac/0xe0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05e4d9e>] xfs_btree_increment+0x21e/0x260 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05fbfd5>] xfs_dialloc_ag_inobt+0x4d5/0x680 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05fc6ee>] xfs_dialloc_ag+0x6e/0x290 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc06024aa>] ? xfs_perag_get+0x2a/0xc0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc05fc96d>] xfs_dialloc+0x5d/0x290 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e881>] xfs_ialloc+0x81/0x590 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc0620887>] xfs_dir_ialloc+0x67/0x1d0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff929b7152>] ? down_write+0x12/0x41
Sep 26 20:30:40 kernel: [<ffffffffc0620e52>] xfs_create+0x462/0x6b0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e433>] xfs_generic_create+0xd3/0x2a0 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e634>] xfs_vn_mknod+0x14/0x20 [xfs]
Sep 26 20:30:40 kernel: [<ffffffffc061e673>] xfs_vn_create+0x13/0x20 [xfs]
Sep 26 20:30:40 kernel: [<ffffffff924695ca>] vfs_create+0x15a/0x1e0
Sep 26 20:30:40 kernel: [<ffffffff9246d835>] do_last+0x11d5/0x1380
Sep 26 20:30:40 kernel: [<ffffffff92435e96>] ? kmem_cache_alloc_trace+0x1d6/0x200
Sep 26 20:30:40 kernel: [<ffffffff9246daad>] path_openat+0xcd/0x5a0
Sep 26 20:30:40 kernel: [<ffffffffc048ed4e>] ? 0xffffffffc048ed4d
Sep 26 20:30:40 kernel: [<ffffffffc0e99242>] ? cshook_security_inode_free_security+0x7592/0x9ac0 [falcon_lsm_serviceable]
Sep 26 20:30:40 kernel: [<ffffffffc0efdf67>] ? cshook_systemcalltable_pre_compat_sys_ioctl+0x2a6e7/0x34460 [falcon_lsm_serviceable]
Sep 26 20:30:40 kernel: [<ffffffffc0efdf89>] ? cshook_systemcalltable_pre_compat_sys_ioctl+0x2a709/0x34460 [falcon_lsm_serviceable]
Sep 26 20:30:40 kernel: [<ffffffff9246f46d>] do_filp_open+0x4d/0xc0
Sep 26 20:30:40 kernel: [<ffffffff9247dcd7>] ? __alloc_fd+0x47/0x180
Sep 26 20:30:40 kernel: [<ffffffff9245ab34>] do_sys_open+0x124/0x220
Sep 26 20:30:40 kernel: [<ffffffff9245ac4e>] SyS_open+0x1e/0x30
Sep 26 20:30:40 kernel: [<ffffffffc0719cac>] unload_network_ops_symbols+0x93bc/0x99d0 [falcon_lsm_pinned_15508]
Sep 26 20:30:40 kernel: [<ffffffffc0783cac>] unload_network_ops_symbols+0x93bc/0x99d0 [falcon_lsm_pinned_15604]
Sep 26 20:30:40 kernel: [<ffffffff929c539a>] system_call_fastpath+0x25/0x2a

[TrevorH]

Right, the missing bits were the important bit. You're hitting a standard stacktrace that is issued when an I/O to a device does not respond for 2 whole minutes. The kernel kicks the device to try to make it respond and the stacktrace is what shows that is being done. This is usually caused by a massively overloaded device that is so busy it never responds in a timely manner to I/O requests. You need to look at the device it's trying to use to see why it isn't responding.