Does anyone have any suggestions to resolve it?
Possibly relevant information gleaned from a bit of googling
- I'm using the default xfs parameters i.e. not messing with the block sizes. http://tracker.ceph.com/issues/6301 talks about an already applied patch to centos7 but seems to boil down to messing with the block sizes.
- single xeon E5-2609v2, 32gb ram, 7x4g sas raid 5 on lsi controller with hot spare (getinfo for disks below)
- not running ceph
- drbd is replicating and status was ok
Information for driver problems.
Code: Select all
== BEGIN uname -rmi ==
3.10.0-123.6.3.el7.x86_64 x86_64 x86_64
== END uname -rmi ==
== BEGIN rpm -qa \*-release\* ==
centos-release-7-0.1406.el7.centos.2.5.x86_64
elrepo-release-7.0-2.el7.elrepo.noarch
epel-release-7-2.noarch
== END rpm -qa \*-release\* ==
== BEGIN cat /etc/redhat-release ==
CentOS Linux release 7.0.1406 (Core)
== END cat /etc/redhat-release ==
== BEGIN getenforce ==
Enforcing
== END getenforce ==
== BEGIN free -m ==
total used free shared buffers cached
Mem: 31980 18651 13328 8 5 16246
-/+ buffers/cache: 2399 29580
Swap: 31999 0 31999
== END free -m ==
== BEGIN rpm -qa kernel\* | sort ==
kernel-3.10.0-123.6.3.el7.x86_64
kernel-3.10.0-123.el7.x86_64
kernel-tools-3.10.0-123.6.3.el7.x86_64
kernel-tools-libs-3.10.0-123.6.3.el7.x86_64
== END rpm -qa kernel\* | sort ==
== Warning: lspci is not installed ==
== Warning: lsusb is not installed ==
== BEGIN rpm -qa kmod\* kmdl\* ==
kmod-libs-14-9.el7.x86_64
kmod-drbd84-8.4.5-1.el7.elrepo.x86_64
kmod-14-9.el7.x86_64
== END rpm -qa kmod\* kmdl\* ==
Code: Select all
== BEGIN uname -rmi ==
3.10.0-123.6.3.el7.x86_64 x86_64 x86_64
== END uname -rmi ==
== BEGIN rpm -qa \*-release\* ==
centos-release-7-0.1406.el7.centos.2.5.x86_64
elrepo-release-7.0-2.el7.elrepo.noarch
epel-release-7-2.noarch
== END rpm -qa \*-release\* ==
== BEGIN cat /etc/redhat-release ==
CentOS Linux release 7.0.1406 (Core)
== END cat /etc/redhat-release ==
== BEGIN getenforce ==
Enforcing
== END getenforce ==
== BEGIN free -m ==
total used free shared buffers cached
Mem: 31980 18156 13824 8 1 15758
-/+ buffers/cache: 2397 29583
Swap: 31999 0 31999
== END free -m ==
== BEGIN cat /etc/fstab ==
#
# /etc/fstab
# Created by anaconda on Tue Sep 9 19:33:28 2014
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs defaults 1 1
UUID=b8c11e24-72ed-4f0a-9fbb-efebc3d67956 /boot xfs defaults 1 2
/dev/mapper/centos-home /home xfs defaults 1 2
/dev/mapper/centos-swap swap swap defaults 0 0
== END cat /etc/fstab ==
== BEGIN df -h ==
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 50G 2.1G 48G 5% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 8.7M 16G 1% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/mapper/centos-home 49G 33M 49G 1% /home
/dev/sda2 497M 126M 372M 26% /boot
/dev/drbd1 6.0T 2.9T 3.2T 47% /mnt/homeagbu
/dev/drbd2 4.0T 2.2T 1.9T 55% /mnt/backdesk
/dev/drbd3 2.0T 322G 1.7T 16% /mnt/backserv
/dev/drbd4 2.0T 1.6T 430G 80% /mnt/genomics
/dev/drbd5 2.0T 224G 1.8T 11% /mnt/agbudata
== END df -h ==
== BEGIN fdisk -lu ==
Disk /dev/sda: 24000.0 GB, 23999998328832 bytes, 46874996736 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sda1 * 1 4294967295 2147483647+ ee GPT
Disk /dev/mapper/centos-swap: 33.6 GB, 33554432000 bytes, 65536000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-root: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/DRBD-homeagbu: 6597.1 GB, 6597069766656 bytes, 12884901888 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/DRBD-backdesk: 4398.0 GB, 4398046511104 bytes, 8589934592 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/DRBD-genomics: 2199.0 GB, 2199023255552 bytes, 4294967296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/DRBD-backserv: 2199.0 GB, 2199023255552 bytes, 4294967296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/DRBD-agbudata: 2199.0 GB, 2199023255552 bytes, 4294967296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-home: 52.4 GB, 52428800000 bytes, 102400000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
== END fdisk -lu ==
== BEGIN parted -l ==
Model: LSI MR9261-8i (scsi)
Disk /dev/sda: 24.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: pmbr_boot
Number Start End Size File system Name Flags
1 1049kB 2097kB 1049kB bios_grub
2 2097kB 526MB 524MB xfs
3 526MB 140GB 140GB lvm
4 140GB 24.0TB 23.9TB
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/centos-home: 52.4GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 52.4GB 52.4GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/DRBD-agbudata: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 2199GB 2199GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/DRBD-backserv: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 2199GB 2199GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/DRBD-genomics: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 2199GB 2199GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/DRBD-backdesk: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 4398GB 4398GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/DRBD-homeagbu: 6597GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 6597GB 6597GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/centos-root: 53.7GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 53.7GB 53.7GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/centos-swap: 33.6GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 33.6GB 33.6GB linux-swap(v1)
Model: Unknown (unknown)
Disk /dev/drbd1: 6597GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 6597GB 6597GB xfs
Model: Unknown (unknown)
Disk /dev/drbd2: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 4398GB 4398GB xfs
Model: Unknown (unknown)
Disk /dev/drbd3: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 2199GB 2199GB xfs
Model: Unknown (unknown)
Disk /dev/drbd4: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 2199GB 2199GB xfs
Model: Unknown (unknown)
Disk /dev/drbd5: 2199GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 2199GB 2199GB xfs
== END parted -l ==
== BEGIN blkid ==
/dev/sda2: UUID="b8c11e24-72ed-4f0a-9fbb-efebc3d67956" TYPE="xfs" PARTUUID="f46afefa-4ee1-43ef-95b9-649cff868e97"
/dev/sda3: UUID="94nxvp-AmhR-DeqH-8kTh-SRY1-pkgD-DNvQhj" TYPE="LVM2_member" PARTUUID="c747b996-7855-41e1-8307-5ac7953f37c7"
/dev/sda4: UUID="z7rrwJ-Wm53-npPk-gNo0-E7qw-ecFl-oOfP3k" TYPE="LVM2_member" PARTUUID="2df9b8e8-9f75-4985-9c57-d2951dccbc3e"
/dev/mapper/centos-swap: UUID="14c07bd7-57d7-42f3-b637-ddee56f4cfeb" TYPE="swap"
/dev/mapper/centos-root: UUID="0899b7b0-7a42-44ab-8e60-193b58907736" TYPE="xfs"
/dev/mapper/DRBD-homeagbu: UUID="d8e829dd339e83a9" TYPE="drbd"
/dev/mapper/DRBD-backdesk: UUID="e5cd491c115d7582" TYPE="drbd"
/dev/mapper/DRBD-genomics: UUID="971797fbff3e2b56" TYPE="drbd"
/dev/mapper/DRBD-backserv: UUID="769898b29d0f3c39" TYPE="drbd"
/dev/mapper/DRBD-agbudata: UUID="22c76dc63c54f5f7" TYPE="drbd"
/dev/mapper/centos-home: UUID="140d6eeb-bf2c-4513-bd74-2e3784b7a6ef" TYPE="xfs"
/dev/drbd5: UUID="a663d717-908b-4cb3-879d-2dd3112fe80e" TYPE="xfs"
/dev/drbd2: UUID="2b490af1-1e8f-4758-9921-ef30fa42b82e" TYPE="xfs"
/dev/drbd4: UUID="5012e580-a9c2-4a25-9a9c-0cb44c69bd73" TYPE="xfs"
/dev/drbd3: UUID="7c92996f-fde8-4df9-9da1-256420f62797" TYPE="xfs"
/dev/drbd1: UUID="5ab7a86c-c8d4-4094-9c1f-ede32c266586" TYPE="xfs"
== END blkid ==
== BEGIN cat /proc/mdstat ==
Personalities :
unused devices: <none>
== END cat /proc/mdstat ==
== BEGIN pvs ==
PV VG Fmt Attr PSize PFree
/dev/sda3 centos lvm2 a-- 130.08g 0
/dev/sda4 DRBD lvm2 a-- 21.70t 5.70t
== END pvs ==
== BEGIN vgs ==
VG #PV #LV #SN Attr VSize VFree
DRBD 1 5 0 wz--n- 21.70t 5.70t
centos 1 3 0 wz--n- 130.08g 0
== END vgs ==
== BEGIN lvs ==
LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert
agbudata DRBD -wi-ao---- 2.00t
backdesk DRBD -wi-ao---- 4.00t
backserv DRBD -wi-ao---- 2.00t
genomics DRBD -wi-ao---- 2.00t
homeagbu DRBD -wi-ao---- 6.00t
home centos -wi-ao---- 48.83g
root centos -wi-ao---- 50.00g
swap centos -wi-ao---- 31.25g
== END lvs ==
Code: Select all
May 6 00:01:01 hans0 systemd: Created slice user-0.slice.
May 6 00:01:01 hans0 systemd: Starting Session 629 of user root.
May 6 00:01:01 hans0 systemd: Started Session 629 of user root.
May 6 01:01:01 hans0 systemd: Created slice user-0.slice.
May 6 01:01:01 hans0 systemd: Starting Session 630 of user root.
May 6 01:01:01 hans0 systemd: Started Session 630 of user root.
May 6 01:17:36 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 01:17:37 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 01:17:39 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 01:17:41 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 01:17:43 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 01:17:45 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
....
May 6 02:07:51 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 02:07:53 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 02:07:55 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 02:07:56 hans0 kernel: nfsd: peername failed (err 107)!
May 6 02:07:56 hans0 kernel: nfsd: peername failed (err 107)!
May 6 02:07:56 hans0 kernel: nfsd: peername failed (err 107)!
May 6 02:07:56 hans0 kernel: nfsd: peername failed (err 107)!
....
May 6 04:07:47 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 04:07:49 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 6 04:07:49 hans0 kernel: net_ratelimit: 55 callbacks suppressed
May 6 04:07:49 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:49 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:49 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:49 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:49 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:50 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:50 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:50 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:50 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:50 hans0 kernel: nfsd: peername failed (err 107)!
May 6 04:07:50 hans0 kernel: rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket
May 6 04:08:05 hans0 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)