I was asked to maintain a small local (without internet connection) linux cluster built in 2015, but I do not have lots of experience of daily maintenance. The cluster contains management, I/O, storage, and computing nodes, all running CentOS 6. After replacing a bad disk of I/O node, the I/O node asks me to run "fsck" on boot:
Code: Select all
Checking filesystems
/dev/sda3 contains a file system with errors, check forced.
/dev/sda3: Resize inode not valid.
/dev/sda3: UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY.
Code: Select all
[root@io32 ~]# fsck
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
/dev/sda3 contains a file system with errors, check forced.
Resize inode not valid. Recreate<y>? yes
Pass 1: Checking inodes, blocks, and sizes
Inode 2490389 has an invalid extent
(logical block 950272, invalid physical block 144233591734273, len 4294017025)
Clear<y>? yes
Inode 2490389, i_blocks is 8, should be 0. Fix<y>? yes
Inode 2490394 has out of order extents
(invalid logical block 2, physical block 9994759, len1)
Clear<y>? yes
Inode 2490394, i_blocks is 24, should be 16. Fix<y>? yes
Inode 2490395, i_size is 201474993487872, should be 0. Fix<y>? yes
Warning... fsck.ext4 for device /dev/sda3 exited with signal 11.
[root@io32 ~]# fsck.ext4 -v /dev/sda3
...
Inode 2490395, i_size is 201474993487872, should be 0. Fix<y>? yes
Segmentation fault
Code: Select all
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda3 162630140 11970676 142398304 8% /
Code: Select all
UUID=... / ext4 defaults 1 1
UUID=... /boot/efi vfat umask=0077,shortname=winnt 0 0
UUID=... swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/mapper/mpathh /backup xfs defaults 0 0
/dev/pvf35 /pvfsDATA gpfs rw,mtime,atime,dev=pvf35,noauto 0 0
Code: Select all
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 222.2G 0 disk
|-sda1 8:1 0 200M 0 part
|-sda2 8:2 0 64.5G 0 part
|-sda3 8:3 0 157.6G 0 part /
sdc 8:32 0 1.8T 0 disk
|-mpathb (dm-0) 253:0 0 1.8T 0 mpath
sdb 8:16 0 1.8T 0 disk
|-mpathc (dm-1) 253:1 0 1.8T 0 mpath
sdf 8:80 0 32.7T 0 disk
|-mpathh (dm-2) 253:2 0 32.7T 0 mpath
sde 8:64 0 2.8T 0 disk
|-mpathe (dm-3) 253:3 0 2.8T 0 mpath
sdd 8:48 0 2.7T 0 disk
|-mpathd (dm-4) 253:4 0 2.7T 0 mpath
sdh 8:112 0 2.8T 0 disk
|-mpathe (dm-3) 253:3 0 2.8T 0 mpath
sdg 8:96 0 2.8T 0 disk
|-mpathd (dm-4) 253:4 0 2.8T 0 mpath
sdi 8:128 0 32.7T 0 disk
|-mpathh (dm-2) 253:2 0 32.7T 0 mpath
sr0 11:0 1 1024M 0 rom
Any advice is greatly appreciated. Thank you!