Posted by nosebreaker on 2011/3/26 16:11:23

I'm trying to build a new system with 3 2tb drives in raid5. I was following the centos raid howto, and it talks about using dmraid and how the boot volume must be on a raid1 array. I'd really rather not do this, since I plan to add 3 more 2tb drives later and expand the array, and it would be nice to add a hot spare and not have to worry that the 2 drives that failed are the boot drives. My second issue is that disk druid doesn't like the GPT partition table on the 2tb drives, but I thought drives of this size had to be GPT due to a 2tb limit of MBR?

1) Is there a way to do software raid5 without the raid1 boot volume?
2) Can anyone comment on the 2tb drive size and GPT partitions?

Posted by pschaff on 2011/3/26 17:10:18

I think you mean mdraid not dmraid (used for hardware/HostRAID/FakeRAID), but you did not say which of the several Wiki RAID articles you are following.

As stated in How to Setup a Software RAID on CentOS 5 you cannot have /boot on RAID other than RAID1. You can make a separate /boot partition and make other partitions RAID5, also as stated in the article.

IIRC the anaconda installer does not understand GPT.

I'd be prone to install the OS on a separate smaller disk (two if you want RAID1) and leave the big RAID5 array to be configured later and mounted where you need it as storage space.

Posted by nosebreaker on 2011/3/27 17:46:03

I was following the following article, also from centos.org!
Configuring Software RAID

The installer did not understand GPT, so I ended up booting off a Ubuntu 9.10 livecd and changed it to msdos, and that worked fine. I created a 100mb /boot on two drives and set for raid1, and then created a 16gb swap partition on all 3 drives as raid5, and then the rest of the 2tb drives became / as raid5. I'd rather not have extra drives in this, the power supply isn't that large.

Once mdadm said the array was done building, I did some speed tests. I got:
Reading was 115MB/s, totally acceptable.
Writing was 55MB/s, kinda not acceptable.

I have another box, similar specs but it had a 3ware 9650 hardware raid card in it and it is set for RAID10 with 6x 500gb drives. I ran the same tests on it and got:

Reading was 161MB/s
Writing was 107MB/s

So I was kind of hoping for 100MB/s write speeds from the new system, since the old one with the old raid card was getting those speeds. I'll post again once I do the tests on a single drive to see what the write speeds are.

Posted by TrevorH on 2011/3/27 17:58:29

You need to wait until it's finished building before you can speed test it. This is a 4 drive RAID 5 array:

Posted by nosebreaker on 2011/3/27 19:17:37

I did wait, it took 10 hours!

Ok, so my tests with a single drive are as follows:
Write speed: 71 MB/s
Read speed: 97 MB/s

So thats not great, but not bad considering these are 5900rpm drives. So why is the RAID5 taking so much overhead? Shouldn't it in theory be twice that for write speeds since it is 2 data drives?

260 MB/s would be fantastic, I only have 3 drives now though. I'm going to start the rebuild of the array to try this again.

Update: After doing some googling, the 55 MB/s write speed may in fact be all I can get from these drives.
http://www.z-a-recovery.com/art-raid.htm

Posted by pschaff on 2011/3/27 20:47:11

Quote:
That is indeed on centos.org, but is an old copy of upstream documentation as the CentOS copies have not been kept up to date. If you want upstream docs I'd just go to the source.

P.S. Please do not change the title in mid-thread to something nondescript. It is very confusing.

Edit: I see some fellow moderator has anonymously acted on that issue - never mind.

Posted by TrevorH on 2011/3/28 1:17:41

What sort of drives are you using? My array here is 4 x 1.5TB Western Digital WD15EADS "Green" 5400 rpm drives and tested individually with hdparm will pull ~95MB/s each. I don't think these are particularly new or fast - they're probably going on 18 months old already. There's nowhere in this thread where you've posted the information about your configuration - which physical disks? what SATA ports are they attached to? Are they seen as /dev/sd* or /dev/hd*? What are you using to measure read/write speeds? What speeds do you get if you test each disk individually - e.g. if the array is /dev/md0 and consists of /dev/sd[bcd] then what speed do you get from /dev/sdb?

Posted by nosebreaker on 2011/3/28 2:19:11

I am using (Seagate Barracuda LP ST32000542AS 2TB 5900 RPM 32MB Cache SATA 3.0Gb/s 3.5" Hard Drive -Bare Drive):
http://www.newegg.com/Product/Product.aspx?Item=N82E16822148413

The motherboard (BIOSTAR A770E3 AM3 AMD 770 ATX AMD)
http://www.newegg.com/Product/Product.aspx?Item=N82E16813138179

Write test (16gb file):
dd if=/dev/zero of=testfile.out bs=1M count=16384
Read test (that same 16gb file you just made):
dd if=testfile.out of=/dev/null bs=1M

When I tested read speeds individually they were 97 MB/s as I would expect, but read speed was only 71 MB/s. I did a basic install of CentOS onto the system to pull these numbers.

They are attached to ports 0 through 2 on the motherboard, they show up as /dev/sda /dev/sdb and /dev/sdc

Posted by TrevorH on 2011/3/28 4:10:05

Ok, so I repeated (more or less) your write test here



As you can see, dd's timing is a little faster than it should be because not all data had hit the disk platters (96 secs vs the actual elapsed 107 so more like 152MB/s). Don't have to bother with that on reading



Even accounting for your 3 spindles vs my 4 I think your rates are low.

Posted by nosebreaker on 2011/3/28 13:43:20

I agree that I would think it would be faster, but if the max write speed is 71 MB/s, then it is possible the 55 MB/s write speed is the max. As I understand it, RAID5 has to do a read on the other drives before it can write the parity bit, even though it should know that anyway. But I am hoping I am wrong!

Do you have any ideas on what I can try to make this work? I suppose I could take some older drives and see if they are any faster.

Posted by nosebreaker on 2011/3/28 21:02:55

I took 3 different drives (80gb, 1tb, 1.5tb - but only 20gb / data size), and ran the same tests. I got:
Read speed: 67 MB/s
Write speed: 43 MB/s

I also ran hdparm:


So this makes me think it might be the motherboard or something else. Any ideas to look for?

Posted by pschaff on 2011/3/28 21:22:46

Are these perchance AFD (Advanced Format Disk) drives? If so, the partition alignment is a critical factor for performance. Also check that you have optimal BIOS settings - i.e. AHCI and not legacy IDE mode.

Posted by nosebreaker on 2011/3/28 21:57:08

I have no idea if they are AFD, how would I align the partitions?

Posted by pschaff on 2011/3/28 22:07:15

This post addresses the issue as well as I have seen. Perhaps an easier way would be to do any partitioning with a later distro (such as recent Fedora) that has partition tools that understand 4k sectors. If I understand correctly it is only an issue if your are not using the whole disk for a single partition, as you may or may not be doing with RAID.

Posted by nosebreaker on 2011/3/29 3:52:10

I may have fixed it!

I used fdisk to change the heads/sectors from 256/63 (old CHS/LBA settings) to 224/56 (4k setting).

In fdisk on each disk:
x for expert mode
h for heads, change to 224
s for sectors, change to 56
w to write and exit

Now speeds on a single drive were 108 MB/s for reading, and 118 MB/s for writing (old was 97/71 MB/s). I am rebuilding the system now, will update tomorrow when its done and I can test.

This post was helpful:
http://www.ibm.com/developerworks/linux/library/l-4kb-sector-disks/index.html?ca=dgr-lnxw074KB-Disksdth-LX

Posted by nosebreaker on 2011/3/29 4:41:36

Right now the recovery on the fresh install is going at 62-65 MB/s, better than before but not the big increase I was hoping for. I'm not sure if this fixed it.

Posted by TrevorH on 2011/3/29 13:02:13

The recovery is, by default, throttled unless you have changed it.



Run sysctl -p to implement any changes you make to /etc/sysctl.conf immediately.

Posted by nosebreaker on 2011/3/30 23:42:41

I found a way to fix it, basically I found a script that calculates the best settings and modifies it for you. I had to modify the script to use /sbin/blockdev and not just blockdev, and my array was /dev/(sda3,sdb3,sdc3) and not sda1/sdb1/sdc1. Once I modified those lines and changed it, it worked!

Before:
Write speed 55 MB/s
Read speed 115 MB/s

After
Write speed is 107 MB/s
Read speed is 164 MB/s

A significant improvement!

Posted by nosebreaker on 2011/3/30 23:45:19

Oh, the only thing I've noticed is, now the drive heads click about every 10 seconds. I found an old post about Ubuntu that does this, but I wanted to ask if anyone here knew of a fix.
http://www.linuxquestions.org/questions/ubuntu-63/hdparm-apm-hard-drive-clicking-sound-588770/

Posted by nosebreaker on 2011/3/31 1:24:16

I added the hdparm -B 255 /dev/sda command to my startup, along with that script to tune the HDD's. Seems to work for now, but I was hoping for something better. That is apparently new firmware for the drive, I'll try that as well.