So I ended up picking up a Broadcom/Avago/LSI MegaRAID 9341-8i 12 Gbps SAS RAID HBA and put the 4x Samsung 860 EVO 1 TB SATA 6 Gbps SSDs on a RAID0 virtual drive and also put my 4x HGST 6 TB 7200 SATA 6 Gbps HDDs on another RAID0 virtual drive as well.
Here are the buffered RAID0 SSD results on local host:
Code: Select all
[root@host home]# cd /home/cluster
[root@host cluster]# time -p dd if=/dev/zero of=10Gfile bs=1024k count=10240; rm -f 10Gfile
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 3.60042 s, 3.0 GB/s
real 3.60
user 0.00
sys 3.59
[root@host cluster]# time -p dd if=/dev/zero of=10Gfile bs=1024k count=10240; rm -f 10Gfile
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 3.63025 s, 3.0 GB/s
real 3.63
user 0.00
sys 3.62
[root@host cluster]# time -p dd if=/dev/zero of=10Gfile bs=1024k count=10240; rm -f 10Gfile
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 3.42913 s, 3.1 GB/s
real 3.43
user 0.00
sys 3.40
And here are the unbuffered results:
Code: Select all
[root@host cluster]# time -p dd if=/dev/zero of=xxx bs=8k count=12500000; rm -f xxx
12500000+0 records in
12500000+0 records out
102400000000 bytes (102 GB) copied, 62.5086 s, 1.6 GB/s
real 62.51
user 3.13
sys 59.37
[root@host cluster]# time -p dd if=/dev/zero of=xxx bs=8k count=12500000; rm -f xxx
12500000+0 records in
12500000+0 records out
102400000000 bytes (102 GB) copied, 63.1949 s, 1.6 GB/s
real 63.36
user 3.20
sys 59.97
[root@host cluster]# time -p dd if=/dev/zero of=xxx bs=8k count=12500000; rm -f xxx
12500000+0 records in
12500000+0 records out
102400000000 bytes (102 GB) copied, 62.3144 s, 1.6 GB/s
real 62.48
user 3.11
sys 59.19
[root@host cluster]# time -p dd if=/dev/zero of=xxx bs=64k count=1562500; rm -f xxx
1562500+0 records in
1562500+0 records out
102400000000 bytes (102 GB) copied, 52.6125 s, 1.9 GB/s
real 52.78
user 0.48
sys 38.05
[root@host cluster]# time -p dd if=/dev/zero of=xxx bs=64k count=1562500; rm -f xxx
1562500+0 records in
1562500+0 records out
102400000000 bytes (102 GB) copied, 52.6867 s, 1.9 GB/s
real 52.85
user 0.49
[root@host cluster]# time -p dd if=/dev/zero of=xxx bs=64k count=1562500; rm -f xxx
1562500+0 records in
1562500+0 records out
102400000000 bytes (102 GB) copied, 53.1124 s, 1.9 GB/s
real 53.11
user 0.51
sys 40.18
[root@host cluster]#
I haven't tuned my NFS settings yet, so it's still only getting about 1.1 GB/s over NFSoRDMA.
Unbuffered local host writes looks pretty decent with an average of 462.35375 MB/s per drive.