By the way,
Are you using NFS v4.2 or older implementation?
I'm aasking because 4.1+ allows pNFS which could increase bandwidth consumption but also performance of the NFS server.
Why is NFSoRDMA in CentOS 7.6.1810 limited to 10 Gbps?
-
hunter86_bg
- Posts: 2019
- Joined: 2015/02/17 15:14:33
- Location: Bulgaria
- Contact:
-
alpha754293
- Posts: 69
- Joined: 2019/07/29 16:15:14
Re: Why is NFSoRDMA in CentOS 7.6.1810 limited to 10 Gbps?
So I did try out 7.7.
No significant differences in the results.
I'm currently using vers=4.1 apparently (as the default).
There are no deployment guides in regards to how to deploy pNFS properly, but RedHat supports it as a pNFS client.
I've tried GlusterFS, but two problems that I encountered was that it wouldn't let me, for some reason, create a distributed striped GlusterFS logical volume, and that with a distributed GlusterFS logical volume that I WAS able to create and test with, it would just write the files in alternating nodes which meant that there would be a requisite of all of the nodes being in an "always-on" state, which may or may not necessarily be true depending on what I need the system for/to do.
In my research regarding this topic, I also found some documentation from storage/deployment experts that do not recommend that the clients and the data servers be on the same systems, and that ideally, they should be on separate systems, which again, I don't have (due to limited capex budget).
(The distributed GlusterFS tests that I ran performed no better than any single RDMA transfer/operation which meant that I wasn't really able to take advantage of it. In theory, had I been able to create at least a striped volume, or distributed striped volume, that should have been better able to take advantage of my 100 Gbps 4x EDR Infiniband connection that I've got, but alas, to no avail.)
In my recent rounds of testing, 10 Gbps was about the best that I was able to get using dd (buffered) and unbuffered was about half that (writing to four Samsung 860 EVO 1 TB SATA 6 Gbps SSDs in RAID0 on a Broadcom/Avago/LSI MegaRAID 9341-8i 12 Gbps SAS HW RAID HBA). md raid was on average, about 5.5% slower than that. (In my actual application that I'm running, I was able to only max out at around 11.2 Gbps, so there is some relatively reasonable alignment in regards to the results -- one based on more theorectical testing (dd) and then my quote "real world" testing that I am actually doing/using with an actual work load that I've got running/solving for a client right now.
No significant differences in the results.
I'm currently using vers=4.1 apparently (as the default).
There are no deployment guides in regards to how to deploy pNFS properly, but RedHat supports it as a pNFS client.
I've tried GlusterFS, but two problems that I encountered was that it wouldn't let me, for some reason, create a distributed striped GlusterFS logical volume, and that with a distributed GlusterFS logical volume that I WAS able to create and test with, it would just write the files in alternating nodes which meant that there would be a requisite of all of the nodes being in an "always-on" state, which may or may not necessarily be true depending on what I need the system for/to do.
In my research regarding this topic, I also found some documentation from storage/deployment experts that do not recommend that the clients and the data servers be on the same systems, and that ideally, they should be on separate systems, which again, I don't have (due to limited capex budget).
(The distributed GlusterFS tests that I ran performed no better than any single RDMA transfer/operation which meant that I wasn't really able to take advantage of it. In theory, had I been able to create at least a striped volume, or distributed striped volume, that should have been better able to take advantage of my 100 Gbps 4x EDR Infiniband connection that I've got, but alas, to no avail.)
In my recent rounds of testing, 10 Gbps was about the best that I was able to get using dd (buffered) and unbuffered was about half that (writing to four Samsung 860 EVO 1 TB SATA 6 Gbps SSDs in RAID0 on a Broadcom/Avago/LSI MegaRAID 9341-8i 12 Gbps SAS HW RAID HBA). md raid was on average, about 5.5% slower than that. (In my actual application that I'm running, I was able to only max out at around 11.2 Gbps, so there is some relatively reasonable alignment in regards to the results -- one based on more theorectical testing (dd) and then my quote "real world" testing that I am actually doing/using with an actual work load that I've got running/solving for a client right now.