One of the nodes runs the subnet manager.
All of the nodes are running CentOS 7.6.1810 with 'Infiniband Support' package group installed.
When I go to run ibdiagnet, it shows:
Code: Select all
I---------------------------------------------------
I- IPoIB Subnets Check
I---------------------------------------------------
I- Subnet: IPv4 PKey:0x7fff QKey:0x00000b1b MTU:2048Byte rate:10Gbps SL:0x00
W- Suboptimal rate for group. Lowest member rate:40Gbps > group-rate:10Gbps
Code: Select all
CA: aes0 mlx5_0:
0x7cfe9003004431f8 1 1[ ] ==( 4X 25.78125 Gbps Active/ LinkUp)==> 3 5[ ] "SwitchIB Mellanox Technol
ogies" ( )
CA: aes2 mlx5_0:
0x248a0703002b1ec6 4 1[ ] ==( 4X 25.78125 Gbps Active/ LinkUp)==> 3 3[ ] "SwitchIB Mellanox Technol
ogies" ( )
CA: aes3 mlx5_0:
0x248a0703002b1eca 2 1[ ] ==( 4X 25.78125 Gbps Active/ LinkUp)==> 3 2[ ] "SwitchIB Mellanox Technol
ogies" ( )
CA: aes4 mlx5_0:
0x248a0703002b1ece 6 1[ ] ==( 4X 25.78125 Gbps Active/ LinkUp)==> 3 1[ ] "SwitchIB Mellanox Technol
ogies" ( )
Switch: 0xec0d9a0300224e70 SwitchIB Mellanox Technologies:
3 1[ ] ==( 4X 25.78125 Gbps Active/ LinkUp)==> 6 1[ ] "aes4 mlx5_0" ( )
3 2[ ] ==( 4X 25.78125 Gbps Active/ LinkUp)==> 2 1[ ] "aes3 mlx5_0" ( )
3 3[ ] ==( 4X 25.78125 Gbps Active/ LinkUp)==> 4 1[ ] "aes2 mlx5_0" ( )
3 4[ ] ==( 4X 25.78125 Gbps Active/ LinkUp)==> 5 1[ ] "aes1 mlx5_0" ( )
3 5[ ] ==( 4X 25.78125 Gbps Active/ LinkUp)==> 1 1[ ] "aes0 mlx5_0" ( )
I tried looking through the documentation and doing research online and it doesn't appear that anybody else has encountered this before?
(I found this out while I was trying to find out what the MTU was on the switch. All of the cards are set to 'connected mode' with a MTU of 4092 (/etc/sysconfig/network-scripts/ifcfg-ib0).)
Your help is greatly appreciated.
Thank you.