Cannot change infiniband adapter from datagram mode to connected mode

Issues related to configuring your network
Post Reply
alpha754293
Posts: 69
Joined: 2019/07/29 16:15:14

Cannot change infiniband adapter from datagram mode to connected mode

Post by alpha754293 » 2019/08/07 03:25:55

I recently download CentOS 7.6.1810 and installed in on my system (dual Xeon E5-2690 (v1), 128 GB of RAM, 1 TB SSD, 3 TB HDD, Mellanox ConnectX-4 dual port 100 Gbps 4x EDR Infiniband NIC).

I got my entire micro cluster (four identical nodes total) set up okay and all of the nodes are connected to a Mellanox MSB-7890 externally managed 36-port 4x EDR IB switch.

I also installed the software package group 'Infiniband Support' after the OS was installed to get everything up and running. OpenSM is running, and all of the nodes can ping each other without any issues.

I'm trying to do some perfomance tuning, especially with the MTU because with the datagram mode, the MTU is maxed out at 2044.

Here are the commands that I am trying to execute:

Code: Select all

# cd /etc/sysconfig/network-scripts/
# ./ifdown ib0
# cat /sys/class/net/ib0/mode
datagram
# echo connected > /sys/class/net/ib0/mode
bash: echo: write error: Invalid argument
How do I resolve this issue?

I tried searching online in regards to changing from datagram mode to connected mode and even tried to do it via the Settings GUI to no avail.

Your help in this matter is greatly appreciated.

Thank you.

chemal
Posts: 776
Joined: 2013/12/08 19:44:49

Re: Cannot change infiniband adapter from datagram mode to connected mode

Post by chemal » 2019/08/07 14:51:52

To enable connected mode, you put the line

Code: Select all

CONNECTED_MODE=yes
in ifcfg-ib0.

alpha754293
Posts: 69
Joined: 2019/07/29 16:15:14

Re: Cannot change infiniband adapter from datagram mode to connected mode

Post by alpha754293 » 2019/08/08 12:08:58

chemal wrote:
2019/08/07 14:51:52
To enable connected mode, you put the line

Code: Select all

CONNECTED_MODE=yes
in ifcfg-ib0.
Thank you.

Do I then restart the network service and/or the interface for it to take effect or would I need to reboot the system for it to take effect?

Thank you.

chemal
Posts: 776
Joined: 2013/12/08 19:44:49

Re: Cannot change infiniband adapter from datagram mode to connected mode

Post by chemal » 2019/08/08 14:47:35

ifdown ib0 && ifup ib0 should be enough.

alpha754293
Posts: 69
Joined: 2019/07/29 16:15:14

Re: Cannot change infiniband adapter from datagram mode to connected mode

Post by alpha754293 » 2019/08/08 21:54:21

chemal wrote:
2019/08/08 14:47:35
ifdown ib0 && ifup ib0 should be enough.
Thank you.

alpha754293
Posts: 69
Joined: 2019/07/29 16:15:14

Re: Cannot change infiniband adapter from datagram mode to connected mode

Post by alpha754293 » 2019/08/10 02:48:08

chemal wrote:
2019/08/08 14:47:35
ifdown ib0 && ifup ib0 should be enough.
How would I be able to verify that it is actually running in connected mode vs. datagram mode?

Also, I was trying to optimize the MTU and I would change it in ifcfg-ib0, ifconfig ib0 down, ifconfig ib0 up, ifconfig ib0 and it would still show the default MTU for datagram mode.

The other thing that I am also noticing is that when I reboot the system, it no longer automatically assigns the IPv4 address to it whereas when it was running in datagram mode, it would.

Any ideas why I might be observing this behaviour?

Thank you.

chemal
Posts: 776
Joined: 2013/12/08 19:44:49

Re: Cannot change infiniband adapter from datagram mode to connected mode

Post by chemal » 2019/08/13 00:51:58

Are you using NetworkManager?

alpha754293
Posts: 69
Joined: 2019/07/29 16:15:14

Re: Cannot change infiniband adapter from datagram mode to connected mode

Post by alpha754293 » 2019/10/06 14:19:34

chemal wrote:
2019/08/13 00:51:58
Are you using NetworkManager?
To be honest -- I am not sure.

I've been editing the network-scripts file for the interface via /etc/sysconfig/network-scripts directly, so I'm not sure.

Post Reply