Problem with 2 phys interfaces -> teamed -> vlan -> bridges

[langesmalle]

Hi,

I have a situation where I would like to combine 2 physical interfaces into a teamed interface (802.3ad mode (lacp)). Followed by a bridged over a vlan linked to the teamed interface:
The configuration I made so far results in all required interfaces in "ip a", however, the brigde interface on which the static IP is configured got the status down:
#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp32s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:10:e0:3c:46:08 brd ff:ff:ff:ff:ff:ff
inet XXX.XXX.XXX.XXX/24 brd 172.16.101.255 scope global enp32s0f0
valid_lft forever preferred_lft forever
inet6 fe80::210:e0ff:fe3c:4608/64 scope link
valid_lft forever preferred_lft forever
3: enp32s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
link/ether 00:10:e0:3c:46:09 brd ff:ff:ff:ff:ff:ff
4: enp51s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master team0 state UP qlen 1000
link/ether 00:c0:dd:25:6b:3c brd ff:ff:ff:ff:ff:ff
5: enp51s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 00:c0:dd:25:6b:3d brd ff:ff:ff:ff:ff:ff
6: enp147s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master team0 state UP qlen 1000
link/ether 00:c0:dd:25:6b:3c brd ff:ff:ff:ff:ff:ff
7: enp147s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 00:c0:dd:25:5e:15 brd ff:ff:ff:ff:ff:ff
8: brcloudstor: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 26:11:54:68:93:fc brd ff:ff:ff:ff:ff:ff
inet YYY.YYY.YYY.YYY/23 brd 10.16.5.255 scope global brcloudstor
valid_lft forever preferred_lft forever
9: team0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 00:c0:dd:25:6b:3c brd ff:ff:ff:ff:ff:ff
inet6 fe80::2c0:ddff:fe25:6b3c/64 scope link
valid_lft forever preferred_lft forever
10: vlan982@team0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 42:7d:84:9a:19:55 brd ff:ff:ff:ff:ff:ff


I also see following message in /var/log/messages:

Jul 25 17:00:22 dc2-kvm01 NetworkManager[2572]: <info> Auto-activating connection 'vlan982'.
Jul 25 17:00:22 dc2-kvm01 NetworkManager[2572]: <warn> Failed to activate 'vlan982': The master connection was not compatible

Anyone who have any experiece with this kind of network-setup?

My config so far:

# cat ifcfg-enp147s0f0
BOOTPROTO=none
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=enp147s0f0
UUID=dee1133b-20a7-4092-b9ea-b5ad750f5226
DEVICE=enp147s0f0
ONBOOT=yes
TEAM_MASTER=d52cd8ce-cf5b-4714-8a1c-880ca909b45a
DEVICETYPE=TeamPort

# cat ifcfg-enp51s0f0
BOOTPROTO=none
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=enp51s0f0
UUID=e2ba89fc-5074-468d-a49c-51863b566c61
DEVICE=enp51s0f0
ONBOOT=yes
TEAM_MASTER=d52cd8ce-cf5b-4714-8a1c-880ca909b45a
DEVICETYPE=TeamPort

# cat ifcfg-team0
DEVICE=team0
DEVICETYPE=Team
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
NAME=team0
UUID=d52cd8ce-cf5b-4714-8a1c-880ca909b45a
ONBOOT=yes
TEAM_CONFIG="{ \"device\": \"team0\", \"runner\": { \"name\": \"lacp\", \"active\": true, \"fast_rate\": true, \"tx_hash\": [\"eth\", \"ipv4\", \"ipv6\"] }, \"link_watch\": {\"name\": \"ethtool\"}, \"ports\": {\"enp51s0f0\": {}, \"enp147s0f0\": {}}}"

# cat ifcfg-vlan982
VLAN=yes
TYPE=Vlan
DEVICE=vlan982
PHYSDEV=team0
MASTER=d7f94d8a-06e3-4573-8be4-4af85cf40800
VLAN_ID=982
REORDER_HDR=0
NAME=vlan982
UUID=5265e229-0e5f-4e5c-b495-0edf8d22949c
ONBOOT=yes
BRIDGE=d7f94d8a-06e3-4573-8be4-4af85cf40800

# cat ifcfg-brcloudstor
DEVICE=brcloudstor
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
BOOTPROTO=none
IPADDR0=10.16.4.101
PREFIX0=23
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=brcloudstor
UUID=d7f94d8a-06e3-4573-8be4-4af85cf40800
ONBOOT=yes

[langesmalle]

I found the solution myself.

For those interested in it, see below.

It was all caused by the NetworkManager.
I disabled it in order to bring up the interfaces, teaming, vlan and bridge up by the default network script (/etc/inint.d/network).
Only changes that were required are:

* interafces: ONBOOT=YES changed to NO -> Teaming requires this and will bring up the physical interfaces with in the script ifup-Team.
* UUID=xxxxx -> al remarked. Were created by the NetworkManager and are no more necessary.
* In vlan-config refer to the bridge-name in stead of bridge UUID
* in the physical interface-config, refer to the team-master-name in stead of the team UUID
* for all interface config -> added NM_CONTROLLED=no

So final working config is, even wit hMTU=9000:

# cat ifcfg-enp147s0f0
BOOTPROTO=none
NM_CONTROLLED=no
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=enp147s0f0
DEVICE=enp147s0f0
ONBOOT=no
TEAM_MASTER=team0
DEVICETYPE=TeamPort
MTU=9000


# cat ifcfg-enp51s0f0
BOOTPROTO=none
NM_CONTROLLED=no
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=enp51s0f0
DEVICE=enp51s0f0
ONBOOT=no
TEAM_MASTER=team0
DEVICETYPE=TeamPort
MTU=9000

# cat ifcfg-team0
DEVICE=team0
NM_CONTROLLED=no
DEVICETYPE=Team
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
NAME=team0
ONBOOT=yes
TEAM_CONFIG="{ \"device\": \"team0\", \"runner\": { \"name\": \"lacp\", \"active\": true, \"fast_rate\": true, \"tx_hash\": [\"eth\", \"ipv4\", \"ipv6\"] }, \"link_watch\": {\"name\": \"ethtool\"}, \"ports\": {\"enp51s0f0\": {}, \"enp147s0f0\": {}}}"
MTU=9000

# cat ifcfg-vlan982
VLAN=yes
NM_CONTROLLED=no
TYPE=Vlan
DEVICE=vlan982
PHYSDEV=team0
MASTER=cloudbrstor
VLAN_ID=982
REORDER_HDR=0
NAME=vlan982
ONBOOT=yes
BRIDGE=cloudbrstor
MTU=9000

# cat ifcfg-cloudbrstor
DEVICE=cloudbrstor
NM_CONTROLLED=no
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
BOOTPROTO=none
DELAY=5
IPADDR=xxx.xxx.xxx.xxx
PREFIX=23
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=cloudbrstor
ONBOOT=yes
MTU=9000