[SOLVED] Weird route at startup both with NetworkManager and without

[giuliano]

I'm rather confused. My setup is a router which has two network interfaces, eth0 to connect to an external modem/router, and eth1 to connect to the internal LAN. It's been working for many years with just an external modem, but recently my provider has replaced it with a modem/router. I set up the external router for a DMZ, i.e. let everything pass through, and configured properly the eth0 interface. I played a little to have it working, and finally everything was OK. ip route show was telling the proper things:

[root@router ~]# ip route show
default via 192.168.1.254 dev eth0 proto dhcp metric 100
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.51 metric 100
192.168.2.0/24 dev eth1 proto kernel scope link src 192.168.2.9 metric 101

eth0 gets the address 192.168.1.51 from the modem/router via DHCP, on subnet 192.168.1.0 while subnet 192.168.2.0 is the local LAN subnet.
All's fine until I do reboot.
At reboot I'm unable to access the network, and ip route shows many lines but doesn't show a path toward the modem/rooter. This happens both using NetworkManager and disabling NetworkManager and using just network.service.
But if, in both cases, I switch off both interfaces, (ifdown eth1 - ifdown eth0), and then switch them on (ifup eth0 - ifup eth1) everything goes in place, routing is correct and WAN becomes reachable.
I have fixed the problem by adding a service which runs at the end of startup, and does exactly that, bring down both interfaces, wait a few seconds, brings up eth0 to get the proper connection to the external modem/router, waits two more seconds and brings up eth1 for the LAN.
What I'm doing wrong which has forced me to resort to that kludge? Any ideas?
Giuliano

[jlehtone]

The assumption is that eth0 gets its config from DHCP, be it in the modem or beyond.
A question is, what does the DHCP offer? If you use NetworkManager, then its logs and/or
nmcli con show $conname
should show all that was received.

[TrevorH]

Don't specify GATEWAY= in the ifcfg file for the interface that should not be the default route.

[giuliano]

The modem/router providing the DHCP server has a fixed address for device eth0 based on its MAC address. Actually in a working condition (i.e. after ifdown - ifup) nmcli shows:
....
802-3-ethernet.auto-negotiate: no
802-3-ethernet.mac-address: 00:00:5A:11:68:4D
802-3-ethernet.cloned-mac-address: --
802-3-ethernet.generate-mac-address-mask:--
....
DHCP4.OPTION[6]: domain_name_servers = 192.168.1.254
DHCP4.OPTION[7]: expiry = 1708353430
DHCP4.OPTION[8]: ip_address = 192.168.1.51
DHCP4.OPTION[9]: network_number = 192.168.1.0
DHCP4.OPTION[10]: next_server = 0.0.0.0
....
192.168.1.51 is the expected IP address.

But a doubt comes to my mind. Does the device eth0 already have received a valid address when eth1 is started? Possibly my kludge has only the effect of delaying start of eth1 after eth0 has been completely set up. Actually ifup returns only after a valid address has been received, if the boot proto is DHCP.
Does it make sense? If such is the case, there is there a way to avoid that network.service/NetworkManager do bring up the two interfaces simultaneously?

[giuliano]

Don't specify GATEWAY= in the ifcfg file for the interface that should not be the default route.

ifcfg-eth1 has:
DEFROUTE=no
and no GATEWAY line.

The issue must be elsewhere.

[jlehtone]

What if you disable eth1 and boot. Does the eth0 get things "right" on (every) boot?

[giuliano]

What if you disable eth1 and boot. Does the eth0 get things "right" on (every) boot?

No, it doesn't.
NetworkManager detects the presence of eth1 and enables it nonetheless. And ip route show tells weird things:

default via 192.168.0.1 dev eth0
default via 192.168.1.254 dev eth0 proto dhcp metric 100
169.254.0.0/16 dev eth0 scope link metric 1002
169.254.0.0/16 dev eth1 scope link metric 1003
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.254
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.9
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.51 metric 100
192.168.2.0/24 dev eth1 proto kernel scope link src 192.168.2.9 metric 101

a default gateway on the non-existing subnet 192.168.0.0
a 169.254.0.0 subnet (virbr0?)
But with the sequence ifdown eth1- ifdown eth0 - ifup eth0 - ifup eth1 everything goes in place, and ip route show tells:

default via 192.168.1.254 dev eth0 proto dhcp metric 100
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.51 metric 100
192.168.2.0/24 dev eth1 proto kernel scope link src 192.168.2.9 metric 101

??????

[giuliano]

For sake of completeness, reversing the order of ifup’s doesn’t change the resulting (correct) routing.

[jlehtone]

What if you disable eth1 and boot. Does the eth0 get things "right" on (every) boot?

No, it doesn't.
NetworkManager detects the presence of eth1 and enables it nonetheless.

How did you disable?

Code: Select all

nmcli con mod eth1 connection.autoconnect no

Yes, NetworkManager does by default generate on the fly connections for all interfaces.
Package NetworkManager-config-server changes that behaviour:

Description :
This adds a NetworkManager configuration file to make it behave more like the old "network" service. In particular, it stops NetworkManager from automatically running DHCP on unconfigured ethernet devices, and allows connections with static IP addresses to be brought up even on ethernet devices with no carrier.

This package is intended to be installed by default for server deployments.

default via 192.168.0.1 dev eth0
default via 192.168.1.254 dev eth0 proto dhcp metric 100
169.254.0.0/16 dev eth0 scope link metric 1002
169.254.0.0/16 dev eth1 scope link metric 1003
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.254
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.51 metric 100

a default gateway on the non-existing subnet 192.168.0.0
a 169.254.0.0 subnet (virbr0?)

The 169.254.0.0/16 is link-local address range https://en.wikipedia.org/wiki/Link-local_address that initscripts used to call "zeroconf".
The virbr0 you would see, if it would exist. IIRC, it defaults to 192.168.122.0/24.


Is it possible that you have multiple configuration files for eth0 -- some for NM, some for initscripts -- that all tell something to kernel on boot?
If so, when you later explicitly take individual connections down and up, you don't take them all up like the boot does.

I would clean all (eth0-related) out and create fresh. Only one -- no mixing of NM and initscripts.

[giuliano]

Is it possible that you have multiple configuration files for eth0 -- some for NM, some for initscripts -- that all tell something to kernel on boot?
If so, when you later explicitly take individual connections down and up, you don't take them all up like the boot does.

THAT'S IT!
I don't have multiple configurations for eth0, but I checked network-script and to my dismay I discovered that I had multiple configurations for the same physical interface: enp2s0 and eth0, enp3s0 and eth1. After installation I changed to the familiar ethx naming, and I wasn't aware that the old naming (generated by NM, I think) was still present in network-scripts, and were used at start-up. I removed the enpxx files, and the weird routing entries have disappeared. No more need for my FixRoute service!
Thanks a lot for your time!
Giuliano