Hello,
I have a server with 3 NICs.
NIC1: ISP1 GW1
NIC2: ISP2 GW2
NIC3: Intranet
I am unable to get traffic to return through the same NIC.
Let's take the case of NIC2.
I have used policy routing as such:
table 2
default via <GW2> dev ens1 src <IP2>
default via <GW2> dev ens1 proto static metric 100
<IP2range(.0/24)> dev ens1 proto kernel scope link src <IP2> metric 100
rules:
0: from all lookup local
32765: from all to <IP2> lookup table 2
32766: from all lookup main
32767: from all lookup default
I used tcpdump and the request to IP2 gets through, but ens1 never replies to it.
However, all requests to IP1 work perfectly. In fact if I change the default gateway to that of ISP2 / GW2, then all requests to IP2 start working and those to IP1 fail.
The ports I am working with are open, so what gives?
Any other ideas how I could make this work?
Returning traffic through the same NIC
Re: Returning traffic through the same NIC
Your rules do not look ok.
rule-<NIC1>:
from <IP-of-NIC1> table 2
route-<NIC1>:
default table 2 via <GW1>
rule-<NIC2>:
from <IP-of-NIC2> table 3
route-<NIC2>:
default table 3 via <GW3>
The point is that when your server replies to something that arrived via NIC1, the incoming packet had IP-of-NIC1 as destination and the reply will have IP-of-NIC1 as aa source. Therefore, our rule must say that if the outgoing packet has IP-of-NIC1 as a source, then use table 2 to do the routing decision. The table 2 has at least "default route" that sets the next-hop to be GW1. A packet destined to GW1 obviously has to be send via NIC1.
Why does the other path work for you currently? All replies are sent via the default gateway. If they are replies to packets that arrived via that same gateway, they are ok.
rule-<NIC1>:
from <IP-of-NIC1> table 2
route-<NIC1>:
default table 2 via <GW1>
rule-<NIC2>:
from <IP-of-NIC2> table 3
route-<NIC2>:
default table 3 via <GW3>
The point is that when your server replies to something that arrived via NIC1, the incoming packet had IP-of-NIC1 as destination and the reply will have IP-of-NIC1 as aa source. Therefore, our rule must say that if the outgoing packet has IP-of-NIC1 as a source, then use table 2 to do the routing decision. The table 2 has at least "default route" that sets the next-hop to be GW1. A packet destined to GW1 obviously has to be send via NIC1.
Why does the other path work for you currently? All replies are sent via the default gateway. If they are replies to packets that arrived via that same gateway, they are ok.
Re: Returning traffic through the same NIC
Yes! It is partially working. Here is my setup:
ip route show table 1
default via <GW1> dev ens1
<IPRANGE-NIC1> dev ens1 scope link src <IP-NIC1>
ip route show table 2
default via <GW2> dev ens2
<IPRANGE-NIC2> dev ens2 scope link src <IP-NIC2>
ip route show
default via <GW1> dev ens2 proto static metric 100
default via <GW2> dev ens1 proto static metric 101
<IPRANGE-NIC1> dev ens2 proto kernel scope link src <IP-NIC1> metric 100
<IPRANGE-NIC3> dev enp4s0f0 proto kernel scope link src <IP-NIC3> metric 100
<IPRANGE-NIC4> dev enp3s0f0 proto kernel scope link src <IP-NIC4> metric 100
<IPRANGE-NIC2> dev ens1 proto kernel scope link src <IP-NIC2> metric 100
ip rule show
0: from all lookup local
32764: from <IP-NIC2> lookup table 2
32765: from <IP-NIC1> lookup table 1
32766: from all lookup main
32767: from all lookup default
So what is not working?
I have an IP tables rule to forward certain ports from <NIC1> or <NIC2> to an IP on the <NIC3> network. Unfortunately those ports that are forwarded can only be accessed through one <NIC1> or <NIC2> depending on which default GW is set. Any solutions for this ? I suspect the only way would be to MARK in iptables the packets depending on which NIC they are coming from and than return them through the appropriate gateway. However, I would very much prefer a simpler solution.
ip route show table 1
default via <GW1> dev ens1
<IPRANGE-NIC1> dev ens1 scope link src <IP-NIC1>
ip route show table 2
default via <GW2> dev ens2
<IPRANGE-NIC2> dev ens2 scope link src <IP-NIC2>
ip route show
default via <GW1> dev ens2 proto static metric 100
default via <GW2> dev ens1 proto static metric 101
<IPRANGE-NIC1> dev ens2 proto kernel scope link src <IP-NIC1> metric 100
<IPRANGE-NIC3> dev enp4s0f0 proto kernel scope link src <IP-NIC3> metric 100
<IPRANGE-NIC4> dev enp3s0f0 proto kernel scope link src <IP-NIC4> metric 100
<IPRANGE-NIC2> dev ens1 proto kernel scope link src <IP-NIC2> metric 100
ip rule show
0: from all lookup local
32764: from <IP-NIC2> lookup table 2
32765: from <IP-NIC1> lookup table 1
32766: from all lookup main
32767: from all lookup default
So what is not working?
I have an IP tables rule to forward certain ports from <NIC1> or <NIC2> to an IP on the <NIC3> network. Unfortunately those ports that are forwarded can only be accessed through one <NIC1> or <NIC2> depending on which default GW is set. Any solutions for this ? I suspect the only way would be to MARK in iptables the packets depending on which NIC they are coming from and than return them through the appropriate gateway. However, I would very much prefer a simpler solution.
Re: Returning traffic through the same NIC
I have been struggling with this problem for 2 weeks now.
I monitor packets with tcpdump and I simply fail to understand what is wrong.
I set GW1 as default gateway, which is assigned to NIC1.
All traffic through NIC1 is working correctly.
All incoming traffic to NIC2 should also be forwarded to internal IPs, including the internal IP of the server which is on NIC3 (internal network) and be returned through the GW2 assigned through the routing rules I've mentioned above..
However all traffic that reaches NIC2, dies in NIC2 and is not sent forward to NIC3.
What gives?
I monitor packets with tcpdump and I simply fail to understand what is wrong.
I set GW1 as default gateway, which is assigned to NIC1.
All traffic through NIC1 is working correctly.
All incoming traffic to NIC2 should also be forwarded to internal IPs, including the internal IP of the server which is on NIC3 (internal network) and be returned through the GW2 assigned through the routing rules I've mentioned above..
However all traffic that reaches NIC2, dies in NIC2 and is not sent forward to NIC3.
What gives?