Glam-routing-problem

I'm in the process of configuring my new Raspberry Pi router 'glam'. I have a situation where I can ping my internet gateway 10.0.0.1 from 'glam', but if I try to ping the internet gateway 10.0.0.1 from my test host 'knowing' which is using 'glam' as its gateway I get 100% packet loss.

Update: this problem is solved! The problem was I was missing the masquerading directive for Netfilter, something like this:

$iptables -t nat -A POSTROUTING -o $BLUE -j MASQUERADE

IPv4 configuration on 'glam'

My router 'glam' is a Raspberry Pi:

root@glam:~# uname -a

Linux glam 6.1.0-rpi6-rpi-v8 #1 SMP PREEMPT Debian 1:6.1.58-1+rpt2 (2023-10-27) aarch64 GNU/Linux

IPv4 is configured like this:

root@glam:~# cat /etc/network/interfaces

# interfaces(5) file used by ifup(8) and ifdown(8)
# Include files from /etc/network/interfaces.d:
source /etc/network/interfaces.d/*

# 2023-11-06 jj5 - BLUE:
#
auto end0
iface end0 inet static
  address 10.0.0.5
  netmask 255.255.0.0
  gateway 10.0.0.1
  dns-nameservers 10.0.0.1

# 2023-11-06 jj5 - RED:
#
auto enx00e099001bf7
iface enx00e099001bf7 inet static
  address 10.1.0.5
  netmask 255.255.0.0

# 2023-11-06 jj5 - ORANGE:
#
auto enx8cae4cdd44a3
iface enx8cae4cdd44a3 inet static
  address 10.2.0.5
  netmask 255.255.0.0

# 2023-11-06 jj5 - GREEN:
#
auto enx8cae4cdd8e63
iface enx8cae4cdd8e63 inet static
  address 10.3.0.5
  netmask 255.255.0.0

root@glam:~# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: end0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e4:5f:01:81:89:01 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.5/16 brd 10.0.255.255 scope global end0
       valid_lft forever preferred_lft forever
3: enx8cae4cdd8e63: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 8c:ae:4c:dd:8e:63 brd ff:ff:ff:ff:ff:ff
    inet 10.3.0.5/16 brd 10.3.255.255 scope global enx8cae4cdd8e63
       valid_lft forever preferred_lft forever
4: enx8cae4cdd44a3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 8c:ae:4c:dd:44:a3 brd ff:ff:ff:ff:ff:ff
    inet 10.2.0.5/16 brd 10.2.255.255 scope global enx8cae4cdd44a3
       valid_lft forever preferred_lft forever
5: enx00e099001bf7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 00:e0:99:00:1b:f7 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.5/16 brd 10.1.255.255 scope global enx00e099001bf7
       valid_lft forever preferred_lft forever
6: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether e4:5f:01:81:89:02 brd ff:ff:ff:ff:ff:ff

root@glam:~# ip route

default via 10.0.0.1 dev end0 onlink 
10.0.0.0/16 dev end0 proto kernel scope link src 10.0.0.5 
10.1.0.0/16 dev enx00e099001bf7 proto kernel scope link src 10.1.0.5 
10.2.0.0/16 dev enx8cae4cdd44a3 proto kernel scope link src 10.2.0.5 
10.3.0.0/16 dev enx8cae4cdd8e63 proto kernel scope link src 10.3.0.5

I have IP forwarding enabled:

root@glam:/srv# sysctl net.ipv4.ip_forward

net.ipv4.ip_forward = 1

For testing purposes I configure Netfilter as per the following script. This ACCEPTs and LOGs all packets.

root@glam:/srv# cat iptables-log.sh

#!/bin/bash

POLICY=ACCEPT

iptables=/usr/sbin/iptables

# 2023-11-16 jj5 - begin by dropping all rules...
$iptables -F

# 2023-11-16 jj5 - apply default policy...
$iptables -P INPUT $POLICY
$iptables -P OUTPUT $POLICY
$iptables -P FORWARD $POLICY

$iptables -A INPUT   -j LOG --log-level warning --log-prefix "$POLICY INPUT: "
$iptables -A OUTPUT  -j LOG --log-level warning --log-prefix "$POLICY OUTPUT: "
$iptables -A FORWARD -j LOG --log-level warning --log-prefix "$POLICY FORWARD: "

As you can see I can ping the internet gateway 10.0.0.1 from 'glam':

root@glam:~# ping -c 3 10.0.0.1

PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.809 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.689 ms
64 bytes from 10.0.0.1: icmp_seq=3 ttl=64 time=0.667 ms

--- 10.0.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.667/0.721/0.809/0.062 ms

IPv4 configuration on 'knowing'

I have a test host 'knowing' which is configured to use 'glam' as its default gateway.

My test host 'knowing' is also a Raspberry Pi:

root@knowing:~# uname -a

Linux knowing 6.1.0-rpi6-rpi-v8 #1 SMP PREEMPT Debian 1:6.1.58-1+rpt2 (2023-10-27) aarch64 GNU/Linux

It's IPv4 config is like this:

root@knowing:~# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e4:5f:01:46:be:25 brd ff:ff:ff:ff:ff:ff
    inet 10.3.14.51/16 brd 10.3.255.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::4b0b:1972:f9d1:12e0/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether e4:5f:01:46:be:26 brd ff:ff:ff:ff:ff:ff

root@knowing:~# ip route

default via 10.3.0.5 dev eth0 proto static metric 100 
10.3.0.0/16 dev eth0 proto kernel scope link src 10.3.14.51 metric 100 

As you can see I can ping 'glam' from 'knowing':

root@knowing:~# ping -c 3 10.3.0.5

PING 10.3.0.5 (10.3.0.5) 56(84) bytes of data.
64 bytes from 10.3.0.5: icmp_seq=1 ttl=64 time=1.63 ms
64 bytes from 10.3.0.5: icmp_seq=2 ttl=64 time=1.59 ms
64 bytes from 10.3.0.5: icmp_seq=3 ttl=64 time=1.78 ms

--- 10.3.0.5 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.585/1.662/1.777/0.082 ms

The problem

Note that I can ping the internet gateway 10.0.0.1 from 'glam', as shown above. However when I try to ping the internet gateway 10.0.0.1 from 'knowing' I get 100% packet loss:

root@knowing:~# ping -c 3 10.0.0.1

PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.

--- 10.0.0.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2043ms

This is confusing because when I watch the logs on 'glam' I see that Netfilter has accepted the packets for forwarding:

jj5@glam:~ $ journalctl | grep SRC=10.3.14.51

Nov 17 11:59:07 glam kernel: ACCEPT FORWARD: IN=enx8cae4cdd8e63 OUT=end0 MAC=8c:ae:4c:dd:8e:63:e4:5f:01:46:be:25:08:00 SRC=10.3.14.51 DST=10.0.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=60461 DF PROTO=ICMP TYPE=8 CODE=0 ID=9 SEQ=1 
Nov 17 11:59:08 glam kernel: ACCEPT FORWARD: IN=enx8cae4cdd8e63 OUT=end0 MAC=8c:ae:4c:dd:8e:63:e4:5f:01:46:be:25:08:00 SRC=10.3.14.51 DST=10.0.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=60649 DF PROTO=ICMP TYPE=8 CODE=0 ID=9 SEQ=2 
Nov 17 11:59:09 glam kernel: ACCEPT FORWARD: IN=enx8cae4cdd8e63 OUT=end0 MAC=8c:ae:4c:dd:8e:63:e4:5f:01:46:be:25:08:00 SRC=10.3.14.51 DST=10.0.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=60890 DF PROTO=ICMP TYPE=8 CODE=0 ID=9 SEQ=3