Concept notes · Networking

Packets, state and translation

What a Linux router does to a packet between arriving and leaving: size, state, translation and table choice.

Addresses below are documentation and private ranges; substitute your own. Most commands need root, so prefix them with sudo.

A stateful firewall passes replies it remembers

Linux connection tracking (conntrack) records each flow when its first packet passes: the original direction (source, destination, ports) and the reply it expects. nftables rules can match that record with ct state: new for a packet starting a flow, established for packets of a flow already seen in both directions, related for traffic tied to a tracked flow, such as an ICMP error about it, and invalid for packets that fit no flow.

A strict forward chain has policy drop, one early rule accepting ct state established,related, then narrow ct state new rules for the connections that may start. Replies pass because this firewall saw the request, not because a rule names the reply direction.

How it fails. The request leaves through the firewall and the answer arrives back at it, then goes no further. The client reports a timeout, not a refusal. A traceroute shows the firewall’s hop and then only *.

Observe it.

nft -a list ruleset

Read each chain’s hook, priority and policy, then the rules in order. -a prints each rule’s handle, which nft insert, nft replace and nft delete use to place, change or remove one rule.

conntrack -L -p tcp --dport 8080

One line per flow: protocol name and number, seconds left, TCP state, the original tuple, then the reply tuple. [ASSURED] appears once traffic has flowed both ways. An entry stuck in SYN_SENT or SYN_RECV never completed its handshake.

tcpdump -ni eth1 'tcp port 8080'

Capture on each side of the firewall at once. A reply (Flags [S.]) seen arriving on one interface and never leaving the other was discarded inside the firewall.

What good looks like. Replies are matched as established and their entries become [ASSURED]. The rules list only who may start a connection, and a connection attempt from the protected side that no rule allows is still dropped.

Source NAT: leaving as the router

Private ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) are not routed on the internet, and providers drop packets that carry them as a source. A router doing source NAT rewrites the source address, and the port if it must, of connections leaving through its outside interface. In nftables this lives in a type nat hook postrouting chain: masquerade uses the outgoing interface’s current address; snat to names one.

The decision is made once, on a connection’s first packet, and stored in its conntrack entry. Every later packet, and every reply, is rewritten from that entry: replies arrive addressed to the router, which maps them back to the inside host. A rule change therefore applies to new connections only. Existing entries keep the decision they were created with until they expire or are removed with conntrack -D.

How it fails. The router itself reaches an outside service, while hosts behind it time out. Or some inside clients work and others fail, depending on when their entries were created.

Observe it.

nft list table ip nat

Read the postrouting chain: which outgoing interface or source each rule matches, and what it translates to.

conntrack -L -d 192.0.2.20

Compare the two tuples. Translated: src=172.16.1.10 dst=192.0.2.20 … src=192.0.2.20 dst=192.0.2.1, so the reply is expected at the router’s outside address 192.0.2.1. If the reply tuple names the inside address itself, the connection left untranslated.

tcpdump -ni eth1 host 192.0.2.20

Capture on the outside interface. Every packet should carry the router’s outside address as its source.

What good looks like. Outside captures show only the router’s address. Reply tuples point at the router. New connections from every inside host succeed, including connections started after the last rule change.

Destination NAT: publishing an inside server

Destination NAT rewrites where an arriving connection is going. A published address and port on the router’s outside become an inside server’s address and port. In nftables it lives in a type nat hook prerouting chain, for example ip daddr 192.0.2.1 tcp dport 8080 dnat to 172.16.1.10:8080. Prerouting runs before the routing decision, so the router routes the rewritten packet towards the inside server.

As with source NAT, the decision is taken on the first packet and stored in conntrack. The server’s replies are rewritten back, so the client only ever sees the published address. Matching address, protocol and port together publishes one service and nothing else on the router.

How it fails. Outside clients time out on the published address, while people inside still reach the server directly at its own address. The router accepted the connection; what happens after the rewrite decides the outcome.

Observe it.

nft -a list table ip nat

Read the prerouting rules: what each matches and where it sends the connection. Handles let you replace or delete one rule.

conntrack -L -p tcp --dport 8080

A translated flow has tuples that disagree: src=192.0.2.20 dst=192.0.2.1 … src=172.16.1.10 dst=192.0.2.20. The original destination is the published address; the reply source is the inside server. [UNREPLIED] means nothing answered at the rewritten destination.

tcpdump -ni eth0 'tcp port 8080'

Capture on the inside interface: do rewritten SYNs leave towards the server, and does a SYN-ACK come back?

What good looks like. Entries for the published port become [ASSURED] with the intended server as the reply source. Only the intended port is published; other ports on the router’s outside address are not forwarded anywhere.

More than one routing table

Linux keeps several routing tables: local (255), main (254), default (253), and any you number yourself. The routing policy database, read with ip rule, decides which table a packet consults. Each rule has a preference number, a selector (source prefix, incoming interface, firewall mark, and more) and an action, usually lookup TABLE.

Rules are tried from the lowest preference upward. When a rule’s selector matches and its table holds a route for the destination, that route is used. When the table has no matching route, the search continues with the next rule. The default rules are 0: lookup local, 32766: lookup main and 32767: lookup default, so any rule you add with a lower number is consulted before main. A common use: traffic from one source range leaves by a second line.

How it fails. One group of hosts behaves differently from the rest towards the same destination: it leaves by an unexpected line, gets “No route to host”, or its replies return on a different interface. Meanwhile ip route looks correct, because it shows only main.

Observe it.

ip rule show

Each line is preference: selector lookup table, in the order the kernel tries them. Check every selector against the real source ranges.

ip route show table 100

Lists exactly what one table holds. Plain ip route shows main only.

ip route get 203.0.113.9 from 172.16.1.10 iif eth0

Asks the kernel how it would route a packet from that source arriving on that interface, which is what forwarding does. The answer names the table when it is not main: … via 198.51.100.42 dev eth2 table 100. Without from and iif you get the decision for the router’s own traffic instead.

What good looks like. For a sample source from each group, ip route get names the intended table and outgoing device, and a source outside every rule still resolves through main.

References