Setting up NetFlow/IPFIX to log NAT translations on Debian 12

Русский · English · Español · Français · Deutsch

Practical Installation Guide ipt_NETFLOW with support for NAT Events, export of broadcast events and customization of the collector nfdump.

All IP addresses in the article are conditional. Public addresses are taken from RFC 5737 documentation ranges.

1. Introduction

When using Linux as a NAT gateway, especially for Internet providers with CGNAT, it is necessary to register the correspondence of internal addresses and ports of subscribers to public addresses and ports. Regular NetFlow takes into account network flows, but does not always allow NAT translation to be restored. For this purpose, NAT Event Logging (NEL) is used based on the events of the conntrack subsystem.

This tutorial uses Debian 12 kernel module ipt_NETFLOW, NetFlow v9/IPFIX and collector nfcapd/nfdump.

2. How NAT Event Logging works

Example of initial connection:

10.10.1.100:50004 -> 198.51.100.20:443

After SNAT:

203.0.113.10:61025 -> 198.51.100.20:443

To identify the subscriber, it is necessary to record the correspondence 203.0.113.10:61025 and 10.10.1.100:50004, protocol, destination address and exact time of the event. The port after NAT may differ from the original one.

Linux stores connection and NAT data in the Netfilter Connection Tracking subsystem (conntrack). ipt_NETFLOW can receive NAT mapping creation and deletion events and export them.

3. Architecture

Subscribers 10.10.0.0/16
        |
Debian 12 NAT gateway (192.0.2.1)
        | SNAT: 203.0.113.10
        +-----------------> Internet
        |
        +--- NetFlow/IPFIX UDP 555 ---> Collector 192.0.2.10
                                        nfcapd -> nfdump -> archive
Component Purpose
Netfilter NAT and packet filtering
conntrack Connection status accounting
ipt_NETFLOW Export streams and NAT Events
nfcapd NetFlow/IPFIX reception
nfdump Analysis and reading of the archive

4. IPFIX fields for NAT

IE Name Meaning
4 protocolIdentifier Protocol
7 sourceTransportPort Source port
8 sourceIPv4Address Source IPv4
11 destinationTransportPort Destination port
12 destinationIPv4Address IPv4 destination
225 postNATSourceIPv4Address Public IPv4 after SNAT
227 postNAPTSourceTransportPort Port after NAT
230 natEvent NAT Event Type

It is important to check the actual template being exported: the presence of NAT Events support does not in itself guarantee the presence of all IEs in one entry.

5. Preparing Debian 12

uname -r
grep CONFIG_NF_CONNTRACK_EVENTS /boot/config-$(uname -r)
grep -E 'CONFIG_NF_NAT=|CONFIG_NF_CONNTRACK=' /boot/config-$(uname -r)
ls -l /lib/modules/$(uname -r)/build

Expected event support: CONFIG_NF_CONNTRACK_EVENTS=y (or corresponding kernel configuration available). Installing tools:

apt update
apt install -y git build-essential dkms pkg-config libxtables-dev linux-headers-$(uname -r)

If the running kernel headers have already been removed from the current repositories, the corresponding packages can be searched in Debian Snapshot Archive. The header and core versions must match.

6. Building ipt_NETFLOW with NAT Events

Source code ipt_NETFLOW

cd /usr/src
git clone https://github.com/aabc/ipt-netflow.git
cd ipt-netflow
./configure --help

In the tested configuration, the call helped:

./configure --enable-natevents

In the version shown --enable-natevents includes support for NAT Events. Check available keys and build result for the selected version.

make -j$(nproc)
make install
depmod -a

In case of error DKMS tree already contains check the registered versions and installed files:

dkms status
find /lib/modules/$(uname -r) -name 'ipt_NETFLOW.ko*'
modinfo -n ipt_NETFLOW

Do not delete the entire DKMS tree without checking. If a signature error appears signing_key.pem, check the Secure Boot mode and module signing requirements.

Loading the module:

modprobe ipt_NETFLOW
lsmod | grep ipt_NETFLOW
ls /proc/sys/net/netflow/
sysctl net.netflow.natevents

Criteria for the presence of NAT Event Logging: in /proc/sys/net/netflow/ parameter should appear natevents. If the module was rebuilt while already loaded, a correct reload of the module will be required. modinfo shows the module on disk, not a guaranteed loaded instance.

7. Export setup

For NetFlow v9:

sysctl -w net.netflow.protocol=9
sysctl -w net.netflow.destination=192.0.2.10:555
sysctl -w net.netflow.natevents=1

For IPFIX:

sysctl -w net.netflow.protocol=10

Check:

sysctl net.netflow.protocol
sysctl net.netflow.destination
sysctl net.netflow.natevents

Constant parameters of the module in /etc/modprobe.d/ipt_NETFLOW.conf:

options ipt_NETFLOW destination=192.0.2.10:555 protocol=10 natevents=1

They will take effect the next time the module is loaded.

8. Are iptables rules necessary?

For NAT Events separate rule -j NETFLOW is not required: events come from conntrack. If you also need statistics for regular threads, you can add:

iptables -I FORWARD 1 -j NETFLOW
iptables -L FORWARD -nvx --line-numbers

Should not be added NETFLOW in several chains without assessing the risk of packet re-counting. No need to add NETFLOW to the NAT table to receive NAT Events.

9. conntrack diagnostics

apt install conntrack
conntrack -L -s 10.10.1.100
conntrack -E -o timestamp,extended
conntrack -E -s 10.10.1.100

Conditional notation:

tcp 6 431999 ESTABLISHED
src=10.10.1.100 dst=198.51.100.20 sport=50004 dport=443
src=198.51.100.20 dst=203.0.113.10 sport=443 dport=61025

The reverse direction shows the public address and assigned port.

10. Statistics ipt_NETFLOW

cat /proc/net/stat/ipt_netflow
watch -n 1 cat /proc/net/stat/ipt_netflow

Depending on the version, statistics contain NAT Events counters (start, stop), export and error parameters. Increasing the counters confirms that the module has processed events, but not delivered all records to the collector.

11. Installing nfdump 1.7.x

nfdump source code

apt update
apt install -y git build-essential autoconf automake libtool pkg-config flex bison libbz2-dev zlib1g-dev liblz4-dev
cd /usr/local/src
git clone https://github.com/phaag/nfdump.git
cd nfdump
git tag --list '*1.7*'

For a working system, select a verified tag and set it in a variable NFDUMP_TAG.

: "${NFDUMP_TAG:?Set NFDUMP_TAG to a verified release tag}"
git checkout --detach "$NFDUMP_TAG"
./autogen.sh
./configure
make -j$(nproc)
make install
ldconfig
/usr/local/bin/nfdump -V
/usr/local/bin/nfcapd -V

If the program does not find libnfdump*.so, check ldd /usr/local/bin/nfdump, library path and configuration ldconfig.

12. Starting the collector

mkdir -p /mnt/flow/nat
/usr/local/bin/nfcapd -w /mnt/flow/nat -p 555 -t 60

For two exporters with division by catalogs (check the syntax at nfcapd -h your version):

/usr/local/bin/nfcapd -p 555 -t 60 -S 1 \
  -n gw1,192.0.2.1,/mnt/flow/1 \
  -n gw2,192.0.2.2,/mnt/flow/2

13. Checking receipt and decoding

tcpdump -ni any udp port 555
find /mnt/flow/nat -type f -name 'nfcapd.*'
/usr/local/bin/nfdump -R /mnt/flow/nat -o raw
/usr/local/bin/nfdump -R /mnt/flow/nat -o 'fmt:%ts %sa:%sp -> %da:%dp %pr'

The latter format is intended primarily for regular streams. NAT Events may appear differently.

To check the real IPFIX Template, capture the packets:

tcpdump -ni any -s0 -w /tmp/nat-ipfix.pcap udp port 555
tshark -r /tmp/nat-ipfix.pcap -V

Check IE 4, 7, 8, 11, 12, 225, 227, 230, as well as real Data Records. Example of logical post content:

Protocol: TCP
Original Source: 10.10.1.100:50004
Destination: 198.51.100.20:443
Translated Source: 203.0.113.10:61025
NAT Event: CREATE

This is an illustration, not a literal output format nfdump.

14. Common mistakes

Symptom What to check
No natevents configure flags, loaded module version, kernel event support
CONFIG_NF_NAT_NEEDED... No This is a test of an old macro, not proof of the absence of NAT
DKMS tree already contains DKMS registered assemblies conflict
Error signing_key.pem Module Signing and Secure Boot
UDP is not received Routes, firewall, destination, exporter meters
UDP is received, but there are no NAT entries natevents, event counters, IPFIX Template, decoding

Checking the route to the collector:

ip route get 192.0.2.10

You should not force define old kernel macros without checking the sources and API compatibility.

15. Performance

Logging load is determined by the number of NAT mappings created, and not just by the speed of the traffic. With 5,000 creations and 5,000 deletions per second, there will be 10,000 events/s, that is, 864 million events per day under constant load.

Recommendations:

  1. Use a separate NAT event collector.
  2. Monitor UDP loss and buffer overflow.
  3. Synchronize gateway and collector clocks.
  4. Control disk space and storage life.
  5. Limit access to subscriber activity logs.
  6. Check operation after updating the kernel and rebooting.
  7. Under load, check the completeness of records and the correctness of the IPFIX Template.

16. Automatic start

File /etc/modules-load.d/ipt_NETFLOW.conf:

ipt_NETFLOW

File /etc/modprobe.d/ipt_NETFLOW.conf:

options ipt_NETFLOW destination=192.0.2.10:555 protocol=10 natevents=1

Example of a systemd collector service /etc/systemd/system/nfcapd-nat.service:

[Unit]
Description=NetFlow/IPFIX Collector
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/nfcapd -w /mnt/flow/nat -p 555 -t 60
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl enable --now nfcapd-nat
systemctl status nfcapd-nat
journalctl -u nfcapd-nat -f

17. Checklist

Conclusion

ipt_NETFLOW allows you to export NAT events directly from the Linux gateway without mirroring all traffic. But three levels of readiness should be distinguished: the module is loaded; NAT Events are generated; the collector accepts and correctly decodes the required fields. To fully account for NAT, you need to check all three levels and the completeness of events under load.

Sources