Iptables firewall rules - NEW TCP connections and the SYN bit. WWW.Smythies.com

The problem: Almost everywhere one searchs for information on the use of the --tcp-flags and --syn extentions to the --protocol tcp or -p tcp check, there is incorrect or misleading information. In particular, the NOT [!] condition is not described well as to what will lead to a TRUE or FALSE result for the expression.

A typical iptables application is to detect and DROP (or log and DROP) NEW TCP connection attempts where the SYN bit is not set. (And these connections are typically not new at all but rather new packets for old connections that have been forgotten.)

Detail:
At least the current version of the man pages for iptables decribes the conditions of the included example correctly, whereas various references on internet are often missing a flag (not always the same one) in the text description. For example (from the man pages):

iptables -A FORWARD -p tcp --tcp-flags SYN,ACK,FIN,RST SYN

Will only match packets with the SYN flag set, and the ACK, FIN and RST flags unset.
O.K. that makes sense and the URG and PSH flags are don't care bits.
Now, the "--syn" extension is equivalent to "--tcp-flags SYN,RST,ACK,FIN SYN", so that part seems simple enough.

Now consider: If the "!" flag precedes the "--syn", the sense of the option is inverted.

What does that actually mean? Does it mean that the SYN bit needs to be reset while all 3 of RST,ACK,FIN need to be set? No.
It merely means NOT --syn, meaning it will only be FALSE when --syn would have been TRUE and it will only be TRUE when --syn would have been FALSE. For example, there are many conditions where the SYN bit can be set but the --syn condition can still be FALSE.



So what is the correct iptables rule to unambiguously perform an incorrect NEW TCP connection check? Virtually all of the references one can find suggest that the rule should similar to:

iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP

Now, the state would never be NEW if the RST or FIN bits were set, so the truth table would be reduced somewhat.

References:
Linuxtopia Firewall Iptables - State NEW but no SYN bit set.