Linux uses 64-bit jiffies now, you can get higher uptimes :)

My term “this more_than_LONG_MAX_jiffies problem” was rather unclear.
The ipt_recent code before his rewrite had several problems related to jiffies values.
With the rewrite, a problem

> if the list of last hits contained empty slots and jiffies is greater than LONG_MAX

seems to be fixed, but another one

> if one of the last hits was more than LONG_MAX jiffies ago

looks not to be fixed yet.
Do you mean the latter fixed too?

The patch that is provided by Juergen is to fix the problem in
the kernel, hence you have to apply the patch to your kernel
and then rebuild the kernel.

To apply the patch refer to Juergen’s comment above, on
October 15th, 2006 at 2:06 pm

I’m running redhat EL 4 with kernel 2.6.9-22.ELsmp

I tryed rebooting and doing the test with in 5 minutes, the rules seem to be working correctly as far as I can tell. Is there any other way to test? or am I safe?
aslo, can some one post on how to actually apply the patch? do i have to recompile the kernel? or is this something applied when i compile iptables?

thanks for your time

BR

J

There is a good news. Just now i was able to reproduce the bug on my i386 box. The results are as follows.

Unpatched kernel: telnetting to the port 1234 for the second time.
The ipt_recent rule kicked in and DROPed the connection request.

Patched kernel: telnet to the port 1234 for the 7th time.
For 6 times the connection refused since we have our hit_count as 6 and for the 7th time the ipt_recent rule kicked in and DROPed the connection request.

Thank you very much for the patch and the iptables rules.

# iptables -P OUTPUT DROP

After this any command execution gives the error and this output for ls.

# ls
RPC: sendmsg returned error 1
nfs: RPC call returned error 1
ash: ls: Operation not permitted

So i feel there is no problem in my iptables setup.

1. Immediately after the setup ive executed the script.
2. After the setup i rebooted the system and executed the script.

There is no difference in the output :(

Did you try running your script again with that setup?

Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT 0 — anywhere anywhere state RELATED,ESTABLISHED
DROP tcp — anywhere anywhere tcp dpt:1234 recent: UPDATE seconds: 60 hit_count: 6 name: foo side: source
ACCEPT tcp — anywhere anywhere tcp dpt:1234 recent: SET name: foo side: source

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

After entering the rules you should see:

$ iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
DROP       tcp  --  anywhere             anywhere            tcp dpt:1234 recent: UPDATE seconds: 60 hit_count: 6 name: foo side: source 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:1234 recent: SET name: foo side: source 

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

For unpatched kernel:
————————-
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

For patched kernel:
———————-
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

There is no difference in the “iptables -L” output for both the unpatched and the patched kernels. I hope u will give me a good solution.

The expected result for you script would be;

• Patched kernel: the 7th telnet call hangs (not the machine, just the command)
• Unpatched kernel: When one of the conditions mentioned in the article is true, the 2nd call hangs. Otherwise it behaves like the patched kernel.

I’ll post a more detailed test setup later. (Might take till tomorrow
evening, I need some sleep tonight :-)

#! /bin/sh

i=1
j=1000
while [ $i -ne $j ] ;
do
telnet 43.88.102.48 1234
i=`expr $i + 1`
done

I got the same messages “unable to connect to remote host” 1000 times.

But what ive noticed is that the system did not hang even after completing 1000 iterations for both bug-fixed and bug-not-fixed kernels.

My system is a 32 bit system and ive applied the patch ipt_recent.patch that contains even the 64 bit fixes as u said. It will be helpful for me if u can tell me a clear method to reproduce the bug.

Of course you can also use a port with some listener (e.g. port 80 if you have web server running) for the test. But using a port with no listener is just as good in this case.

“unable to connect to remote host”

My /etc/hosts contains the follwoing ip addresses
127.0.0.1
43.88.102.48

Hence i did a telnet as follows
# telnet 43.88.102.48 1234

and i got the error.

iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 1234 -m recent --update --seconds 60 --hitcount 6 --name foo -j DROP
iptables -A INPUT -p tcp --dport 1234 -m recent --set --name foo -j ACCEPT

Then, within the first five minutes after booting, try telnetting to port 1234. With a working version of ipt_recent, the rules won’t block unless you connect more than six times within 60 seconds. With a broken version, the second connect will be blocked.

$ cd linux-2.6.11
$ patch -NEp1 < ~/ipt_recent-fix.patch
patching file include/linux/netfilter_ipv4/ipt_recent.h
patching file net/ipv4/netfilter/ipt_recent.c

Let me know if you need some help with the iptable rules.
(If you’re using shorewall, I’ve posted two rules which use ipt_recent in another article.)

The test case for the problem actually is pretty simple, just write a ipt_recent rule that blocks after receiving a specific number of packets within 60 seconds on some port. The first jiffies rollover happens five minutes after booting, so there’s enough time to recreate the described problem by telnetting to the port. With a broken version of ipt_recent, the rule will hit after receiving the first packet. With a fixed version it won’t hit until the number of packets you’ve specified is reached.

It will be useful to accept the patch if any one can provide a
testcase.

http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.18

commit 404bdbfd242cb99ca0e9d3eb5fbb5bcd54123081
Author: Patrick McHardy
Date: Mon May 29 18:21:34 2006 -0700

[NETFILTER]: recent match: replace by rewritten version

Replace the unmaintainable ipt_recent match by a rewritten version that
should be fully compatible.

Jürgen, do you think you will take on this at some point? Or do you consider this module a lost case?

http://article.gmane.org/gmane.comp.security.firewalls.netfilter.devel/14931/match=ipt+recent

Sam

http://www.kd.cz/~martin/kernel-recent/readme

These are his patches using 64 bit counters for kernels 2.6.9 and 2.6.14:

http://www.kd.cz/~martin/kernel-recent/

Claims to have worked well on 2.6.9 since February 2005.

Juergen, would you consider maintaining it? You seem to have a very good grasp of the issues.

For example, on a 3.0Ghz cpu with 100 interrupts per sec. for the timer, the correct value is 100, correct? Jiffies appear to get incremented 100 times per second.

The proc interface for ipt_recent just shows non-empty slots (ie. only slots with value different from zero) but the hit-count calculation looks at all slots unconditionally. The hit-count algorithm compares the stored time with current jiffies using time_before_eq(). time_before_eq(ul, 0UL) returns true for ul values greater than LONG_MAX + 1.

For ipt_recent’s hit-count algorithm this means, that as soon as jiffies (plus the time value given in the rule) reaches LONG_MAX + 2 every empty slot counts as a hit!

What are empty slots and how do we detect them?

hourly cron script to cleanup SSHA file in ipt_recent (script pseudocode):

get current jiffies by parsing /proc/interrupts
if jiffies close to rollover then
   issue iptables commands to wipe out entries in /proc/net/ipt_recent
else
  for each entry in /proc/net/ipt_recent/SSHA
   age = current jiffies - last_seen
   if age > MAX_AGE
       issue iptables commands to clear this entry from ipt_recent/SSHA
   endif
  endfor 
endif

The kernel initializes jiffies to a value that makes the first rollover happen 5 minutes after boot. This is done to catch rollover problems.

After that, rollovers happen all ULONG_MAX / HZ seconds.

The ipt_recent problem shows up earlier: It happens after LONG_MAX / HZ seconds. E.g., on a 32-bit system with HZ=1000 that means after (2³¹-1)s/1000 = 2147483.647s ≈ 24.855 days.

On 64-bit systems you’re safe. On 32-bit kernels you can either use a kernel with a lower HZ setting (e.g. 100 or 250, which gives you about 249 or 99 days before getting problems) or use my patch (I really should upload an updated version).

Is the time required for the jiffies rollover predictable?

Would rebooting the server once per week avoid this problem?

I’ll post an updated patch for just the jiffies problem in the next days. And, maybe, I’ll start a rewrite when I have some free days.

Is there an update to this? It seems like Dave Miller didn’t like the proposed fix, and thinks that the only solution is a complete rewrite of ipt_recent? This is somewhat annoying because this seemed like the best solution to this problem I’ve seen so far, and to find that the kernel needs a module re-written before this can be useful is disheartening.