The kernel-side firewall is the fastest and the most secure software solution (difficult to kill the kernel isn't it?). Using it have also the advantage to use the hardware firewall found on some network controllers. Iptables is the primary tool for controlling it, but there are many others frontends with easier syntax.
If you want to configure easier, you should use this :.
Keep in mind tracking byte count for each IP can use lot of memory.
In your case I would install ipset, which is developed by the same team of iptables :
#create ipset for accounting with default lifetime 300 secs
ipset create IP_QUOTA_SET hash:ip timeout 300 counters
#create separated rule chain
iptables --new-chain PER_IP_QOUTING
#send packets to chain
iptables -t filter -A INPUT \
-i <in-iface> --dst <ip> \
-p tcp --dport <dstport> \
-j PER_IP_QUOTING
#if ip doesn't exist in the set, add it
iptables -t filter -A PER_IP_QUOTING \
-m set ! --match-set IP_QUOTA_SET src \
-j SET --add-set IP_QUOTA_SET src --timeout 300
#if packet exists in the set, check bytes
#if byte counter > quota then drop packet
iptables -t filter -A PER_IP_QUOTING \
-m set --match-set IP_QUOTA_SET src \
--bytes-gr 1000 -j DROP
#pass other packets (for debug purpose)
iptables -t filter -A PER_IP_QUOTING \
-j RETURN
In this case you can check the list and edit it by ipset command.
To show current list with counters and timeouts :ipset list IP_QUOTA_SET
.
STRONG NOTE : iptables is Linux specific and is available since linux 2.4. The kernel implementation along the userspace tools did change in 2.0 and 2.2 previously.
The 3.13 version introduced a new change which will replace ipset; arptables; ebtables; ip6tables, and iptables with a single tool.
As with previous versions, their will be a transition period where frontends like vuurmuur will remain compatible with the kernel, but don't expect to use iptables in the future.