[H-GEN] IP Traffic Monitoring

[Tony Nugent]

[ Humbug *General* list - semi-serious discussions about Humbug and     ]
[ Unix-related topics. Posts from non-subscribed addresses will vanish. ]

On Wed Sep 10 2003 at 17:34, Russell Stuart wrote:

> > oh, you have :)  ok, these packets must be getting rejected/dropped
> > at the kernel level[2] - but perhaps you are not looking at the
> > right things?  For example, icmp or arp (or other) packets being
> > might be useful as symptoms to diagnose the problem (indeed, an
> > over-generation of such traffic may *be* the problem).
> 
> Its a bit hard to miss icmp and arp requests as tcpdump prints them
> out.

By default, yes, everything is dumped, but you obviously know how
to filter that.  If you want to see just icmp and arp, then tell
that to tcpdump:  "tcpdump -ni ippp0 arp or icmp"  (for the benefit
of the lurkers:)  Or perhaps "tcpdump -ni ippp0 arp or icmp or port 135"

> Actually almost all of out legitimate traffic is ssh.  So this
> command:
>   tcpdump -nl -i ippp0 not port 22
> should log just the garbage.

True... (although I also need to ignore traffic I "trust" and
already know about, like ipip [proto 4], ipsec etc).

> In actual fact the BadIp packet counts go
> up in iptraf, and this tcpdump command prints nothing.  This leaves
> three possibilities:
> 
> (a) The packets with bad checksums are ssh packets.
> (b) The packets do not really have bad checksums.

my personal suspicion.

> (c) The packets with bad checksums are dropped by the kernel before
>     they get to tcpdump's packet filter.
> 
> I don't believe it is (a).  There are no "hangs" on the ssh sessions
> indicative of TCP retries, and ping never drops packets.  (b) is a
> possibility, but I have never seen iptraf do that before.  This leaves
> (c).

But if tcpdump can't "see" these (filtered?) packets, why does
iptraf see it?  tcpdump is usually regarded as the semi-official
reference tool for analysing network traffic...  :)

> > Also, you should be able to gather "real" (crude raw) statistics
> > from the relevant /proc/ files (eg "watch cat /proc/net/dev" and
> > other similar spells).  You might also need to look at doing some
> > tweaking to the kernel settings in /proc/sys/net/ipv4/*[2].
> 
> There was nothing obvious under /proc.

Perhaps nothing *obvious* :)

  [I assume that you know that hardware and settings along with all
  sorts statistics can be read (eg, with cat) from the files in
  /proc, including the ability to set run-time behaviour by
  modifying the values in the various files in the /proc/sys/
  heirachy.]

> > Finally, who is your idsn provider?  Do they have technical staff
> > who can help, or web management pages that allow you to check on
> > things like your network statistics?  (Telstra does, on both
> > counts... once you finally reach them, I have generally found their
> > tech guys very helpful).
> 
> Telstra is our provider, and I did contact them.  They were fairly
> certain were were being DOS'ed by one or more Windows worms.  They were
> able to fix most of the problem by blocking the packets upstream.  That
> means I don't really have an urgent problem any more.

That would be ports 135 and 593 (along with 137-139).  And you
should be seeing this sort of traffic in your packet dumps (if it is
there).

  BTW, the latest rumours/information about the msblaster worm (and
  who *didn't* see it? :) is that it was the cause of (or at the
  very least a major contributing factor to) the recent massive
  power failures in north america.  Idiots for using windows in
  mission-critical situations... the US navy is still using winnt,
  isn't it?  a bit of worry :)

> But according to
> iptraf, roughly 50% of all incoming packets _still_ have bad checksums.

Still?  And as high as 50%?  I'm suspicious... was it flagging bad
checksums before?  (You probably don't know).

> I am still curious about where they are coming from.  My current theory
> is that they are bad packets generated by the worms using a raw socket.

Perhaps... if you have those ports blocked, then perhaps this is why
tcpdump isn't seeing them?  If you are, then you might want to log
them with iptables just before the drop/reject rule.

  I'm not sure at what "level" of the kernel packet handling tcpdump
  and other such tools get their information, can anyone answer
  that?  Eg, does it see all the traffic (raw), or only after
  kernel-level and/or netfilter-level and/or routing-level
  filtering.  Would promiscious mode make a difference? etc).

FWIW... I've just tried tcpdump and iptraf[1] on an ippp0 interface
and saw nothing like you describe, all looks well.  There's a lot of
one-hit port 135 traffic coming in[2], but nothing at all being
flagged as having bad checksums.

  [1] rh7.3, kernel 2.4.20-20, hisax isdn driver, tcpdump 3.6.3-17,
  iptraf 2.5.0-3

  [2] I wish telstra would block it everywhere (all netbios ports
  mentioned above - if anyone needs it, well that's what VPNs are
  all about).  I also wish telstra would filter viruses and spam out
  of their customers' email... but I digress :)  

Good luck with figuring this out.

Cheers
Tony

--
* This is list (humbug) general handled by majordomo at lists.humbug.org.au .
* Postings to this list are only accepted from subscribed addresses of
* lists 'general' or 'general-post'.  See http://www.humbug.org.au/