Be clued on Protocol Analysis

It pays to know how to read packet captures when troubleshooting network problems like latency problems. Like there was one time when we were investigating why a server response became slow after moving it behind a firewall. Finger pointing ensues until we decided to do sniff the traffic in the firewall to know whats happening. From the capture, we see the server sending reverse DNS lookup for every client request it receives. So before processing the client requests, it waits for the reverse DNS result until it times out because the firewall is blocking the DNS requests. After understanding what was going on, the administrator turned off the reverse DNS lookup in the server.

Most of the time, people will be looking at bandwidth utilization issues/network congestion but it is rarely the cause if application response over the network is slow. Sure there are peer-to-peer traffic and worm scans to watch out for especially on slow links but there are more possible factors that can cause slow application access. Slow backend server response, problematic SQL queries (those long, long SELECT statements) and chatty applications are some possibilities that I can think of. The point is you should know how to push the blame on others. :)

Here's a helpful presentation slide from Laura Chappell that should help you get started in analysing slow networks. While you're at it, browse through the Protocol Analysis Institute website and you'll see more articles from Laura Chappell.