Ubuntu 12.04 http connection refused

Question

Today all of our Ubuntu 12.04 installations have developed a wierd bug. Whenever you try to access the internet either in a browser (Chrome & Firefox tested), apt-get/Software Centre or wget/curl it throws up "Connection Refused" errors.

It can be intermittent; if I disconnect and reconnect the network it works once or twice then dies. HTTPS also appears to work OK, or at least is failing less.

The problem is the same across 4 Ubuntu machines; we have Mint (11 & 13) and Windows (Server 2003, XP, Vista and 7) on the same network which are working perfectly. This is completely Ubuntu specific.

Things I have tried:

• None of the Ubuntu installs have similar non-core packages installed, and none appear to have been updated in the last three days (the problem started sometime in the last 10 hours)
• Nothing is reported as wrong in the logs
• Netstat -s (see below) reports large numbers of failed connection attempts, this is the only "error" I can track down
• DNS is resolving 100% fine, that does not appear to be an issue (if I try to hit IPs directly they exhibit the same error).
• The one caveat to that is that local hostnames are fully accessible (e.g. server1.theoffice.domain is accessible but not via its IP)
• There are no commonalities in hardware, and the network infrastructure has been swapped out for known working kit
• I ran wireshark to see if I could snaffle an error there. It all seemed to be A-OK (i.e. sending out packets properly). I did see a lot of (ACK, RST) packets coming back, but couldn't track down if that was a major issue or not.

Anything I've missed? Any known issues appearing in the last day or so?

Netstat output for TCP:

Tcp:
    3325 active connections openings
    3 passive connection openings
    2393 failed connection attempts
    167 connection resets received
    4 connections established
    52388 segments received
    36962 segments send out
    246 segments retransmited
    0 bad segments received.
    230 resets sent
TcpExt:
    153 TCP sockets finished time wait in fast timer
    1045 delayed acks sent
    Quick ack mode was activated 126 times
    5 packets directly queued to recvmsg prequeue.
    10 bytes directly received in process context from prequeue
    36099 packet headers predicted
    2718 acknowledgments not containing data payload received
    1136 predicted acknowledgments
    5 times recovered from packet loss by selective acknowledgements
    Detected reordering 1 times using time stamp
    1 congestion windows fully recovered without slow start
    1 congestion windows partially recovered using Hoe heuristic
    96 congestion windows recovered without slow start after partial ack
    6 TCP data loss events
    TCPLostRetransmit: 1
    1 timeouts after SACK recovery
    10 fast retransmits
    5 forward retransmits
    13 retransmits in slow start
    156 other TCP timeouts
    126 DSACKs sent for old packets
    44 DSACKs received
    42 connections reset due to unexpected data
    30 connections reset due to early user close
    8 connections aborted due to timeout
    TCPDSACKIgnoredNoUndo: 16
    TCPSpuriousRTOs: 3
    TCPSackShifted: 4
    TCPSackMerged: 14
    TCPSackShiftFallback: 23
    IPReversePathFilter: 92

further informations

• Telnet on port 80 has the same issue (connection refused).

Answer 1

step 1

I assume that you have done successful tests with ping. traceroute and telnet.

With telnet somehost 80 the accessible of the http service should be ensured.

If this all works the problem should be a proxy problem. Assuming that the non-ubuntu installations are working a misconfiguration of the proxy will be occurred locally in the ubuntu installations.

step 2

The failing telnet shows that the problem is below the application layer. Ping is working?
May you check with nmap which ports are reachable?
May you check your local firewall rules with ufw status.

Resolution

"firewall was disabled on all units, ping was working & nmap didn't show up anything abnormal. We tracked this down to a bug in the application layer which creates malformed packets and that led to our network abuse monitor intercepting HTTP requests & halting them. Nearly killed me finding that out! Thanks for your replies"