Today all of our Ubuntu 12.04 installations have developed a wierd bug. Whenever you try to access the internet either in a browser (Chrome & Firefox tested), apt-get/Software Centre or wget/curl it throws up "Connection Refused" errors.
It can be intermittent; if I disconnect and reconnect the network it works once or twice then dies. HTTPS also appears to work OK, or at least is failing less.
The problem is the same across 4 Ubuntu machines; we have Mint (11 & 13) and Windows (Server 2003, XP, Vista and 7) on the same network which are working perfectly. This is completely Ubuntu specific.
Things I have tried:
- None of the Ubuntu installs have similar non-core packages installed, and none appear to have been updated in the last three days (the problem started sometime in the last 10 hours)
- Nothing is reported as wrong in the logs
- Netstat -s (see below) reports large numbers of failed connection attempts, this is the only "error" I can track down
- DNS is resolving 100% fine, that does not appear to be an issue (if I try to hit IPs directly they exhibit the same error).
- The one caveat to that is that local hostnames are fully accessible (e.g. server1.theoffice.domain is accessible but not via its IP)
- There are no commonalities in hardware, and the network infrastructure has been swapped out for known working kit
- I ran wireshark to see if I could snaffle an error there. It all seemed to be A-OK (i.e. sending out packets properly). I did see a lot of (ACK, RST) packets coming back, but couldn't track down if that was a major issue or not.
Anything I've missed? Any known issues appearing in the last day or so?
Netstat output for TCP:
Tcp: 3325 active connections openings 3 passive connection openings 2393 failed connection attempts 167 connection resets received 4 connections established 52388 segments received 36962 segments send out 246 segments retransmited 0 bad segments received. 230 resets sent TcpExt: 153 TCP sockets finished time wait in fast timer 1045 delayed acks sent Quick ack mode was activated 126 times 5 packets directly queued to recvmsg prequeue. 10 bytes directly received in process context from prequeue 36099 packet headers predicted 2718 acknowledgments not containing data payload received 1136 predicted acknowledgments 5 times recovered from packet loss by selective acknowledgements Detected reordering 1 times using time stamp 1 congestion windows fully recovered without slow start 1 congestion windows partially recovered using Hoe heuristic 96 congestion windows recovered without slow start after partial ack 6 TCP data loss events TCPLostRetransmit: 1 1 timeouts after SACK recovery 10 fast retransmits 5 forward retransmits 13 retransmits in slow start 156 other TCP timeouts 126 DSACKs sent for old packets 44 DSACKs received 42 connections reset due to unexpected data 30 connections reset due to early user close 8 connections aborted due to timeout TCPDSACKIgnoredNoUndo: 16 TCPSpuriousRTOs: 3 TCPSackShifted: 4 TCPSackMerged: 14 TCPSackShiftFallback: 23 IPReversePathFilter: 92
- Telnet on port 80 has the same issue (connection refused).