3

Hy!

First off, the tl;dr explanation of the issue:

Canonical's IP 91.189.88.162 (one of the six IPs security.ubuntu.com resolves to) is returning a HTTP 302 redirect to "http://179.184.158.89:80/pdata/05f7e7f89ba2302b/security.ubuntu.com/ubuntu/dists/xenial-security/InRelease". That IP does not belong to any official Ubuntu mirror and the URI contains some sort of identifier (it may be a tracking code, it changes every request). Investigation led to the conclusion that this is not a software/OS-related issue (easily reproducible from a different laptop, booted from a Live USB stick wired to Telefonica's modem. It can't be reproduced anywhere else apparently).

Are Canonical's repositories supposed to behave like that in any circumstance whatsoever? Pcap sample attached at the end of this message. Manual HTTP request mimicking "apt-get update":

george@workstation-04:~$ GET -USed -H "Host: security.ubuntu.com" -H "User-Agent: Debian APT-HTTP/1.3 (1.2.24)" http://91.189.88.162/ubuntu/dists/xenial-security/InRelease
GET http://91.189.88.162/ubuntu/dists/xenial-security/InRelease
Host: security.ubuntu.com
User-Agent: Debian APT-HTTP/1.3 (1.2.24)

302 Found
Cache-Control: no-cache
Connection: close
Location: http://179.184.158.89:80/pdata/05f4cdcfbada50fc/security.ubuntu.com/ubuntu/dists/xenial-security/InRelease
Content-Length: 0
Content-Type: text/html
Client-Date: Sat, 25 Nov 2017 20:44:04 GMT
Client-Peer: 91.189.88.162:80
Client-Response-Num: 1
X-Content-Type-Options: nosniff

GET http://179.184.158.89:80/pdata/05f4cdcfbada50fc/security.ubuntu.com/ubuntu/dists/xenial-security/InRelease
User-Agent: Debian APT-HTTP/1.3 (1.2.24)

200 OK
Cache-Control: max-age=1144, s-maxage=3300, proxy-revalidate
Connection: close
Date: Sat, 25 Nov 2017 20:13:04 GMT
Accept-Ranges: bytes
ETag: "18ef2-55ed440fdb600"
Server: nginx
Content-Length: 102130
Expires: Sat, 25 Nov 2017 21:05:00 GMT
Last-Modified: Sat, 25 Nov 2017 20:10:00 GMT
Client-Date: Sat, 25 Nov 2017 20:44:05 GMT
Client-Peer: 179.184.158.89:80
Client-Response-Num: 1
X-OC-Service-Type: re

Now for the long, comprehensive version (thorough analysis, unnecessary read if you figured out what's wrong based on the info above):

During a routine check in one of my VMs yesterday, something odd caught my attention:

root@workstation-03:/# apt-get update ; apt-get upgrade -y
Hit:1 http://br.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://br.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]                                                         
Get:3 http://br.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]                                                                 
Get:5 http://br.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [668 kB]                                                                          
Ign:6 http://winswitch.org xenial InRelease                                                            
Get:7 http://br.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages [630 kB]
Hit:8 http://winswitch.org xenial Release                                                           
Get:10 http://br.archive.ubuntu.com/ubuntu xenial-updates/main amd64 DEP-11 Metadata [307 kB]
Get:11 http://br.archive.ubuntu.com/ubuntu xenial-updates/main DEP-11 64x64 Icons [227 kB]
Get:4 http://179.184.158.91:80/pdata/03ee5d7e461a5049/security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]**
Get:12 http://br.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [555 kB]            
Get:13 http://br.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages [527 kB]   
Get:14 http://br.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 DEP-11 Metadata [185 kB]          
Get:16 http://br.archive.ubuntu.com/ubuntu xenial-updates/universe DEP-11 64x64 Icons [263 kB]                       
Get:17 http://br.archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 DEP-11 Metadata [5.888 B]                                             
Get:18 http://br.archive.ubuntu.com/ubuntu xenial-backports/main amd64 DEP-11 Metadata [3.324 B]                                                  
Get:19 http://br.archive.ubuntu.com/ubuntu xenial-backports/universe amd64 DEP-11 Metadata [4.588 B]                                            
Get:15 http://179.184.158.91:80/pdata/03ee827eaa21046b/security.ubuntu.com/ubuntu xenial-security/main amd64 DEP-11 Metadata [60,3 kB]          
Get:20 http://179.184.158.91:80/pdata/03ee977e21229dae/security.ubuntu.com/ubuntu xenial-security/main DEP-11 64x64 Icons [57,6 kB]
Get:21 http://179.184.158.91:80/pdata/03eeaa7e9723e1be/security.ubuntu.com/ubuntu xenial-security/universe amd64 DEP-11 Metadata [51,4 kB]
Get:22 http://179.184.158.91:80/pdata/03eef57e5c24a1d6/security.ubuntu.com/ubuntu xenial-security/universe DEP-11 64x64 Icons [85,1 kB]**
Fetched 3.937 kB in 3s (1.115 kB/s)    
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
...

I don't see where that "179.184.158.91" came from, there is only one non-official repo installed in this VM (winswitch), but xenial-security is what is calling this IP address. Additionally, it appears to carry unique identifiers like "03ee827eaa21046b".

Additional info:

cat /etc/apt/sources.list

deb http://br.archive.ubuntu.com/ubuntu/ xenial main restricted
 deb-src http://br.archive.ubuntu.com/ubuntu/ xenial main restricted

deb http://br.archive.ubuntu.com/ubuntu/ xenial-updates main restricted
 deb-src http://br.archive.ubuntu.com/ubuntu/ xenial-updates main restricted
deb http://br.archive.ubuntu.com/ubuntu/ xenial universe
 deb-src http://br.archive.ubuntu.com/ubuntu/ xenial universe
deb http://br.archive.ubuntu.com/ubuntu/ xenial-updates universe
 deb-src http://br.archive.ubuntu.com/ubuntu/ xenial-updates universe

deb http://br.archive.ubuntu.com/ubuntu/ xenial multiverse
 deb-src http://br.archive.ubuntu.com/ubuntu/ xenial multiverse
deb http://br.archive.ubuntu.com/ubuntu/ xenial-updates multiverse
 deb-src http://br.archive.ubuntu.com/ubuntu/ xenial-updates multiverse

deb http://br.archive.ubuntu.com/ubuntu/ xenial-backports main restricted universe multiverse
 deb-src http://br.archive.ubuntu.com/ubuntu/ xenial-backports main restricted universe multiverse

deb http://security.ubuntu.com/ubuntu xenial-security main restricted
 deb-src http://security.ubuntu.com/ubuntu xenial-security main restricted
deb http://security.ubuntu.com/ubuntu xenial-security universe
 deb-src http://security.ubuntu.com/ubuntu xenial-security universe
deb http://security.ubuntu.com/ubuntu xenial-security multiverse
 deb-src http://security.ubuntu.com/ubuntu xenial-security multiverse

cat /etc/apt/sources.list.d/*

root@workstation-03:~# cat /etc/apt/sources.list.d/*
deb http://winswitch.org/ xenial main

None of the installed repos resolve to that IP:

george@workstation-03:~$ dig A security.ubuntu.com +short
91.189.91.23
91.189.88.149
91.189.91.26
91.189.88.152
91.189.88.162
91.189.88.161

george@workstation-03:~$ dig A winswitch.org +short
78.129.163.65

Reverse IP lookup tools fail to find any FQDN address resolving to that IP as well.

That IP is announced by my ISP (Telefonica) but seems to be in use by a small provider I never heard of until now:

george@workstation-03:~$ whois 179.184.158.91 | grep owner:
owner:       TELEFÔNICA BRASIL S.A

george@workstation-03:~$ host 179.184.158.91
91.158.184.179.in-addr.arpa domain name pointer imaxima.static.gvt.net.br.

I tried to reproduce that behavior immediately after, but that IP no longer showed up during apt-get update.

There is no proxy of any sort in this VM (system-wide or application-specific). This VM connects to the internet via an OPNSense firewall, no outbound filtering is setup. I did attempt to tcpdump the traffic in order to check the HTTP Headers sent and received by apt-get as soon as I noticed the suspicious address but, since I was no longer able to reproduce that behavior, nothing would come up with it. Luckily, however, I ran an apt-get update again last night and that strange IP showed up once again, although in only one line this time around.

At that moment I fired up Wireshark and went through the entire capture. As it turns out this is not a DNS-related anomaly (I had previously confirmed this by querying every resolver I have set up here, none of them returned that "179.184.158.89"), at least one out of the six IPs that security.ubuntu.com resolves to (91.189.88.162) is returning that unknown URI via HTTP 302. Here is the list I tested against:

security.ubuntu.com.    383     IN      A       91.189.88.149
security.ubuntu.com.    383     IN      A       91.189.88.162    X
security.ubuntu.com.    383     IN      A       91.189.88.152
security.ubuntu.com.    383     IN      A       91.189.91.26     
security.ubuntu.com.    383     IN      A       91.189.91.23
security.ubuntu.com.    383     IN      A       91.189.88.161

I can now consistently reproduce the behavior manually. I set User-Agent to "Debian APT-HTTP/1.3 (1.2.24)" in order to divert any sort of attention from a hypothetical honeypot that might be sitting somewhere in between, just in case:

george@workstation-04:~$ GET -USed -H "Host: security.ubuntu.com" -H "User-Agent: Debian APT-HTTP/1.3 (1.2.24)" http://91.189.88.162/ubuntu/dists/xenial-security/InRelease
GET http://91.189.88.162/ubuntu/dists/xenial-security/InRelease
Host: security.ubuntu.com
User-Agent: Debian APT-HTTP/1.3 (1.2.24)

302 Found
Cache-Control: no-cache
Connection: close
Location: http://179.184.158.89:80/pdata/05f4cdcfbada50fc/security.ubuntu.com/ubuntu/dists/xenial-security/InRelease
Content-Length: 0
Content-Type: text/html
Client-Date: Sat, 25 Nov 2017 20:44:04 GMT
Client-Peer: 91.189.88.162:80
Client-Response-Num: 1
X-Content-Type-Options: nosniff

GET http://179.184.158.89:80/pdata/05f4cdcfbada50fc/security.ubuntu.com/ubuntu/dists/xenial-security/InRelease
User-Agent: Debian APT-HTTP/1.3 (1.2.24)

200 OK
Cache-Control: max-age=1144, s-maxage=3300, proxy-revalidate
Connection: close
Date: Sat, 25 Nov 2017 20:13:04 GMT
Accept-Ranges: bytes
ETag: "18ef2-55ed440fdb600"
Server: nginx
Content-Length: 102130
Expires: Sat, 25 Nov 2017 21:05:00 GMT
Last-Modified: Sat, 25 Nov 2017 20:10:00 GMT
Client-Date: Sat, 25 Nov 2017 20:44:05 GMT
Client-Peer: 179.184.158.89:80
Client-Response-Num: 1
X-OC-Service-Type: re

All of those five remaining IPs return HTTP 200, which is the expected behavior for all of them as far as I know. For instance:

george@core-workstation:~$ GET -USed -H "Host: security.ubuntu.com" -H "User-Agent: Debian APT-HTTP/1.3 (1.2.24)" http://91.189.88.161/ubuntu/dists/xenial-security/InRelease
GET http://91.189.88.161/ubuntu/dists/xenial-security/InRelease
Host: security.ubuntu.com
User-Agent: Debian APT-HTTP/1.3 (1.2.24)

200 OK
Cache-Control: max-age=1271, s-maxage=3300, proxy-revalidate
Connection: close
Date: Sun, 26 Nov 2017 23:34:48 GMT
Accept-Ranges: bytes
ETag: "18ef2-55eeac2604300"
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 102130
Expires: Sun, 26 Nov 2017 23:56:00 GMT
Last-Modified: Sun, 26 Nov 2017 23:01:00 GMT
Client-Date: Sun, 26 Nov 2017 23:33:34 GMT
Client-Peer: 91.189.88.161:80
Client-Response-Num: 1

As you can see, Canonical's IP 91.189.88.162 itself is returning an HTTP 302 redirect to that suspicious IP. Since this anomaly is reproducible from any one of my VMs and even from a live OS on an old laptop I have laying around while plugged straight into Telefonica's modem, I came to the conclusion that this is not related to my firewall either. Even though Telefonica's gear sits in my living room, it's outside my network's secure perimeter so it's not audited or anything. Either way, I don't believe that's the culprit since it's a dirt cheap DSL-2730E ADSL2+ Router with no custom static routes configured. I will try bridging it one of these days to see if things change.

Oddly enough, that 302 response doesn't carry the webserver signature unlike all others requests towards it, which led me to believe something in between might be intercepting packets to that particular IP and port. Here's an example taken from a server I own in Canada, which clearly shows the "Server" header:

root@server-1:~# GET -USed -H "Host: security.ubuntu.com" -H "User-Agent: Debian APT-HTTP/1.3 (1.2.24)" http://91.189.88.162/ubuntu/dists/xenial-security/InRelease
GET http://91.189.88.162/ubuntu/dists/xenial-security/InRelease
Host: security.ubuntu.com
User-Agent: Debian APT-HTTP/1.3 (1.2.24)

200 OK
Cache-Control: max-age=0, s-maxage=3300, proxy-revalidate
Connection: close
Date: Mon, 27 Nov 2017 05:48:29 GMT
Accept-Ranges: bytes
ETag: "18ef2-55eef9eebc700"
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 102130
Expires: Mon, 27 Nov 2017 05:48:29 GMT
Last-Modified: Mon, 27 Nov 2017 04:49:00 GMT
Client-Date: Mon, 27 Nov 2017 05:48:29 GMT
Client-Peer: 91.189.88.162:80
Client-Response-Num: 1

However, to make things short, further attempts to fingerprint that server behind 91.189.88.162 failed to prove that the traffic is being tampered with. TCPtraceroute and mtr both show the expected route, HTTP latency is within a 5% margin compared to 91.189.88.161 (which is one one those six and is also located in London). Nmap probing also suggests I'm reaching Canonical's server (unless this is a very carefully crafted MITM attack, which doesn't seem to be the case). I also see no evidence of BGP Hijacking and the route is fine:

show route protocol bgp 91.189.88.162 | no-more 

inet.0: 688953 destinations, 2269968 routes (688942 active, 0 holddown, 4860 hidden)
+ = Active Route, - = Last Active, * = Both

91.189.88.0/24     *[BGP/170] 2w0d 17:19:51, MED 0, localpref 85, from 94.142.108.190
                      AS path: 3356 41231 I, validation-state: unverified
                      to 5.53.3.85 via ae11.0
                      to 5.53.3.79 via ae17.0
                    > to 5.53.3.223 via et-9/3/0.0
                    [BGP/170] 2w0d 17:20:31, MED 0, localpref 85, from 94.142.108.210
                      AS path: 3356 41231 I, validation-state: unverified
                      to 5.53.3.85 via ae11.0
                      to 5.53.3.79 via ae17.0
                    > to 5.53.3.223 via et-9/3/0.0
                    [BGP/170] 5d 02:34:24, MED 0, localpref 85, from 94.142.108.193
                      AS path: 3356 41231 I, validation-state: unverified
                      to 5.53.3.85 via ae11.0
                      to 5.53.3.79 via ae17.0
                    > to 5.53.3.223 via et-9/3/0.0

Tcptraceroute against port 80:

george@workstation-04:/$ tcptraceroute -w1 91.189.88.162 80
Selected device enp0s3, address 10.4.4.119, port 47917 for outgoing packets
Tracing the path to 91.189.88.162 on TCP port 80 (http), 30 hops max
 1  10.4.4.200  0.295 ms  0.220 ms  0.306 ms
 2  192.168.25.1  1.938 ms  2.106 ms  1.883 ms
 3  gvt-b-sr01.cta.gvt.net.br (179.184.120.13)  20.106 ms  22.429 ms  19.890 ms
 4  201.22.69.21.dynamic.adsl.gvt.net.br (201.22.69.21)  20.087 ms  20.514 ms  20.716 ms
 5  201.22.64.99.dynamic.dialup.gvt.net.br (201.22.64.99)  27.391 ms  28.520 ms  28.410 ms
 6  213.140.39.82  27.475 ms  28.178 ms  27.646 ms
 7  5.53.3.143  143.486 ms  142.026 ms  142.533 ms
 8  * * *
 9  ae-126-3512.edge5.london1.Level3.net (4.69.166.45)  271.503 ms  268.769 ms  268.715 ms
10  SOURCE-MANA.edge5.London1.Level3.net (212.187.138.82)  291.886 ms  271.616 ms  273.050 ms
11  yukinko.canonical.com (91.189.88.162) [open]  281.588 ms  273.400 ms  273.496 ms

While I was beginning to dig into this issue the power went out in my neighborhood (which is extremely rare by itself. Go Murphy!). Once it came back 3 hours later up I booted all my VMs back, except one which failed to boot properly. Guess which one? That's right.

By the way, that happens to be the only Ubuntu 16.04 VM I have at home. This type of failure is unprecedented for any of my VMs, which makes it one hell of a coincidence. I immediately proceeded to snapshot its disk and decommissioned the said VM for an in-depth forensic analysis later on just to be on the safe side. Apparently something broke X11-Server, which was last updated on 2017-11-07. I'll look into it later but I can't see how this redirect alone would cause this to happen.

It should be noted that, once the power came back up I was no longer facing that issue, that is, every request to the six IPs would return HTTP 200 as it should. So earlier today I kept forcing my modem to restart its PPPoE authentication in order to obtain a new dynamic IP and then I'd check again against those 6 IPs to see if that behavior would come back. 11 reconnects later, bingo! 91.189.88.162 started throwing me those HTTP 302. Here is the list of IPs I was using after each reconnect (in case there's an ACL in Canonical's frontend that happens to be purposefully manipulating the behavior based on source IP for whatever reason). All but the very first and last didn't experience any issues with security.ubuntu.com:

191.250.187.149 (The one I was using when I started this topic)
177.132.10.0
187.112.57.37
186.212.197.62
177.132.109.6
187.112.135.18
201.86.5.226
201.86.5.226
201.86.5.226
189.115.80.207
177.133.196.249
177.16.143.235
177.204.139.117
179.182.184.0/24 (The IP I'm using now belongs to this subnet)  

I queried Canonical's six IPs from another ISP in Brazil as well as several servers worldwide, always performing 5 queries per destination IP in order to rule out inconsistencies. None of them returned that HTTP 302. Ever. Not even once. In my home connection, however, every single attempt against 91.189.88.162 yields HTTP 302 unless I keep trying a few seconds apart, in which case it will return HTTP 200 as if I was bypassing some sort of cache despite the fact that "Cache-Control: no-cache" is set in the 302 response. Weird, huh?

I invite anyone to attempt to reproduce that behavior, please let me know if you come across that 302 redirect:

GET -USed -H "Host: security.ubuntu.com" -H "User-Agent: Debian APT-HTTP/1.3 (1.2.24)" http://91.189.88.162/ubuntu/dists/xenial-security/InRelease

I must reiterate that this destination IP in my case (179.184.158.89) does not belong to any company in Ubuntu's mirrors list, I just don't see why it should be serving me the InRelease file. This address is allocated to a small ISP in a different state 400km+ away from here.

Bottom line is, If this is intentionally set up to collect statistics (which wouldn't even make much sense), it's safe to say this has been extremely poorly implemented since it's fairly erratic and only works for one out of the 6 IPs (Which will be picked randomly or round-robin since they can't manage how the local DNS client will handle them and all 6 A records share the same TTL). This leaves a lot of room for speculation.

In case someone wants to see it, here is he pcap file taken during an apt-get update. I filtered out anything but HTTP Requests for the sake of simplicity, but I will be happy to upload the whole thing if necessary for debugging. Packets of interest are #6, #7 and #9:

https://mega.nz/#!NORi2ALA!njJRKZ4i26GHXaET_ZA6Z4ymWYKhccTGNiEBCcGtwbA

SHA256: 40fc995e505ee2dcaf9aa3e23961a757f628224e3fca83d0deed6374ba9a3fbe

I must be missing something... PS: I assume this is more of a privacy concern rather than a security incident, either way this behavior is undocumented (Google won't find anything similar to it) so I decided to check with you people just to be on the safe side.

Any help is much appreciated! Thanks for sticking this far! :)

  • 1
    Have you configured a system proxy by any chance? Have you tried obtaining a packet capture with Wireshark (use the display filter http and check for example the User-Agent header). – Lekensteyn Nov 25 '17 at 1:32
  • 1
    Maybe just ISP http hijack – alfred Nov 27 '17 at 10:16
  • 1
    @Mr.Ash rather than using the display filter "http" for making an export, you should right-click on the packet and use "Follow > TCP Stream", otherwise the TCP stream will not be complete (can you do that and upload a new pcap?). Not sure if coincidence, but the IP ID field (ip.id) for the malicious response is very close to the request (22430 vs 22429). (At first I was looking at the TTL field, if that is relatively high it may indicate a local attacker, but that does not seem to be the case here.) – Lekensteyn Nov 27 '17 at 21:48
  • 1
    If you want to do more forensics on it, you could capture more traffic and look at the timings. The suspicuous response has a tcp.time_delta of about 30ms, this could be an upper bound of the location of your adversary. – Lekensteyn Nov 27 '17 at 23:10
  • 1
    @Mr.Ash the destination is not interesting, but the redirect source. The destination could be an unsuspecting victim. I wonder if it is a misconfigured server, an infected one, or perhaps one of the hops between (and including your device) and the server. If you take more samples of the HTTP queries, is there any difference in the timings? Is there any chance that you can find a friend with the same ISP and test this? What if you replace your router and try the query again? Perhaps something is modifying the response on the fly. (And it is also possible that the server is just misconfigured.) – Lekensteyn Nov 28 '17 at 3:06
2

The X-OC-Service-Type indicates that the response came from an Open Cache node.

ISPs use Open Cache servers provided by companies such as Qwilt as transparent caching proxies on their networks. Your request was intercepted by their routing node, and redirected to a cache node. The cache node did not have the object cached ('re'=relay for miss; it would be 'lo'=local for a hit), so it theoretically went to the original host+path you requested to get the object.

It is possible that it was actually some sort of malicious interception, but my guess is it was just the ISP's proxy.

  • I Googled around for the "X-OC-Service-Type" and no results came up other than this thread, so I assume this is something new. Thanks for the info! But why is it that this OpenCache will only cache the InRelease file? It isn't even large anyway. I tried to pull some deb files from the cache IP with no avail, it'd return another HTTP 302 instead of fetching from cache or backend. Counter productive, isn't it? Not to mention that there's a 10gbps public mirror in my town, the cache is 400km+ away. Also, it always returns "X-OC-Service-Type: re" here. Very poorly implemented caching maybe? – Mr.Ash Nov 30 '17 at 6:29
  • 1
    Sorry that's about as much as I know - I found this page via googling that header as well (looking for docs). The ideal situation is that the OC server is closer to you than whatever it is proxying for; but it's a new system getting rolled out so I'm guessing they are just getting it set up (or perhaps just doing some random testing with it). – Taylor Dec 4 '17 at 21:38

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.