16

On Saturday (May 18) I started one of my VMs (running Ubuntu 18.04 Server).

To my surprise, the VM almost immediately attempted to connect to d16r8ew072anqo.cloudfront.net:80, something that I have never seen before--it's a pretty "pristine" installation of Ubuntu, with no custom apt repositories, and just a few packages installed. It had never connected to anything suspicious before--mostly to Ubuntu and Snap domains. (I use Little Snitch to monitor network traffic so I see connections in real time and can deny them as well.)

I spent some time trying to figure out what happened and I believe I narrowed it down to unattended-upgrades installing security patches. Specifically, when I manually reinstalled intel-microcode:amd64 package I was able to reproduce the strange connection to CloudFront (although it might have been just a coincidence).

Then, on Monday, I wanted to document the issue in case something similar happens again, but to my surprise I couldn't reproduce the strange connection anymore.

And the only observable difference on Monday was that the output from sudo apt-get install --reinstall intel-microcode:amd64 [1] did't have the Ign:1 line.

I searched the web, including http://archive.ubuntu.com/ubuntu, grep-ed the VM's disk, checked the DNS records of misc. ubuntu.com subdomains, tried wget-ting different URLs to find a redirect to the suspicious domain--but I couldn't find any clue about the strange connection to CloudFront.

My question is: does anyone know what happened, or at least noticed the same connection in their logs?

(By the way, I'm aware of one example where the Ubuntu team used CloudFront to relieve their servers: MD5 mismatch on my 12.04 ISO, what is going on? --so I'm hoping that maybe this is a similar case?)


[1]:

$ sudo apt-get install --reinstall intel-microcode:amd64
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 1,801 kB of archives.
After this operation, 0 B of additional disk space will be used.
Ign:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 intel-microcode amd64 3.20190514.0ubuntu0.18.04.2
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 intel-microcode amd64 3.20190514.0ubuntu0.18.04.2 [1,801 kB]
Fetched 1,801 kB in 20s (89.2 kB/s)          
(Reading database ... 107477 files and directories currently installed.)
Preparing to unpack .../intel-microcode_3.20190514.0ubuntu0.18.04.2_amd64.deb ...
Unpacking intel-microcode (3.20190514.0ubuntu0.18.04.2) over (3.20190514.0ubuntu0.18.04.2) ...
Setting up intel-microcode (3.20190514.0ubuntu0.18.04.2) ...
update-initramfs: deferring update (trigger activated)
intel-microcode: microcode will be updated at next boot
Processing triggers for initramfs-tools (0.130ubuntu3.7) ...
update-initramfs: Generating /boot/initrd.img-4.15.0-50-generic
23

I made some inqurires to the Security and Archive teams about this, as they'd be the authoritative source on whether this was expected behavior or not. The following is a summary of what they shared with me:

This observed behavior was offloading traffic load from the archive mirrors to Cloudfront, and was done between Wednesday May 15th and Monday May 20th in order to alleviate load from the Archive Servers, specifically for the Kernel and Microcode updates.

According to those teams, this is the first time such offloading was done via CloudFront, and in this particular case was expected behavior.


While it looks suspicious, the teams have confirmed with me via IRC that this was expected behavior, and they are surprised that more people hadn't noticed the behavior in the past.

ULTIMATELY: Not malicious behavior, but instead was expected behavior, and was required in order for things to not overload on the archive servers. (It was also the first time they've had to do this so at least nothing blew up heh)

Standard behavior of NOT offloading to Cloudfront should be back in place now.

  • Thank you, that's very good news! So I guess that on Monday, May 20th, my trials happened after CloudFront was turned off and that's why I no longer could reproduce the (non-)issue. – Tomasz Zieliński May 23 at 16:30
  • Debian (of all distros) have been using CloudFront and Fastly CDNs since 2016, so Ubuntu doing the same is nothing new there... – grawity May 24 at 19:56
  • @grawity except that Ubuntu has never seemed to need to offload this. But you're not wrong it's 'nothing new' in the grand scheme of things but for Ubuntu has not been done much. (And is an atypical observation in Ubuntu.) – Thomas Ward May 25 at 0:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.