How do apt and wget find URI locations in the web and fetch files? How are these methods different?

Question

I am using Ubuntu 18.04LTS, behind a proxy. This question was sparked by (but it is completely independent of) apt-get install behind proxy - Failed to fetch

When I check the location of package yudit-common with apt install --print-uris (e.g.; actually, as a prerequisite for feh), I get http://us.archive.ubuntu.com/ubuntu/pool/universe/y/yudit/yudit-common_2.9.6-7_all.deb as the location in the web of the deb file. See Note 1 below.

Then when I install yudit-common, everything works fine, which means my PC knows where is the deb file http://us.archive.ubuntu.com/ubuntu/pool/universe/y/yudit/yudit-common_2.9.6-7_all.deb in the web, and it can download it prior to installing. See Note 2 below.

But if I try to wget the file, my PC doesn't even know where is the deb file (in this case, of course it would not be able to get it). See Note 3 below.

Where are apt and wget looking for the information?
Can I force wget to use the same domain lookup as apt?
In the case it is a firewall who is letting apt succeed in knowing ip addresses and pulling files from the web, and precluding wget from doing that, can I somehow "channel" wget requests via apt?

---

Notes

1. Location of the .deb package file

    $ apt install --print-uris feh 2> /dev/null
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    The following additional packages will be installed:
      yudit-common
    Suggested packages:
      yudit-doc yudit
    The following NEW packages will be installed:
      feh yudit-common
    0 upgraded, 2 newly installed, 0 to remove and 51 not upgraded.
    Need to get 1.760 kB of archives.
    After this operation, 9.775 kB of additional disk space will be used.
    'http://us.archive.ubuntu.com/ubuntu/pool/universe/y/yudit/yudit-common_2.9.6-7_all.deb' yudit-common_2.9.6-7_all.deb 1637644 MD5Sum:987ef69fa59b1c3034bfa28955e612d9
    'http://us.archive.ubuntu.com/ubuntu/pool/universe/f/feh/feh_2.23.2-1build1_amd64.deb' feh_2.23.2-1build1_amd64.deb 122020 MD5Sum:b5f42ee280ee4aada1cbd93ec0007e68

2. Installing yudit-common

    $ sudo apt-get install yudit-common
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Suggested packages:
      yudit-doc yudit
    The following NEW packages will be installed:
      yudit-common
    0 upgraded, 1 newly installed, 0 to remove and 51 not upgraded.
    Need to get 1.638 kB of archives.
    After this operation, 9.411 kB of additional disk space will be used.
    Get:1 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 yudit-common all 2.9.6-7 [1.638 kB]
    Fetched 1.638 kB in 9s (174 kB/s)      
    Selecting previously unselected package yudit-common.
    (Reading database ... 488875 files and directories currently installed.)
    Preparing to unpack .../yudit-common_2.9.6-7_all.deb ...
    Unpacking yudit-common (2.9.6-7) ...
    Setting up yudit-common (2.9.6-7) ...

3. wgetting yudit-common .deb file

    $ wget http://us.archive.ubuntu.com/ubuntu/pool/universe/y/yudit/yudit-common_2.9.6-7_all.deb
    --2020-09-20 06:32:21--  http://us.archive.ubuntu.com/ubuntu/pool/universe/y/yudit/yudit-common_2.9.6-7_all.deb
    Resolving us.archive.ubuntu.com (us.archive.ubuntu.com)... failed: Name or service not known.
    wget: unable to resolve host address ´us.archive.ubuntu.com´

4. Interesting:

https://unix.stackexchange.com/questions/168584/wget-is-unable-to-resolve-host-address-80-of-the-time

Answer 1

I don't know the exact relation and interoperation of my PC, the proxy, the firewall, and the DNS server.

But I found that the cause apt was succeding in my case, and wget was not, is that I was missing a proper configuration of the proxy for wget. When I added file ~/.wgetrc with

http_proxy = http://username:password@proxy:port
ftp_proxy = http://username:password@proxy:port
proxy_user = username
proxy_password = password 
use_proxy = on

It started working. (Note it is somewhat insecure, as the password is stored in plain text... I wonder if there is any way to have wget ask for the password; or perhaps setting a global .wgetrc in a directory with the same level of security as apt.conf). On the other hand, apt relies on /etc/apt/apt.conf, which was in place.

As for the questions:

Where are apt and wget looking for the information?
Part of the sequence involves setting the proxy. apt and wget have different mechanisms for that, as mentioned above (I know one can set environment variables as well, and that would work at least for wget).

Can I force wget to use the same domain lookup as apt?
I conceive a caveman-way... parsing apt.conf and using that info for wget. Perhaps there is something cleaner.

In the case it is a firewall who is letting apt succeed in knowing ip addresses and pulling files from the web, and precluding wget from doing that, can I somehow "channel" wget requests via apt?
In the light of findings above, this question is replaced by the previous one.

Answer 2

Yes, your Ubuntu system knows where to find .debs. It's a three-step process.

1. Your system knows where archives (sources) are located. Its in /etc/apt/sources.list.

   $ cat /etc/apt/sources.list
   deb http://us.archive.ubuntu.com/ubuntu/ focal main universe
   deb http://us.archive.ubuntu.com/ubuntu/ focal-updates main universe
   deb http://security.ubuntu.com/ubuntu focal-security main universe
   

   • The initial sources are placed by the Ubuntu installer. You can change these sources; it's just a text file owned by root. However, you also risk breaking your system, so be sure you know how to restore the originals.

2. Apt keeps a database of all known packages from all known sources. Name, description, version, size, URI, etc. Apt uses it's database to calculate available upgrades, and to know the URIs to fetch packages from.

   Each time you run sudo apt update, apt downloads a Release file from each source, and uses that Release file to updates apt's database.

   • The database is at /var/lib/apt/lists. It's designed to be machine-readable, not human-readable. Use apt commands (apt search or apt list or apt-cache, etc) to query the database.

   • This is why most instructions say to run apt update right before installing new software -- so apt's database has the correct URI.

3. When you tell apt to sudo apt install foo, apt consults it's database to look for the right version, uninstalled dependencies, and the URIs of all the packages that it needs.