0

I am using Ubuntu 18.04LTS, behind a proxy. This question was sparked by (but it is completely independent of) apt-get install behind proxy - Failed to fetch

When I check the location of package yudit-common with apt install --print-uris (e.g.; actually, as a prerequisite for feh), I get http://us.archive.ubuntu.com/ubuntu/pool/universe/y/yudit/yudit-common_2.9.6-7_all.deb as the location in the web of the deb file. See Note 1 below.

Then when I install yudit-common, everything works fine, which means my PC knows where is the deb file http://us.archive.ubuntu.com/ubuntu/pool/universe/y/yudit/yudit-common_2.9.6-7_all.deb in the web, and it can download it prior to installing. See Note 2 below.

But if I try to wget the file, my PC doesn't even know where is the deb file (in this case, of course it would not be able to get it). See Note 3 below.

Where are apt and wget looking for the information?
Can I force wget to use the same domain lookup as apt?
In the case it is a firewall who is letting apt succeed in knowing ip addresses and pulling files from the web, and precluding wget from doing that, can I somehow "channel" wget requests via apt?

Notes

  1. Location of the .deb package file
    $ apt install --print-uris feh 2> /dev/null
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    The following additional packages will be installed:
      yudit-common
    Suggested packages:
      yudit-doc yudit
    The following NEW packages will be installed:
      feh yudit-common
    0 upgraded, 2 newly installed, 0 to remove and 51 not upgraded.
    Need to get 1.760 kB of archives.
    After this operation, 9.775 kB of additional disk space will be used.
    'http://us.archive.ubuntu.com/ubuntu/pool/universe/y/yudit/yudit-common_2.9.6-7_all.deb' yudit-common_2.9.6-7_all.deb 1637644 MD5Sum:987ef69fa59b1c3034bfa28955e612d9
    'http://us.archive.ubuntu.com/ubuntu/pool/universe/f/feh/feh_2.23.2-1build1_amd64.deb' feh_2.23.2-1build1_amd64.deb 122020 MD5Sum:b5f42ee280ee4aada1cbd93ec0007e68
  1. Installing yudit-common
    $ sudo apt-get install yudit-common
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Suggested packages:
      yudit-doc yudit
    The following NEW packages will be installed:
      yudit-common
    0 upgraded, 1 newly installed, 0 to remove and 51 not upgraded.
    Need to get 1.638 kB of archives.
    After this operation, 9.411 kB of additional disk space will be used.
    Get:1 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 yudit-common all 2.9.6-7 [1.638 kB]
    Fetched 1.638 kB in 9s (174 kB/s)      
    Selecting previously unselected package yudit-common.
    (Reading database ... 488875 files and directories currently installed.)
    Preparing to unpack .../yudit-common_2.9.6-7_all.deb ...
    Unpacking yudit-common (2.9.6-7) ...
    Setting up yudit-common (2.9.6-7) ...
  1. wgetting yudit-common .deb file
    $ wget http://us.archive.ubuntu.com/ubuntu/pool/universe/y/yudit/yudit-common_2.9.6-7_all.deb
    --2020-09-20 06:32:21--  http://us.archive.ubuntu.com/ubuntu/pool/universe/y/yudit/yudit-common_2.9.6-7_all.deb
    Resolving us.archive.ubuntu.com (us.archive.ubuntu.com)... failed: Name or service not known.
    wget: unable to resolve host address ´us.archive.ubuntu.com´
  1. Interesting:

https://unix.stackexchange.com/questions/168584/wget-is-unable-to-resolve-host-address-80-of-the-time

Improve this question
8
  • 1
    #3 look like a DNS or other networking failure, not magic. When I tried it, it worked properly. – user535733 Sep 20 '20 at 15:07
  • @user535733 - But why would this fail for wget and not for apt? Note that I am behind a proxy. – sancho.s ReinstateMonicaCellio Sep 20 '20 at 16:00
  • It's not apt vs. wget, it's DNS lookups at Time 1 vs. DNS lookups at Time 2. – waltinator Sep 20 '20 at 16:09
  • @waltinator - I'm sorry, I don't understand the expression "DNS lookups at Time 1". Therefore I don't understand this as an explanation for any of the two related questions in the OP. – sancho.s ReinstateMonicaCellio Sep 20 '20 at 16:17
  • A "DNS lookup" is how a site's name (us.archive.ubuntu.com) is translated to an IP address. "Time 1" and "Time 2" are different times. I'm suggesting that your environment changed in between. /etc/resolv.conf, dnsmasq, connectivity,... – waltinator Sep 20 '20 at 16:25
1

I don't know the exact relation and interoperation of my PC, the proxy, the firewall, and the DNS server.

But I found that the cause apt was succeding in my case, and wget was not, is that I was missing a proper configuration of the proxy for wget. When I added file ~/.wgetrc with

http_proxy = http://username:password@proxy:port
ftp_proxy = http://username:password@proxy:port
proxy_user = username
proxy_password = password 
use_proxy = on

It started working. (Note it is somewhat insecure, as the password is stored in plain text... I wonder if there is any way to have wget ask for the password; or perhaps setting a global .wgetrc in a directory with the same level of security as apt.conf). On the other hand, apt relies on /etc/apt/apt.conf, which was in place.

As for the questions:

Where are apt and wget looking for the information?
Part of the sequence involves setting the proxy. apt and wget have different mechanisms for that, as mentioned above (I know one can set environment variables as well, and that would work at least for wget).

Can I force wget to use the same domain lookup as apt?
I conceive a caveman-way... parsing apt.conf and using that info for wget. Perhaps there is something cleaner.

In the case it is a firewall who is letting apt succeed in knowing ip addresses and pulling files from the web, and precluding wget from doing that, can I somehow "channel" wget requests via apt?
In the light of findings above, this question is replaced by the previous one.

Improve this answer
2
  • If you need to set a proxy, it's best to set it via environment variables http_proxy and ftp_proxy. Then all programs should be able to use them. – raj Sep 22 '20 at 19:53
  • As for your question about the domain lookup, if proxy is set, neither apt nor wget (nor any other program that uses the proxy) does the domain lookup on the target domain. It onl does the domain lookup of the proxy, and then passes the target domain name to proxy which does the lookup. On the other hand, if proxy is not set, the program does the DNS lookup by itself, using your system resolver configuration (/etc/resolv.conf). – raj Sep 22 '20 at 19:56
0

Yes, your Ubuntu system knows where to find .debs. It's a three-step process.

  1. Your system knows where archives (sources) are located. Its in /etc/apt/sources.list.

    $ cat /etc/apt/sources.list
deb http://us.archive.ubuntu.com/ubuntu/ focal main universe
deb http://us.archive.ubuntu.com/ubuntu/ focal-updates main universe
deb http://security.ubuntu.com/ubuntu focal-security main universe
    • The initial sources are placed by the Ubuntu installer. You can change these sources; it's just a text file owned by root. However, you also risk breaking your system, so be sure you know how to restore the originals.

  2. Apt keeps a database of all known packages from all known sources. Name, description, version, size, URI, etc. Apt uses it's database to calculate available upgrades, and to know the URIs to fetch packages from.

    Each time you run sudo apt update, apt downloads a Release file from each source, and uses that Release file to updates apt's database.

    • The database is at /var/lib/apt/lists. It's designed to be machine-readable, not human-readable. Use apt commands (apt search or apt list or apt-cache, etc) to query the database.

    • This is why most instructions say to run apt update right before installing new software -- so apt's database has the correct URI.

  3. When you tell apt to sudo apt install foo, apt consults it's database to look for the right version, uninstalled dependencies, and the URIs of all the packages that it needs.

Improve this answer
8
  • Even if generally useful, I know this explanation. But I am not seeing here an explanation of who is in charge of telling apt where to find us.archive.ubuntu.com in the web (which ip), and then why that same actor isn't conveying the same information to wget. I guess that would answer the question. – sancho.s ReinstateMonicaCellio Sep 20 '20 at 15:59
  • DNS lookup (resolver) is responsible. I'm simply going to repeat what @waltinator said: You had a coincidental network problem when using wget, not a wget or Ubuntu problem. Without the (non-Ubuntu) network problem, wget works as you expect. – user535733 Sep 20 '20 at 19:08
  • Please see my comment below @waltinator. I doubt the coincidence you mention is the cause. – sancho.s ReinstateMonicaCellio Sep 20 '20 at 22:56
  • See my first comment from your question: When I tried wget on the URL, it works properly. – user535733 Sep 21 '20 at 0:27
  • Ok, I understand wget should work, and then there is something "strange" in my case. That is what I'm trying to find out, and what I am asking about. – sancho.s ReinstateMonicaCellio Sep 21 '20 at 7:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.