10

TL;DR

I've got a strange situation where I can do DNS lookups on some hosts but not others. This seems to be related to resolv.conf having one nameserver entry pointing to my nameserver and another that is presumably to do with docker, but I'm not sure how to fix it.

The problem

I've been reading Stéphane Graber's excellent introduction to LXD and wanted to try it out. Therefore, I did:

$ sudo usermod -a -G lxd <myusername>
$ newgrp lxd
$ sudo lxd init

I configured it with all of the default settings. I then typed:

$ lxc image list images:
error: Get https://images.linuxcontainers.org/streams/v1/index.json: lookup images.linuxcontainers.org: no such host

Some tests

I tried accessing that address from a web browser on another PC and it worked fine. So I figured something must be wrong with the DNS set-up, but:

$ host images.linuxcontainers.org
images.linuxcontainers.org is an alias for canonical.images.linuxcontainers.org.
canonical.images.linuxcontainers.org has address 91.189.91.21
canonical.images.linuxcontainers.org has address 91.189.88.37
canonical.images.linuxcontainers.org has IPv6 address 2001:67c:1560:8001::21
canonical.images.linuxcontainers.org has IPv6 address 2001:67c:1562::41

So I tried wget:

$ wget https://images.linuxcontainers.org/streams/v1/index.json
--2016-11-10 15:56:22--  https://images.linuxcontainers.org/streams/v1/index.json
Resolving images.linuxcontainers.org (images.linuxcontainers.org)... failed: Name or service not known.
wget: unable to resolve host address "images.linuxcontainers.org"

which made me think that there was a problem with my internet connection, but if I use us.images.linuxcontainers.org (which I saw mention of somewhere on the web):

$ wget https://us.images.linuxcontainers.org/streams/v1/index.json
--2016-11-10 15:57:26--  https://us.images.linuxcontainers.org/streams/v1/index.json
Resolving us.images.linuxcontainers.org (us.images.linuxcontainers.org)... 91.189.91.21, 2001:67c:1562::41
Connecting to us.images.linuxcontainers.org (us.images.linuxcontainers.org)|91.189.91.21|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3086 (3.0K) [application/json]
Saving to: "index.json"

index.json                                100%[==================================================================================>]   3.01K  --.-KB/s    in 0s

2016-11-10 15:57:26 (8.36 MB/s) - "index.json" saved [3086/3086]

I also tried canonical.images.linuxcontainers.org, which (according to host above) is what images.linuxcontainers.org is an alias of and that worked too, so it looks like host can lookup images.linuxcontainers.org, while wget and lxc cannot, but wget can access canonical.images.linuxcontainers.org and most other sites I've tried.

$ wget https://canonical.images.linuxcontainers.org/streams/v1/index.json
--2016-11-10 16:02:28--  https://canonical.images.linuxcontainers.org/streams/v1/index.json
Resolving canonical.images.linuxcontainers.org (canonical.images.linuxcontainers.org)... 91.189.91.21, 91.189.88.37
Connecting to canonical.images.linuxcontainers.org (canonical.images.linuxcontainers.org)|91.189.91.21|:443... connected.
ERROR: no certificate subject alternative name matches
        requested host name "canonical.images.linuxcontainers.org".
To connect to canonical.images.linuxcontainers.org insecurely, use `--no-check-certificate'.

$ wget --no-check-certificate https://canonical.images.linuxcontainers.org/streams/v1/index.json
--2016-11-10 16:02:37--  https://canonical.images.linuxcontainers.org/streams/v1/index.json
Resolving canonical.images.linuxcontainers.org (canonical.images.linuxcontainers.org)... 91.189.88.37, 91.189.91.21
Connecting to canonical.images.linuxcontainers.org (canonical.images.linuxcontainers.org)|91.189.88.37|:443... connected.
WARNING: no certificate subject alternative name matches
        requested host name "canonical.images.linuxcontainers.org".
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://uk.images.linuxcontainers.org/streams/v1/index.json [following]
--2016-11-10 16:02:37--  https://uk.images.linuxcontainers.org/streams/v1/index.json
Resolving uk.images.linuxcontainers.org (uk.images.linuxcontainers.org)... 91.189.88.37, 2001:67c:1560:8001::21
Connecting to uk.images.linuxcontainers.org (uk.images.linuxcontainers.org)|91.189.88.37|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3086 (3.0K) [application/json]
Saving to: "index.json.1"

index.json.1                              100%[==================================================================================>]   3.01K  --.-KB/s    in 0s

2016-11-10 16:02:38 (96.5 MB/s) - "index.json.1" saved [3086/3086]

I've also tried wget -4 and wget -6 to rule out IPv6 issues, but results were the same either way. Finally I tried some other programs like w3m but no difference there either.

I'm obviously missing something; can anyone offer any advice on why I can't get lxc to download the image list?

The PC

The PC is a relatively new install running Ubuntu Server 16.10 with very few additional packages installed in the main host. Docker is installed and running but no containers are up. Interestingly, I recently rebooted into kernel 4.8.6 to test another problem I was having and with that kernel I could access images.linuxcontainers.org, but docker wouldn't start, so I'm wondering if this might be to do with docker.

Configuration

/etc/resolv.conf looks like this (but for some reason I'm not aware of, is actually a symlink to /run/resolvconf/resolv.conf):

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 192.168.1.254
nameserver 127.0.0.53
search lan

If I do query the first listed nameserver I get the same result as if I don't append the nameserver address:

$ host images.linuxcontainers.org 192.168.1.254
images.linuxcontainers.org is an alias for canonical.images.linuxcontainers.org.
canonical.images.linuxcontainers.org has address 91.189.91.21
canonical.images.linuxcontainers.org has address 91.189.88.37
canonical.images.linuxcontainers.org has IPv6 address 2001:67c:1560:8001::21
canonical.images.linuxcontainers.org has IPv6 address 2001:67c:1562::41

If I do query the second, it times out without reaching a server:

$ host images.linuxcontainers.org 127.0.0.53
;; connection timed out; no servers could be reached

If I query the second but use the canonical name it works and then times out???

$ host canonical.images.linuxcontainers.org 127.0.0.53
Using domain server:
Name: 127.0.0.53
Address: 127.0.0.53#53
Aliases:

canonical.images.linuxcontainers.org has address 91.189.88.37
canonical.images.linuxcontainers.org has address 91.189.91.21
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached

Edit 1:

/etc/nsswitch.conf looks like this:

# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:         compat
group:          compat
shadow:         compat
gshadow:        files

hosts:          files resolve [!UNAVAIL=return] dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis

Edit 2

The modified nsswitch.conf now looks like this:

# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:         compat
group:          compat
shadow:         compat
gshadow:        files

hosts:          files resolve dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis

Edit 3

Contents of /etc/systemd/resolved.conf:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See resolved.conf(5) for details

[Resolve]
#DNS=
#FallbackDNS=8.8.8.8 8.8.4.4 2001:4860:4860::8888 2001:4860:4860::8844
#Domains=
#LLMNR=yes
#DNSSEC=no
#Cache=yes
12
  • 1
    /etc/resolv.conf is usually a symlink on Ubuntu, that's normal. What does /etc/nsswitch.conf contain?
    – muru
    Commented Nov 11, 2016 at 8:43
  • @muru I've added it above (this one isn't a symlink)
    – DrAl
    Commented Nov 11, 2016 at 8:45
  • 2
    @ThatGuy Thanks for that. This won't work with every server (as it depends on whether the server looks out for the domain name) - in this case I could just use canonical.images.linuxcontainers.org anyway as this seems to work. However, I also would need to persuade lxc image list and lxc launch to use this instead of the built-in address.
    – DrAl
    Commented Nov 11, 2016 at 13:07
  • 1
    @DrAl, This is the first time to see resolve hosts keyword, it seems wrong. You should have something like hosts: files dns [!UNAVAIL=return] or if you have mDNS installed files dns mdns4_minimal [NOTFOUND=return] mdns4 . you can remove [NOTFOUND=return] or [!UNAVAIL=return] , it's the default action anyway if nothing left to query.
    – user.dz
    Commented Nov 20, 2016 at 10:14
  • 1
    @user.dz, I've tried removing this (although it is the default set up by my installation as I've never previously touched this file). This seems to allow wget to work (albeit after a few seconds delay "resolving images.linuxcontainers.org). Thank you! Any idea why this might have been added to my /etc/nsswitch.conf? The list of packages I've installed onto the server is listed in a link in the "The PC" section above.
    – DrAl
    Commented Nov 27, 2016 at 12:44

1 Answer 1

Reset to default
2

  • This is the first time to see resolve hosts keyword, it seems wrong. You should have something like

    hosts: files dns [NOTFOUND=return]

    or if you have mDNS installed

    hosts: files dns mdns4_minimal [NOTFOUND=return] mdns4

    You can remove [NOTFOUND=return] or [!UNAVAIL=return] , it's the default action anyway if nothing left to query.

  • Well, after some digging I could find that there is an NSS module new to me

    libnss-resolve

    nss module to resolve names via systemd-resolved

nss-resolve is a plugin for the GNU Name Service Switch (NSS) functionality
of the GNU C Library (glibc) providing DNS and LLMNR resolution to programs via
the systemd-resolved daemon (provided in the systemd package).

Installing this package automatically adds resolve to /etc/nsswitch.conf.

    You could end up installing it somehow and not with the packages you mentioned. There is no package depends on it.

    ~$ apt-cache rdepends libnss-resolve
libnss-resolve
Reverse Depends:

    But I can't say anything right now, why systemd-resolved was unreliable. It is better to report it as bug in systemd.

13
  • 1
    Interesting, thank you. I tried removing "resolve" and the wget got a lot quicker, so it looks like that was what was causing the problem. My hosts line in the nsswitch.conf now just has "files dns".
    – DrAl
    Commented Nov 27, 2016 at 14:27
  • 1
    I used dpkg-query -l '*resolve*' to check and found that libnss-resolve is installed, so I installed aptitude and ran aptitude why libnss-resolve and it says i ubuntu-standard Recommends libnss-resolve.
    – DrAl
    Commented Nov 27, 2016 at 14:28
  • 1
    @DrAl, which release is this? I've U16.04 no such dependency between ubuntu-standard & libnss-resolve .
    – user.dz
    Commented Nov 27, 2016 at 14:33
  • 1
    That gives us some light about the current setup, but you have the last fix , which means this another bug (systemd-resolved/nss-resolve is unreliable). Do you have its service running systemctl status systemd-resolved.service
    – user.dz
    Commented Nov 27, 2016 at 14:45
  • 1
    @DrAl, so it does use defaults, from man resolved.conf , that means we are back to /etc/resolv.conf :/ . resolve should have same effect as dns in nsswitch.conf.
    – user.dz
    Commented Nov 27, 2016 at 15:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .