How to get access from nginx on host OS to files inside lxc-container?

Question

I have Ubuntu 14.04 and lxc-container inside it. Both of host OS and container have nginx 1.4.6 installed.

Container has ip-address 10.0.3.100 and located in /home/romka/.local/share/lxc/php55/rootfs/ (it's a unprivileged container).

Nginx in host OS has following configuration:

upstream php55 {
        server  10.0.3.100;
}

server {
        # some not important directives are skipped

        # serve non static request
        location / {
            proxy_pass          http://php55/;
            proxy_set_header    X-Real-IP       $remote_addr;
            proxy_set_header    Remote-Addr     $remote_addr;
            proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header    Host            $host;
        }

        # process static request
        location ~* ^.+\.(ico|jpg|jpeg|gif|png|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|swf|flv|htc|js|css)$ {
            root /home/romka/.local/share/lxc/php55/rootfs/var/www/do.kece.ru/;
        }

        # some not important directives are skipped
}

This config means that requests to php-scripts are passed to nginx inside the container. Requests to static files should be processed by nginx in host OS.

Requests to php-files are correctly passes to container, but requests to static files are return error "403 Forbidden" and in logfile on host OS I see line:

2014/12/06 15:56:40 [error] 966#0: *9 open() "/home/romka/.local/share/lxc/php55/rootfs/var/www/do.kece.ru/1.txt" failed (13: Permission denied), client: 84.23.63.51, server: do.kece.ru, request: "GET /1.txt HTTP/1.1", host: "do.kece.ru"

Inside the container nginx works on behalf of user and group www-data:www-data with uid:gid - 33:33, inside host OS nginx works on behalf of the same user with uid:gid - 100033:100033. In container configuration I have lines:

lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536
lxc.rootfs = /home/romka/.local/share/lxc/php55/rootfs

When I run 2 commands: "ps uax | grep nginx" and "lxc-attach -n php55 -- ps uax | grep nginx" I see that both instances of nginx has the same uids.

romka@kece:~/.local/share/lxc/php55$ ps uax | grep nginx
root       965  0.0  0.1  85872  1364 ?        Ss   15:29   0:00 nginx: master process /usr/sbin/nginx
www-data   966  0.0  0.2  86512  2804 ?        S    15:29   0:00 nginx: worker process
www-data   967  0.0  0.2  86164  2036 ?        S    15:29   0:00 nginx: worker process
www-data   968  0.0  0.2  86164  2036 ?        S    15:29   0:00 nginx: worker process
www-data   969  0.0  0.1  86164  1784 ?        S    15:29   0:00 nginx: worker process
100000    2763  0.0  0.1  85868  1388 ?        Ss   16:21   0:00 nginx: master process /usr/sbin/nginx
www-data  2767  0.0  0.1  86256  1812 ?        S    16:21   0:00 nginx: worker process
www-data  2768  0.0  0.2  86256  2320 ?        S    16:21   0:00 nginx: worker process
www-data  2769  0.0  0.1  86256  1812 ?        S    16:21   0:00 nginx: worker process
www-data  2770  0.0  0.1  86256  1812 ?        S    16:21   0:00 nginx: worker process
romka     2917  0.0  0.0  11748   916 pts/8    S+   16:24   0:00 grep --color=auto nginx

romka@kece:~/.local/share/lxc/php55$ lxc-attach -n php55 -- ps uax | grep nginx
root       409  0.0  0.1  85868  1388 ?        Ss   13:21   0:00 nginx: master process /usr/sbin/nginx
www-data   413  0.0  0.1  86256  1812 ?        S    13:21   0:00 nginx: worker process
www-data   414  0.0  0.2  86256  2320 ?        S    13:21   0:00 nginx: worker process
www-data   415  0.0  0.1  86256  1812 ?        S    13:21   0:00 nginx: worker process
www-data   416  0.0  0.1  86256  1812 ?        S    13:21   0:00 nginx: worker process

Files has correct chmod and owner:

ls -lh /home/romka/.local/share/lxc/php55/rootfs/var/www/do.kece.ru/
total 8.0K
-rw-r--r-- 1 www-data www-data  6 Dec  5 23:37 1.txt
-rw-r--r-- 1 www-data www-data 17 Dec  5 23:48 index.php

Why nginx from host OS can't get access to files from container? And what can I do to solve the problem?

Answer 1

What I think the issue is is that a parent directory is blocking the permission to the LXC rootfs. So you should move the /var/www directory in the container to /var/local/www/ in the host, then bind mount it into the container and change your host nginx config to point to that new location.

In the host run:

install -m 750 -o 100000 -g www-data -d /var/local/www/
cp -R /home/romka/.local/share/lxc/php55/rootfs/var/www/ /var/local/www
rm -rf /home/romka/.local/share/lxc/php55/rootfs/var/www/

Then edit your container config to add the line

lxc.mount.entry=/var/local/www /home/romka/.local/share/lxc/php55/rootfs/var/www none bind 0 0

Then finally change your nginx config to point to /var/local/www for the static stuff.