Sharing folder with VM through libvirt, 9p, permission denied

Question

An Ubuntu Server 14.04 Host hosts an Ubuntu Server 14.04 guest via libvirt/qemu-kvm. The system runs fine, but - as guest - I have issues writing to a shared folder (<filesystem>) that drive me nuts. Both machines are relative vanilla installations.

I attached the given folder like this:

[host] $ virsh edit guest-vm-name
# ...
<filesystem type='mount' accessmode='mapped'>
  <source dir='/data'/>
  <target dir='/data'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</filesystem>
# ...

From the guest I mount the filesystem as following:

[guest] $ sudo -u www-data mkdir /tmp/mnt
[guest] $ sudo mount -t 9p -otrans=virtio,rw,version=9p2000.L /data /tmp/mnt

I use the www-data user as that will be the efftive user later on, and group and user ids have to match if p9 is used, afaiu. That also means that on the host, /data (which is ext4 partition, LVM on RAID btw) looks like

[host] $ ls -lha /data
[host] $ drwxrwxr-x  4 www-data www-data 4.0K Nov 11 08:34 .
[host] $ drwxr-xr-x 24 root     root     4.0K Nov  7 16:58 ..
[host] $ drwxr-xr-x  2 www-data www-data 4.0K Nov 11 08:34 jail
# ...

In the guest, if I try to write to anything on the shared filesystem, I get permission errors (irrespective of the used user):

[guest] $ sudo -u www-data touch /tmp/mnt/jail/letmeout
touch: cannot touch ‘/tmp/mnt/jail/letmeout’: Permission denied

I can read files though

[guest] $ cat /tmp/mnt/jail/throughthewindow
Great Weather!

I tried various things, especially:

• stopped apparmor service and called aa-complain (i hope that was effective)
• set security to none in /etc/libvirt/qemu.conf
• set user and group to root in /etc/libvirt/qemu.conf

/var/log/syslog and dmesg do not show anything suspicous.

Any pointers?! Thanks.

Answer 1

I know, that this is an old thread, but I just came across a similar problem and found a solution that works at least in parts.

I also changed the user and group values in /etc/libvirt/qemu.conf to root, like you did.
But I also changed the dynamic_ownership setting, because the description sounded promising:

# Whether libvirt should dynamically change file ownership
# to match the configured user/group above. Defaults to 1.
# Set to 0 to disable file ownership changes.

My setup is:

• Host: Debian 8 (Jessie)
• Guest: Debian 8 (Jessie)
• shared folder on host belongs to root
• local user w/ id 1000 is member of the group libvirt (might be important)
• mount point on the guest (/mnt/data) belongs to user 1000 ("alexander")

I can now finally write files in the mounted shared folder with both root(0) and alexander(1000). (before I did that, only root was allowed to write files here)

Setting dynamic_ownership to 0 is what has to be done in "mapped" accessmode.

Here's some more information about my setup:

/etc/libvirt/qemu.conf:

user = "root"
group = "root"
dynamic_ownership = 0

filesystem (virsh edit):

<!-- ... -->
    <filesystem type='mount' accessmode='mapped'>
      <source dir='/share/vm.localdomain/owncloud_data'/>
      <target dir='/data'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </filesystem>
<!-- ... -->

Host:

ls -al /share/vm.localdomain/owncloud_data/
total 8
drwxr-xr-x 2 root      root      4096 Sep  8 00:15 .
drwxr-xr-x 7 root      root      4096 Sep  8 00:10 ..

fstab on guest:

/data  /mnt/data/  9p  trans=virtio  0  0

Guest:

root@debian:~# cd /mnt/data
root@debian:/mnt/data# touch touched_by_root
root@debian:/mnt/data# su - alexander
alexander@debian:~$ cd /mnt/data
alexander@debian:/mnt/data$ touch touched_by_user
alexander@debian:/mnt/data$ ls -al
total 16
drwxr-xr-x 2 alexander alexander 4096 Sep  8 00:30 .
drwxr-xr-x 6 root      root      4096 Sep  7 18:13 ..
-rw-r--r-- 1 root      root         0 Sep  8 00:30 touched_by_root
-rw-r--r-- 1 alexander alexander    0 Sep  8 00:30 touched_by_user

back on host:

root@Host /share/vm.localdomain # ls -al /share/vm.localdomain/owncloud_data/
total 16
drwxr-xr-x 2 root      root         4096 Sep  8 00:30 .
drwxr-xr-x 7 root      root         4096 Sep  8 00:10 ..
-rw------- 1 root      libvirt-qemu    0 Sep  8 00:30 touched_by_root
-rw------- 1 root      libvirt-qemu    0 Sep  8 00:30 touched_by_user

conclusion

The strange thing is that on the guest these two files belong to different users (root vs. alexander), whereas on the host, both files belong to the same user (root:libvirt-qemu). :-O
I don't know exactly how this magic works, but apparently it does.

Hope this helps,
Alexander

Answer 2

@Alexander gave a great answer. I didn't do everything that he did, but this is what I did do to get same-user rights working both ways across the 9p file system. (This way is intended to just get 9p to work, both ways, without security hassles. For more security, you will need a different method or settings)

HOST (order is unimportant)

• I added my host user to the following group: <kvm> (I am also a member of <libvirtd>). This step may be unnecessary because:

• in the file /etc/libvirt/qemu.conf you can change the user and group with which all of your VMs authenticate and execute.

(this is a powerful little change, and the repercussions need to be mapped out if you are trying to accomplish this on something like a production server)

user = "billy"    #### No, my name isn't billy, but it's cute. 
                  #### Alternatively you can declare your <uid>, like
                  ## user = "+1000" ####        << That's what I did.
                  #### (They tell you everything you need to know about
                  #### this stuff inside /etc/libvirt/qemu.conf)
group = "billy"
dynamic_ownership = 1

(What the above change does is tells the virtual machine host to convert all libvirt cross-VM-requests for any guest VMs you are running to the <user> that you set; including guest-VM <root>. *Note again that it is a system-wide setting with regard to libvirt and by extension qemu if libvirt is your sole qemu interface.)

• Setup you host as found in this excellent official document https://www.linux-kvm.org/page/9p_virtio

(by the way, "Squash" references a 9p security model. It means "none" and is the most permissive setting for this context)

GUEST

• Inside the guest VM, mount the 9p share using <root> privileges, also found at the bottom of the document https://www.linux-kvm.org/page/9p_virtio

(I also added one option as explained in the document below) My mount command is:

mount -t 9p -o trans=virtio,access=any,version=9p2000.L /hostshare /tmp/host_files

Nore that if you don't have permission to write to the share or have limited permission after you have set this up, the excellent suggestion by @randomocean 's answer should help. That is, have <root> create a folder under the share, and set chmod 777 to it.

-Even More Info: https://wiki.qemu.org/Documentation/9psetup

Answer 3

I came across the same problem when I was trying to mount a directory from Ubuntu host to Minikube's VM. The question does not specify Minikube, but it does not matter since it is a VM run by QEMU/KVM. The instructions below should apply to any QEMU/KVM VM. I could not find specific instructions on how to mount host directories into KVM VMs. I searched a lot and it was quite tiresome. I will explain what I did to handle permission errors as well as symlink errors or "Unknown error 526" errors. I am going to mention briefly why it was necessary to have this setup, just to include a few keywords that someone having the same issues might use when searching for a solution.

Problem

I was running Minikube via VirtualBox. VirtualBox can mount /Users on MacOS, or /home on Linux, into the VM, and you do not need to tackle mounting problems as much as you have to do with KVM. I needed to develop an Android application whose backend should run on Kubernetes. To test the Android application, I need to run virtual devices, which are run via KVM on Linux. One cannot run a VM on VirtualBox and a VM on KVM at the same time, because only one of them can use the CPU's virtualization technology at a time. It would be quite an inconvenience to shut down Minikube when I would need to check something in the Android application or vice versa. Since it is possible to run many VMs on KVM at the same time, I decided to run Minikube on KVM just for the sake of running both Minikube and Android virtual devices at the same time. It is also possible to run many VMs at the same time with VirtualBox, but the default Android virtual devices are not run on VirtualBox. It looks like Genymotion uses VirtualBox but it is a paid solution.

Goal

My goal was to mount the directory that contains my application files just like VirtualBox mounts. Minikube can mount directories from host via minikube mount command, but, at the time of writing, it is not reliable for some reason. It gave me Unknown error 526 while I tried to access the files. It is also very slow.

@Alexander's answer helped me solve the issue, in fact. But, it was not quite enough, since I did not want to run the VMs as root. When they are run as root, the files and/or directories created inside the VMs have root:root permission in the host, which I do not want.

Solution

Defining the user that runs VMs

First of all, I edited /etc/libvirt/qemu.conf file (on the host) to change the user who runs the VM processes as my user, which is ubuntu, and to disable dynamic_ownership:

user = "ubuntu"
group = "ubuntu"
dynamic_ownership = 0

After making the change above, I restarted my host machine, since I do not know how to apply the changes otherwise. Without a restart, the VMs kept being run as the previous user, which was libvirt-qemu. To ensure the change has been applied, do the following:

1. On your host machine, run ps aux | grep kvm
2. Check the user that runs the VM processes. You should see ubuntu in this case.

Defining the mount volume

Next, add a Filesystem hardware in Virtual Machine Manager:

1. On the host, open Virtual Machine Manager by running virt-manager
2. Double click on the VM and select View > Details
3. In the Details window, click Add Hardware.
4. Select Filesystem and configure it like so:

Type: mount (only option)
Driver: Path
Mode: Squash
Write Policy: Immediate
Source Path: <absolute path to the directory you want to mount into the VM>
Target Path: hosthome # Or any other keyword or path that will be used to define the mount so that we can reference it when mounting it in the VM

# Do not check "Export filesystem as readonly mount"

5. Click Apply. If the VM was running, the changes will be applied at the next boot.

Squash mode is important, because it can handle symbolic links. The other modes gave me too many levels of symbolic links errors. Passthrough mode solved the issues that Mapped mode gave, but it was not enough. Squash mode is the mode that did not give me any errors.

Solving permission issues that might occur when running the VM

Now, start (or restart if it was running) the VM. If you get permission errors while starting the VM, then you need to make sure that the files used to run the VM belong to the user you defined in qemu.conf file. In my case, I changed the following files' permissions on the host by examining the error messages:

chown ubuntu:kvm ~/.minikube/machines/minikube/boot2docker.iso # Previously belonged to "libvirt-qemu" user
chown ubuntu:root ~/.minikube/machines/minikube/minikube.rawdisk # Previously belonged to "root" user

Mounting the volume to the VM

Next, the only thing left to do is to mount the volume into the VM. Get into the VM, in my case via minikube ssh, and then mount the volume:

# First, create the directory to which the the volume will be mounted. This might not be necessary in your case.
sudo mkdir -p /hosthome/ubuntu/Projects

# Then, mount the volume
sudo mount -t 9p -o trans=virtio,version=9p2000.L,cache=none,msize=262144,rw hosthome /hosthome/ubuntu/Projects

• cache=none: This is for improved performance. You can change this to another value depending on your case. The values are explained here.
• msize=262144: This is for improved performance as well. This defines "the number of bytes to use for 9p packet payload", as explained here. This is the default value used by minikube mount.
• rw: "Mount the filesystem read-write", as explained in man mount. This is an option of mount command.

That's it. You should be able to read and write files/directories from within the VM without any issues. If /etc/fstab file inside the VM can be modified, the mounting logic can be added into the file so that the volume is mounted automatically after the VM starts. Unfortunately, this is not possible for Minikube (See Disadvantages section).

What I gained with this setup

The problems this setup solved for me are:

• minikube mount was slow and unreliable, as it is explained in the documentation. The documentation says that 9p mounts are unreliable, but this setup is more reliable than minikube mount.
• The files/directories created by Docker containers inside the mounted directory have ubuntu:ubuntu ownership on the host. So, I can access them from within the IDE and change them as I please. Previously, they had root:root ownership, which my user on the host did not have.
• I no longer get Unknown error 526 which I got with minikube mount
• The performance is higher than minikube mount
• I can run virtual Android devices with hardware acceleration as well as a Kubernetes cluster at the same time such that the development is much more convenient. Actually, the development is "possible" now.

Disadvantages

• For Minikube, the volume should be mounted manually each time minikube is started. This is because Minikube does not persist the changes made to /etc/fstab file. As a convenience workaround, a shell script that contains the mount command can be created inside /data directory, which survives through VM restarts, and it can be executed from within the VM, after the VM is started.

References

• The answers to this question
• man mount
• Sharing directories with virtual machines and libvirt
• v9fs: Plan 9 Resource Sharing for Linux

Answer 4

I had this problem too, created a folder named /home/user/shared on the host and then used virt-manager to add the folder and mounted it as a 9p virtio on the qemu VM.

I checked out the settings in /etc/apparmor.d/libvirt and it has entries for the new /home/user/shared, and I noticed it only had 'r' permission for /home/user/shared but it has rw for all files in /home/user/shared/. I tried adding a w for write permissions, but that didn't seem to get saved so I went into the host /home/user/shared and created a sub directory and did a chmod 777 on that sub directory. That worked and the guest VM was able to write to the su director any create and edit files.

tldr: create a subdirectory in the shared folder with 777 permissions and see if you can write to that.

Answer 5

For me, adjusting a proper ACL mask on the host machine did it:

sudo setfacl -Rm u:libvirt-qemu:rwx /data

(Ubuntu Server 20.04 host, Debian 10 guest, and passthrough access mode)