You made some mistakes, but the main problem is in ubiquity and grub. Basically, when you set / to be an encrypted partition and don't create a separate partition for /boot, grub gives an error message like:
I know /boot is encrypted. You need to set GRUB_ENABLE_CRYPTODISK=y in /etc/default/grub. I won't do it for you, so I'm going to fail and your installation will stop.
An overview of the process
We use EFI mode.
We install to an unencrypted /boot partition and an encrypted btrfs / using the standard installer.
After the installer finishes, we chroot, make some important configuration changes, and re-install grub to the EFI System Partition and re-create initrd.
The detailed steps
Boot from Ubuntu 16.04 install disk (tested with Xubuntu).
Connect to the Internet and run sudo apt update && sudo apt upgrade to update the installer components
Use fdisk, gparted, or another tool to create 3 partitions:
A GPT partition table
A 200MB partition that we will use for the EFI System Partition
A multi-gigabyte partition that we will eventually use as our encrypted swap partition, but which will function as our temporary unencrypted /boot
An encrypted partition that uses the rest of the space
Choose "Something else" when asked about installation type.
Configure /dev/sda1 as EFI System Partition
Configure /dev/sda2 as ext2, formatted, with mount point of /boot
Configure /dev/mapper/sda3_crypt as btrfs with mount point of /
Continue with the installation.
After it finishes, choose to stay in the live system (no reboot).
Copy the contents of /boot and do a chroot
sudo mount -o subvol=@ /dev/mapper/sda3_crypt /target
sudo mount /dev/sda2 /mnt
# (Watch those trailing slashes! rsync is very sensitive to them.)
sudo rsync -aXAH /mnt/ /target/boot/
sudo mount /dev/sda1 /target/boot/efi
sudo mount --bind /dev /target/dev
sudo mount --bind /proc /target/proc
sudo mount --bind /sys /target/sys
sudo chroot /target
(Everything is now happening as chroot inside your new system.)
Add line to /etc/default/grub
Add line to /etc/crypttab. You will need to first run sudo blkid to find the UUID of /dev/sda3 (NOT /dev/mapper/sda3_crypt).
sda3_crypt UUID=<UUID of /dev/sda3> none luks,discard
Edit /etc/fstab and delete the line for /boot. The other entries are correct.
Install grub to the EFI System Partition, generate a new grub.cfg, and prepare initrd.
Optional double-check: Double-check that /boot/efi/EFI/ubuntu/grub/grub.cfg contains lines that include insmod luks, cryptomount -u <UUID>, the correct boot entries, etc. And double-check that your initrd contains the cryptsetup binary. If these things are missing, it is because grub-mkconfig and/or update-initrd couldn't figure out how the volumes that you've mounted or specified in fstab relate to the encrypted volume in crypttab. (There's a lot of magic autoconfiguration that they do.) This may happen if you diverge from this guide by, for example, using ZFS or by trying to partition sda3_crypt.
(If using ZFS instead of btrfs) grub-mkconfig and update-initrd won't recognize ZFS. The workaround involves (during chroot, prior to grub-mkconfig/update-initrd) editing /usr/sbin/grub-mkconfig to add || true to line 139 (which starts with GRUB_DEVICE=), adding GRUB_DEVICE="/dev/mapper/sda3_crypt" to /etc/default/grub, creating file /usr/share/initramfs-tools/conf-hooks.d/forcecryptsetup with contents export CRYPTSETUP=y and file /etc/initramfs-tools/conf.d/cryptroot with contents target=sda3_crypt,source=UUID=<UUID of sda3>,key=none,discard. All of this is in addition to steps that you would take if you were not encrypting the ZFS partiion (like installing zfs userspace utils in both the live system and during chroot and deleting the line that mounts / in fstab).
You should see grub asking for your password. Then you'll get the boot menu. After choosing Ubuntu you'll be asked for your password again. Then you'll be in your system. Read more about how Ubuntu uses BTRFS.
TODO: Create encrypted swap (hint: it involves editing crypttab, fstab, and re-running update-initrd).
TODO: Save your password so you only need to enter it once into grub. This is detailed here.
Every time you install a new kernel, you should run the custom grub-mkconfig command.
Every time you update grub, you should run the custom grub-installcommand.
It's tempting to create a single encrypted volume and partition it to create the swap partition (and possibly others), but this does not work. Both grub-mkconfig and update-initrd will misbehave. However, I haven't tried LVM.
grub+ your title. I suppose you found your procedure for encrypting at pavelkogan.com/2014/05/23/luks-full-disk-encryption (which is the first google search on this topic). if not, maybe that will help. Good Question! and Good Luck!
/bootas the mount point for everything. I'm hoping that's a typo. 2. You did not mount
/devwhile chrooting, but you did mount devpts. O.o