FYI, you don't need a separate /boot partition on a raid1; grub2 can boot directly from lvm on raid5.
If your raid5 is partitioned and currently LVM is using /dev/md1p1 as the PV, then you can shrink the PV, shrink the partition, and create a new partition using the free space to use for btrfs.
You will need to use
parted to shrink the lvm partition after using
pvresize to have LVM release part of it. You will need to have
parted use sectors for the units and delete the LVM partition, and recreate it with the same start sector, but a shorter length. This will require either that the partition not be in use ( do it from a livecd ), or will complain that it could not update the kernel, and you will have to reboot for it to take affect.
After that, you can use
gparted to create a new btrfs partition using the free space, then add it to /etc/fstab, mount it, and transfer some files. To move more space, you will need to repeat the
parted shrink step, then use
gparted to move the btrfs partition to the left, and expand. This will take a long time because it has to copy all of the existing data in the btrfs partition.
It would be much easier, safer, and faster to simply create a new LV using whatever space is currently free in the VG, and format it with btrfs, then transfer some files, shrink an existing LV with
resize2fs ( assuming it's ext4, and it will need to be unmounted to do this ) and
lvresize, then add the space to the btrfs lv with
lvresize, and tell btrfs to use the new space with
btrfs filesystem resize max /path/to/btrfs-mount.