Just a data point really...
I just tried converting a 200G partition and using the btrfs-tools V4.0 and it failed - actually it seems it crashed - after reporting 'not enough free space' (partially overwriting the progress info).
# btrfs-convert -l trial /dev/trial/home
creating btrfs metadata.
not enough free space244027/ 240502]
extent-tree.c:2628: btrfs_reserve_extent: Assertion `ret` failed.
fsck reported no problems and the filesystem still mounts:
# fsck -f /dev/trial/home
fsck from util-linux 2.26.2
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/trial-home: 240502/13107200 files (19.2% non-contiguous), 52237869/52428800 blocks
One interesting point is that the progress was reporting more than 100%, i.e 244027/240502. Perhaps someone could clarify what it was counting?
Anyway, the conclusion is that 744M out of 200G (99.7% used) is not enough free space!
FWIW, I would thing that btrfs-convert would get all the metadata in place before destroying (copying and overwriting) the 1Mb which indicated it's an EXT4 filesystem. Until it does that step, there's nothing to undo; it's just been writing in free space. Perhaps this could also be confirmed?