A while back when I had Windows and Ubuntu on the same hard drive, I defragged from windows and it messed ubuntu up. Is there a way so that I can defrag it without touching the ext4 partition?
In short - perfectly safe.
If you installed Ubuntu with Wubi - then Ubuntu installation is same as every other file in Windows. Defragmentation won't change contents and any installation image moves would be detected by boot loader.
If you have installed Ubuntu in separate partition, then it is even better - no Windows defragger would touch ext4 partition ad performance should be the best.