I tried installing MatLab from the software center out of curiosity, without having installed the executables. This led to the install unable to complete. I closed the GUI of the installer window, but that did not cancel the install. I see no cancel button.
I assume this is what you are talking about?
Near the top of the Software Center is a bar with "All Software, Installed, and History" When you have a install working, there is also an In progress to the right of History
Click on the In Progress tab and you will see Matlab with the progress bar, next to the progress bar will be an X if you click the X it will cancel the install.
If you have already installed it, you can not cancel it but you can completely remove it from your system. The easiest way to get rid of it completely would be in a terminal, enter this command
This method does not install MatLab, but rather tries to configure it for better integration with the Ubuntu/Debian standard (for applications installation).
Do you already have MatLab (non-free software)?
Have you tried going back into Software Center, searching installed software for 'MatLab', and then click on 'Remove'?