There is another way if you want to get rid of it permanently. If you are happy with Matt2's solution there is no need to do so. I wanted remove it totally, because the mouse gestures provide by the extension All-in-one Gestures don't work on new tabs with the new startpage (by the way, I think its official name is new tab page).
Enter this address to the location bar and confirm that you will be careful:
Search for the preference with name:
Update: you can set browser.newtabpage.enabled (Step 2) to False (Step 3) instead of changing browser.newtab.url.
Double click to modify and set the value to
if you prefer to see your own startpage (set its url in the standard preferences dialog).
Close the window, you're done!