My ssh server is set up on the standard port, and I forwarded the correct port in my router, but I'm still getting a "connection refused" message when trying to connect from the outside. I can connect from the inside just fine.
If you can connect from your local network, Ubuntu most likely has nothing to do with it - it's a network configuration issue.
Many ISPs block incoming connections on "unusual" ports (or all incoming connections). Some ISPs have a setting in their control panel enabling/disabling port blocking.
As a diagnostic - try exposing port 80 (Apache running some webpage) - I'm almost sure you'll find it doesn't work either.
You can also fiddle with modem settings (how do you know you forwarded "the correct port"?), but this would only help after you triple-checked your provider doesn't block you.