The nmap tool will help you to draw a map of the services available from the outside world to your server. As such, it won't help you to protect your server but will tell you where to pay attention. The more open ports you have, the more potential problems you may have.
An open source tool like OpenVAS will be certainly more helpfull to you. OpenVAS is vulnerability scanner that will not only show which ports are open on your server(s) but also try various kind of well-know attacks on these services and report what he found, with link to official security news explaining what the vulnerability is and how to overcome it.
Running this tool on your server will certainly help you to understand how it is possible that your server is regularly hacked.
To secure the servers I'm responsible for, I follow these guidelines :
- Reducing the number of open ports
- Reducing the number of installed software to what is stricly necessary
- Removing unwanted users and groups
- Be sure that each users has a good password or are locked in case of system accounts
- For each application running, review the security guidelines available in their official documentation
- Performing software update when available
- Enabling a valid logging and auditing mechanism to be able to determine how the hacking was done
An successful attack can be a combination of multiple factors : e.g. : weak password allowing unwanted FTP access, bad FTP configuration allowing to go the web root, bad permissions on filesystem allowing to write in the web root.
At your place, I would review all my users password, my FTP setup, my web server setup, the permissions in use and also I would have a look to the log files to see if I can find back a trace of the commands that were run to try understand where the weakness really lay.
A tool like OpenVAS will help you to figure out what is possible to do on your server you'd never thinked about.