ntp does well at keeping your computer set to the right time, and does that without ever running it backwards--which would be a bad thing for some programs you might be running.
It not only sets the time, but it makes continuous adjustments to how your computer keeps time so that its time is not only right at one instant in time, but stays close to the real time (within dozens of milliseconds, not dozens of minutes). It adjusts both phase (the time) and rate (how fast the clock "ticks"). ntp never makes the clock run backwards. It can take a long time to establish how fast to tick the clock after a reboot, so ntp keeps track of the drift in a file called /var/lib/ntp/ntp.drift . Since you aren't running ntp none of this happens.
ntp is not as popular as it once was because sleeping laptops and desktops, and virtual machines stop it from running some of the time. It's designed to run every once in a while on its own schedule on a computer that is running all of the time, and in the real world where time is continuous. That's probably why it isn't installed by default on the modern workstation. [For Vmware, see this]
Instead ntpdate is run when the network interface is brought up. When a sleeping laptop is awakened it reestablishes the network connection, ntpdate is run, and the time is again correct. If the machine's hardware clock is pretty accurate, and the network is brought up and down fairly often, that's generally good enough for most people.
For some reason stock ntpdate doesn't alway run. Use ntpdate-debian instead in this case. Syntax for the for.mer is something like ntpdate ntp.ubuntu.com, for the latter it is ntpdate-debian
In the absence of one of those things, then ntp is a better way to keep time.
Systems are designed to take a timer interrupt every so often and update its idea of the time every interrupt. As long as the hardware timer is running to spec. the time doesn't drift too much. If the hardware timer isn't, the time will drift more (all such clocks will drift some, for the same reason your wristwatch or battery controlled clock will. Clocks plugged into the wall are synced to the time by the frequency and phase kept by your power company).
Most computer timers are controlled by a crystal controlled oscillator circuit on its integrated circuits. Despite the crystal, they run faster and slower depending on the environment, mostly the temperature. Unless you have some time syncing software installed we don't know about, I'd say your system's clock is off spec.
If you were to run ntp for a day or two it would store information in /var/lib/ntp/ntp.drift that would indicate how much it would have to adjust the rate at which your operating system time advances per interrupt in order to match your hardware clock rate with the real time it gets over the internet. Keeping the file the same and just starting and stopping ntp after a minute after that (assuming you keep the /var/lib/ntp/ntp.drift file unchanged) might do a lot to correct for this if the clock bias ntp sets remains after ntp ends. I'm not sure of this detail.
I suspect the value ntp would store in /var/lib/ntp/ntp.drift is far different than mine.
If this machine is kept running all of the time, however, the best thing to do is to install ntp and let it do its thing. See the other answers for details on getting the time right before starting it. I run ntp on my desktop and ntpdate on my laptop.
An interesting possible alternative, adjtimex, is mentioned in this answer by nealmcb.
If your system is not kept running all of the time, running ntpdate at boot time seems like a good option.
Warning some software can freak if the computers time goes backwards. Running ntpdate after boot could cause this to occur.
One gotcha, that can be a problem: As I recall ntp expects the time to not be too far off. If it is, trying to act conservatively, ntp won't adjust the time at all. If you are in this situation it makes sense to do both--run ntpdate at boot to get the time initialized to the right time, and then let the ntp run to keep it running to provide accurate timekeeping. In particular a bad motherboard battery can cause this error, as can booting a computer that's been off for a long time.