Ntp does well at keeping your computer set to the right time, and does that without ever running it backwards--which would be a bad thing for some programs you might be running.
It not only sets the time, but it makes continuous adjustments to how your computer keeps time so that its time is not only right at one instant in time, but stays close to the real time (within dozens of milliseconds, not dozens of minutes). It adjusts both phase (the time) and rate (how fast the clock "ticks"). It can take a long time to establish how fast to tick the clock after a reboot, so ntp keeps track of the drift in a file called /var/lib/ntp/ntp.drift . Since you aren't running ntp none of this happens.
It is not as popular as it once was because sleeping laptops and desktops, and virtual machines stop it from running some of the time. It's designed to run every once in a while on its own schedule on a computer that is running all of the time, and in the real world where time is continuous. That's probably why it isn't installed by default on the modern workstation. [For Vmware, see this]
Instead ntpdate is run when the network interface is brought up. When a sleeping laptop is awakened it reestablishes the network connection, ntpdate is run, and the time is again correct. If the machine's hardware clock is pretty accurate, and the network is brought up and down fairly often, that's generally good enough for most people.
In the absence of one of those things, then ntp or regular ntpdate runs are a better way to keep time.
Systems are designed to take a timer interrupt every so often and update its idea of the time every interrupt. As long as the hardware timer is running to spec. the time doesn't drift too much. If the hardware timer isn't, the time will drift more (all such clocks will drift some, for the same reason your wristwatch or battery controlled clock will. Clocks plugged into the wall are synced to the time by the frequency and phase kept by your power company).
Most computer timers are controlled by a crystal controlled oscillator circuit on its integrated circuits. Despite the crystal, they run faster and slower depending on the environment, mostly the temperature. Unless you have some time syncing software installed we don't know about, I'd say your system's clock is off spec.
If you were to run ntp for a day or two it would store information in /var/lib/ntp/ntp.drift that would indicate how much it would have to adjust the rate at which your operating system time advances per interrupt in order to match your hardware clock rate with the real time it gets over the internet. Keeping the file the same and just starting and stopping ntp after a minute after that (assuming you keep the /var/lib/ntp/ntp.drift file unchanged) might do a lot to correct for this if the clock bias ntp sets remains after ntp ends. I'm not sure of this detail.
I suspect the value ntp would store in /var/lib/ntp/ntp.drift is far different than mine.
If this machine is kept running all of the time, however, the best thing to do is to install ntp and let it do its thing. See the other answers for details on getting the time right before starting it. I run ntp on my desktop and ntpdate on my laptop.
An interesting possible alternative, adjtimex, is mentioned in this answer by nealmcb.
If your system is not kept running all of the time, running ntpdate from a cron job every hour or two sounds very reasonable.