Sometimes internet routers have problems passing through NTP traffic. The reason is that UDP is a bit more trickier to forward than TCP and sometimes the port is even used on the device itself for an NTP daemon.
In my case it was DD-WRT which blocked all of my NTP traffic. And as you might have noticed: NTP is not very good in saying that something went wrong. And especially not good in saying what went wrong. Here is the link to a thread about DD-WRT: https://superuser.com/questions/386240/allow-lan-clients-to-sync-time-through-dd-wrt
You can easily check if your Router is blocking NTP with the tool ntpdate (which is kind of a single use version of ntpd):
ntpdate pool.ntp.org - this call behaves like ntpd as long you run it as root and ntpd is not running at the same time. This command fails if your router blocks NTP
ntpdate -u pool.ntp.org - this call uses an unprivileged port, which much more likely works.
In case your router blocks NTP, what can you do?
Unfortunately, ntpd doesn't support any other way of communication than opening the port 123.
Fortunately, there are alternative implementations which use different ports:
I got good results with OpenNTPD, but in Forums people also recommended chrony
You can also setup just one OpenNTPD host and let all other machines in your network sync to this machine with their default NTP implementation.