Knockd listens to the link layer, in the OSI model, this is layer 2. Layer 3 is where ip addresses live. Therefore, you will not be able to listen to only a single ip address.
The work around would be to create a MACVTAP, therefore creating a new layer2 device. Move the IP address of 124.124.124.124 to that interface, and running Knockd on the new interface eg macvtap0.
ip link add link eth0 macvtap0 type macvtap
ip address del 124.124.124.124/<insert the subnet> dev eth0
ip address add 124.124.124.124/<insert the subnet> dev macvtap0
Note: creating a macvtap, creates an interface with a new mac address, depending on your ISP/Upstream provider this may create an issue. If this occurs, add a route for outbound traffic from 124.124.124.124, to route through one of your other external IP address. So your upstream provider will see the mac address of the physical device. (this is the easiest way, I know there are other ways to have outbound traffic use the main mac address)