I had the exact same problem.
Sometimes I'd scroll down and the screen would jump up.
OS: VERSION="20.10 (Groovy Gorilla)"
Mouse: some generic USB wireless mouse
I tried another generic USB wireless mouse (from a different manufacturer) and the problem was even worse.
My up wheel is Button 5 so:
xev | grep -B 2 "button 5"
which reliably outputs two events everytime I scroll up one notch.
E.g.
--
ButtonPress event, serial 37, synthetic NO, window 0x7000001,
root 0x7a5, subw 0x0, time 126819928, (119,90), root:(264,229),
state 0x0, button 5, same_screen YES
--
ButtonRelease event, serial 37, synthetic NO, window 0x7000001,
root 0x7a5, subw 0x0, time 126819928, (119,90), root:(264,229),
state 0x1000, button 5, same_screen YES
It was odd that one mouse was worse than the other. Note that the bad mouse is so bad that sometimes you can be constantly scrolling up and it just keeps jumping around in the middle.
So, I then tried another mouse (identical to the first) and that worked perfectly.
I then tried measuring the battery voltage in the mouse to check if it could have been the signal.
Good mouse: 1.24V 40%
Medium mouse: 1.32V 60%
Bad mouse: 1.40V 70%
So the battery has nothing to do with the problem.
I even swapped the battery in the good mouse with the battery in the bad mouse just in case there was a problem with my battery tester but that did not fix things.
Interestingly, on the bad mouse I found I was getting occasional "button 4" events, even when I was scrolling up. E.g. I was seeing many events with:
xev | grep -B 2 "button 4"
I opened up the mouse (not easy as it seems to clip together) and blew on the wheel.
Putting it back together it was significantly better which points to maybe dust on the scroll wheel. i.e. there was only the very occasional "button 4" event when scrolling up which could have been due to some residual dust I missed as I couldn't get into the mouse easily.
In summary, with the 3 mice I had, the problem turned out to be the mouse and dust on the wheel rather than Ubuntu. That is, the bad mouse which was struggling to even scroll at all is now working well.