Now when I scroll up or down by one tick, the text moves up and down by 5 lines, which is not what I want. How do I change that number to 1? Is it even possible?
The scrolling speed is hardwired in the source. There's no config option to modify it, your only chance is to modify the source and rebuild it.
You'll need to modify this line of Ubuntu package libvte-2.91-0 (source package: vte2.91) to contain
v = 1.0; instead.
Please refer to other sources describing how to patch and recompile an Ubuntu package. Once the new package is installed, you need to quit all gnome-terminal windows at once for the change to take effect.