What you're asking for is split tunneling with OpenVPN for the torrent client Transmission and the complexity of that depends on the firewall rules you prefer. I'd have written the basics but since there were already really good detailed articles on how to setup a split tunneling credential with fallback and make only Transmission to follow that credential, I'm only giving some pointers assuming you already have a working OpenVPN config and proper DNS restriction (no leaks).
sudo apt install iptables resolvconf apt-utils while you may already have some of the programs. Now add a new user to your system that won't have any superuser ability whatsoever,
sudo adduser --disabled-login zzz. zzz is the name of this new user. To avoid service permission headache add zzz to your group and yours to zzz.
sudo usermod -aG zzz <username>
sudo usermod -aG <group_username> zzz
ip route list and find the line that looks like
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.101.
eth01 is the active network interface and
192.168.1.101 is the inet. Yours could be different. Watch out for these as you follow the upcoming scripts.
sudo iptables --flush. Append a rule that restricts the zzz user to use regular network interface, only works through the OpenVPN tunnel.
sudo iptables -A OUTPUT ! -o lo -m owner --uid-owner zzz -j DROP
sudo apt install iptables-persistent and press YES for any permission it asks for.
4. Create a new script file on openvpn config directory
sudo nano /etc/openvpn/iptables.sh and add this script.
export VPNUSER="zzz" # watch out.
# Look up the first article on "iptables Script for vpn User" section for the script, I don't know if I'm allowed to paste that.
# The script simply uses iptables built-in chains to redirect network traffic to VPN.
Make this script executable
sudo chmod 755 /etc/openvpn/iptables.sh.
Create an IP routing script
sudo nano /etc/openvpn/routing.sh and type
# Look up the first article on "Routing Rules Script for the Marked Packets" section for the script, I don't know if I'm allowed to paste that.
# The script simply puts some routing commands to block the traffic when VPN goes down.
Make this script executable
sudo chmod 755 /etc/openvpn/routing.sh
5. Add the new zzz user value to the IP routing table
sudo nano /etc/iproute2/rt_tables. Just add
200 zzz at the last line.
Additionally you can add a new kernel parameter config as
sudo nano /etc/sysctl.d/zzz.conf and put
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.eth0.rp_filter = 2
This may not be necessary since it's just hardening the security to kernel level.
Reload sysctl and systemd
sudo sysctl --system,
sudo systemctl daemon-reload. Now check if openvpn is running okay,
systemctl status firstname.lastname@example.org and your openvpn end-IP as user zzz,
sudo -u vpn -i -- curl ifconfig.co
curl shows the right VPN IP you're good to go for the next steps. You shouldn't be disconnected from the server.
6. Stop the transmission-daemon service,
sudo systemctl stop transmission-daemon. Create a new directory and a
systemd config at
sudo mkdir -p /etc/systemd/system/transmission-daemon.service.d && sudo nano /etc/systemd/system/transmission-daemon.service.d/local.conf. Paste the config
ExecStart=/usr/bin/transmission-daemon -f --log-error -g /etc/transmission-daemon
This will allow
transmission service to run only after openvpn service has run. Reload systemd
sudo systemctl daemon-reload.
7. Adding proper user, group and permission to transmission
sudo chown -R zzz:zzz /etc/transmission-daemon/ && sudo chmod -R 775 /etc/transmission-daemon/
sudo chown -R zzz:zzz /var/lib/transmission-daemon/ && sudo chmod -R 775 /var/lib/transmission-daemon/
Do the same for all download folders of transmission like
sudo chown -R zzz:zzz ~/Downloads && sudo chmod -R 775 ~/Download.
Add the changed directory to transmission settings
sudo nano /etc/transmission-daemon/settings.json,
8. Finally turn on
sudo systemctl start transmission-daemon.service. If you're using the Transmission Web GUI follow the nginx method. You can try checking if the torrent is working with the openvpn IP here.