0

Problem statement: gitlab VM isn’t reachable via HTTP after upgrade/reboot to KVM host.

Topology: Topology

Troubleshooting notes:

The gitlab VM is reachable via SSH from all hosts in the topology. The gitlab VM is reachable via HTTP from itself, from several interfaces on the KVM host (wspbm) but not from the PCs in the topology. IPTABLES rules were added to the INPUT and FORWARD chains to permit everything, no change.

SSH from PC to gitlab VM:

[root@los-alamos]$ ssh gitlab ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:52:92:86 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.86/24 brd 192.168.1.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe52:9286/64 scope link 
       valid_lft forever preferred_lft forever
[root@los-alamos]$ 

Netcat commands from KVM host wspbm:

[root@wspbm]# nc -s 192.168.1.17 192.168.1.86 80
get
HTTP/1.1 400 Bad Request
Server: nginx
Date: Sat, 11 Aug 2018 14:13:36 GMT
Content-Type: text/html
Content-Length: 166
Connection: close

<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>
[root@wspbm]# nc -s 192.168.5.1 gitlab 80
get
HTTP/1.1 400 Bad Request
Server: nginx
Date: Sat, 11 Aug 2018 14:13:41 GMT
Content-Type: text/html
Content-Length: 166
Connection: close

<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>
[root@wspbm]#

Telnet from the gitlab host itself:

root@gitlab:~# telnet 192.168.1.86 80
Trying 192.168.1.86...
Connected to 192.168.1.86.
Escape character is '^]'.
get
HTTP/1.1 400 Bad Request
Server: nginx
Date: Sat, 11 Aug 2018 16:32:27 GMT
Content-Type: text/html
Content-Length: 166
Connection: close

<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>
Connection closed by foreign host.
root@gitlab:~# 

Though we’re getting 400s back, it proves that the application is up and responding.

Running tcpdump shows that kvm host wspbm sees a reset coming from the gitlab kvm guest:

[root@wspbm]# date ; tcpdump -vvv -i any host 192.168.1.86 and host 192.168.1.33 and port 80
Sat Aug 11 10:07:45 MDT 2018
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
10:07:47.940406 IP (tos 0x10, ttl 64, id 3351, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.1.33.39392 > gitlab.wsp.local.http: Flags [S], cksum 0x49b5 (correct), seq 475015689, win 29200, options [mss 1460,sackOK,TS val 13313822 ecr 0,nop,wscale 7], length 0
10:07:47.940492 IP (tos 0x10, ttl 64, id 36097, offset 0, flags [DF], proto TCP (6), length 40)
    **gitlab.wsp.local.http > 192.168.1.33.39392: Flags [R.], cksum 0x4b7e (correct), seq 0, ack 475015690, win 0, length 0
10:07:47.940501 IP (tos 0x10, ttl 64, id 36097, offset 0, flags [DF], proto TCP (6), length 40)**
    gitlab.wsp.local.http > 192.168.1.33.39392: Flags [R.], cksum 0x4b7e (correct), seq 0, ack 1, win 0, length 0 (NOT SURE WHY A SECOND RESET IS SEEN)
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel
[root@wspbm]# 

The host sourcing the SYN just sees a single reset back:

root@los-alamos:~# date ; tcpdump -vvv -i enp3s0 host 192.168.1.86 and port 80
Sat Aug 11 10:07:47 MDT 2018
tcpdump: listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:07:47.919351 IP (tos 0x10, ttl 64, id 3351, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.1.33.39392 > gitlab.wsp.local.http: Flags [S], cksum 0x49b5 (correct), seq 475015689, win 29200, options [mss 1460,sackOK,TS val 13313822 ecr 0,nop,wscale 7], length 0
10:07:47.919838 IP (tos 0x10, ttl 64, id 36097, offset 0, flags [DF], proto TCP (6), length 40)
    **gitlab.wsp.local.http > 192.168.1.33.39392: Flags [R.], cksum 0x4b7e (correct), seq 0, ack 475015690, win 0, length 0**
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
root@los-alamos:~# 

But the gitlab kvm guest never sees any traffic:

root@gitlab:~# date ; tcpdump -vvv -i ens3 host 192.168.1.33 and dst port 80
Sat Aug 11 10:07:46 MDT 2018
tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
root@gitlab:~# 

The above results were the same after adding blanket accept rules to the INPUT and FORWARD iptables chains:

[root@wspbm]# iptables -L | egrep '(INPUT|FORWARD)' -A 2
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
--
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
[root@wspbm]# 

Thanks in advance for any suggestions. I’m happy to provide any other output.

EDIT: 8/16/18

I spun up an nginx docker container on the KVM host OS (not within the gitlab VM) listening on port 80 just to test. The service is reachable from the KVM host:

[root@wspbm]# docker run --name nginx -d -p 80:80 nginx
f8e91cfec019e42354b4c3d7dac09947bb3b7f6ba6f75c2965b1524d6dc69e4a
[root@wspbm]# telnet 192.168.1.17 80
Trying 192.168.1.17...
Connected to 192.168.1.17.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
[root@wspbm]# 

But not reachable from the desktop in the topology:

[root@los-alamos]$ telnet 192.168.1.17 80
Trying 192.168.1.17...
telnet: Unable to connect to remote host: Connection refused
[root@los-alamos]$ 

EDIT: 8/17/18

I changed the port used on the gitlab VM from 80 to 8888 and the service is now reachable from everywhere.

Gitlab now listening on 8888:

root@gitlab:~# lsof -i :8888
COMMAND  PID       USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
nginx   3255       root    7u  IPv4 5570784      0t0  TCP *:8888 (LISTEN)
nginx   3256 gitlab-www    7u  IPv4 5570784      0t0  TCP *:8888 (LISTEN)
nginx   3257 gitlab-www    7u  IPv4 5570784      0t0  TCP *:8888 (LISTEN)
root@gitlab:~# 

Connection attempt from desktop:

root@los-alamos:~# telnet gitlab 8888
Trying 192.168.1.86...
Connected to gitlab.wsp.local.
Escape character is '^]'.
get
HTTP/1.1 400 Bad Request
Server: nginx
Date: Fri, 17 Aug 2018 11:36:42 GMT
Content-Type: text/html
Content-Length: 166
Connection: close

<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>
Connection closed by foreign host.
root@los-alamos:~# 

While this unblocks the side project I was working on, I don't feel like it's a true fix to whatever the underlying issue was here. I'm happy to leave this issue open and continue to troubleshoot if anyone has any suggestions on how to make port 80 on the VM reachable in case the solution is useful to anyone else in the future.

0

Figured out what was going on here. Redirection in iptables happens within the nat table:

[root@wspbm]# iptables -L --line-numbers -t nat
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:http redir ports 3129

Removed the above rule and everything started working. I think what happened here is work on a side project with squid was put aside and rules were removed but iptables-persist didn't forget about my redirection rule so my reboot stung me!

[root@wspbm]# cat /etc/iptables/rules.v4 | grep 3129
-A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 3129
[root@wspbm]#

In order to resolve this, the active rule was removed:

[root@wspbm]# iptables -t nat -D PREROUTING -p tcp --dport 80 -j REDIRECT --to 3129
[root@wspbm]# 

Then the stored rules were updated with the active rules which now don't contain the problematic redirection rule:

[root@wspbm]# cat /etc/iptables/rules.v4 | grep 3129
-A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 3129
[root@wspbm]# cd /etc/iptables/
[root@wspbm]# cp rules.v4 rules.v4.08212018
[root@wspbm]# iptables-save > rules.v4
[root@wspbm]# cat rules.v4 | grep 3129
[root@wspbm]# 

All connectivity is now working as expected. Mark this issue as PEBCAK.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.