CentOS cannot be ping and ssh

[james.grezer]

Hello Team,

Good day!

May I request for any help that you can provide me regarding one of our server.

One of our server cannot be ping nor ssh. We already check all possibilities but still the issue persist. Please be guided below on our troubleshooting steps.

• Check the IP configuration if correct : Correct

• Check the IP ethernet port status : UP and Running then restart

• Check if sshd port is listening and service is enable and running : Port is Listening, Services are UP and Running

• Ping,SSH, and Telnet other servers : Working

• Ping, SSH, and Telnet to the server : Not reachable

• Perform tcpdump in the server and confirm the connection of other server : Outgoing is working but Incoming is not

• Firewalld, selinux and iptables are disabled

• Security agents are in-place the same with the working same subnet server

• IP Route and Route -N is identical with the working same subnet server

• Firewall and Network Team already confirmed that connection is allowed in the Server, confirmed because 1 same subnet server is accessible

As per our checking, the server has multiple operating system kernels (RHEL 8.3, RHEL 8.4 and CentOS). Would that be a possible cause of the connection issue?

Server is CentOS Linux 8 with Kernel Linux 4.18.0-305.el8.x86_64

I hope you can help me as I am really out of option here right now. Thanks!

[TrevorH]

If you run tcpdump against the interface on the CentOS box that you are expecting to respond to these attempts and you see no traffic then the problem is further up the network chain. Tcpdump will show the packets if they arrive on the interface at all - even if they are blocked by the firewall on CentOS - i.e. it "sees" all packets that arrive on that interface before they are processed by anything so even if they are blocked then tcpdump will see them.

If the packets do not show up then the network switch that it is connected to is not sending them down the piece of wire connecting you to it.

[james.grezer]

Hello TrevorH,

Good day!

Based on the tcpdump that we conducted, the server was able to respond to the Incoming connection of the "Windows server/Working Server" the tcpdump has traffic going to the working server/windows server. That is confirming that the network switch is sending down the connection to the server.

Apparently, in the Windows/Working server cannot still access the Proxy server which inaccessible via ssh and ping through its Hostname/IP Address.

Is there anything that I need to check on my end? Please let me know.

Thank you for the response and inputs.

[TrevorH]

So just to clarify, if you fire up tcpdump -i eth0 (or whatever interface) and then attempt to send a packet to the ip address on that interface from outside, you see the packet arrive? If the answer is yes then the problem is either firewall or your application is not listening on that port.

[james.grezer]

Hello TrevorH,

Yes, the packet arrive. Application is listening on that port and firewall is okay as well. Here's the new discovery. Upon checking with other teams, they experienced it as well with the same server but different virtualization since they are in RHV instead of VMWare.

So here's what I found, the / mount points in two different disk (/sda2/vgfrel-root and sda4/vgfrel-root) and based on the fstab. The server mounts in /dev/mapper/vgfrel-root that might be causing the issue.

Based on the blkid, the server has different UUID for the list below.

• /dev/mapper/vgfrel-root: UUID="64063b35..."

• /dev/sda2: UUID="jvBE9N...."

• /dev/sda4: UUID="DqGPCt...."

Is there any way I can test both /sda2/vgfrel-root and sda4/vgfrel-root to check the correct root mount point?

[james.grezer]

Okay upon testing out in the my local lab. It seems that it's not the case since the disk is only expanded and they re-add the PVS sda4 to its VG and extend the LV. So I'm back to square one.

[jlehtone]

Yes, the packet arrive. Application is listening on that port and firewall is okay as well. Here's the new discovery. Upon checking with other teams, they experienced it as well with the same server but different virtualization since they are in RHV instead of VMWare.

Do I read that right? Your "server" is a VM that was running on VMWare hypervisor and had network issue and has been moved to run on RHV hypervisor and still has the same issue?

You say that you can see an incoming packet on this server's interface, but it does not "get in" -- no reply is seen going back.

Interfaces have statistics that you can see with:

Code: Select all

ip -s li

You said that firewall is "okay", but also "down". I'd rather ask, what you get with:

Code: Select all

sudo nft list ruleset

You did refer to this server as "proxy". Does it have any other "data" than the configuration of the proxy service?
You also show that the OS packages (e.g. kernel) are old. Could you create a fresh new VM with up-to-date distro and install the proxy service into it?

[james.grezer]

I already attached the requested information.

To clarify, no they are not migrated from VMWare to RHV Hypervisor. Sorry for that confusion.

Yes, incoming packets are seen in the tcpdump while simultaneously pinging the server but still cannot ssh nor ping the server.

[jlehtone]

You nft output shows that:

1. You have firewall rules. The firewall is not "down"
2. These rules are not generated by firewalld.service. Something/someone else has added them
3. The shown rules drop everything that tries to come in (but do not block existing connections)

That explains why ping and ssh from outside never get reply from the server.

[james.grezer]

Hello Jlehtone,

Good day!

Thank you for the inputs. As per checking the not working servers (FRE001 and FRE002) it seems that they have both rules set.

Just to clarify since I haven't had any experience on this nfts, based on this article. This will backup and re-initiate the nft ruleset in case that we need to revert it.

You can combine these two commands above to backup your ruleset:

% echo "flush ruleset" > backup.nft
% nft list ruleset >> backup.nft
And load it atomically:

% nft -f backup.nft

Reset/Wipe/Flush

% nft flush ruleset

Please let me know.