Shutdown appears to 'kill' everything forcably?

Issues related to applications and software problems
Post Reply
mmiller7
Posts: 3
Joined: 2019/08/18 17:11:48

Shutdown appears to 'kill' everything forcably?

Post by mmiller7 » 2019/08/18 17:26:13

I'm finally building a new server with CentOS 7 to replace an old CentOS 6 virtualization host.

After googling and trying things for most of this weekend (Friday night, all Saturday, more than half of Sunday) I am yet to find anything that helps. I've tried making custom systemd scripts (with so many suggestions I lost track of which came from what site) and I've tried enabling and tweaking services...I'm really out of ideas, and if I can't figure out a solution the only thing I can think of would be going back to CentOS 6 and ignoring that it will EOL next year.

When I shut down the host (using 'reboot', 'shutdown -r now', 'shutdown -h now') it instantly drops all network connectivity and I see the monitor (within about 1 second) flash up that it looks like the mdadm disks are stopping and then 2-3 seconds later its halted or rebooted POST-ing.

Initially I wanted to get an email notification on shutdown (I've got postfix working to email me thru gmail - it works for startup, apcupsd, and other things fine) and I can't figure out any way to make it successfully send one.

I finally gave up on that and moved to setting up my VMs...except when the host goes down for any reason it does the same thing where it is "down" in just a second or two. When I boot it up again the guest VMs act like "the plug was pulled" and now I've also got corrupt databases in my VMs to deal with too.

I've already tried to enable the service 'libvirt-guests' and configured /etc/sysconfig/libvirt-guests to do a shutdown and 300 second timeout.

This is really boggling my mind, I've never had any of these problems with CentOS 6, the only thing I can think of is something is horribly broken with systemd...but I also can't figure out how to even diagnose it. The logs seem to just stop...and even trying to put a line in scripts that writes to like /root/debug.txt it looks like nothing is ever run when it goes to do a shutdown.

Can anyone please help me figure out where I begin to fix this mess???

User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: Shutdown appears to 'kill' everything forcably?

Post by avij » 2019/08/18 17:59:48

It does not kill everything forcibly, it's up to the daemons to decide what they want to do when shutting down.

https://centos.org/forums/viewtopic.php?t=50851 may be related, if you wish to have your VMs shut down in an orderly fashion, although you may have already tried some of those settings. Your VMs may need to have acpid running so that they'd notice they are being shut down.

https://access.redhat.com/documentation ... n_settings may also be informative.

mmiller7
Posts: 3
Joined: 2019/08/18 17:11:48

Re: Shutdown appears to 'kill' everything forcably?

Post by mmiller7 » 2019/08/18 22:43:51

Yes, I did see both of those...that first person sounds more or less exactly like what I'm seeing (or think I'm seeing).

I just tried changing parallel shutdown from 3 to 0 (I'm running on a hex-core Xeon so I thought 3 should be easy) and oddly it seems a bit better now, and I'm also suddenly getting emails from it with it set to 0 parallel. I can't really explain why though.

Is there a logfile that may help me see exactly what order the system is "coming down" to try and identify what (if any) of the commands it may be running or failing with? I can't watch anything on SSH because it immediately dies and I plugged in a monitor but it blanks the display as soon as the shutdown command is given.

Seems like some bizarre interactions that make me wonder if there's a race condition the way it doesn't always behave the same.

chemal
Posts: 776
Joined: 2013/12/08 19:44:49

Re: Shutdown appears to 'kill' everything forcably?

Post by chemal » 2019/08/18 22:52:55

mmiller7 wrote:
2019/08/18 17:26:13
I finally gave up on that and moved to setting up my VMs...except when the host goes down for any reason it does the same thing where it is "down" in just a second or two. When I boot it up again the guest VMs act like "the plug was pulled" and now I've also got corrupt databases in my VMs to deal with too.
You should enable the libvirt-guests service on the host. It will take care of your VMs when the host is shut down or rebooted.

mmiller7
Posts: 3
Joined: 2019/08/18 17:11:48

Re: Shutdown appears to 'kill' everything forcably?

Post by mmiller7 » 2019/08/19 00:30:17

Huh...I swear I enabled that after configuring it but apparently somehow I got the wrong command. Finally went into webmin instead of command line sure enough the service wasn't actually auto-starting. Maybe when I was trying to enable it I just started instead of setting on boot? Dunno.

Now I just have to figure out why my send-email-on-shutdown also seems to randomly work and randomly get killed before it can send (I know it gets queued in postfix because it goes out the next boot-up)...so clearly the unit is running, just not consistently finishing?

I rather liked the old initV way of doing it, much easier to wrap my head around and debug than this systemd stuff...and was slow enough I could actually see what it did during bootup and during shutdown.

chemal
Posts: 776
Joined: 2013/12/08 19:44:49

Re: Shutdown appears to 'kill' everything forcably?

Post by chemal » 2019/08/19 04:00:42

mmiller7 wrote:
2019/08/19 00:30:17
Now I just have to figure out why my send-email-on-shutdown also seems to randomly work and randomly get killed before it can send (I know it gets queued in postfix because it goes out the next boot-up)
Postfix doesn't get killed, it's asked to shut down gracefully. Which it does. As you found out, this doesn't mean it has to send your mail before shutting down.

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: Shutdown appears to 'kill' everything forcably?

Post by hunter86_bg » 2019/08/21 04:04:06

When my system was a KVM host I also made some tweaks.
1. I used a custom script that puts systemd inhibit if there is a vm running on the system - used to prevent automatic sleep that I have configured
2. KVM comes with some services - one of them starts on boot all previously running (before the shutdown) VMs, and even change the behaviour on shutdown . It's like putting the VM to sleep and after a reboot -to resume.

So check the systemd's inhibit or you can prevent accidental shutdown via systemd conflict ( check it here: 1580343

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Shutdown appears to 'kill' everything forcably?

Post by TrevorH » 2019/08/21 09:40:57

If you shutdown the system with running KVM guests then you need to have enough disk space available in the storage pool for it to be able to save a copy of each guests RAM. It does the equivalent of a suspend on each guuest, saves the RAM to disk then suspends the guest. No space, no suspend and the guest is killed instead.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply