PTY allocation request failed on channel 0

[mathog]

In order to run "yum" on a chroot disk image this is done:

Code: Select all

ROOTDIR=/var/lib/systemimager/images/centos7.x.beowulf
TARGDIR=$ROOTDIR/tmp/yum_rpms
/usr/bin/rm -rf $TARGDIR
/usr/bin/mv $HOLDDIR $TARGDIR
mount -o bind /sys  $ROOTDIR/sys #for network
mount -o bind /proc $ROOTDIR/proc #for network
mount -o bind /dev  $ROOTDIR/dev  #for urandom, needed by yum
chroot $ROOTDIR /usr/bin/bash -c 'yum -y install /tmp/yum_rpms/*.rpm'
/usr/bin/rm -rf $TARGDIR #DO THIS ONLY AFTER ALL OTHER MACHINES UPDATED!!!
umount $ROOTDIR/sys 
umount $ROOTDIR/proc
umount $ROOTDIR/dev 

The problem is that sometimes yum has a problem, or maybe it is unrelated to that, but when all is said and done afterwards /dev/pts/0
is gone, so on an ssh login this happens:

Code: Select all

ssh auser@machine
auser@machine's password: 
PTY allocation request failed on channel 0

Other info:

Code: Select all

fuser  /var/lib/systemimager/images/centos7.x.beowulf/dev
/var/lib/systemimager/images/centos7.x.beowulf/dev:    48rc
ps -ef | grep 48
root        48     2  0 Aug20 ?        00:00:00 [kdevtmpfs]

I thought at first that /dev was not unmounted properly, but

Code: Select all

umount /var/lib/systemimager/images/centos7.x.beowulf/dev
umount: /var/lib/systemimager/images/centos7.x.beowulf/dev: target is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

On a normally working system:

Code: Select all

lsof /dev/pts/0
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
bash    26732 root    0u   CHR  136,0      0t0    3 /dev/pts/0
bash    26732 root    1u   CHR  136,0      0t0    3 /dev/pts/0
bash    26732 root    2u   CHR  136,0      0t0    3 /dev/pts/0
bash    26732 root  255u   CHR  136,0      0t0    3 /dev/pts/0
xterm   26790 root    0u   CHR  136,0      0t0    3 /dev/pts/0
xterm   26790 root    1u   CHR  136,0      0t0    3 /dev/pts/0
xterm   26790 root    2u   CHR  136,0      0t0    3 /dev/pts/0
lsof /sys/fs/cgroup
#nothing

#but on the broken system

lsof /dev/pts*
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/systemd
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/devices
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/blkio
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/perf_event
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/memory
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/cpuset
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/net_cls,net_prio
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/hugetlb
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/cpu,cpuacct
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/freezer
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/pids
      Output information may be incomplete.
COMMAND     PID           USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
systemd       1           root    0u   CHR    1,3      0t0  1028 /dev/null
systemd       1           root    1u   CHR    1,3      0t0  1028 /dev/null
...

lsof /sys/fs/cgroup
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/systemd
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/devices
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/blkio
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/perf_event
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/memory
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/cpuset
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/net_cls,net_prio
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/hugetlb
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/cpu,cpuacct
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/freezer
      Output information may be incomplete.
lsof: WARNING: can't stat() cgroup file system /sys/fs/cgroup/pids
      Output information may be incomplete.
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
systemd-l 2010 root    6r   REG   0,17     4096 7711 /sys/devices/virtual/tty/tty0/active

Nothing stands out in /var/log/messages or dmesg.

What is messed up here and how does one fix it? (Other than a reboot, which works of course, but is an awfully crude way to resolve this.)

Thanks.

[mathog]

I spoke too soon - this time rebooting was not so easy. As root

Code: Select all

sync
reboot

but it hung part way down. The network was still up and from another machine

Code: Select all

nmap name.privatenet
Not shown: 998 closed ports
PORT     STATE SERVICE
2049/tcp open  nfs
4001/tcp open  newoak
MAC Address: 84:2B:2B:08:49:DB (Dell)

nmap name.publicnet
Not shown: 982 filtered ports
PORT      STATE  SERVICE
22/tcp    closed ssh
111/tcp   closed rpcbind
139/tcp   closed netbios-ssn
445/tcp   closed microsoft-ds
2049/tcp  open   nfs
4000/tcp  closed remoteanything
4001/tcp  open   newoak
4002/tcp  closed mlchat-proxy
4003/tcp  closed pxc-splr-ft
4004/tcp  closed pxc-roid
6000/tcp  closed X11
6001/tcp  closed X11:1
6002/tcp  closed X11:2
6003/tcp  closed X11:3
6004/tcp  closed X11:4
6881/tcp  closed bittorrent-tracker
19283/tcp closed keysrvr
19315/tcp closed keyshadow
MAC Address: 84:2B:2B:08:49:DA (Dell)

And it stayed that way for 10 minutes. Plugged in a monitor and a keyboard (it is normally headless) but pounding away on the keyboard didn't bring anything up on the monitor. Since the power button shut down is enabled, or maybe holding that button just cut the power, but anyway pressing the power button brought it down and doing so again let it reboot.

The last part of the shutdown in /var/log/messages had:

Code: Select all

Aug 22 16:46:15 machinename sedispatch: Connection Error (An SELinux policy prevents this sender from sending this message to this recipient, 0 matched rules; type="method_call", sender="(null)" (inactive) interface="org.freedesktop.DBus" member="Hello" error name="(unset)" requested_reply="0" destination="org.freedesktop.DBus" (bus)): AVC Will be dropped
Aug 22 16:47:12 machinename sedispatch: Connection Error (An SELinux policy prevents this sender from sending this message to this recipient, 0 matched rules; type="method_call", sender="(null)" (inactive) interface="org.freedesktop.DBus" member="Hello" error name="(unset)" requested_reply="0" destination="org.freedesktop.DBus" (bus)): AVC Will be dropped
Aug 22 16:47:15 machinename sedispatch: Connection Error (An SELinux policy prevents this sender from sending this message to this recipient, 0 matched rules; type="method_call", sender="(null)" (inactive) interface="org.freedesktop.DBus" member="Hello" error name="(unset)" requested_reply="0" destination="org.freedesktop.DBus" (bus)): AVC Will be dropped
Aug 22 16:48:12 machinename sedispatch: Connection Error (An SELinux policy prevents this sender from sending this message to this recipient, 0 matched rules; type="method_call", sender="(null)" (inactive) interface="org.freedesktop.DBus" member="Hello" error name="(unset)" requested_reply="0" destination="org.freedesktop.DBus" (bus)): AVC Will be dropped
Aug 22 16:48:15 machinename sedispatch: Connection Error (An SELinux policy prevents this sender from sending this message to this recipient, 0 matched rules; type="method_call", sender="(null)" (inactive) interface="org.freedesktop.DBus" member="Hello" error name="(unset)" requested_reply="0" destination="org.freedesktop.DBus" (bus)): AVC Will be dropped
Aug 22 16:48:50 machinename systemd: Stopping Authorization Manager...
Aug 22 16:48:50 machinename systemd: Removed slice system-systemd\x2dfsck.slice.
Aug 22 16:48:50 machinename systemd: Stopped Session 356 of user modules.
Aug 22 16:48:50 machinename systemd: Removed slice User Slice of modules.
Aug 22 16:48:50 machinename systemd: Removed slice system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice.
Aug 22 16:48:50 machinename systemd: Stopping Availability of block devices...
Aug 22 16:48:50 machinename systemd: Stopped Session 357 of user root.
Aug 22 16:48:50 machinename systemd: Stopped Stop Read-Ahead Data Collection 10s After Completed Startup.
Aug 22 16:48:50 machinename systemd: Stopped target Multi-User System.
Aug 22 16:48:50 machinename systemd: Failed at step CGROUP spawning /usr/sbin/blkdeactivate: No such file or directory
Aug 22 16:48:50 machinename systemd: Stopping Job spooling tools...
Aug 22 16:48:50 machinename ModemManager[1288]: <info>  Caught signal, shutting down...
Aug 22 16:48:50 machinename systemd: Stopping Dynamic System Tuning Daemon...
Aug 22 16:48:50 machinename systemd: Stopping Run automatic yum updates as a cron job...
Aug 22 16:48:50 machinename systemd: Failed at step CGROUP spawning /bin/rm: No such file or directory
Aug 22 16:48:50 machinename systemd: Stopping Modem Manager...