Page 1 of 2

httpd can't start due to ports 80 and 443 magically being in-use...why?

Posted: 2009/12/04 01:28:22
by torontob123
Hi Guys,

Recently I installed an SSL certificate from GoDaddy on my Amazon EC2 instance which runs CentOS 5. All has been good till I installed the SSL certificate. Everytime I restart the system it fails to have httpd service working because:

[root@ip-]# service httpd start
Starting httpd: (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs


I looked on the net and found a solution which says I should change port 80 and port 443 in files httpd.conf and ssl.conf to port 88 and 553 and restart apache, change back the ports to 80 and 443 and restart again. This method worked before but now I am stuck with it not starting with ports 80 and 443 and if I change these ports then I can't reach my site and I have to add port number wtih : which is really not good.

1- What causes this problem and where can I look to debug it?
2- What can I do to permenently stop this problem? is there any place that I can dedicate these two ports to apache user?

Thanks a bunch

httpd can't start due to ports 80 and 443 magically being in

Posted: 2009/12/04 18:40:28
by michaelnel
I would use the "fuser" command to find out what is using ports 80 and 443, then proceed from there.

[code]
# fuser 80/tcp
80/tcp: 2850 12763 15537 18030 30044 32226
[/code]

The return from fuser shows that processes with pids 2850 12763 15537 18030 30044 32226 are using port 80.

[code]
# ps aux | grep 2850 | grep -v grep
root 2850 0.0 0.4 318772 30428 ? Ssl Oct22 0:03 /usr/sbin/httpd
[/code]

So, in this case it is httpd process listening on port 80. Look for children:

[code]
# pgrep httpd
2850
12763
15537
18030
30044
32226
[/code]

So those are all instances of httpd.

There is probably a more elegant way to do it, but I am stupid and tend to use primitive tools.

Re: httpd can't start due to ports 80 and 443 magically being in-use...why?

Posted: 2009/12/04 21:25:16
by pschaff
Looks like the traditional *nix approach of using a set of small sharp tools to me. Pretty smart! :-)

Re: httpd can't start due to ports 80 and 443 magically being in-use...why?

Posted: 2009/12/04 22:10:37
by michaelnel
Yeah, but I didn't use awk or sed or pipelines or, most importantly, FONTS!

Re: httpd can't start due to ports 80 and 443 magically being in-use...why?

Posted: 2009/12/04 22:19:31
by pschaff
Well, the pipe was mental in this case. Awk and sed are good tool to have in the bag. You lost me on [size=150][color=FF0000][font=Courier]FONTS[/font][/color][/size] though. ;-)

Re: httpd can't start due to ports 80 and 443 magically being in-use...why?

Posted: 2009/12/05 04:01:46
by torontob123
Thanks for the detailed response with examples. I think this is a bit complex issue here though. I have read about this problem on other forums as well and it seems that everyone is suggesting a different thing but none of them points to the real root of the problem. This problem happend again after two restarts and here is what I did as you suggested, but as you can see httpd is running on those ports but it's just confused and can't stop or start and is reserving the ports:

[root@ ~]# service httpd status
httpd (pid 1974) is running...

[root@ ~]# service httpd stop
Stopping httpd: [FAILED]
[root@ ~]# service httpd start
Starting httpd: (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
[FAILED]

[root@ ~]# fuser 80/tcp
80/tcp: 1974
[root@ ~]# fuser 443/tcp
443/tcp: 1974

[root@ ~]# ps aux | grep 1974 | grep -v grep
root 1974 0.0 0.3 19496 6508 ? S 22:22 0:00 /usr/sbin/httpd

[root@ ~]# pgrep httpd
1970
1974

[root@ ~]# service httpd restart
Stopping httpd: [FAILED]
Starting httpd: (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
[FAILED]
[root@ ~]# killall apache
apache: no process killed

[root@ ~]# service httpd start
Starting httpd: (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
[FAILED]
[root@ ~]# killall apache
apache: no process killed

[root@ ~]# service httpd stop
Stopping httpd: [FAILED]

[root@ ~]# killall apache
apache: no process killed

So, as it seems that everything is in order in terms of ports being bint to right program, I have no choice but to run my trick to get this back to working. So, I went in and changed the port 80 in httpd.conf to port 88 just to unbind it.
[root@ ~]# locate httpd.conf
/etc/httpd/conf/httpd.conf
/etc/httpd/conf/httpd.conf.orig
[root@ ~]# nano /etc/httpd/conf/httpd.conf


[root@ ~]# service httpd restart
Stopping httpd: [FAILED]
Starting httpd: (98)Address already in use: make_sock: could not bind to address 0.0.0.0:443
no listening sockets available, shutting down
Unable to open logs
[FAILED]
[root@ ~]# locate ssl.conf
/etc/httpd/conf.d/ssl.conf
[root@ ~]# nano /etc/httpd/conf.d/ssl.conf
[root@ ~]# service httpd restart
Stopping httpd: [FAILED]
Starting httpd: Apache/2.2.9 mod_ssl/2.2.9 (Pass Phrase Dialog)
Some of your private key files are encrypted for security reasons.
In order to read them you have to provide the pass phrases.
Server .ec2.internal:443 (RSA)
Enter pass phrase:

OK: Pass Phrase Dialog successful.
[ OK ]
[root@ ~]# service httpd stop
Stopping httpd: [ OK ]
[root@ ~]# pgrep httpd
1970
1974
[root@ ~]# fuser 80/tcp
80/tcp: 1974
[root@ ~]# fuser 443/tcp
443/tcp: 1974

[root@ ~]# ps aux | grep 2850 | grep -v grep

[root@ ~]# killall apache
apache: no process killed


Changing ports back to 80 and 443 in both files ssl.conf and httpd.conf:
[root@ ~]# nano /etc/httpd/conf.d/ssl.conf
[root@ ~]# nano /etc/httpd/conf/httpd.conf

[root@ conf.d]# lsof -i :443
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
httpd 1974 root 4u IPv4 6323 TCP *:https (LISTEN)

[root@ conf.d]# lsof -i :80
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
httpd 1974 root 3u IPv4 6321 TCP *:http (LISTEN)


[root@ conf.d]# kill -9 1974

[root@ conf.d]# lsof -i :80
[root@ conf.d]# lsof -i :443

[root@ conf.d]# service httpd start
Starting httpd: Apache/2.2.9 mod_ssl/2.2.9 (Pass Phrase Dialog)
Some of your private key files are encrypted for security reasons.
In order to read them you have to provide the pass phrases.

Server .ec2.internal:443 (RSA)
Enter pass phrase:

OK: Pass Phrase Dialog successful.
[ OK ]
[root@ conf.d]#



**************************

Now, is there anything I can do to debug this? What is next?

Thanks a lot

Re: httpd can't start due to ports 80 and 443 magically being in-use...why?

Posted: 2009/12/05 15:18:10
by pschaff
[quote]
torontob123 wrote:
...
[root@ ~]# killall apache
apache: no process killed
...[/quote]

Should be "killall httpd".

Looks like you successfully started Apache in the last step. What is not working now?

Re: httpd can't start due to ports 80 and 443 magically being in-use...why?

Posted: 2009/12/05 16:05:33
by torontob123
Thanks for the 'killall apcahe' I was wondering why it wasn't working.

So, the problem now is that this has the potential to happen again. Because this happened over dozen times already in the past week if server is restarted.

Some suggest that maybe there is a two or more of the .conf file of httpd or ssl and hence apache gets confused trying to load the settings and ports again. I check and did a locate, as you can see there is no such thing.

I want to know what causes this occasionally so I can stop it from happening. It's embarressing because I was doing a demo for someone the other day and this happened. I had to stop everything and go after this problem.

Thanks again.

Re: httpd can't start due to ports 80 and 443 magically being in-use...why?

Posted: 2009/12/05 16:34:16
by pschaff
Your first post shows a manual attempt to start httpd. Is is also being started at boot time?[code]chkconfig --list httpd[/code]

If so, you should either disable the start at boot time[code]chkconfig httpd off[/code]and start manually with the passphrase, or make provisions for the passphrase to be automatically provided.

Re: httpd can't start due to ports 80 and 443 magically being in-use...why?

Posted: 2009/12/06 15:26:41
by torontob123
:-) I am laughing at my own stupidity now. Of course it's the passphrase. So, how can I incorporate that passphrase? In fact, now that I am thinking of it, it's a useless security feature for me but I think I had no choice but to chose a passphrase.

So, httpd does start at boot and that's how it should be.
Wondering if I can add --passphrase "password" or something like that?


P.S. Here is a way to remove passphrase from SSL-key:
Ref. http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#removepassphrase

How can I get rid of the pass-phrase dialog at Apache startup time?
The reason this dialog pops up at startup and every re-start is that the RSA private key inside your server.key file is stored in encrypted format for security reasons. The pass-phrase is needed to decrypt this file, so it can be read and parsed. Removing the pass-phrase removes a layer of security from your server - proceed with caution!

Remove the encryption from the RSA private key (while keeping a backup copy of the original file):

$ cp server.key server.key.org
$ openssl rsa -in server.key.org -out server.key

Make sure the server.key file is only readable by root:

$ chmod 400 server.key

Now server.key contains an unencrypted copy of the key. If you point your server at this file, it will not prompt you for a pass-phrase. HOWEVER, if anyone gets this key they will be able to impersonate you on the net. PLEASE make sure that the permissions on this file are such that only root or the web server user can read it (preferably get your web server to start as root but run as another user, and have the key readable only by root).

As an alternative approach you can use the ``SSLPassPhraseDialog exec:/path/to/program'' facility. Bear in mind that this is neither more nor less secure, of course.

Thanks a bunch for the tips.