Designing Log Server for Web Farms

Hello all,
I am going to explain how to configure a Log Server in order to send apache log files to this Log server (Part 2). This will keep the log files for all web servers in one location behind the firewall in a separate machine. For more information about part 1 and configuration of Firewall, Gateway, and Load balancer, please look at my previous comments that I posted in Feb 5 or click on this link: http://ktaraghi.blogspot.ca/2012/02/designing-firewall-gateway-and-load.html.

Configuring Log Server:

This server has two responsibilities. First, it keeps the error_log and access_log of both apache servers in one location at the same time. It is completely synchronized and very useful since the logs data are centralized in one place and easy to administer by administrator. Second, it acts as a backup server to keep a copy of web servers in case of disaster recovery for web servers and logs (Would be part 3). It has been scheduled to make a copy in regular bases.

Now, let's configure the Web servers and Log server to send the system logs from Web server to Log server. In web servers, we should change the following line in rsyslog.conf:

vi /etc/rsyslog.conf 
syslog.* @@192.168.56.104:514

which 192.168.56.104 is the ip address of Log server. To forward messages to Log server via UDP, we should use the hostname or ip address with the at sign ("@"). To forward it via plain tcp, we should use two at signs ("@@"). The destination port defaults to 514. Now, we should restart rsyslog to take effect the above configuration.
  
service rsyslog restart

 In Log server, we should change the followings in /etc/rsyslog.conf file:

vi /etc/rsyslog.conf

change these two line
#ModLoad imtcp.so
#$InputTCPServerRun 514
to  
ModLoad imtcp.so 
$InputTCPServerRun 514

which imtcp is a module and it is an input plugin for plain TCP syslog.
Now, we should configure firewall to accept port 514:  

iptables -I INPUT -p tcp –dport 514 -j ACCEPT

Now restart rsyslog service:  

service rsyslog restart

At this point, I configured syslog to route system log files to a remote host (Log Server). Now, in order to redirect access_log and error_log logs in Web servers to Log server, I should change and add some lines in httpd.conf. Hopefully, the new versions of apache support redirection of access_log. You can find the following paragraph in apache documentation at http://httpd.apache.org/docs/2.1/logs.html


“Apache httpd is capable of writing error and access log files through a pipe to another process, rather than directly to a file. This capability dramatically increases the flexibility of logging, without adding code to the main server. In order to write logs to a pipe, simply replace the filename with the pipe character "|", followed by the name of the executable which should accept log entries on its standard input.”

To achieve this goal, I changed and added the following lines to httpd.conf in web servers:

vi /etc/httpd/conf/httpd.conf 

change
ErrorLog logs/error_log
to

ErrorLog "|/usr/bin/logger -p syslog.info -t ***WebServerNo X***" 

and also change
CustomLog logs/access_log combined
to

CustomLog "|/usr/bin/logger -p syslog.info -t ***WebServerNo X***" combined 

which X is “1” for web server 1 and “2” for web server 2.

Let's explain these two important lines in httpd.conf. According to apache documentation, these two lines send their output to system log in “/var/log/messages” by using pipe (redirection) and “logger” command. -t (tag) marks every line in the log with the specified tag. So, I used two different tag to distinguish their logs easily. -p enters the message with the specified priority. The priority may be specified numerically or as a ‘‘facility.level’’ pair. In this case, ‘‘-p syslog.info’’ logs the message(s) as informational level in the syslog facility. After redirection output by this method to “/var/log/message” file, rsyslog send those information and logs to remote host (Log Server).
Now, httpd service should restart for new configuration:

service httpd restart

The following picture shows the content of my “/var/log/message” in Log server. Also, I made some mistakes by purpose in the url in browser to show you all error and access log in this file:

And, That's all for Log server. Please follow up my next comments for configuring Backup Server (Part 3) that I will post it soon.
Regards,
Khosro Taraghi