Designing Log Server for Web Farms

Hello all,
I am going to explain how to configure a Log Server in order to send apache log files to this Log server (Part 2). This will keep the log files for all web servers in one location behind the firewall in a separate machine. For more information about part 1 and configuration of Firewall, Gateway, and Load balancer, please look at my previous comments that I posted in Feb 5 or click on this link: http://ktaraghi.blogspot.ca/2012/02/designing-firewall-gateway-and-load.html.

Configuring Log Server:

This server has two responsibilities. First, it keeps the error_log and access_log of both apache servers in one location at the same time. It is completely synchronized and very useful since the logs data are centralized in one place and easy to administer by administrator. Second, it acts as a backup server to keep a copy of web servers in case of disaster recovery for web servers and logs (Would be part 3). It has been scheduled to make a copy in regular bases.

Now, let's configure the Web servers and Log server to send the system logs from Web server to Log server. In web servers, we should change the following line in rsyslog.conf:

vi /etc/rsyslog.conf 
syslog.* @@192.168.56.104:514

which 192.168.56.104 is the ip address of Log server. To forward messages to Log server via UDP, we should use the hostname or ip address with the at sign ("@"). To forward it via plain tcp, we should use two at signs ("@@"). The destination port defaults to 514. Now, we should restart rsyslog to take effect the above configuration.
  
service rsyslog restart

 In Log server, we should change the followings in /etc/rsyslog.conf file:

vi /etc/rsyslog.conf

change these two line
#ModLoad imtcp.so
#$InputTCPServerRun 514
to  
ModLoad imtcp.so 
$InputTCPServerRun 514

which imtcp is a module and it is an input plugin for plain TCP syslog.
Now, we should configure firewall to accept port 514:  

iptables -I INPUT -p tcp –dport 514 -j ACCEPT

Now restart rsyslog service:  

service rsyslog restart

At this point, I configured syslog to route system log files to a remote host (Log Server). Now, in order to redirect access_log and error_log logs in Web servers to Log server, I should change and add some lines in httpd.conf. Hopefully, the new versions of apache support redirection of access_log. You can find the following paragraph in apache documentation at http://httpd.apache.org/docs/2.1/logs.html


“Apache httpd is capable of writing error and access log files through a pipe to another process, rather than directly to a file. This capability dramatically increases the flexibility of logging, without adding code to the main server. In order to write logs to a pipe, simply replace the filename with the pipe character "|", followed by the name of the executable which should accept log entries on its standard input.”

To achieve this goal, I changed and added the following lines to httpd.conf in web servers:

vi /etc/httpd/conf/httpd.conf 

change
ErrorLog logs/error_log
to

ErrorLog "|/usr/bin/logger -p syslog.info -t ***WebServerNo X***" 

and also change
CustomLog logs/access_log combined
to

CustomLog "|/usr/bin/logger -p syslog.info -t ***WebServerNo X***" combined 

which X is “1” for web server 1 and “2” for web server 2.

Let's explain these two important lines in httpd.conf. According to apache documentation, these two lines send their output to system log in “/var/log/messages” by using pipe (redirection) and “logger” command. -t (tag) marks every line in the log with the specified tag. So, I used two different tag to distinguish their logs easily. -p enters the message with the specified priority. The priority may be specified numerically or as a ‘‘facility.level’’ pair. In this case, ‘‘-p syslog.info’’ logs the message(s) as informational level in the syslog facility. After redirection output by this method to “/var/log/message” file, rsyslog send those information and logs to remote host (Log Server).
Now, httpd service should restart for new configuration:

service httpd restart

The following picture shows the content of my “/var/log/message” in Log server. Also, I made some mistakes by purpose in the url in browser to show you all error and access log in this file:

And, That's all for Log server. Please follow up my next comments for configuring Backup Server (Part 3) that I will post it soon.
Regards,
Khosro Taraghi

Designing Firewall, Gateway, and Load Balancer Server for Web Farms with iptables

Hello all,

Today, I am going to show you how we can configure a Firewall, Gateway, and Load Balancer Server for your Web Farm Servers with iptables in Linux (part 1). In part 2, I will explain how to configure a Backup/Log Server in order to send apache log files to this server. This will keep the log files for all web servers in one location behind the firewall in a separate machine. Also, it acts as a backup server to keep a copy of web servers in case of disaster recovery for web servers.

                                                          Figure 1

When a single Web Server machine isn’t enough to handle the traffic on your Web site, it’s time to look into building a Web Farm that uses multiple machines on the network acting as a single server. Also, Web Farms are an obvious choice if you’ve hit the limits of your single machine hardware. Although the process of creating a Web Farm isn’t difficult, administration of two or more servers and keeping them properly synchronized is actually a lot more work than administering a single server. Now, to see the functionality of a Web Server Farm, I am going to demonstrate and configure a Web Server Farm with two separated machine, which is running Apache version 2.2.17, behind a firewall. To demonstrate this scenario, I used four virtual machines all with Fedora 13 operation system. Figure 1 shows the details of this scenario.

First of all, let's configure the firewall/Gateway machine. The Firewall/Gateway machine has two network interfaces. One network interface has an ip address of 192.168.56.101 for internal network and another network interface has an ip address of 192.168.2.X ,which X is a number that my router at home gives this number, to connect to Internet. Our gateway machine has to have the ability to forward the packets. In order to achieve this goal, we should enable IP forwarding on this machine with following command:

   echo 1 > /proc/sys/net/ipv4/ip_forward  

You can put this command in /etc/rc.local file. So, when you restart the computer, it loads and runs automatically and it doesn't need to enter this command manually.

Now, the next step is configuring firewall and load balancing. Before configuring firewall, I loaded the default Fedora Server firewall by clicking System->Administration-> Firewall and then Options->Load Default Configuration->Server. Now, in order to balance the loads across the web servers in a round robin fashion, I added these two lines to the PREROUTING table in ip tables.

iptables -I PREROUTING -i eth0 -p tcp -m tcp --dport 80 -m statistic  --mode nth --every 2 -j DNAT --to-destination 192.168.56.102:80

iptables -A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j DNAT  --to-destination 192.168.56.103:80

Let's discuss about these two important lines in iptables. First, because we want to forward the http packets with destination port 80 to web servers, we have to change the destination address for packets to web servers in PREROUTING tables. That's why I added these lines in PREROUTING table. According to man page, statistic module matches packets based on some statistic condition.It supports two distinct modes settable with the --mode option which are nth and random. The random mode is not in our case since we want to know exactly which packet goes to which server. --every n match one packet every nth packet and it works only with the nth mode.

In this case, it matches every 2 packets. --packet p set the initial counter value for the nth mode and because the default value is 0, I didn't mention in the command. Therefore, the first command says match the tcp protocol with destination port 80 that comes from eth0 for every 2 nth packet (even packets) and do DNAT to address of the first web server which is 192.168.56.102. For other packets, just do the regular DNAT to the second web server like the second line (odd packets). So, with these iptables commands, we have a round-robin method. Because the packets are prerouted to web servers, we should accept packets in FORWARD table in firewall server. Also, for returning packets from web servers to firewall server, we should accept packets in FORWARD table as well. So, we should add two more rules to firewall:

iptables -I FORWARD -p tcp -m tcp --sport 80 -j ACCEPT
iptables -A FORWARD -d 192.168.56.0/24 -p tcp -m tcp --dport 80 -j ACCEPT

                                                    Figure 2


Figure 2 shows the output of iptables-save command on Firewall Server. There is one more rule that I should add to iptables when the packets go out from firewall in POSTROUTING table. 

Iptables -I POSTROUTING -o eth0 -p tcp -j MASQUERADE 

Here is a screen shot from browser that shows the round-robin method works (Figure 3):

                                                           Figure 3

Also, the following picture show my log in Log Server to prove that round-robin method works. I will talk about Log Server and its configuration later in part 2. Notice that I added a log prefix for each web server to distinguish the related log easily.

                                                                     Figure 4

Here is my bash script that I wrote to use wget command to bombard my web servers:

#/usr/bin/bash 

for((i=0;i<=100;i++)) 

do 

wget http://192.168.2.7/index.html 

done 

rm -f index.html*

Here is the result of running my script. I just captured last 10 lines:

                                                                  Figure 5

Web Servers

The following steps show how I configured the web servers: 

 1. I added a route to Gateway/Firewall machine with this command: 

        route add default gw 192.168.56.101

You can add this command to /etc/rc.local file to run it automatically when you restart computer 

 2. To install apache server, I ran this command: yum install httpd  

 3. chkconfig –levels 2345 httpd on To turn on the httpd service in run level 2,3,4, and5 

 4. I created a simple webpage in the following path: /var/www/html/index.html which is the default Document Root for apache server. The following shows the content of “index.html” code and as I said, it is a simple html code just for demonstration:

 
For web server 1:
[root@f13-ws1 ~]# cat /var/www/html/index.html 
<html>
 <head> 
  <title>My Website Number 1</title> 
</head> 
<body> 
 <center> 
  <h2>Welcome to my webpage</h2> 
</body> 
</html>

For web server 2:  
[root@f13-ws2 ~]# cat /var/www/html/index.html 
<html> 
 <head> 
  <title>My Website Number 2</title> 
</head> 
<body> 
  <center> 
     <h2>Welcome to my webpage</h2> 
</body> 
</html> 
4. service httpd start To start apache server  
5. Next, I added a rule to firewall to accept tcp connection for destination port 80 (http):
iptables -I INPUT -p tcp –dport 80 -m state –state NEW -j ACCEPT 
The next two screen shot are the output of iptables-save command in both web servers:

                                                                    Figure 6

                                                                   Figure 7
In part 2, I will discuss and continue with Log/Backup Server and its configuration. 
Regards,
Khosro Taraghi